Some thoughts-

You could add some semantics to the vanity urls, for example /u/:username would reduce the possibility of collisions between usernames and module/controller names, after all that is the biggest problem with using top level names for vanity’s sake.

You could also go the facebook-like route: force people to opt into vanity urls, what this would allow you to do is check that the name requested is not dispatchable (like a user named “user”). The problem with this is you wont be able to ensure that your future code (modules and controllers) collide with an existing users vanity url. This also allows you to do whatever caching of the list at registration time. (You might consider invalidating the cache when a new url is registered and letting a plugin rebuild the cache only when it needs to, again lots of options here.)

One option might be to use a different 3rd level domain, like member.mysite.com/username, that would redirect to http://www.mysite.com/user/username, etc.

Either way, its interesting code you’ve got there, as for improvements, I’d try to localize everything to inside a route (even if you have to pull in the standard dispatcher). And like you said, a cache is the best way to go since the collection of possible routes only changes when a new user comes into the system.