I've found that it's usually as fast to use the Vim Command-T plugin to switch files even once they're opened than it is to switch back to that buffer directly.

Main exception being the alternate buffer for which I have a quick shortcut - (<leader>, - <leader> being set to ,) The consequence is I usually have piles of buffers open at once.

Edit: sjs puts it better above. Having to think 'is this file already opened or not' and then using that information to determine the command you use to switch to it is rather annoying.