We're using vagrant to a very limited extent. It's kinda slow (thanks RubyGems!) and overly complicated (why the hell does it use that huge COM-like API thing when it could just shell out to the command line utils?) It's been mostly replaced with two or three short bash scripts now... but they are relatively specific to our app.
In short, the world needs a Vagrant-clone that's < 500 lines of bash.
It's funny you should mention the virtualbox gem. Originally the library did just what you described but the switch back to shelling out was made for windows compat in 0.9
I'm scanning the GitHub repository and it doesn't seem much different from when I last looked. The code is extremely Java-y. I wanted to make a small change to some command (I forget what), but I had to wade through 4+ levels of indirection simply to find a single line of code doing the real work.
In short, the world needs a Vagrant-clone that's < 500 lines of bash.