The problem with Vagrant was always that the other backends besides VirtualBox were second class citizens because of image and config file incompatibility between backends. And VB is pretty bad, the driver side was always crashy on Linux and it conflicts with the native virtualization drivers on Mac, Windows and Linux.
So there were good reasons to avoid it before Docker too.
i used vagrant with libvirt/qemu at my last job and it worked just fine though...
the only problem i had was updating base boxes, but that was self inflicted because it was easier to make a function to delete them from the libvirt cache than maintaining proper versioning.
/edit: i just realized from your other comments that you're probably constraining yourself to prebuild baseboxes. you really shouldn't. its trivial to built them with packer [1] and there are lots of config files on github[2] to do just that. This makes it possible to really tweak them for full integration right after 'vagrant up'
So there were good reasons to avoid it before Docker too.