> Alternatively, we could have used a conventional hypervisor. Nvidia suggested VMware (heh). But they could have gotten things working had we used QEMU. We like QEMU fine, and could have talked ourselves into a security story for it, but the whole point of Fly Machines is that they take milliseconds to start.
Someone could implement virtio-cuda (there are PoCs on github [1] [2]), but it would be a huge maintenance burden. It should really be done by Nvidia, in lockstep with CUDA extensions.
Nvidia vCS makes use of licensed GPGPU emulation code in the VM device model, which is QEMU in the case of KVM and Xen. Cloud Hypervisor doesn't use QEMU, it has its own (Rust?) device model, https://github.com/cloud-hypervisor/cloud-hypervisor/blob/ma...
So the question is, how to reuse Nvidia's proprietary GPGPU emulation code from QEMU, with Cloud Hypervisor? C and Rust are not friends. Can a Firecracker or Cloud Hypervisor VM use QEMU only for GPGPU emulation, alongside the existing device model, without impacting millisecond launch speed? Could an emulated vGPGPU be hotplugged after VM launch?
There has been some design/PoC work for QEMU disaggregation [3][4] of emulation functions into separate processes. It might be possible to apply similar techniques so that Cloud Hypervisor's device model (in Rust) process could run alongside a QEMU GPGPU emulator (in C) process, with some coordination by KVM.
If this approach is feasible, the architecture and code changes should be broadly useful to upstream for long-term support and maintenance, rather than custom to Fly. The custom code would be the GPGPU emulator, which is already maintained by Nvidia and running within QEMU on RedHat, Nutanix, etc.
It would also advance the state of the art in security isolation and access control of emulated devices used by VMs.
> Someone could implement virtio-cuda (there are PoCs on github [1][2]
Any company (let alone Fly) doing this won't go against Nvidia Enterprise T&C?
> how to reuse Nvidia's proprietary GPGPU emulation code from QEMU
If it has been contributed to QEMU, it isn't GPL/LGPL?
> Could an emulated vGPGPU be hotplugged after VM launch
gVisor instead bounces ioctls back and forth between "guest" and host. Sounds like a nice, lightweight (even if limited & sandbox-busting) approach, too. Unsure if it mitigates the need for the "licensing dance" tptacek mentioned above, but I reckon the security posture of such a setup is unacceptable for Fly.
NVIDIA support is not special as far as QEMU is concerned—the special parts are all in their proprietary device driver, and they talk to QEMU via the VFIO infrastructure for userspace drivers. They just reimplemented the same thing in Cloud Hypervisor.
Red Hat for one doesn't ship any functionality that isn't available upstream, much less proprietary, and they have large customers using virtual GPU.
Not NVIDIA; fly.io reimplemented the parts that CH didn't already have. I know that CH is developed on GitHub but I don't know whether the changes are public or in-house.
That said, the slowness of QEMU's startup is always exaggerated. Whatever they did with CH they could have done with QEMU.
Do you mean vCS [1], which is already integrated and licensed by KVM/RedHat/Nutanix, Xen/Citrix and VMware?
It's distinct from SR-IOV, PCI passthrough, vGPU-for-VDI, and MIG.
[1] https://blogs.nvidia.com/blog/virtualcomputeserver-expands-v...