You can give both hyperthreads in a physical core to the same tenant, no?
Scheduling different VMs to run on the same hyperthreaded core at once seems like it can't be good for either VM's performance, even if there were no security concerns. Hyperthreading is much more useful for running multiple threads of the same app, accessing similar instruction caches etc.
(There's also a question of safety within the VM, but a huge number of cloud users are running effectively one user within their VM.)
Yes, you can isolate hyperthread siblings to the same VM but you also need to ensure no host code (userspace or kernel) runs on that core, or the untrusted guest may be able to read values stored in L1 by that code. This is harder to do and likely would result in large performance drops for some workloads (because you are essentially disabling the advantage of locality for data that needs to be accessed from both guest and host environment).
Other things require sandboxed multitenancy than just full-on VMs. Database queries against a "scale-free" database like BigQuery/Dynamo, for example, where two queries from different tenants might actually be touching the same data, with much the same operations, and therefore you'd (naively) want to schedule them onto the same CPU for cache-locality benefits.
Okay, so many tennants are on the same BigQuery/Dynamo machine sharing cores.
If the API is "SQLish queries", I have a hard time believing you are going to be able to trigger these kind of attacks. You need a tight loop of carefully constructed code to flip them, no?
The latter question is very important indeed. If you for instance render websites in your vm they, if i understand correctly, can potentially read secrets from other processes, like db credentials and other stuff...
If the only real solution is to turn off HT/SMT that, seen positively, should net us a lot faster VMs then...
If you render websites that run code in your VM (e.g., you're running a traditional shared hosting infrastructure where mutually-untrusted users can upload PHP scripts, or you're doing something serverless / FaaS / Cloudflare Workers / etc. where mutually-untrusted users can upload functions), then yes. If you're rendering websites in the sense of hosting WordPress for lots of people but not permitting custom plugins, then no.
Scheduling different VMs to run on the same hyperthreaded core at once seems like it can't be good for either VM's performance, even if there were no security concerns. Hyperthreading is much more useful for running multiple threads of the same app, accessing similar instruction caches etc.
(There's also a question of safety within the VM, but a huge number of cloud users are running effectively one user within their VM.)