- containers are each isolated in a VM (aka virtualized)
- workloads are not homogenous and change often (your neighbor today may not be your neighbor tomorrow)
I believe these are fair assumptions if you’re running on generic infrastructure with kubernetes.
In this setup, my concerns are pretty much noisy neighbors + throttling. You may get latency spikes out of nowhere and the cause could be any of:
- your neighbor is hogging IO (disk or network)
- your database spawned too many threads and got throttled by CFS
- CFS scheduled your DBs threads on a different CPU and you lost your cache lines
In short, the DB does not have stable, predictable performance, which are exactly the characteristics you want it to have. If you ran the DB on a dedicated host you avoid this whole suite of issues.
You can alleviate most of this if you make sure the DB’s container gets the entire host’s resources and doesn’t have neighbors.
> - containers are each isolated in a VM (aka virtualized)
Why are you assuming containers are virtualized? Is there some container runtime that does that as an added security measure? I thought they all use namespaces on Linux.
Not so; neither Kata containers nor Firecracker are in widespread public use today. (Source: I work for AWS and consult regularly with container services customers, who both use AWS and run on premise.)
- lots of containers running on a single host
- containers are each isolated in a VM (aka virtualized)
- workloads are not homogenous and change often (your neighbor today may not be your neighbor tomorrow)
I believe these are fair assumptions if you’re running on generic infrastructure with kubernetes.
In this setup, my concerns are pretty much noisy neighbors + throttling. You may get latency spikes out of nowhere and the cause could be any of:
- your neighbor is hogging IO (disk or network)
- your database spawned too many threads and got throttled by CFS
- CFS scheduled your DBs threads on a different CPU and you lost your cache lines
In short, the DB does not have stable, predictable performance, which are exactly the characteristics you want it to have. If you ran the DB on a dedicated host you avoid this whole suite of issues.
You can alleviate most of this if you make sure the DB’s container gets the entire host’s resources and doesn’t have neighbors.