The problem is k8s is both a orchestration system and a service provider.
Grid/batch/tractor/cube are all much much more simple to run at scale. More over they can support complex dependencies. (but mapping storage is harder)
but k8s fucks around with DNS and networking, disables swap.
Making a simple deployment is fairly simple.
But if you want _any_ kind of ci/cd you need flux, any kind of config management you need helm.
K8s has swap now. I am managing a fleet of nodes with 12TB of NVMe swap each. Each container gets (memory limit / node memory) * (total swap) swap limit. No way to specify swap demand on the pod spec yet so needs to be managed “by hand” with taints or some other correlation.
The problem is k8s is both a orchestration system and a service provider.
Grid/batch/tractor/cube are all much much more simple to run at scale. More over they can support complex dependencies. (but mapping storage is harder)
but k8s fucks around with DNS and networking, disables swap.
Making a simple deployment is fairly simple.
But if you want _any_ kind of ci/cd you need flux, any kind of config management you need helm.