Kubernetes' secrets management is nominal at best. It's basically just another data type that has K8S' standard ACL management around it. With K8S, the cluster admin has access to everything, including secrets objects. It's not encrypted at rest by default, and putting all the eggs in one basket (namely, etcd) means they're mixed in with all other control plane data. Most security practitioners believe secrets should be stored in a separate system, encrypted at rest, with strong auditing, authorization, and authentication mechanisms.
It's "good enough" for most and extension points allow for filling the gaps.
This also dodges the crux of GP's argument -- instead of running 1 cluster with 10 components, you now need a half dozen clusters with 1 component each, but oops they all need to talk to each other with all the same fun TLS/authn/authz setup as k8s components.
I'm a little confused. Why does the problem with K8S secrets necessitate having multiple clusters? One could take advantage of a more secure secrets system instead, such as Hashicorp Vault or AWS Secrets Manager.
The point is that once you're talking about comparable setups, you need all of Vault/Nomad/Consul and the complexity of the setup is much more than just "one binary" as hashi likes to put it.
> So now you have 3 separate Consul clusters, a Vault cluster, and a Nomad cluster. So what did you gain really?
GP's point was already talking about running Vault clusters, not sure you realized we aren't only talking about nomad.
The only thing I was trying to say is that although K8S offers secrets "for free," it's not best practice to consider the control plane to be a secure secrets store.
