Did you run into ipv4 subnet limitations wrt to node count or something else? 5k is max supported node count for a while now but running it with a tunnel-based cni (as is typical in cloud) would probably be too wasteful so it’s likely only an option for when you can configure l3 on your own switches. Or maybe using ENIs in ec2. I found that ipv4 space requires very careful planning especially if you plan on peering clusters together.
> weird network setup issues, daemon sets for storage/logs, persistent volumes, rough edges of the tooling operating the cluster (people tell you that with k8s you write less scripts! But in reality depending on your cluster size the scripting moves to the admin side.
Network with Cilium has been mostly “setup once and forget”. But yeah all the addons will require constant upkeep in a self-managed scenario. As far as configuration management goes I agree it’s a big hole which we had to fill with our own DSL configuration framework (think Helm3 but with Starlark). Now there are products like Pulumi which can offer similar API
> weird network setup issues, daemon sets for storage/logs, persistent volumes, rough edges of the tooling operating the cluster (people tell you that with k8s you write less scripts! But in reality depending on your cluster size the scripting moves to the admin side.
Network with Cilium has been mostly “setup once and forget”. But yeah all the addons will require constant upkeep in a self-managed scenario. As far as configuration management goes I agree it’s a big hole which we had to fill with our own DSL configuration framework (think Helm3 but with Starlark). Now there are products like Pulumi which can offer similar API