> Swarm works, but has poor support for volumes - which means it's tricky to run legacy applications on swarm (which eg uploads files to local disk, not s3 - or keeps state/cache on disk, not a database).

One way round that is to use an NFS volume. However, I've hit problems with too many client NFS connections on a docker swarm and so found it better to mount the NFS volume on each host and use a bind mount instead.

CSI support will hopefully make this easier. But: there are quite some options once you Look deeper.

Also the volume plugin spec is so simple that it is possible to maintain your own plugin (even without csi).

My general feeling about adding an NFS depency to multinode swarm is that it's effectively adding a single point of failure to a system which is otherwise somewhat robust against single node failure...

Traefik is really simple to set up and I’d bet setting up s3 is 100x easier than kubernetes no? Unless I’m missing something.

Fwiw I found swarm lovely and just so much easier to work with than anything else solving the same problems.

Porting a ten year old app from local file storage to s3 might not be trivial.

For a new app, one generally should and can embrace 12-facors, and delegate state to stateful services (managed databases, key-value stores, s3 etc).

Do note that for simple services, local disk can be very hard to beat for low complexity, extremely high performance - with the caveat that horizontal scaling might be tricky.

Ed: also depending on privacy requirements - self- hosted s3 (eg minio) might be tricky for a production load. OTOH self-hosted Kubernetes is no walk in the park either!