Docker defaults to overlayfs but you don't have to use it. If you use ZFS or another storage driver it will leverage their capabilities to provide the same functionality: https://docs.docker.com/storage/storagedriver/select-storage...

Docker on ZFS is a pain in the ass and slow as hell. And I say this as both a happy docker/container and ZFS user. But combining them is a bad idea.

I definitely noticed slowness as well when I tried it.

I'm currently using overlay because I leave my docker mount on my root partition since ZFS has always been a pain for root. I hear Ubuntu 19.10 is smoothing that over a bit though.

It's too bad because I've ran into issues on lower memory boxes where the Linux block cache and the arc compete for memory and cause performance slowdowns or errors. If all the filesystems were ZFS this wouldn't be an issue.

Also all the other ZFS benefits like resisting bitrot etc.

I just use overlay on top of ZFS. It's just the native ZFS docker backend that is very slow.

Why didn't I think of that?

So all you need to do is manually specify the volume driver to overlay2 when your FS is ZFS, and this just works?

Facebook, for one, uses Btrfs for its containers: https://facebookmicrosites.github.io/btrfs/docs/btrfs-facebo...

fyi, btrfs as /var/lib/docker/btrfs has a disadvantage under certain circumstances: running commands (du,rsync,..) on the subvolume folders altering the access time will incur a storage cost if the filesystem is mounted without noatime option: https://github.com/moby/moby/issues/39815 - not sure if it applies to ZFS as well (see https://lwn.net/Articles/499648/)

Another difference that may matter for some:

on block based CoW FS, shared executables and libs will not be shared among containers, because file cache is file based not block based.

Using Overlayfs - or other file level CoW storage, when one starts several containers from the same image, shared libraries and executables will be loaded only once(so will be actually shared) across containers.

In fact, this is explained in the article which also has an interesting anecdote about btrfs.