Yes, and this is what LXD does. I think I mentioned it in the article, but basically the issue is that it requires one of:
1. A clever server, which asks you which version do you have so it can generate a diff for you. This has quite a few drawbacks (storage and processing costs as well as making it harder to verify that the image you end up with is what was signed by the original developers). But this will guarantee that you will always get de-duplication.
2. Or you could pre-generate diffs for a specific set of versions, which means it's a lottery whether or not users actually get transfer de-duplication. If you generate a diff for _every_ version you're now back to storage costs (and processing costs on the developer side that increase with each version of the container image you have). You could make it so that the diffs only step you forward one version rather than instantly get you the latest, but then you now have clients having to pull many binary diffs again.
This system has existed for a long time with BSD as well as distributions having delta-RPMs (or the equivalent for debs). It works _okay_ but it's far from ideal, and the other negatives of using loopback filesystems only make it less ideal.
I could be technically inaccurate, but my understanding is that it's rsync but with the server serving a metadata file which allows the rsync-diffing to happen from the client side rather than the server side - hence no clever server required.
It also doesn't require diffing particular revisions; but only the different blocks will be fetched. It does require server the metadata file, but they're note very large afaik.
1. A clever server, which asks you which version do you have so it can generate a diff for you. This has quite a few drawbacks (storage and processing costs as well as making it harder to verify that the image you end up with is what was signed by the original developers). But this will guarantee that you will always get de-duplication.
2. Or you could pre-generate diffs for a specific set of versions, which means it's a lottery whether or not users actually get transfer de-duplication. If you generate a diff for _every_ version you're now back to storage costs (and processing costs on the developer side that increase with each version of the container image you have). You could make it so that the diffs only step you forward one version rather than instantly get you the latest, but then you now have clients having to pull many binary diffs again.
This system has existed for a long time with BSD as well as distributions having delta-RPMs (or the equivalent for debs). It works _okay_ but it's far from ideal, and the other negatives of using loopback filesystems only make it less ideal.
In my view, dumb formats are best.