What aspects of NFS do you think break half of the important guarantees of a file system?

Well, at least O_APPEND, O_EXCL, O_SYNC, and flock() aren't guaranteed to work (although they can with recent versions as I understand it).

UID mapping causing read() to return -EACCES after open() succeeds breaks a lot of userland code.

Lack of inotify support is one that has annoyed me in the past. It not only breaks some desktop software, but it also should be possible for NFS to support (after all, the server sees the changes and could notify clients).

Thanks for this, it's helpful. Totally heard about O_APPEND and read() returning -EACCESS. The other ones, I agree, should be fixed in later versions of the Linux kernel/NFS client.

Just ran into this one recently trying to replace Docker w/ Podman for a CICD runner. Before anyone protests we have very strong, abnormal requirements on my project preventing most saner architectures. It wasn’t the root cause but the failure behavior was weird due to the behavior you just described.

This is kind of an interesting thought that more mirrors how Docker uses OverlayFS to track changes to the entire file system. No need for new file APIs.

It can also use ZFS to do this.

I take a different view on this. IMO the tricks that existing file systems play to get more performance (specifically around ordering and atomicity) make it extra hard for developers to reason about. Obviously, you can't do anything about fsync dropping error codes, but some of these failure modes just aren't possible over file systems like NFS due to protocol semantics.

Here's how I would think about this. Regatta isn't the best way to add synchronization primitives to S3, if you're already using the S3 API and able to change your code. Regatta is most useful when you need a local disk, or a higher performance version of S3. In this case, the addition of these new primitives actually just makes Regatta work better for our customers -- because we get to achieve even stronger consistency.

That’s true, we could just release the software open source, but that doesn’t help our customers who don’t want to run and manage their own infrastructure. Our customers tell us that the value of the product comes from it being fully managed — they simply need to click a button, and all of this works out of the box.

It’s not as clear, but it’s certainly something we are considering. If you’d like to use us on-prem, I’d love to hear more. Can you shoot me an email with details at hleath [at] regattastorage.com?

I spent several years working in the Northeast, and I developed an appreciation (but not a skill) for sailing. In some sense, I think of Regatta as high-speed sailing on top of customer data lakes.

Sailing, I was hoping it was a rowing metaphor.

Thank you so much! It’s been amazing to see what you all have built over the years, and it’s (of course) been inspirational for me.

Awesome! Great to meet you, so happy that so many folks in the space are here.

This is true, and I think that there are consumer (or at least "run on your laptop") versions of this that could make sense. However, the technology underneath would have to look very different. For example, these protocols are designed for online file systems (e.g. you must be connected to the file system directly in order to list what's in a directory). This works great in a data center, but doesn't work great on your laptop.

On the other hand, something like Dropbox is actually a program running on your laptop that simulates a file system, and then does the synchronization at the file level as needed. I think that there's probably some latent demand for a similar product for developers to access their S3 buckets easily from their laptops, and it's something we might look into as we get farther along.