The answer for successfully applying VCS to higher-dimensional spaces will demand more mathematically-elegant intermediate representations. Most source code files are highly structured by default. Image files are mostly feasible to diff as-is. Typical 3d models, not so much. 3d models with animation, even less so.
To be clear - the problem isn't that we cant detect a difference, it's that we cannot produce a useful view of the difference such that a human can make an informed decision. With images/audio/code, you can still extract useful knowledge as long as you know the shape of the difference relative to the whole, even if the difference itself is a meaningless mesh of colors between 2 image files.
Writing a useful diff engine for 3d models represented using constructive solid geometry would probably be substantially easier than with other approaches. I don't know if CSG is actually constrained to 3 dimensions either... I feel like GitHub actually tried to do something like this but I don't know if it went very far.
I've been working in the collaborative editing space for a decade now (!) and I think this problem is simply a consequence of an age old architectural mistake that we persist because we don't know any better.
The problem is that the filesystem acts as a dumb blob store that doesn't understand changes. And it doesn't understand any of the semantics of what its storing. Its a lovely abstraction - but it has bullied us into the age old "load / edit / save" workflow. And that workflow just isn't very good:
- Its slower for the computer - because no matter how incremental, the computer has to rewrite the entire file to disk
- Its buggy by default. A power loss or malfunction can and will result in a corrupt save file. You can fix this with careful use of fsync() and renames, but almost nobody does that correctly because its complicated and confusing. And it makes saving slower. You see this all the time in video games - "When you see this symbol, don't turn off your console".
- Because you only give the OS snapshots in time, you end up discarding information about the changes that have happened along the way.
With code editing we work around this with diff-match-patch, but its lossy, not realtime, and not good enough for collaborative editing.
With more complex data (like 3d models) this problem is much more egregious. There's no reason the modelling program couldn't output a fine grained sequence of changes as they're made by the user. Any collaborative editor already does this.
Having a rich set of changes gives us way more options for merging - eg, they could be merged in a conflict-free way if we want. Or with a custom UI or whatever. Its strictly more information that you can generate with diff-match-patch.
The problem is there's no obvious place on the filesystem to put that information, or way to tell the OS about changes as they happen. And so long as the file system is the way that disparate programs talk to each other, and the file system can't handle this information, programs ends up just discarding any detailed change sets. And we end up with ugly JSON merge conflicts in 3d files. Collaborative editing (if it happens at all) ends up done on a per-app basis rather than just working by default across the whole OS - like it should.
And the result of all that is that the desktop is being slowly sidelined in favour of proprietary web apps (like figma or google docs). Why? Because web apps aren't constrained to the paradigm of saving and loading files.
It won't be easy, but it seems to me like a no-brainer for us to fix.
> Having a rich set of changes gives us way more options for merging - eg, they could be merged in a conflict-free way if we want. Or with a custom UI or whatever. Its strictly more information that you can generate with diff-match-patch.
this reminds me of an event store from event sourcing. the file itself is just the on-disk persisted format of the 'permanent undo log'.
Depending on how fancy the system is, one could view all of the changes, the mark a particular change in the past as 'deleted', or modify that change's parameters, or maybe even do a 'this sequence of changes replaces that change there' operation. on disk, there are just more events added to the event stores, while onscreen, the file is re-rendered / re-projected anew to reflect what it would look like as ifthose changes were original. Surely i'm just poorly reinventing CRDTs or DAGs or similar, right?
granted, i don't know how easy it would be to not cascade subsequent edit events if one of their past dependencies was gone, so we'd need to make the events be 'pure functions' (the X-eith thru X+N-eth words equal Y) rather than imperative statements (add text at offset X with content Y)
> web apps aren't constrained to the paradigm of saving and loading files.
How is reading and writing bytes to FS that different from doing the same on a network stack? In fact I'm puzzled by your post which seems to imply that a 'database' -- a store for structured data -- can't be written over a file system.
> The problem is there's no obvious place on the filesystem to put that information, or way to tell the OS about changes as they happen.
A thin layer of abstraction over FS seems reasonable, possible, and frankly ancient tech: they are called databases.
Of course you can build things on top of files. But most people don't, they use files directly, and therefore get "bullied" (which I think is a very apt term) into load/edit/save cycles of file-shaped chunks.
Application programmers could be provided with a better storage API, either as a library or as a standard from the platform. The result would be better software with little work from the application programmer's side. I'm not a Mac person anymore, but I believe they have something like this in the form of CoreData (no idea of it's good or not), and I imagine Windows has seven things that you can either use or not.
An analogy would be how it's possible to build something that resembles concurrent processing and top of any single-tasking system, but a platform that doesn't provide it as a built-in will be very different to a platform that does.
Non-UNIX operating systems had structured data as a core part of their libraries. This was probably a good idea that got lost thrown aside by some people call "the minimalist elegance of UNIX", which other people, including I imagine my GP post, have less endearing terms for.
The GP I replied to starts with "I've been working in the collaborative editing space for a decade now (!)" and claiming a network interface somehow magically does something that an FS can not (and speaking of Unix, entirely ignoring that both mechanisms use file descriptors). I mean, what to make of statements such as "[t]he problem is there's no obvious place on the filesystem to put that [meta-data] information"?
As for the generic "application programmers", possibly you are right, and doing 'literal' mappings between their memory and persistent models is to be expected.
Yeah, just like "web based" systems use a protocol to shuffle bits around. So an FS will in fact support building higher semantics on top of files and folders, and a visit to your local .git directory will provide an example of a contemporary popular tool that does precisely that.
Now possibly the GP is upset that random app A can't just use the files spit out by app B, which is as reasonable an expectation as having random client C use a universal driver to talk to various servers.
The problem isn’t that doing this stuff is impossible. It obviously is - databases work well. The problem is the point of interoperability between applications in Unix is the file. It’s not the database - applications don’t pass data to each other via Postgres. The pass data with files. And that means if you want multiple programs (eg, a 3D modelling program and git) to share information, the way they do so is by the modelling program saving flat files and git reading them (and diffing them). That just isn’t very good.
We can work around this on a per application basis. My code editor could output all the changes to a custom SQLite database. But then it won’t work with all the other tools on the system. I can't use grep or awk. I can't use git properly. We end up with isolated bundles bits that can't easily be used together.
Doing that goes against platform conventions. It’s not the Unix way. So people don’t do it. And as a result, for interoperability reasons applications don't save their data to databases. They use flat files, hope nothing gets corrupted and discard data about changes. The reason isn’t technical. It’s because conforming your software to the platform convention is what people want. I want to be able to just copy my data to a usb key without thinking about postgres's data format and extraction tools.
There's a parallel in programming. Languages define common types - like strings. But they don’t need to. Rust could leave the String implementation details to 3rd party crates. The problem is if they do that, libraries wouldn’t be able to share strings easily with each other because they might each be using a different string data type. So standard APIs would probably avoid strings whenever they can. This is the problem we have on Unix. There’s no interoperable, platform-standard way to store changes on disk. So all the apps I use don’t. When git needs change feeds, it’s forced to use diff-match-patch instead. But it’s not as good: You don’t get real-time feeds. And it doesn’t work properly with non-text data. Conflicts are unavoidable. And patches are non canonical.
You’re right - we can solve all this on a per application basis. I don’t want individual apps to have their own hand rolled data silos and their own crappy git replacements. I want a platform where those tools work with any kind of data. Even with apps that haven’t been written yet. That’s the Unix way. I love the filesystem. I just think that after decades of experience we can do it better.
Put on your OS designer hat and let's think this through. A generalized capability for persistent data with universal semantics -- the analog of the magical universal driver for networked IPC -- is a design brief for a generalized database. So your first decision is 'how is the data organized'? Given the generality of the brief, the optimal candidate is a graph of documents. So a graph database that would allow arbitrary, semantic, linkages between files.
So now we need to get a handle on the semantics. At the individual document level, the OS will need to provide some sort of type system for data, the equiv of your example of types in languages.
Then we need a generalized mechanism to allow for creation of higher level semantics -- the bits that semantically relate doc A to doc B. These would be the arcs of the graph. So now we're looking at something like RDF.
Will this be a freaking cool OS? Definitely. Will it be performant? TBD, but obviously not as performant as the current state of affairs. And that's only the mechanisms aspect of it. You will still need to address all the issues related to 'shared/universal semantics' that the Semantic Web people have to deal with.
At that point you the OS designer need to ask yourself if this is a reasonable task for the OS.
It’s easy to represent 3D data as JSON polygon soup, which can then be diffed and merged as text. A lot of game engines store geometry data in this way. It often works fine until there’s a merge conflict, at which point things become difficult.
Revisions A and B have, as part of their changes, both modified vertices originally on lines 437 and 7689, and A has changed 5678 while B has deleted it. It’s often easier to discard one revision and redo the work in-order, rather than try to resolve.
> To be clear - the problem isn't that we cant detect a difference, it's that we cannot produce a useful view of the difference such that a human can make an informed decision.
The comment is not talking about diffing JSONs but being able to understand what the difference means.
To be clear - the problem isn't that we cant detect a difference, it's that we cannot produce a useful view of the difference such that a human can make an informed decision. With images/audio/code, you can still extract useful knowledge as long as you know the shape of the difference relative to the whole, even if the difference itself is a meaningless mesh of colors between 2 image files.
Writing a useful diff engine for 3d models represented using constructive solid geometry would probably be substantially easier than with other approaches. I don't know if CSG is actually constrained to 3 dimensions either... I feel like GitHub actually tried to do something like this but I don't know if it went very far.