I thought so, too and developed SIT (https://sit.fyi). An interesting thought that I had initially helped me make it even more future-proof: Git is likely not here forever, it'll probably get replaced with something else in due time. So I designed SIT to be both merge-conflict-free and SCM-independent by relying just on additive sets of files.

A VCS might be a good storage location for these 'objects', but it certainly doesn't provide the structure to manage them.