The downside of building this directly on Matrix events is that I don’t think it would work offline, right? (Until p2p matrix gets more common, that is).

My biased suggestion would be to see how far you get with storing the data to be synced in SyncedStore, and connecting Matrix-CRDT and y-indexeddb for local storage.