Neat project. I'd love to see it combined with something like Project North Star AR.

But for expert use... When smartphones were new, the user population untrained, and onboarding was critical, there was the idea of skeuomorphic user interface design. The app should resemble the real-world object. So affordances would transfer from novice user experience. The calendar app should have lined pages, a spiral binding down the center to hint at horizontal flipping, and textured leather margins. :) We don't do that anymore.

For expert users, what are the odds that the severe constraints of physical reality line up nicely with the domain constraints desired for maximum productivity? How long before people start complaining "I don't like reality - no stretchy arms - you actually have to walk over to something to grab it - yuck".

I joke that I currently don't do V of VR, because I almost always have camera passthrough, don't do A of AR, because I'm usually ignoring the world other than for user input, and don't do R of either, because the UI is so often aphysical. Resemblance to physical reality as design smell.

Though how to "think with your hands", when HIDs are still so crude? Tracked/instrumented manipulatives could be neat.