Ha! You're kinda right and I like it. I guess thinking of literal files floating in space isn't a unique human experience eh? Good to know I've got someone like me out there at least =)
Well then I have to say well done, haha. I had an old version in SceneKit / ARKit that worked in AR, but performance was limited based on my implementation and I could only get a few hundred files on screen on at once before you started losing too many frames.
I wish you luck in your endeavors, and maybe one day we can chat about your experience, and what you've done =)
And I didn't make anything that automatically populates files in VR, but I did use a program (XSOverlay) that lets me use 2D desktop windows in VR, and I would open a crap ton of files at once and have them positioned all around me (along with the directory listings they came from).
Stay tuned! This is a planned future I had temporarily removed while I worked on performance in the Metal engine, and is coming back very soon =) I'd love the opportunity to pick your brain about potential intermediate formats as well - e.g., if you have specific languages or datasets that you'd want to be able to operate on. I have some ideas about adding a tiny runtime scripting layer in this as well to operate on text JIT, instead of just laying it out.
> Can this work with visionos?
Theoretically yes, but I haven't done the work to composite the two rendering layers together. Like a lot of Metal, there is a massive amount of documentation that would certainly be helpful if written, but currently isn't. I do plan on this, however, in the future.
I built an interactive call graph interface targeting visionOS that I'll be open sourcing soon. My approach is to use the Language Server Protocol(LSP) as the itermediate format as many LSP implementations support the call hierarchy method. You can then proxy the commands to the headset for rendering. Using the LSP is nice because you can integrate the controls into small editor plugins and get live code analysis.
Most code analysis programs fail in some combination of editor integration, language support, iteration speed, or interactivity.
One of the big issues with visionOS right now is the divide between a full immersive space and a volume. There is a dev option to enable the virtual display to work in an immersive space, but normally a full metal rendered scene will hide your mac monitor. The volume requires RealityKit and provides no hand tracking data. My approach is to run the graph fruchterman-reingold spring embedder in a metal compute kernel updating a LowLevelMesh in a volume. The biggest limit I've faced is around 1000 input targets (graph nodes) is the most the RealitKyt renderer can support before the frame rate dips under 90Hz.
Good luck if you attempt visionOS, it is nearly undocumented.
Your implementation sounds great, and I'm looking forward to seeing it. Yes, the SDK is extremely obtuse and really not that easy to play with. I kinda was stubborn and refused to pick up the new rendering tool after just having learned SceneKit and then Metal, so I'm sure it's gotten at least a little easier, but the tools that Metal exposes just don't have analogues from what I can tell.
Keep me on your short list for beta users when the time comes =)
I thought Unity's Polyspatial was supposed to make this easier but definitely interested to see what you come up with..... AVP is a but underutilized by apps.
I'm obsessed with visual space representations of word as well. My application has a test mode where I render an entire dictionary (like, Webster) and then "plays" a book word by word, highlighting the word, then it's definition, and those words definitions and so on and so on, and it creates a trippy and distinct visualization kinda like an audio visualizer.. but with words. We should chat if you're interested in this kinda thing!
https://github.com/tikimcfee/LookAtThat
I'm working on a 3D infinite canvas of text, focusing on code. Runs on iPhone, iPad, macOS so they can all act as separate viewports into a space. You can point the app at a repository, download a copy locally, instantly render the entire repository into space in less than a second in most cases, and then fly around and search for text. I just got an optimization working for larger files and it's kinda fun how much even an iPhone can do with instanced rendering.
Im developing it with the use in mind of flying through your code to show others relationships, or edit with a visuospatial look at your code instead of basic 2D tabs and a mind map of which one had the thing you're working on. It's kinda fun to work on the project In the project!
It's built on Swift and Metal but can ready any utf8 text file, minus a few subsections of the Unicode spec (for now).
You should add another use to it which is exploring live in-memory object graphs of applications. Make some adapter library to allow getting those objects from apps that want to use and then you can fly through the data being processed. For debugging, exploration and education. I'm using a basic version of this idea for debugging: https://github.com/Quiark/overlog
I've thought about this a lot, and `overlog` looks awesome! I appreciate you validating the use case. I've seen a lot of apps that have something like this - either an open socket that runs that allows the app to consume data, or like you mentioned, something that you install via a small library and output something readable. Either one sounds feasible, honestly. I'd love to chat with you about this, maybe see if we can some up with something - it's super useful to have lots of data like this on screen at once!
How did you learn Metal? The documentation on it from Apple leaves a lot to be desired for learning it from scratch and all the books I’ve encountered look woefully out of date.
Well first and foremost, by doing exactly what you're doing now and asking a bunch of people for help too, haha. It's not been easy, and I'm truly still terrible at it.
However, honestly, most came from following this tutorial series on YouTube which broke down building a basic game engine, and I stopped about 20 or so videos in once I had the tools I needed. I highly recommend!
And hey feel free to DM me on something if you'd like - I'm happy to answer questions and help where I can!
Not yet, but I was planning on putting up at least a beta within the next few days or so. Most of the features about editing, searching, colorizing, syntax analysis, etc., are in some almost-finished stated or another, but I'd love to publish something for people to play with! If you drop a line to my email on GitHub, I can make sure to add you to the beta, or at least send ya a link when it's up =)
If you've got an iPhone or iPad, I've got an AR prototype for glyph based rendering like this you might enjoy playing with: https://github.com/tikimcfee/LookAtThat
Alpha release builds an AR app to pull code, render in space.
This seems to be Swift-exclusive, but the idea is absolutely there. Imagine when this gets expanded to any codebase. Imagine all the spaghetti code you could visualize as real spaghetti!
Bingo. Every C-suite egotistical narcissist is seeing that the one thing they do - talk- is even more worthless than it was to begin with. The number of conversations I’ve had in the last year that reek of “terrified upper management” is both cathartic and depressing.
Hey there! Sincerely cool stuff, I’m glad it’s fun for you.
It’s actually quite approachable to play with, and some of the comments about, “wut?” may be best answered by a little more experimentation on the user’s side, haha. I think the content itself is tricky, which may trip people up.
Something I’ve seen before that may be interesting is doing something with the definitions of words. ATM, you’re using a source list of words and using the embedded vectors to visualize. But what if you visualized not just the words themselves, but the ordered list of words that make up the definition(s) of that word visible in some spatial relationship. This would look interesting because around (connected to?) each word is its meaning in this case; changing the definition (the context use of the word) would also change the definition… and also change the connected word nodes in the graph. I envision ordered lines and colored words in this style.
If you end up doing something like that, start with like.. a “sentence player”. At the moment you show the words at once. What would it look like to “animate” the appearance of the words and their relationships by definition?
Anyway. Thanks for getting this far, haha. This is a really fascinating project and I’m glad you shared it. Please do tell if any of this is close or far off from something you might be interested in!
I wonder what patterns we could create from the word glyphs that produce meaningful patterns. You said “edge of the galaxy” based on the position of the words. I wonder what else you’d come up with different embeddings and organizations.
