Ha! First: I appreciate the detailed and thoughtful reply, even if I feel wildly judged.
It's distinctly possibly that you're simply "better" at reverse engineering than I am, which really just means that you might do it frequently and I might do it a few times a decade. This isn't going to keep me up tonight, because my identity isn't tied to being someone who reverse engineers things.
That said, I am pretty thrilled with this solution. I launched a web-enabled version last night and so far about 1100 people have used it to convert 6800 files after I replied to some posts on relevant musician forums around the web.
In my defense, what you're not taking into any consideration is that until 48 hours ago, I'd never looked at the MIDI spec or opened a MIDI file, before. You clearly have a huge amount of domain knowledge that I don't pretend to have.
I also, shocking as it may seem, haven't worked with binary formats in over a decade. I'm a web developer. Binary formats aren't an alien mystery to me, but all of the tools for working with them had to be re-learned as I was working on this.
Anyhow, don't fall into the trap of equating typing speed with the time it takes to learn a domain and consider (design) an approach. If I could think at the speed I can type, John Carmack would have nothing on me.
In the end, I absolutely did get lucky. The proprietary format was, as you proposed, a bunch of 1 track/format 0 MIDI files, bounded by hierarchy metadata that was discarded.
Curiosity did get the better of me and you seem to be spamming every fucking forum so WTH. The longest time it took for anything was waiting for the file to download. The rest of this was about 5 minutes of effort. For the record I do not know the MIDI spec, but it must be one of the most easy to google and well documented things out there, and it's pretty simple likely because it's nearly 40 years old and had to run on potatoes.
The file you could have reverse engineered is not enough of a challenge for an interview question in a low level/systems field and yet you didn't even attempt to do that part. The file hierarchy is absolute offsets and sizes in 4 byte little endian quantities. It took less than 30 seconds to figure that out. How? Find MThd string, what offset is it at, search for that offset as a value. Notice that it is adjacent to a null terminated string with a name ending in .mid and a small quantity. Is that small quantity added to the original offset the start of the next MThd, yes? Done. Anyone with the shittiest hex editor and Ctrl-F can do this in a minute.
Rebuilding the hierarchy is quite simple since the absolute file offset to the entries is adjacent to every directory name.
These midinfo things are interesting they also have some sidecar data. Their content might also be something novel to reverse, that might be worth bragging about.
> I spent almost a week (!) reverse engineering their absurd proprietary format using a hex editor and the MIDI spec.
Since this whole affair seems to have just boiled down to naively iterating for the 4 byte MIDI file magic and then examining the MIDI file metadata you didn't even need all the bluster of breaking out the hex editor and a calculator... Bam https://github.com/jimm/midilib, done (the actual get the MIDI data part can be done in 1 line with a string split).
It's too bad ChatGPT didn't suggest that. That should extract the 3 pieces of metadata you attempted to store in the directories.
Sidebar, nothing here stands out as absurd. It just looks like the obvious solution some working stiff would put together to bundle some data up. They don't obfuscate or do anything that stands out as fucky. Since it's read only its not like not using sqlite is some cardinal sin and they probably gave it all of 3 seconds of thought it deserved.
If you were using this as a learning exercise, fine, but then go back and check your work because your tempos and key signatures are off. eg the tempos that you categorize as 104 have an actual encoded tempo of 571429 µs/beat, or 1.7499 bps or 104.9999 BPM, ie it's what humans would call 105 BPM. Not the end of the world but this is pretty rookie floating point mistake.
And I'm pretty sure you bungled the key signature because at least ones that say Abm are Fm. Why is that... ah you have ignored relative keys. For instance since the meta event FF 59 02 FC 01 has a sf value of 0xFC which is 2's complement -4 that's 4 flats. If this were a major key that would be Ab, but it's a minor key so it's Fm. Oh no, even simple keys are fucked. FF (1 flat, or F major is coming out as Cb, 7 flats), seems like a bungling of 2's complement.
My music education culminated in 2nd chair trombone in the 8th grade and everything I know about the MIDI spec I got from the top 3 hits of google, so caveat emptor.
The only reason I knew it was wrong was because I checked the files with mido (I know you claim ruby is superior, but I seem to be doing ok with dull old python like a rube).
Also in the file I count 640 MIDI headers and I only got 639 files out of your thingy so you might want to revisit that part I mentioned about bugs.
It might actually be interesting if you reconstructed the hierarchy in the metadata. That is not hard, it is all absolute offsets, sizes and null terminated ASCII strings.
But what you did seems to have written a (possibly buggy) loop that calls basic standard library functions in ruby, and attempt to poorly parse/convert a few MIDI meta-events. It's not reverse engineering if you have a nearly perfect spec in front of you, you completely skipped the (simple) reversing part and bungled the easy part. And this task shouldn't take a few days nor hours for someone experienced with basic scripting. https://adventofcode.com/2023
> the entire process of iterating through the binary blob, pulling out the MIDI header/track chunks, and then creating entries with smart naming in a zip archive is done entirely in-memory. For the non-programmers, this is otherwise known as "hard" or "showing off".
We have very different definitions of hard. Unless you were writing a shell script doing it any other way is bananas. These are like a few hundred bytes a piece.
Your technical chops are discordant with the arrogance and incivility you've displayed towards others on this forum. (And name dropping Carmack, that's bold)
eg: "Serious question: are you wickedly combative in all discussions, or just when you get the cold feeling that perhaps you might not know as much as you think you do?" in response to someone who simply had a differing opinion to this: "you keep implying HTMX is a good choice, when there are vastly more powerful solutions in play."
"I assume that the person who downvoted my comment is a bootcamp grad. Good luck with your future endeavors."
It's distinctly possibly that you're simply "better" at reverse engineering than I am, which really just means that you might do it frequently and I might do it a few times a decade. This isn't going to keep me up tonight, because my identity isn't tied to being someone who reverse engineers things.
That said, I am pretty thrilled with this solution. I launched a web-enabled version last night and so far about 1100 people have used it to convert 6800 files after I replied to some posts on relevant musician forums around the web.
In my defense, what you're not taking into any consideration is that until 48 hours ago, I'd never looked at the MIDI spec or opened a MIDI file, before. You clearly have a huge amount of domain knowledge that I don't pretend to have.
I also, shocking as it may seem, haven't worked with binary formats in over a decade. I'm a web developer. Binary formats aren't an alien mystery to me, but all of the tools for working with them had to be re-learned as I was working on this.
Anyhow, don't fall into the trap of equating typing speed with the time it takes to learn a domain and consider (design) an approach. If I could think at the speed I can type, John Carmack would have nothing on me.
In the end, I absolutely did get lucky. The proprietary format was, as you proposed, a bunch of 1 track/format 0 MIDI files, bounded by hierarchy metadata that was discarded.