How does someone learn this skill? I get that it obviously takes a ton of work, but how do you even prepare/study for (whatever it takes work to do) in the first place? The intuition required seems so different from what my intuition of computer systems would allow even imagining.
Broadly, the basis for this is an understanding of how computers actually work - if you learned about it in school, the class would have started with Von Neumann vs Harvard architecture graduating to building a CPU from scratch (well, logic gates, anyway, possibly with Verilog), and followed by another class about writing an operating system; kernel, drivers, then a (rudimentary) user land - basic implementation of syscalls, a glibc-type library to make syscalls, basic shell on top of that. That knowledge and a perseverant attitude (getting exploits to work can involve trying to debug code with zero feedback).
As far as this hack goes, broken down into three separate pieces, the exploits should be understandable as being sandbox escapes, but for three different sandboxes (Browser, OS, VM). Sandboxes are implemented in code, the code is going to have bugs, and exploits are "merely" a case of figuring out how to use the bugs to your advantage. No one's made a totally hack-proof sandbox. Not Apple (this year's contest featured a touchbar takeover), not Google (Chrome's fallen in years past), apparently not Microsoft or VMware (nor Linux/Canonical/Ubuntu, either).
If you're interested in learning about this stuff, microcorruption is an online security CTF "game" that starts off fairly easily. https://microcorruption.com/login
where he describes the hack of the Nintendo Switch, because they shipped an old version of Webkit. It's pretty impressive to see how "just" Javascript can jump from browser to native.
I have two recommendations that anyone with a CS degree or equivalent knowledge should be able to jump into with 0 other training, and come out with a solid understanding of how to find vulnerabilities and exploit them.
* Hacking: The Art Of Exploitation
This is an excellent book. It's lab-oriented, so you will be getting hands-on experience with reverse engineering and exploiting software.
They have a graph of classes somewhere, and you can see "Oh I want to get to advanced exploitation, what should I take?" and you just follow along. Start with intro to x86, intermediary x86, then move on to the software vulnerability classes, life of a binary, etc.
Really, with only these skills alone, you can self-teach the rest of the way. There's a book called The Art Of Kernel Exploitation, if I'm remembering right, that would take you into a more advanced but more niche area, and it's really up to you where to go after you get the basis (just like with programming).
I too would like to know. One recommendation I saw in regards to learning to hack stuff like the Nintendo Switch is, first learn to write a kernel, and also learn about compilers. Also, personally I've been considering working through Bunnie Huang's "Hacking the Xbox".
Just finished reading "Hacking the Xbox," per your suggestion - agree that it's quite good at explaining the process. The actual presentation does a good job of condensing what I imagine was an arduous adventure. Thanks for the recommendation.
Look at all the historical claims and the techniques involved. It takes a lot of tenacity and trying things that shouldn't work but on extremely rare occasions somehow do.
A lot of them publish source code that demonstrates how the exploit worked, plus when paired with an open-source project that has a vulnerability you can see how that matches up to the target.
There's a large toolbox of techniques to learn, but in this fashion they're surprisingly well documented. Using these techniques you can find other exploits if you're creative.
an experienced programmer can simply read through the code and see edge cases that migt be explotable. often decompiled/asembled code and use debuggers.
