You can start by learning JVM bytecodes [0], and play around with Jasmine[1].

Even today many cool performance tricks require understanding the bytecodes of the underlying language runtimes.

[0] - https://docs.oracle.com/javase/specs/

[1] - https://jasmin.sourceforge.net/