While it doesn't look like it's true anymore (http://www.oracle.com/technetwork/java/hotspotfaq-138619.htm...), I sure seem to remember the Hotspot JVM having an option for 32 bit pointers for ... code, I think it was. Since few apps would need more than 4 GiB of code. And as you note, as the FAQ notes, this doesn't change much for most programming.
If your heap is under 32G Java uses pointer arithmetic to fit pointers into 32 bits. "UseCompressedOops" is your google term. The reduction in cache misses offsets a lot of the overhead.
