You're exactly correct. This is why the browsers decreased timing resolution in javascript so that you couldn't time memory accesses accurately enough to tell if the address was cached or not.
What does that do, besides turn the exfiltration problem from an immediate one into a statistical one?
IF it can be turned into a statistical problem, it may become an infeasible attack. You'd have to run the whole attack (not just the last reading bit since that would bring it into the cache after the first read) many times to be able to ascertain the difference. Even then, the difference might be less than the noise from other processes on the system (I think 80 cycles was used in the PoC?).
Maybe there will end up being a new Jumping Around Kernal Address Space System (JAKASS - a cousin of Linux's FUKWIT patch) that periodically resets kernel ASRL to make it fully impossible.
I think this is what all sandboxes have to do: set the TSC disable flag, restrict system timer precision (make it configurable per sandbox: web servers generally don't need more than 1ms precision), make system timer report fuzzy (randomized) time. Heck, why not also make the CPU run at randomized frequency to mess with busy loop timers.
What does that do, besides turn the exfiltration problem from an immediate one into a statistical one?