Privileged memory access only happens in meltdown, spectre is much more difficult to prevent (seems impossible short of killing speculation altogether) but can only access arbitrarily in its own address space.
Spectre is simple to prevent: just don't modify the cache while speculating and the CPU can still do all the same branch prediction and speculative execution pipelines that exist today.
It doesn't even have to be that drastic, the CPU can still fetch any memory that's not calculated. Both sides of a non-computed jump for instance, or the next line of an array that's being scanned in order.
Some memory references would be sequential instead of overlapped, but it wouldn't be the end of the world.
What I'm saying is that speculative execution without modifying the cache is safe from a Spectre reading arbitrary memory, the main concern.
So start from there. That's not a terrible starting position, with all of today's predictive logic and speculative pipelines working at full speed on the fastest codes and partially working as codes use more uncached memory accesses.
Then you add back in things like speculatively prefetching code and global variables (addresses hardcoded in the instruction not calculated) and you're nearly back to 2017 performance. The problem is widespread and severe, but we're not going back to some dark ages of computing.
