The scary thing is that without mitigations, it is shockingly easy to pull off. WebKit engineers made multiple fully working exploits internally and we would be hard pressed to do that for other kinds of vulnerabilities. Building full exploits out of your run of the mill use-after-free bug is much harder and requires specific expertise.
Not that Spectre attacks are going to be easy to pull off, but it really makes you reconsider everything that you thought you could take for granted.