So why draw the line there? Why is it ok to abstract away modern hardware, but not ok to abstract away memory management or pointers?

You can write a useful program that only uses one core. You can't write a useful program that doesn't use memory.

But you can write a great deal of useful code that never does any manual memory management.

You asked where you draw the line. My point is that there's a real difference between not using a resource (more than one core) or using it without understanding how it works, because it's hidden behind a paper curtain.

Because understanding pointers makes understanding references a lot easier.