In my case a cache could be something remote, having any calls directly to that will slow down unit tests a lot, having one or two calls to it unit testing the cache implementation it self is OK, but having every method which makes use of it make those calls isn't going to work.

In reply to your original question, its not extremely bad if you can't unit test that class alone , but in this case you wouldn't be able to unit test anything which uses it either.