If you aren't part of the university then Oxford and Cambridge aren't great. I think the pressures from tourism force them to isolate themselves. They are expensive as well.

"Pursuing your own intellectual thing by yourself will be incredibly tough and lonely."

Yes.