Probably, but it is a good point: premature optimization is easy and fun because you're doing it while writing the code and it's already in your head. Optimizing a large, working, congealed codebase that you didn't write is often hard and feels risky, even with good test coverage. Very local optimizations, fine, but those requiring refactoring? Less likely to happen.
Maybe there's a golden opportunity somewhere inbetween where optimization is not premature but also not painful to do.
Though it does ultimately boil down to no “fearless” refactoring being possible in real life. Unfortunately we seldom have such systems and test coverage/quality.
Maybe there's a golden opportunity somewhere inbetween where optimization is not premature but also not painful to do.