To me, mining has always been about having resources, to buy and maintain hardware, server space, electricity, etc.

What’s the real difference, other than lag time for PoW?

Mining costs money and those with more money can have more miners and more power over the rules.

Is it actually true that stakers can collude to change the consensus rules of the Ethereum network?

No, just like bitcoin you need the cooperation of the community/users too

Miners/stakers do not make the rules and never have.

Except, that is not how PoS works.