That's really where ActivityPub starts to rub me the wrong way. Server admins really need moderation power since everything is hosted on their hardware, but it also is a poison pill for decentralization.

I can host my own server and make my own rules, but every other admin can just ban my instance.