If you study Anarchist philosophy "mutually agreeing on a set of rules etc..." is effectively the syndicalist answer to this problem. However as you point out, history seems to show that this is incredibly unstable and is easily dominated by someone breaking the rule of non-violence.
The radical monarchist-libertarians answer is to eliminate the commons, such that everything has an explicit pricing mechanism. No less problematic, or unstable.
I think in the end, there is something about humanity that does not allow for stable equilibrium over extended periods (100s of years). Not sure what.
The problem with consensus based rules is that it takes way too long to do anything. Around the turn of the century, when anarchism was still opposed to capitalism, rather than just people swinging bike locks in the name of identity politics, any kind of anarchist style organizing basically amounted to spending the whole day sitting in a giant meeting, and then leaving with nothing important decided on.
Not sure which century you're referencing - 2000s?
I concur though. The Occupy Wall Street meetings I had the luck of attending at Zuccotti Park reflected as much and nearly all of the other anarcho-syndicalist meetings I've seen do work at the snails pace of consensus.
The adherents would argue that this speed is a feature not a bug. I would generally agree with the argument that slowing down decision making is a good thing. Not sure it's a survivable strategy as a group however.
The radical monarchist-libertarians answer is to eliminate the commons, such that everything has an explicit pricing mechanism. No less problematic, or unstable.
I think in the end, there is something about humanity that does not allow for stable equilibrium over extended periods (100s of years). Not sure what.