That's why many people say "Everything is an exit scam in waiting". Even your legitimate meatspace bank will cheat if it thinks the economics make sense.
You could use transparency to tell how much a given escrow agent is holding and don't use them if it approaches the value of their reputation. And/or use a system where they don't hold the funds, they merely choose to unlock a transaction to pay to A or B.
You could use transparency to tell how much a given escrow agent is holding and don't use them if it approaches the value of their reputation. And/or use a system where they don't hold the funds, they merely choose to unlock a transaction to pay to A or B.