There is a very, very long history of that being a bad idea. There are some situations (commodity exchanges) it’s necessary, but they are heavily regulated and audited for that reason. They also have non trivial storage costs.
Gold depositories have the same issues as crypto exchanges have typically had - it’s almost inevitable that someone absconds with the actual gold without telling anyone, or ‘prints’ extra without backing assets, etc. and until someone does an audit or everyone tries to withdraw their gold/tokens/whatever, no one knows.
How you can transfer bitcoin between exactly two people with zero outside involvement from other entities? I suppose you might be able to have a bunch of pre-filled wallets of various amounts that you can transfer via thumb drive, but in a practical sense your transaction is going over various networks and through other machines.
In order for a balance to be transferred on the chain, it MUST be included in a mined block (the longest chain one/aka the primary blockchain).
So there is always a middleman, unless you’re on your own chain. But it’s distributed, no one can reliably pick who is going to be the miner (or always be the miner) because of the POW algorithm, and everyone can trust the outcome because of the way everything is structured. There are very strong incentives for everyone to try to mine, for instance, and the more mining power there is, the more difficult it is for anyone to have a controlling interest or monopoly on mining, barring special economies of scale in mining hardware, anyway. (Which do exist, but not enough to make it doable to get a monopoly right now)
Worst case, the miners can ignore your particular transaction indefinitely, but the counterparty can see that too, and it would require active collusion from a very large percentage of miners to even attempt that.
I thought of a random analogy here - the block chain miners are the equivalent of a common carrier parcel service here (albeit with far stronger guarantees against tampering).
So it’s not a middleman like a bank or a payment processor. Rather a bulk processor who moves a bunch of opaque chunks somewhere for money without any particular regard for who or what as long as they get paid for it.
