Is that on theoretical or practical grounds? I would love to learn how you would approach the challenge (no snark). I feel lots of developers miss the needed background, pour in a lot of work and then be stuck with it. How can we avoid that?
Is there some settled science, some principles and patterns in distributed security? Like, you want A, now you can only have option 1 or 2. But if you want B too, this only leaves you with option 2, provided you satisfy D too. But the combo D+B rules out any E.
No, there's no science to it at all; it's just a collective action problem. You saw it in Matrix's effort to get all their clients encrypted by default (they were hamstrung by an installed base of popular clients that didn't work that way), and again in the response to Nebuchadnezzar.
That sounds like a social problem again. What foundational materials would you recommend to read though for anyone trying to build something secure and non-centralized? It is a pity that everyone spends a lot of effort in this area, only to learn they did it wrong and having to deal with unfortunate design decisions. That is, if they are honest about it.
You can build secure and noncentralized! What you can't do is build secure and federated, where everyone lives in a shared, broadly reachable namespace comprised of independently operated instances.
I simply wouldn't build a secure group message system to begin with. It's a treacherously hard problem and the very few people who have done it well accomplished that with major UX compromises that put them at long-term disadvantage against schlock like Telegram, and survived mostly due to force-of-nature word of mouth.
If you're going to try, and you want to be rigorous about it, I'd say you need to start by reading the whole history of cryptanalyses of secure messaging systems, even systems you don't care about. Read the papers carefully and work out the attacks for yourself. It's a little like math in that you're only going to figure it out by actually working the examples yourself.
Is there some settled science, some principles and patterns in distributed security? Like, you want A, now you can only have option 1 or 2. But if you want B too, this only leaves you with option 2, provided you satisfy D too. But the combo D+B rules out any E.