It's really a confluence of factors. There are even elements rooted from the specification itself that contribute, for example requiring many round trips in network interactions with a lot of different servers; requiring large amounts of random disk access to implement core features... These things tend to shape the experience born out of client and server implementations even after a fair amount of performance work. I hope that Matrix-org makes an effort to consolidate the specification into something more streamlined for any upcoming versions.