That does not sound different from what Windows does. By default all programs running under the same user can access all windows of other applications (except UAC elevated ones). It's a relic from when OLE and Clipboard in Windows 3 just was (very simplified) a pointer to RAM.
The only reason it is worse with X11 is that it is an inherently networked protocol, so the same statements also apply to any remote connections you might allow to your X server. It also makes it somewhat easier to capture Xkb / Xinput events purely through API, without need for any elevation or excessive polling of the devices ("it just works").
That includes any systems you might have SSHed into with X forwarding enabled, as it automatically extends the trust there. Yes, your ssh client might try to enable X SECURITY extension (which clamps acesss to just the current window), but it is disabled by default or bypassed anyway by the users as that extension is known to crash quite a few programs.
Both are a product of their time when the prevailing approach was to trust the programs you run.
