The difference between X and Wayland is way more than needing window-manager integration. You are forgetting that keyboard and mouse inputs run through Xorg so the problem goes way deeper.
You don't have to worry about that if you use GTK or SDL. They support both. The other really important thing in the age of console ports is support for game controllers, and that isn't handled by the display server at all. It has a separate API, which SDL provides a nice high-level interface for.
