You don't have to worry about that if you use GTK or SDL. They support both. The other really important thing in the age of console ports is support for game controllers, and that isn't handled by the display server at all. It has a separate API, which SDL provides a nice high-level interface for.