There's more to it - X.Org is based on lowest-common-denominator code from early days of X11, and the internal driver system despite upgrades is a bit lacking.
There's glamor, but AFAIK it's not as tested as it should, and is still shoehorned into old model.
An example of not following the old model is Xsgi, which was (hw) compositing and quite ingenious in many ways.
Many distros do this already. Both for graphics drivers (xf86-video-modesetting) and input drivers (xorg-input-libinput)