Depends on what you mean by "run a different program". You can't really replace WindowServer on macOS, as it is pretty much macOS itself. macOS is not really as modular as Linux or even Windows. There are things that hook into it to give extra behavior (Helium, Afloat, Magnet, Amethyst, chunkwm, several others in the AppStore), but replacing it is not really practical, I believe.
> You can't really replace WindowServer on macOS, as it is pretty much macOS itself.
I find this very foreign. As if a unix system didn't let you change the shell!
This is sad to hear, because I'm a happy user of a headless macOS (not set up by me), to which I ssh almost daily for testing purposes, and it is a fairly decent unix system. It has a few idiosyncrasies, sure, but those help to make your pipelines more portable. I always supposed that the GUI would be just a regular program running on top of that unix.