It was 20 years ago, I tried running GIMP over remote X on a 100base-T LAN. It was pretty much un-useable. Granted modern day Gigabit networks might have made it more palatable, but I suspect with all the modern UI toolkit assumptions about the X server being on the same machine, the end result is that it will still be worse than an optimized VNC.
As a comparison, windows remote desktop on the same network was pretty snappy at the time.
VNC or something similar might be a better choice