While that's true, you really only need to blast out RGB data to get an image on screen. Most of what you are talking about is layered on top and optional.

I did a tiny HDMI implementation in an FPGA for a project, the TMDS implementation was what took the longest.