It's not a "planned conspiracy", but rather experienced DOM/CSS/JS technicians have no incentive to "fix" it because the arcane nature of it keeps them in demand, busy, and well-paid. A horse farmer is unlikely to invent a car or try to perfect one.
I just don't buy that narrative. It's not my experience, and it doesn't really make sense. Most frontend devs would love a more sane ecosystem and tools. It's not like it would put them out of business... it would just make their work more pleasant.
Also, rank and file "dom diddlers" wouldn't be the ones "fixing" this problem anyway. There's a ton of economic incentive to fix it, but the problem of entrenchment is powerful when essentially the entire web is run on this mess. That's why it hasn't happened yet.
I mean, it's arcane all around. Ever tried developing for Android or iOS? Webdevs have the extra burden of "having" to support an infinite variety of form factors, aspect-ratios, input types etc. That kind of flexibility does not come for free.
It's not like these arcane technicians don't already try to "fix" things, with the plethora of JS/HTML frameworks-of-the-day to solve all use-cases (must...resist...urge to post xkcd link....)
Really, webdevs are not malevolently guarding their jobs by jealously protecting their dark web secrets like some private government contractor. They are just doing what they have to do in a highly complex field where laypeople still think a 16 year old on a weekend energy drink bender should be able to push out a finished Facebook clone for $50 on upwork.com
The borderline between those who can do something about it and those who are just "worker bees" who are or feel "stuck" with it is probably rather blurry.
Why can't a bunch of talented and experienced UI developers get together with university researchers to propose and test candidate HTTP-friendly GUI standards? (Let's limit the scope to "productivity" applications so we don't try to redo the entire Web.)
Asking others about it, UI "specialists" don't seem to care for what feels like job security reasons. That's just the vibe I get. It may be wrong, but I report it as I read it.
Those who DO care are often non-UI specialists who have to make a UI as part of the project but don't want to micromanage UI details that should be commonplace and road-tested by now being they are 30-year-old GUI idioms.
Re: a highly complex field where laypeople still think a 16 year old on a weekend energy drink bender should be able to push out a finished Facebook clone for $50...
Well there are probably some prodigies who just about could, partly because they don't have to scale up to gajillion customers just yet and don't have to support backward data/content compatibility.