I would recommend reading the issue I linked for why it's necessary. JS needs special access to resources that can't be provided without proxying.

You can see this yourself in a DASH player[0] and testing with e.g. https://invidio.us/api/manifest/dash/id/CvFH_6DNRCY. It will fail with

> Access to XMLHttpRequest at 'https://r5---sn-ab5l6nzy.googlevideo.com/videoplayback' from origin 'http://reference.dashif.org' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

0. http://reference.dashif.org/dash.js/nightly/samples/dash-if-...

The most likely reason for the error is badly decoded signature. YT uses &sig as a "crypto" access token (lasting couple of hours last time I checked). I have zero trouble playing dash videos in standalone mplayer (after manually extracting two video audio sources and decoding proper signatures).

Only browsers have this restriction.

The issue I linked above doesn't appear to explain this. I would instead recommend reading this[0] comment.

0. https://github.com/omarroth/invidious/issues/177#issuecommen...

Ok, so CORS prevents from manual muxing/proper DASH player playback. How about two separate video audio tags with javascript synchronized playback/seeking?