So what I did was have setup a proxy server in cloudflare workers.
proxy.yourdomain.com?url=websitetocrawl
Now in the worker, I replaced all the external resources of a html (CSS URLs, scripts) via regex to also go through the proxy.
Since the proxy is under the same cross-origin you're able to do whatever you want. And even if it was a different domain, window.postMessage has a domain argument that allows you communicate explicitly with the parent window.
So what I did was have setup a proxy server in cloudflare workers.
proxy.yourdomain.com?url=websitetocrawl
Now in the worker, I replaced all the external resources of a html (CSS URLs, scripts) via regex to also go through the proxy.
Since the proxy is under the same cross-origin you're able to do whatever you want. And even if it was a different domain, window.postMessage has a domain argument that allows you communicate explicitly with the parent window.