Can't (AI) crawlers just use a browser user agent? TL;DR: Yes, but with some more work that the big AI companies can surely manage to implement.

    To bypass detection, a crawler would need to mimic not just the user agent, but also the full set of browser behaviors, headers, and environmental fingerprints—a much more complex task that Cloudflare’s systems are designed to counter.