I'm still not sure this is going to be very effective,
as so many of the worst offenders don't identify themselves as bots,
and often change their user agent.
Has Cloudflare said anything about identifying the bad actors?
i've mentioned this in a couple replies so maybe i'm wrong but it's up to the client to obey robots.txt. Why would they not just ignore it? Unless there's some legal consequence not complying with robots.txt then why even follow it? There's no technical enforcement of the policies in the file, it's up to the client to honor them.
> There's no technical enforcement of the policies in the file, it's up to the client to honor them.
That's incorrect. Cloudflare does in fact enforce this at a technical level. Cloudflare has been doing bot detection for years and can pretty reliably detect when bots are not following robots.txt and then block them.
