Are the big players (minus Google since no one blocks google bot) actively taking measures to circumvent things like Cloudflare bot protection?
Bot detection is fairly sophisticated these days. No one bypasses it by accident. If they are getting around it then they are doing it intentionally (and probably dedicating a lot of resources to it). I'm pro-scraping when bots are well behaved but the circumvention of bot detection seems like a gray-ish area.
And, yes, I know about Facebook training on copyrighted books so I don't put it above these companies. I've just never seen it confirmed that they actually do it.
If you enable Cloudflare Captcha, you'll see basically no more bots, only the most persistent remain (that have an active interest in you/your content and aren't just drive-by-hits).
It's just that having the brief interception hurts your conversion rate. Might depend on industry, but we saw 20-30% drops in page views and conversions which just makes it a nuclear option when you're under attack, but not something to use just to block annoyances.
There was no hard block, just everyone getting the brief JS-captcha-thing.
The reason I'm fairly confident that it wasn't bots that were blocked is that we run quite a bit of bot-filtering in our analysis (and only track via delayed JS, which already gets rid of of 80-90% of bots; we don't care about bots viewing our pages, but we don't want them to mess up our stats) and all other metrics (conversion-rate, device + browser-distribution, country-mix) still lined up, they were just significantly lower than what expectations + search analytics said they should be.
Bot detection is fairly sophisticated these days. No one bypasses it by accident. If they are getting around it then they are doing it intentionally (and probably dedicating a lot of resources to it). I'm pro-scraping when bots are well behaved but the circumvention of bot detection seems like a gray-ish area.
And, yes, I know about Facebook training on copyrighted books so I don't put it above these companies. I've just never seen it confirmed that they actually do it.