The simplest and most effective trick is to have a link that is not visible to humans but only to crawlers. When an IP ends up on that link you know it's a crawler. You can then proceed to block it. A honeypot in other words.
Typically display none will do but you can also have white on white text with no tabindex or off screen absolute positioning, etc.
This is harder to do than you think. An industrious developer who works on these things will quickly notice that their distributed crawler has started failing, look through logs and ultimately identify the problem. Then they'll switch to another set of IPs and continue, this time without requesting that link. The other issue is that you need to set an explicit rule that is aware of how each and every API endpoint should be accessed, and whether or not it should logically be directly accessible.
I say this as someone who has had to combat this specific technique - I'd suggest that if you believe it works, it's probably because you saw obvious scraping activity stop when you did it, but you were never aware of the more professional scraping that adapted to it or was never caught by it in the first place.
Whenever I've scraped a website, I am extremely careful not to request more pages than I need. A better method for blocking them is to flag requests for resources that do not proceed in a logical manner. For example, if you have an API endpoint that displays the information scrapers want, that endpoint should have a specific "route" through the user interface. If you find requests directly to that resource without first proceeding through the typical UI flow, that is more accurate for identifying a scraper.
This is still not foolproof, because the scraper can just script requests to the requires series of pages in order. But it's a good start for getting rid of most scrapers. The most effective method for getting rid of scrapers is IP agnostic behavior analysis, because it can catch e.g. scrapers trying to parallelize requests that increment across a proxy farm or requests that don't obey typical behavior constraints in the UI.
Could you have the content of the page explain that its a bot trap (so screen reader users know its not worth visiting) and only block if, say, an IP visits it multiple times in a short window?
Wouldn't block specialised scrapers as they would know to avoid that URL/link (though they'd probably work it out anyway), but would still limit more broad crawlers.
I do something similar to this on PPC campaigns, but not for the purpose of blocking IPs. Bots tend to come in waves from a given site. If more than X% of clients from a given referrer wind up on the fake link (or other techniques show that the clients are bots), my code can automatically suspend the display of ads on the offending site for a period of time. If it continually happens, then the site's owner is the likely culprit, so I have a threshold at which that site is automatically and permanently blacklisted from all of my campaigns.
I have found that advanced bot mitigation is the single most significant determining factor of PPC ROI, which is a sad commentary on the current state of the paid advertising ecosystem.
Depends entirely on the niche and traffic source. I have abandoned some niches because even with automated, realtime suspension of campaigns/referrers, there were so many bots (more than 50%) that I couldn't make the niche profitable. There are two primary sources for bots that click on PPC ads: competitors looking to drain budgets, and site owners looking to profit by having bots click on their ads. The latter is the easiest to defend against, since you can simply blacklist their site(s), plus any other sites that are likely on the same server (fortunately, most criminals are pretty lazy/cheap in this regard). I know one marketer that auto-blacklists all domains that have private Whois information after 1 click, and he has done very well with that.
To answer your question, based on my personal experience, I'd say on average, for display campaigns (specifically not referring to Google search ads, which tend to have a lower percentage of bot traffic - while Bing is the Wild Wild West)...I'd say overall it's somewhere in the 30% neighborhood. Not all of those have malicious intent, but in PPC, every non-human click on your ads is malicious.
Regardless of niche, realtime bot mitigation is probably the best competitive advantage that one can have in the PPC arena.
That sounds like a great trick, but I'd want a solution against someone getting large swathes of IP's banned, because the bot crawled my site and posted the link somewhere.
Determining link visibility is a standard crawler thing. So you'd need to be way more clever than that to succeed against someone who is moderately clueful.
Typically display none will do but you can also have white on white text with no tabindex or off screen absolute positioning, etc.