Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
btilly
20 days ago
|
parent
|
context
|
favorite
| on:
Nepenthes is a tarpit to catch AI web crawlers
It would be more efficient for them to spin up a team to study this robots.txt thing. They've ignored that low hanging fruit, so they won't do the more sophisticated thing any time soon.
tgv
20 days ago
[–]
You can't make money out of studying robots.txt, but you can avoid costs skipping bad web sites.
xeromal
20 days ago
|
parent
[–]
Sounds like a benefit for the site owner. lol. It accomplished what they wanted.
Consider applying for YC's Spring batch! Applications are open till Feb 11.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
