How are you this naive? Do you really think scrapers give a damn about your robo... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		Sohcahtoa82 10 days ago \| parent \| context \| favorite \| on: Cloudflare outage on December 5, 2025 How are you this naive? Do you really think scrapers give a damn about your robots.txt?

cpncrunch 10 days ago [–]

The legitimate ones do, which is what I was referring to. Obviously there are bastard ones as well.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact