Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

How are you this naive? Do you really think scrapers give a damn about your robots.txt?




The legitimate ones do, which is what I was referring to. Obviously there are bastard ones as well.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: