Google indexers respect robots.txt, so there goes the irony. | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		avip on June 1, 2018 \| parent \| context \| favorite \| on: Bypassing website anti-scraping protections Google indexers respect robots.txt, so there goes the irony.

p49k on June 2, 2018 | [–]

They also provide something of value to the operators of the websites they scrape, namely search traffic.

pbhjpbhj on June 1, 2018 | [–]

I've heard that they [sometimes] visit but don't index, is that true?

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact