Hacker News new | past | comments | ask | show | jobs | submit login

Websites like WSJ manage to get listed high in the Google rankings by presenting the actual article, instead of a paywall, to the Googlebot and to requests with Google in the Referer field (as I understand it). I gather that they do the same thing to the Archive scraper. As far as I'm concerned, that's a cheap trick the website performs, and using a cheap trick to get around it seems fine to me.



Consider applying for YC's Summer 2025 batch! Applications are open till May 13

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: