Soon, quick and dirty solution: User-Agent: \* Disallow: / | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

DataDaemon on May 26, 2025 | parent | context | favorite | on: Google is burying the web alive

Soon, quick and dirty solution: User-Agent: * Disallow: /

cr125rider on May 26, 2025 [–]

So then the internet police know to zoom up the fiber line and arrest the user-agent for not following robots.txt!

Consider applying for YC's Fall 2026 batch! Applications are open till July 27.
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact