Hacker News new | past | comments | ask | show | jobs | submit login

https://web.archive.org/web/*/snopes.com

> Sorry.

> This URL has been excluded from the Wayback Machine.

They also do not exclude the archive.org bot in https://www.snopes.com/robots.txt




That only shows that it's excluded, not for what reason. In 2017 Internet Archive announced it will start to ignore robots.txt in the future. When I tried to archive random facebook page (it was not allowed in robots.txt), it archived it happily. Afaik current way to exclude you site requires contacting info@archive.org and proving that the site is your.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: