Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

>Your site has been assigned special crawl rate settings. You will not be able to change the crawl rate.

That's the messed up part. I guess the question is, does robots.txt override that or not? If it does, fine. If not, all you need to do is make a few "google ignores robots.txt" posts and the problem solves itself.



Googlebot doesn't use the crawl delay line in robots.txt if that's what you're asking. I guess theoretically it's because you can normally change it in the webmasters interface.

It's not really ignoring robots.txt either, as crawl delay is not an 'official' setting




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: