Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

You're at least partly right about sites being crawled differently depending on popularity. I think the factors may not be limited to popularity alone, but we see this behavior documented in the crawling rate documentation Google provides its users/clients so there is no reason it couldn't apply to other "expensive" actions their crawlers do.

How this might work with the most popular sites out there?

We see it in on-page answers that provide extracts of pages with the answers to questions asked in search phrases that include a reference to the document they were sourced from.

Matt Cutts used to qualify sites like Wikipedia as "reputable" to the eyes of the search engine.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: