Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I've seen this on a site that I built for a freelance client a couple years ago. At first they only indexed the homepage and some sporadic articles. Eventually one specific article was linked on an .edu website, and now that article is reliably indexed as well.

If I manually request that they index a page, it always succeeds and shows up on Google within a few days, but the page gets pruned from the index within a month or two.

The weird thing is that within the past couple years I've also seen unwanted indexing, e.g. low-spec staging servers getting wrecked by Google crawler traffic. I don't think the "no automatic indexing" treatment is standard for every site, so there must be something that triggers it, but I've spent a long time unsuccessfully trying to pin down the cause.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: