Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I have always found crawling harder personally. At least with indexing you can switch to batch processing, or split the index into real-time and stale portions to give that fresh feel. Just my personal opinion though. I have never had the chance to index billions of pages. Several hundred million is about as far as I have gone.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: