If you were able to index the same amount of content you'd have a damn good alternative. And that's the level of this experiment.

It is not trivial at all. That is a huge amount of scraping.

Any machine that can do that will be a similar spec to what you need for serving queries. Not as fast as google does it, but a good amount of them.

It is trivial compared to the task of translating a user query into search results.

It's akin to saying the magic behind OpenGPT is the dataset.

Keyword search is not that hard. Pagerank is not that hard.

The comparable goal to the article is to be a search engine, not to fight google for best results.

> Keyword search is not that hard. Pagerank is not that hard.

George Hotz, is that you ?

Keyword search and pagerank had working solutions decades ago.

Hotz was trying to make a car controller that had never been done before, by himself, and then he wanted to """improve""" search with no explanation of what that meant that I saw.

I think if he was tasked with taking twitter from no search to "has a search" he probably could have managed it. A team of five people definitely could have managed it.

I think you severely underestimate the effort search engines have to put to filter out all the SEO garbage and come up with some sensible results

I don't underestimate the effort, I claim that it's not strictly necessary to have more than a couple people working on it for a minimum viable product, and less for a beta.

What? The OP covers much more than "just shuffling tweet sized buffers to network offload cards"