Indexing isn't the hard part. Ranking is. For any given query, there are probably millions of web pages that match. How do you pick the best ones to show the user? How do you do that even when most people are really bad at expressing what they want?
Even if all websites were acting in good faith, this would be a really hard problem. Now add in the fact that there's an $80 billion industry devoted to gaming search results, and it suddenly becomes much harder.
Would it be impossible to build a search engine as good as Google? No. But you could very well spend billions of dollars to match or even slightly surpass Google's performance and still lose. Why? Because unless it's clear and obvious that you're superior to Google, people are still going to use Google because that's what they're used to.
That's why the smart players in the space like DDG aren't competing head-to-head in search, and are instead focusing on areas, like privacy, where Google can't compete. As for the others, I suggest you try Googling "cuil" some time.
Then Google’s ranking algo must be pretty bad because all I have is a first page full of bullshit Advertorial articles when I’m looking for serious stuff.
(Maybe that’s what the average users are looking for, though.)
Even if all websites were acting in good faith, this would be a really hard problem. Now add in the fact that there's an $80 billion industry devoted to gaming search results, and it suddenly becomes much harder.
Would it be impossible to build a search engine as good as Google? No. But you could very well spend billions of dollars to match or even slightly surpass Google's performance and still lose. Why? Because unless it's clear and obvious that you're superior to Google, people are still going to use Google because that's what they're used to.
That's why the smart players in the space like DDG aren't competing head-to-head in search, and are instead focusing on areas, like privacy, where Google can't compete. As for the others, I suggest you try Googling "cuil" some time.