TBH if you're looking to do something serious & consumer facing you may want to look into paying for Google's Custom Search API, or if your data set differs dramatically in shape from webpages, hiring a really good data scientist and rolling your own:
I'm an ex-Googler who's now working fairly extensively with ElasticSearch. While I've been impressed at how easy it is to get started, all the published academic ranking algorithms are utter crap compared to what Google has. Okapi BM25 is mentioned as the state-of-the-art up-thread; it was developed in the 80s, long before Google was born, and most of the advances in search since the 90s have happened behind closed corporate doors inside Google, Yahoo, and Bing.
This is solid advice. At my last company we had SOLR. I think we would have even qualified for what most people would call a "fair" use case of a search stack. Millions of items in the catalog, and millions of users to go along with it. Lucene is light years faster than a SQL engine with such requirements for search.
What is it not better at though? Pretty much everything else. GIS, Cross joins, parent child indexing, etc. Postgres can do it all too with a little digging.
The thing neither of them has? Semantic search. If I ever hit that scale again at another company, the jump won't be from SQL to Lucene, but from SQL to Google.
You can do semantic search with Solr and ES, it just isn't built-in. But it has support for stemming and synonyms which are the building blocks. You have create the synonyms file yourself with word2vec or a similar algorithm.
Has Google and/or Microsoft made significant advances in keyword relevance algorithms specifically? Google is a combination of keyword relevance + reputation (PageRank) + semantics (word2vec) + AI + other stuff when combined is the best search engine in world. Is the keyword relevance part really that much better than BM25?
This is a great suggestion. If site requires SEO why bother rolling another search engine. Everyone is just going to complain that it isn't google anyways.
https://developers.google.com/custom-search/
I'm an ex-Googler who's now working fairly extensively with ElasticSearch. While I've been impressed at how easy it is to get started, all the published academic ranking algorithms are utter crap compared to what Google has. Okapi BM25 is mentioned as the state-of-the-art up-thread; it was developed in the 80s, long before Google was born, and most of the advances in search since the 90s have happened behind closed corporate doors inside Google, Yahoo, and Bing.