Isn't it actually against Google's policy to show different content to the crawler than to actual users? I might be misremembering, but I thought at one point this practice could get you delisted.
You can't search for stuff on your company's intranet, can you? Or your private bank statement?
If someone wants to make something private, it's definitely their right to do so. But they should not attempt to mislead me by offering it as a search result of the public web.
Why not? The information is available to me if I pay. If it's not indexed, how am I supposed to know it's even there? And if I'm already paying for access to a site, then I certainly want it to show up in Google search results.
Uhm, you go to a web site you subscribe to, and look for it there.
Google is not your private search engine. There is a lot of paid content that Google cannot index and thus search for you: that's a natural order for me.
Now, I am fine if Google shows those results for you. But it also shows it for me, and I ain't a subscriber.
Now, I understand Google is not what it used to be, so it's simply a bad search engine for me. Writing is on the wall, though: they are losing the geek out there, and while they won't be gone anytime soon, they are Microsoft of the 90s (living off of their monopoly).
I'd remind you that Google isn't your private search engine either. It's an index of public-facing websites, some with content that requires payment. The job of filtering and discernment is yours.
Perhaps there are search engines that do some of that legwork and you'd prefer to use those, or you could use some of the advanced search techniques available on Google. But suggesting they remove any listings of subscriber-only material seems impractical -- also not helpful to the many people who discover content through search results.
I agree with your last comment:
they are Microsoft of the 90s (living off of their monopoly)
Oh, most definitely they are not my search engine either (and I am not talking about any absolute right or wrong, just what I consider better).
It's just that it shouldn't be presented different content than what I can see browsing to the page myself: if both Google and me can only see abstracts, that's fine to me too.
If not, I am happy to use paywall-avoiding sites so publishers would reduce power Google has over them sooner.
Basically, it is my initiative to drive behaviour that I want in the market: either offer content for free, or restrict commercial search engines too.
Are you comparing paid/commercial content to private data that requires authentication? They might both require a login, but c'mon man. Not the same at all. Of course Google doesn't index my company's private intranet. But yes, they do index content that's available to the public but requires a subscription.
They are both behind a login for all practical purposes, right?
If you want to see search results on stuff only you have access to, that should be a special feature by Google: you let it log in using your credentials and crawl away.
I don't have access to it, so I'd rather get results that I care about. This is SEO at its worst, masking results I do want to artificially inflate their ranking.
Maybe I am not looking for a NYT article on it, but instead a wikipedia page. Or a university's history department coverage. Or an actual scientific study. Guess which one will rank higher today while having less objective information?
Now guess why that is? Artifical SEO for paywalled content.
Call me a cynic but they probably just want you to pay for their journalism. Looks like a cool product, and it's also probably piracy.