"I believe Google Adwords killed the web because it incentives SEO optimized garbage", then proceeds stripping publishers of their ability to get paid for quality non ads-financed content. The motivation makes little sense to me.
It's 100% their own fault. They could block the Google spider from getting the full content for free just like any other non-paying rando, and just serve it the same teaser content.
So the argument is, that ‘because Google Bot can read the content for free, then I should be allowed to do so too’? This seems like a pretty week excuse for bypassing a paywall. I think the reason we have annoyances like ‘please sign up for our news letters’ is that news sites are struggling to make money. A service like 12ft.io is only making the situation worse.
Suppose someone puts free stuff on the sidewalk and says, "provided free-of-charge to clowns; everyone else $5".
Then if I put on a clown nose and take the stuff, I'm completely in the right.
If the purveyor does not agree then they can:
1. Not put the stuff on the public sidewalk, but offer it inside their store only.
2. Have the cashier, validate that the the customer holds a clowning license issued by the state and is a member in good standing of some clowning association.
The argument is not at all anyone is entitled to the stuff no matter what, even in the face of protective measures like (1) and (2). The argument is that if some server discriminates what information it put out to the unauthenticated public according to some superficial browser client indications that are easily donned by any client, then it's perfectly fair game for anyone to manipulate those indications in order to be served any/all thus available versions of the content.
If that doesn't reflect the purveyor's intent, then they have a clear technological avenue for securing the protected content to authenticated users only.
I see it in reverse: the reason some news sites struggle to make money is their belief in a broken business model that's perpetuated by mega-corps like Google. The rules of the game are rigged; house always wins. The only way to win is to not play.
I wouldn't say this "justifies" a service like 12ft.io, but such services are inevitable.
> So the argument is, that ‘because Google Bot can read the content for free, then I should be allowed to do so too’? This seems like a pretty week excuse for bypassing a paywall. I think the reason we have annoyances like ‘please sign up for our news letters’ is that news sites are struggling to make money. A service like 12ft.io is only making the situation worse.
Yes, because I will subscribe to your site because I want to read one article on it. Next, supermarket wants to sell me a caw instead of carton of milk I came for.
It’s not my job as an individual to solve systemic issues. I’m not willing to pay for news. Usually I don’t bypass paywalls, but when I do I don’t feel bad about it
Crawled paywall content is also SEO optimized garbage. The argument I assume is that paywalls that can’t be crawled (and thus can’t be SEO optimized) are therefore better.
Not that I particularly agree. I just don’t want to see ads or pay for content.
Shameless plug (Founder here), but I’d highly recommend checking out [redacted] as an alternative!
We work on a number of major publications that 12ft does not, including NYT, WSJ, and several others. We have a full list of supported publications here: [link redacted]
Stopping the page load halfway through defeats the NYT paywall, last I remember. Either that or switching to Firefox reader mode, or combination of both.
Edit: Just retested. Went to nytimes.com, clicked on a story, got the paywall. Turned on FF reader mode and reloaded. Blam, there is the whole article.
This is so very easy to do correctly in the back end, such that without an authenticated session from a paying user, the full content simply does not leave the server.
I think they are leaving it easily bypassable on purpose. They know that some "piracy" is beneficial, like it is in music. The paywall has to have finely tuneed "difficulty dial", such that they get revenue, without throttling the exposure (which will negatively act on the same revenue).
A "Fort Knox" paywall is probably not revenue-maximizing, that is to say.
> i wonder how you see this playing out in the future
In the near future, we're planning to expand into more general adblocking; we're currently focused on growth & there's still a lot of internal discussion happening, but we're most seriously considering a Spotify-style or micropayments model... very early days though - we just officially launched last week!
It's not immediately clear from the page, but this app performs MITM on your local machine, requires sudo, and also updates automatically using super user privileges.
I don't really like that Apple has such privileges on my machine, but trusting paywall blocker with them seems absurd.
That's all true - we mention these directly on our FAQ ([redacted]) -- many are issues we plan to address as soon as possible -- the application 100% works in terms of functionality, but is definitely still under heavy development!
> Note that the first time you run Incoggo, you will need to enter your computer password.
That's a very understated way to say "we run stuff on your computer that require root privileges". It is explained later but there's no such thing as too many warnings about running 3rd party code under root on my computer.
> If you’re a very security-minded individual (or you use your computer for very sensitive tasks),
Who doesn't? Like, accessing my bank account? Accessing my email, where my bank account would sent 2FA/password reset? Uploading my ID photo to one of the dozens sites now that require that? I'm not sure we know each other well enough so I'd entrust you access to all that. Not even if that allows me to read Vanity Fair. I guess I'm one of those "very" individuals?
> Incoggo adds a file to your system’s sudoers.d folder that whitelists specific commands from requiring a sudo password to perform
Erm, does that mean any other tool would be free to use those commands under root now? Probably not the best idea.
> Incoggo also installs a trusted root certificate in your system keystore
Do I understand right that this gives it full untraceable MITM capability to any site I visit? I remember the government of Kazakhstan tried to do that to their citizens. They did it in a stupid way. They should have promised all citizens free access to Wall Street Journal.
I don't feel comfortable installing your application on the desktop. Why it should be a desktop app? What exactly is it doing and why can't it be just an extension for the browser?
There is this browser extension called Bypass Paywalls by magnolia1234. Looks like I'm sticking with it for now.
For folks with an Amex Platinum, consider using the new $240 digital entertainment credit for a free NYT subscription. [1] I got tired of trying to dodge it, and none of the other options were compelling. Plus I'm led to believe democracy dies in darkness. Or is that WaPo?
The ad-popup actually mentions that signing up for a free account with them bypasses the restriction, I'm not sure to what degree but I have noticed it works whenever I log in.
Furthermore, your AmEx option is not free, and for those who would rather not support an organization like the NYT, is not an option either. For those who don't even want a free account with NYT either, Private Browsing mode in Safari on iOS with AdGuard works every time.
It's included in the annual fee of an Amex Platinum so it is no additional cost if you elect this option. It covers the cost of an NYT subscription completely. This is a new credit, and not one that factors into my cost-benefit analysis on keeping the card. Probably true of most folks. I mention it because I suspect many cardholders are in the same boat and do not know about this.
You are correct that it is not "free" but rather "included at no additional charge" should you choose to elect it.
ah ok, I just meant more not free in the sense that I assume the digital credit is probably already spoken for between; Netflix, Spotify, Disney, Audible, Amazon Prime, etc etc.
Unless I'm mistaken they only support Peacock, Audible, SiriusXM, and The New York Times! Were it to include the ones you mention, I suspect I'd feel quite differently haha.
I'd never pay money for reading NYT (or learning why nobody bothered to turn the lights on for the poor dying democracy), I like the idea per se (actually, Audible sounds as a better option).
Or just use a client that does not have a JavaScript engine.
From the perspective of this text-only browser user, there is no "paywall" there are just complaints from people who are using browsers that run JavaScript.
From where I sit, it looks like JavaScript is the problem, not "paywalls". Specifically, browsers that run it automatically and web developers that use it irresponsibly. It seems the JS interpreter is not in the browser for the user to write and run her own scripts to improve her UX but for website operators to run their own scripts, without any prior selection or approval from users, in order to benefit their own interests.
Allow me to explain. A true access control would require an approved source IP, a password or something similar. Thus, this so-called "paywall" is not an access control to exclude non-subscribers. It is more like some sort of deliberate annoyance, forced hassle as a weasel-like form of "marketing". It is implemented using JavaScript. The question to ask is whether NYT could and would carry out the same sort of intentional annoyance without the use of JavaScript. They probably could, yet the fact is they don't. And so we are left to wonder, perhaps they wouldn't, even if they could. JavaScript clearly enables, and seems to me may actually inspire, shenanigans like this "paywall". (Why annoy non-subscribers. "Because we can. It's easy.") Take JavaScript away from web developers and what would happen. Would they reimplement this "paywall" in another way, without using JavaScript. How easy would that be, compared to using JavaScript.
What we have now is NYT 1. collecting data on non-subscribers, 2. showing them ads (thus potentially producing revenue from non-subscribers) and 3. annoying non-subscribers (after collecting data on them and showing ads) to try to get them to subscribe. #3 is a new low.
Turn off JavaScript and this does not work. Or use a client that does not support JavaScript and one never even knows this is happening to other users. I only know about a NYT "paywall" because I read about it. I have never experienced myself. I can read the entire online edition fast, in one go. Apparently the only reason is because I am not using a browser released by an entity that collects data on users, a browser that runs other peoples' code, more or less indiscriminantly, by default. This leads me to have a different perspective than the users of popular browsers. The popular browsers and JavaScript seem like they are the problem, or at least a fundamental part.
For nyt a simple trick that always works for me at least on iOS device is to open it in private mode in safari and set the website settings to enable “use reader automatically”. In fact this trick works for many behind paywall but not all like ft.com.
For sites that this and other methods don't work on like NYTimes, WSJ, and WaPo, I use Incoggo (https://www.incoggo.com). I saw them on HN a few weeks ago and haven't really had issues. They're only out on Mac though.
The method you used (Whois for the IP address) doesn't tell you where it's hosted, just the address the IP is registered to.
It's being announced by Amazon, has nearly 100k other domains associated with it [0] and a <3ms ping for me in Europe so I think this is just CloudFront.
I also have an address at 340 S Lemon Ave haha. I guess we're neighbors.
Speaking of, I highly recommend virtualpostmail. I've used them as my personal mailing address for years now. I did a USPS change of address once, and now any time I move, I no longer have to update my address with anyone. Also, I get PDFs of my letters in my inbox, and can reship them to wherever I happen to be that week, even if I'm away from home.
So happy to see other users. I was an expat for 3 years (including during Covid) and they sent me all my care packages at super reasonable rates. I’m keeping them now that I’m back at home so I don’t have to give my home address to everybody. Highly recommended for all!
Extra protip - I’d you’re traveling, they can ship to any FedEx office store and “hold for pickup” and most full-service hotels will receive packages for you, even a day or two in advance.
I know where you are coming from, but ultimately its another blow against quality journalism.
It started with ad revenue moving from media to tech, and now the hacking community makes it harder for them to sell their products on the Internet.
The deck is stacked against media - think about how much they need to to exactly right in order to survive the Internet era
Or, they could focus on actually doing quality journalism, picking their niche areas of expertise and producing interesting material, rather than trying to win at "most clicks first."
Then, importantly, set expectations for reach and profits appropriately and use relevant advertising to their services/topics which also has value, not stupid algorithm/'targeted' advertising.
It's not easy, but working to actually build trust and value is far better than joining the race to the bottom.
The mass media as it exists today was made possible by a combination of certain technological and societal advancement. If further evolution on either or both counts makes the business model underlying it non-viable - that's too bad, but there's no right to industry survival.
This is great, because when I search something, can see a sentence in the search results relevant to my search, then click the link and cannot see the portion of the page relevant to the search, I can go to 12ft.io.
The showing of a special site to the search bot to add noise to search results and waste my time, I happily bypass with a clear conscience.
It’s inconvenient but why do we have the expectation that people will give away their work for free?
Journalism is not easy and journalists currently don’t have any other way to collect income other than publishing their work through a medium.
For some of us we afford to give away work for free because our universities (work=scientific publications) or our employer (work=open source software) pay for for it to be free for their own reasons and because they have the funding to do so.
We don’t want journalism to have questionable sources of funding, just so that we click open the stories more easily.
We want journalism to be profitable by direct funding from the consumers.
Ads was the unfortunate middle ground.
But taking Ads away will only push media towards getting funded by forces that exclusively want to control the media.
TLDR: pay your content creators even if it seems tedious to you
I pay for the content I think is important that it exists, but the web working is also important. If they are giving Google the content I see no issue with using that against them. It's up to them, either the content is private or public it can't be both.
This doesn't seem to work on Firefox mobile for me. I was trying with nyt. Similar services like outline.com also have a difficult time with nyt, so unsurprising but would love a workaround.
Do sites still show google the full articles? Why would they want that, they should be relevant enough with just the first few paragraphs and not need the complexity.
I’ve not tried this service yet, I’m curious though.
However, the easiest trick to bypass paywalls in my opinion is to switch to ‘reader’ mode in your browser. This works for most news sites I visit, including NYT. There are exceptions, but I reader mode should be your go to first attempt if you ask me.
It’s built into every browser now, and if it doesn’t work, then check archive.org. Using these two methods alone will get you most anything you want to read.
There are many paywalls not so easy to bypass as removing a hovering modal, spoofing User-Agent and Referrer or redirecting to Wayback Machine or Google Cache.
One more point: I predict in 2022 that sites stop allowing Google to index their content via bot. Paywall sites are going to ride that business model all the way down in flames.
Do you think they'll do it manually or just not allow their pages to be indexed? That doesn't seem like something a publisher would want to do / happen as I'm sure a lot of their traffic comes from search.
It is both. Surely the business are responsible for coming up with a model. Similarly the community is responsible for setting rules so that community members have their needs met.
You can ban sandwich shops in your city, and there wont be a viable model for sandwich shops.
> You can ban sandwich shops in your city, and there wont be a viable model for sandwich shops.
No one banned sandwich shops (or paywalled newspapers) in this context, though.
The problem is that some "sandwich shops" wants to give out free samples (i.e. keeping paywall open for bots and other clients that don't run js), because they know it's good for business, and then complain that people are taking free samples.
Well... if this "handing out free samples" approach is not working out for them, it is up to them (and not everyone else) to figure out something that works for them.
If not enough people are willing to pay full-price for their (I assume) high-quality sandwhiches, and much rather source their sandwhiches from somewhere else (if there are no free samples to be had), that's not anyone's problem but their own.
You're making it seem like there is some regulamentory issue that prevents them from having/deploying a successful business model, when that's not the case: the problem is something else.
I agree that regulation was a bad example. I think a more accurate example is that consumers want high quality content, but dont want to pay and don't want advertising either.
Consumers saying come up with a business plan that gives me want for free in return for nothing are in for disappointment
I don't necessarily disagree with what you are saying. But, when the price of ignoring the problem for the consumer is "disappointment", rather than "bankrupcy", it still seems to me that it is not consumers that should be particularly worried.
As a consumer, I am more interested in avoiding my disappointment, than their bankruptcy. I also understand that there are some problems that can not be solved by business alone, some will require changing consumer attitudes.
In this case the business is proving the general public with information. Which is of great concern for all of us living in society, it has an influence on what people know and think.
Another user asked but failed to get a reply. What should they do instead?
The old business model failed when people stopped buying physical newspaper copies and this is what they came up with. It was unpopular from the start and lead to many papers struggling, so I'm sure they've thought about alternatives harder than the average social media user. I'm all ears in case you have a good idea though. Remember the goal is getting money to pay the journalists and other expenses.
You might see the other direction happen too. Surely Google understands the disappointment for their end users clicking a search result and landing on a teaser paywall article that has less content than the search snippet.
ISTR that was seen as huge leverage for a long time.
If people were hitting a hard paywall, Google tended to penalize the result. A lot of sites ended up with a looseness in the paywall basically to ensure people landing from a search result got "first page free" style visibility.
I feel like the key is to realize the difference between "news" and "editorial" content. A lot of "news" is begging for a race to the bottom-- I can get the latest Covid numbers, sportsball score, or the fact a water main burst on 12th street from 5 local TV channels or two or three newspaper-format sites. You can't really compete by providing a premium product for those things. On the other hand, the editorial content-- the op/eds, the bespoke columns-- that's where you can justify a paywall.
The more important question is why people feel the need to implement adblockers. If ads were less obnoxious and intrusive, people wouldn’t block them.
In short: current site monetization tactics are objectionable and deserve to be thwarted. Rather than huffing about people who block ads, publishers and advertisers should consider that an incentive to come up with something better.
That’s a valid point. I’ll admit I am concerned about the survival of competent journalism, and as far as paywalls go, I can’t consider them unreasonable or offensive.
Ads/trackers are a different beast entirely. Sure, they also provide funding, but it’s a model that’s rooted in obnoxiousness, invasiveness, and disrespect. That approach needs to be punished, and I think loss of revenue is entirely justifiable; even beneficial, if it incentivizes more reasonable and respectful monetization strategies.
It is possible to agree that content creators should be paid and that ads are bad. Substack, for example, allows you to pay and doesn't have ads. Ads are one business model, not the only one.
Nope, making a paywall not show content to Google bots is a (slow) way to get us out of our misery.
Basically, as long as search traffic is more important to publishers, Google will have an upper hand.
If paywalls are in our future, getting there through Google should not be.
By working around paywalls, you are asking publishers to not mess the search results or not to give Google preferential treatment (depending on your angle).
Exactly this, the "why?" section sounded weird to me. Paywalls (specifically subscriptions and paying for a service directly) are a way to not need AdWords, if those don't work then what can websites do to make back their hosting, dev, writing and all other costs than using Ads
Isn't it actually against Google's policy to show different content to the crawler than to actual users? I might be misremembering, but I thought at one point this practice could get you delisted.
You can't search for stuff on your company's intranet, can you? Or your private bank statement?
If someone wants to make something private, it's definitely their right to do so. But they should not attempt to mislead me by offering it as a search result of the public web.
Why not? The information is available to me if I pay. If it's not indexed, how am I supposed to know it's even there? And if I'm already paying for access to a site, then I certainly want it to show up in Google search results.
Uhm, you go to a web site you subscribe to, and look for it there.
Google is not your private search engine. There is a lot of paid content that Google cannot index and thus search for you: that's a natural order for me.
Now, I am fine if Google shows those results for you. But it also shows it for me, and I ain't a subscriber.
Now, I understand Google is not what it used to be, so it's simply a bad search engine for me. Writing is on the wall, though: they are losing the geek out there, and while they won't be gone anytime soon, they are Microsoft of the 90s (living off of their monopoly).
I'd remind you that Google isn't your private search engine either. It's an index of public-facing websites, some with content that requires payment. The job of filtering and discernment is yours.
Perhaps there are search engines that do some of that legwork and you'd prefer to use those, or you could use some of the advanced search techniques available on Google. But suggesting they remove any listings of subscriber-only material seems impractical -- also not helpful to the many people who discover content through search results.
I agree with your last comment:
they are Microsoft of the 90s (living off of their monopoly)
Oh, most definitely they are not my search engine either (and I am not talking about any absolute right or wrong, just what I consider better).
It's just that it shouldn't be presented different content than what I can see browsing to the page myself: if both Google and me can only see abstracts, that's fine to me too.
If not, I am happy to use paywall-avoiding sites so publishers would reduce power Google has over them sooner.
Basically, it is my initiative to drive behaviour that I want in the market: either offer content for free, or restrict commercial search engines too.
Are you comparing paid/commercial content to private data that requires authentication? They might both require a login, but c'mon man. Not the same at all. Of course Google doesn't index my company's private intranet. But yes, they do index content that's available to the public but requires a subscription.
They are both behind a login for all practical purposes, right?
If you want to see search results on stuff only you have access to, that should be a special feature by Google: you let it log in using your credentials and crawl away.
I don't have access to it, so I'd rather get results that I care about. This is SEO at its worst, masking results I do want to artificially inflate their ranking.
Maybe I am not looking for a NYT article on it, but instead a wikipedia page. Or a university's history department coverage. Or an actual scientific study. Guess which one will rank higher today while having less objective information?
Now guess why that is? Artifical SEO for paywalled content.
As much as I hated AMP this is one thing I loved about it. Click an article, get paywalled… Google the article headline and it’d take you to a paywall-free AMP version of the full article.
AMP’s one redeeming feature—a shit-free browsing experience—is, as expected, being eroded bit by bit. I’m seeing things like autoplaying videos and popovers in AMP pages now.
My go-to is readability mode, Instapaper, and turning JS off.
