Hi, I'm the CEO & Founder of DuckDuckGo. To be clear (since I see a ton of confusion going around about this story), when you load our search results, you are anonymous, including ads. That is, this news cycle is not about our search engine, it's about our browsers -- and, contrary to the headlines, we actually do restrict Microsoft trackers there today a great deal, and more than the major browsers do.
When most other browsers on the market talk about tracking protection they are referring to 3rd-party cookie protection e.g., blocking them) and fingerprinting protection (i.e., restricting APIS scripts can use), and our browsers impose these restrictions on all third-party tracking scripts, including those from Microsoft. We also have a lot of other web protections that also apply to Microsoft scripts (and everyone else) that most browsers don't do, including Global Privacy Control, 1st-party cookie expiration, referrer header trimming, new cookie consent handling (in our Mac beta), fire button (one-click) data clearing, and more.
What this article is talking about specifically is one web protection that the major browsers don't even attempt to do — stopping third-party tracking scripts from even loading on third-party websites. See for yourself the bottom 'tracker content blocking' section of this audit site: https://privacytests.org/ios.html (scroll to the very bottom).
This web protection is a particular challenge to get right because websites can easily break when scripts that they depend on don't load, but because it makes for better privacy and faster page loads, we've taken it on while still trying to not break sites. As a result, it is far from perfect, as we can't block all scripts do to persistent breakage issues (which we try to workaround), and domains change all the time (and we continuously crawl sites looking for new tracking domains).
Though because we're doing this protection where we can, and also offer many other unique protections (e.g., Google AMP protection, smarter HTTPS upgrading, tracking protection for other apps in Android, email protection to block trackers for emails sent to your regular inbox, etc.), users get way more privacy protection with our app than they would using other browsers. Our goal has always been and remains to provide the most privacy we can in one download. And we have a lot more planned.
The issue at hand is all of our web protections apply to Microsoft scripts on 3rd-party sites (again, this is off of DuckDuckGo.com, i.e., not related to search) except this one around full content blocking, as we are currently contractually restricted by Microsoft there. However, this limited contractual restriction (about this one web protection) is the only one we have, and we have been and are working with them as we speak to reduce or remove it.
I understand this is all rather confusing because it is a search syndication contract that is preventing us from doing a non-search thing. That's because our product is a bundle of multiple privacy protections, and this is a distribution requirement imposed on us as part of the search syndication agreement that helps us privately use Bing results to provide you with better private search results overall. While a lot of what you see on our results page privately incorporates content from other sources (including our own indexes), e.g., Wikipedia, local listings, sports, stocks, lyrics, weather, quick answers, etc., etc., we source most of our traditional links and images privately from Bing (though because of other search technology our link and image results still may look different). Really only two companies (Google and Microsoft) have a high-quality global web link index (because I believe it costs upwards of a billion dollars a year to do), and so literally every other global search engine that wants to offer a search product competitive with Google for mainstream users, needs to bootstrap with one or both of them. The same is true for maps btw -- only the biggest companies can similarly afford to accurately map every neighborhood. And we are still a very small company relative to these companies -- literally on the order of 1,000 times smaller.
Anyway, I hope this provides some helpful context. I understand why people are upset with us on this, and we will do better. We're working on updates to our app store descriptions and other materials.
(Also FYI -- this was discussed extensively at https://news.ycombinator.com/item?id=31490515). This is also a somewhat misleading title since it's not about our search engine, which people would assume, and also as noted above Microsoft scripts are actually restricted in our browser a lot, so I would suggest changing it.)
So many words to minimize what happened, makes the conclusion of "I understand why people are upset with us on this" seem a little insincere.
To me, this is really simple. You entered a contract that allowed a company to circumvent tracking protections. the extent and which product this happened on are only semantic details. It fundamentally undermines your trustworthiness as a supposedly privacy-oriented company.
I am very sincere about this and I understand we have to do a lot more to rebuild trust. I also understand that will take a lot of time.
I do think the details matter though because based on the comments across the web, most people seeing this news cycle are coming to false conclusions, e.g., that this is about our search engine (it's not), that we give 3rd-party Microsoft scripts a free pass on 3rd-party websites (we don't), that we don't block 3rd-party cookies in our browsers (we do), or that any of the major browsers offer better web protection (they don't, and in fact they don't even attempt this particular type of protection).
Unfortunately this takes a few paragraphs to unpack fully. Probably someone else can explain it better and more succinctly, but I'm trying my best here to be succint.
> This is also a somewhat misleading title since it's not about our search engine, which people would assume
I understand the controversy (or at least I think I do), and I also respect what you're trying to explain.
But if DuckDuckGo has to have a browser (instead of simply recommending to its users to change their default search engine on their current browser), then that browser needs to live up to its promises, and also to very high standards.
On the current page for the DDG browser [1] it says at the top:
> Seamlessly take control of your personal information, no matter where the Internet takes you.
If this means anything, it means the DDG browser blocks all trackers, always, in all places, no?
No browser can provide 100% protection if for no other reason than the tracker landscape is constantly evolving as it works to evade our protections. We will re-evaluate our marketing messages, though as a comparison look at how the major browsers market their tracking protections and yet do not even attempt to provide any script blocking, let alone have major holes in their cookie protection (see https://privacytests.org/).
Very interesting link, thanks. Didn't know that localStorage and sessionStorage could be read across domains!
But the comparison should include (browser)+uBlock Origin. I wonder if there are many users who care about privacy and don't install uBlock Origin by default.
> If this means anything, it means the DDG browser blocks all trackers, always, in all places, no?
You've created a completely unrealistic standard. Nothing meets the standard of blocking all trackers everywhere as that isn't possible today or in the foreseeable future.
Well then, they shouldn't promise what they can't deliver.
Also, I don't think that's true. And it's part of the problem with the current DDG defense. They have an agreement with MS to not block 3rd party trackers, and they say "in any case it's very difficult to do and nobody else even tries to do it".
The best tools like uBlock, Umatrix, Privacy Badger, etc. all miss plenty of trackers, but not for a lack of trying. A complete solution doesn't exist today, nor is there anything promising on the horizon that I'm aware of. I am very curious to know the source of your optimism (I don't have much at the moment).
> They have an agreement with MS to not block 3rd party trackers, and they say "in any case it's very difficult to do and nobody else even tries to do it". It's a misdirection.
They shouldn't have done that to be sure, they've said as much, and are paying the price. They also said that future agreements won't include that carve out. They screwed up and will hopefully learn a valuable lesson from all this.
I will say, you should have used more precise language and said something like, "they have an agreement to not block 3rd party trackers in their browser" (emphasis mine). Your statement suggests that they don't block those sorts of things across their entire line of products, which isn't true. You're adding to the confusion by using overly broad language that is not precise and not 100% truly accurate.
Finally, I'm not aware of anything that is widely available that is as good privacy-wise as DDG and is actually usable by the most people. Until something exists that's just as good privacy-wise and is somewhat useful, I see very little choice but to use DDG. We'll see how DDG behaves in the coming months and years. Long term, they really should figure out something besides ads as a business model, since the incentives don't align with privacy.
“[W]hen you search, you expect unbiased results, but that’s not what you get on Google,” @matthewde_silva quotes @yegg"
Seems you're backtracked on being unbiased with the recent announcement of DuckDuckGo censoring sources on the back of the Ukrainian War. Any comment on that - seems like the bigger DDG gets, the further you stray from your initial values
No, there hasn't been any backtracking on this. The full quote continues "On Google, you get results tailored to what they think you’re likely to click on, based on the data profile they’ve built on you over time."
Unlike other search engines (like Google), we don’t alter search results based on someone’s previous search history. In fact, since we don’t track our users we don’t have access to search histories at all. Those other search engines show you results based on a data profile about you and your online activity (including your search history), and so can be slanted towards what they think you will click on the most based on this profiling. This effect is commonly known as the search filter bubble, but using DuckDuckGo can help you escape it.
This does not mean our search results are generally “unfiltered” because, for every search you make online, a search engine’s job is to filter millions of possible results down to a ranked order of just a handful. In other words, a search engine has to use algorithms programmed by people to determine what shows up first in the list of results, what shows up second, and so on. Otherwise, for every search you’d just get a completely random set of results, which of course wouldn’t be very useful.
However, we do this ranking in a strictly non-partisan manner, and not based on my politics (or anyone's for that matter). I left another comment here on how that works: https://news.ycombinator.com/item?id=31557837
Your last statement directly contradicts your tweet on March 10th.
"Like so many others I am sickened by Russia’s invasion of Ukraine and the gigantic humanitarian crisis it continues to create. #StandWithUkraine
At DuckDuckGo, we've been rolling out search updates that down-rank sites associated with Russian disinformation."
We get it - you are making these moves because of financial reasons or political pressure....but don't try and play to both sides with the corporate speak.
**
We actually do not intentionally censor any news results, meaning media outlets are not being removed or their stories displayed so far down in the results they are effectively removed. That is, unless legally prohibited, you should find all media outlets in our results, and they should generally show on top if you search for them by name or domain name. If you are seeing otherwise, please let me know and we will investigate.
A search engine's primary job is to rank results, trying to put results that most quickly and accurately answer the query on top. We do this ranking in a strictly non-partisan manner. Ranking for news-related searches is particularly difficult because for most news stories there are often hundreds of media outlets covering the same story, many with similar relevancy in terms of keyword matching and popularity. As such, we look to another ranking factor to ensure just the top of the results aren't taken by obviously very low-quality news results so that users have more sources of relevant, high-quality news results to compare and choose between.
The non-partisan factor we've found to help accomplish this is a rare, but well-documented history of a site's complete lack of news reporting standards, such as routinely using spam or clickbait to artificially inflate traffic, consistently publishing stories without citing sources, censoring stories due to operating with very limited press freedom, or misleading readers about who owns, funds, and authors stories for the site. And since we do not censor sites, even state-sponsored media in countries with very limited press freedom, these sites will still show up in results, and even on top like when you search for them directly.
**
With that context, RT is a media outlet with "very limited press freedom" where journalists must censor their articles or else face jail time, or worse. And RT still shows up in our results, and on top if you search for it directly, e.g., https://duckduckgo.com/?q=rt+ukraine&ia=web
The concern comes down to how that evaluation of a website is made. It’s the same way people can’t really accept fact checkers because “who fact checks the fact checkers?” With the news ranking it is who evaluates the evaluators. Without an option for users to switch off the part of the algorithm that includes evaluating for journalistic truth (as I will summarize those non partisan factor’s goal) people will always feel like the possibility of manipulation will always be there.
> And RT still shows up in our results, and on top if you search for it directly, e.g
Strange, but RT does not show up in DDG for me. Not by using your link, not even by searching "Russia Today" on it. The first results I get are Al Jazeera or Fox News. No Russia Today at all. How's that?
I'm in the EU, yes, but I don't understand that either. Because there are search engines, freely available in the EU, that didn't honour that questionable request about banning websites.
So, for me, DDG is hiding some results while other search engines that I can use in the EU, do not.
I was not talking about EU-based search engines, I was talking about search engines not applying censorship that I can use from the EU. And as far as I know, DDG is not EU-based either.
If that's the case, why would you accede to the EU's demands? Would you geoblock search results in China if the CCP deemed certain content or content providers illegal?
You're splitting hairs when you say down ranking is not censorship - you know full well the impact to a websites traffic once it gets out of the top 3 results let along the 1st page of results.
The fact you started censoring "Russian media outlets" right after the start of the war shows the political nature of your decision, those "limited press freedoms" weren't a concern until Mar 22.
Do you publish the decision making process for how the "non-partisan" decisions on censorship are made - or is this another blackbox? I'm sure Dailymail, New York Post and Al Jazeera are all down-ranked too, considering they meet your "non-partisan factors" right?
> Really only two companies (Google and Microsoft) have a high-quality global web link index ...
Yandex and Baidu also have pretty good index, and perhaps would be even cheaper than Bing. I'd say they only lack in terms of the data (search queries, search result links clicked etc). It's just US protectionist policy (on behalf of the BigTech) at play here to avoid these services. Both Yandex and Baidu outperform Google when it comes to Russian and Chinese search queries, clearly indicating they are at par (or even better) than Google in terms of search algorithms. The only thing they lack is better data, and that's why they struggle with foreign search queries. (Though I find Yandex has improved a lot here, enough to make it my regular search engine).
LOL. Have you actually used Baidu? It is trash. Results are full of heavily SEO-ed sites and sites that grab other websites' content and present as if it's their own. I say this as someone who has used Baidu for many years, including before Google was forced to leave China in 2020.
No, I don't use Baidu as it performs poorly for english queries. From what I have read online, the popular sentiment is that Baidu is better than Google when it comes to chinese search queries. Ofcourse, like Google they have the monopoly advantage (atleast in China). Yandex, I do use daily - it's good with queries it knows about, but struggles with new / unknown queries (for e.g. when I search something geographically related or local, at which Google is obviously better because of its userbase).
"literally every other global search engine that wants to offer a search product competitive with Google for mainstream users, needs to bootstrap with one or both of them."
I don't want to read too far into the specific wording, but you mentioned "bootstrap" and this feels like a good opportunity to ask the question. Is DDG planning or thinking about how it might move towards it's own index for general results? I feel like most of the criticism DDG receives these days is because it's backed by another, less privacy concerned, mega-corp. While it may not be something that's feasible today, and not in it's entirety, are there moves being made towards building an independent global web link index?
When most other browsers on the market talk about tracking protection they are referring to 3rd-party cookie protection e.g., blocking them) and fingerprinting protection (i.e., restricting APIS scripts can use), and our browsers impose these restrictions on all third-party tracking scripts, including those from Microsoft. We also have a lot of other web protections that also apply to Microsoft scripts (and everyone else) that most browsers don't do, including Global Privacy Control, 1st-party cookie expiration, referrer header trimming, new cookie consent handling (in our Mac beta), fire button (one-click) data clearing, and more.
What this article is talking about specifically is one web protection that the major browsers don't even attempt to do — stopping third-party tracking scripts from even loading on third-party websites. See for yourself the bottom 'tracker content blocking' section of this audit site: https://privacytests.org/ios.html (scroll to the very bottom).
This web protection is a particular challenge to get right because websites can easily break when scripts that they depend on don't load, but because it makes for better privacy and faster page loads, we've taken it on while still trying to not break sites. As a result, it is far from perfect, as we can't block all scripts do to persistent breakage issues (which we try to workaround), and domains change all the time (and we continuously crawl sites looking for new tracking domains).
Though because we're doing this protection where we can, and also offer many other unique protections (e.g., Google AMP protection, smarter HTTPS upgrading, tracking protection for other apps in Android, email protection to block trackers for emails sent to your regular inbox, etc.), users get way more privacy protection with our app than they would using other browsers. Our goal has always been and remains to provide the most privacy we can in one download. And we have a lot more planned.
The issue at hand is all of our web protections apply to Microsoft scripts on 3rd-party sites (again, this is off of DuckDuckGo.com, i.e., not related to search) except this one around full content blocking, as we are currently contractually restricted by Microsoft there. However, this limited contractual restriction (about this one web protection) is the only one we have, and we have been and are working with them as we speak to reduce or remove it.
I understand this is all rather confusing because it is a search syndication contract that is preventing us from doing a non-search thing. That's because our product is a bundle of multiple privacy protections, and this is a distribution requirement imposed on us as part of the search syndication agreement that helps us privately use Bing results to provide you with better private search results overall. While a lot of what you see on our results page privately incorporates content from other sources (including our own indexes), e.g., Wikipedia, local listings, sports, stocks, lyrics, weather, quick answers, etc., etc., we source most of our traditional links and images privately from Bing (though because of other search technology our link and image results still may look different). Really only two companies (Google and Microsoft) have a high-quality global web link index (because I believe it costs upwards of a billion dollars a year to do), and so literally every other global search engine that wants to offer a search product competitive with Google for mainstream users, needs to bootstrap with one or both of them. The same is true for maps btw -- only the biggest companies can similarly afford to accurately map every neighborhood. And we are still a very small company relative to these companies -- literally on the order of 1,000 times smaller.
Anyway, I hope this provides some helpful context. I understand why people are upset with us on this, and we will do better. We're working on updates to our app store descriptions and other materials.
(Also FYI -- this was discussed extensively at https://news.ycombinator.com/item?id=31490515). This is also a somewhat misleading title since it's not about our search engine, which people would assume, and also as noted above Microsoft scripts are actually restricted in our browser a lot, so I would suggest changing it.)