Google forgets so much, and has a tendency to bias results, while DuckDuckGo is so incompetent at times. Add the walled gardens that can’t be searched unless you are part of them, and we have a worst search experience today than over a decade ago.
I am confident that the amount of data that needs to be indexed plays a huge factor too. It just isn’t as easy as it used to be.
All of those things, and don't forget the steady arms race: the contemporary Web is stunningly adversarial to good search results.
Ten years ago, when Google screwed up, you'd land on a really obvious linkfarm page with a bunch of SEO spam: it was literally just a collection of keywords repeated over and over.
Now, when $search-engine screws up, which it often does, you land on a plausible-looking page, with bad but not terrible English, that just... doesn't tell you anything worth knowing. I know the smell of them, and it can still take me a section or two of precious time to figure out what's going on.
There's often not that much link juice, either: as far as I can tell, the business model is purely to put an ad or two in front of my eyeballs.
One of my issues with instagram is it cant be searched other then using hashtags. Those are just really crappy key words. So many artists put their art on there and its just sucked into a blackhole of pictures to never be seen again. It's impossible to search and find anything specific or anything that occurred more then a few hours ago
There is no sorting. Nothing. The comments are ranked by the AI, the posts are ranked by the AI, the explore section is ranked by the AI.You can't sort by time or likes.
> "Manually changing region to Russia gives many more results."
This probably explains it: DuckDuckGo indexes different languages together in one big index internally(?). "VK" returns many results in Russian, filling some maximum number of results. And then a filter just removes all the Russian ones if you're searching from a non-Russian locale?
> Might be a partnership with Yandex to not show russian results outside Russia?
Quite the opposite.
My search results from DDG over the past year or so have been infested with Russian sites, articles and even twitter account titles of English language accounts, despite not living there and not having any preference for Russian in any browsers settings or regions and never issuing a single query in Russian.
"infested" is a little much, don't you think? I can sympathise with frustration on having results that don't match your language preferences, but "infested"? Come on.
I mean the sentiment lacks nuance and it gives context to why someone would use pejorative adjectives about Russian-anything even when referring to search engine results
And you can pinpoint which generations of people are likely speaking by their use of said pejoratives on otherwise benign subjects
They're only against Communism. Russians with money are just fine, apparently. The logical conclusion of "money is speech" is that foreign intelligence agencies can buy as much speech as they like.
Protip: its not just Russians playing around with sentiment in other countries for their own gains, and it is extremely easy to deflect all attention towards Russia. It can be long time allies that you harbor the least feelings about.
Searching for ”vk” should return a lot of things beside the Russian social network. Vodka Kick, Vulkan, Voight-Kampff, Västerbottens-kuriren, etc. etc. I don’t think this is intentional.
If it is a blacklist, it's not a very good one. "pedobear." and "pedophile." work fine. I don't understand why they would want to block these anyway. One's just a meme, and the other is a bunch of mugshots of strange looking men.
They have been censoring any results vaguely related to pedophilia for the past year. If you search for "lolis" or "lolicon" which are fictional cartoons, that will be censored.
Is it picking up VK profile pictures? Sort of reminds me of myspace but with, err, Russian sensibilities. I also tried it with Bing and got literally the exact same results.
Google is definitely different, it shows mostly logos instead of photos.
I never used VK but I got the impression it is a real social network, for better or for worse. Should Bing drop them from its index? Now I'm morbidly curios what kind of weird crap searches of social networks from other cultures might dredge up. Like a Chinese VK.
huh... looks like the default search is now showing multiple things for me (showed just one result first time I checked)
However I'm seeing lots of slightly uncomfortable pictures of young girls (not porn, but didn't want to scroll too far because it felt like there could be some mixed in) and tanks in the image search. This doesn’t change noticeably using the three strictness settings.
We hired a baby sitter once that was from Ukraine. I made the mistake of Googling her unique name. Her VK profile was a whole different person from the sitter at my house.
Haha her name is probably far less unique than you think. My Slavic name is fairly unique in the Western world but there are literally thousands of people with it on VK.
When you want to add a friend on VK, you have to ask them for things like city, school, and age just to narrow down the results!
It's unfortunately not uncommon for such images to appear on the Russian internet. The Youtube thumbnails are possibly things that were linked to from VK communities. Or maybe there are links in the content somewhere.
If I go on Yandex and search "img " and then a bunch of random numbers, I get almost all pictures of young women, and many pictures of underage girls in provocative poses.
This sort of thing does happen. A few months ago, people discovered similar results on Google Images(!) by searching "TV television and film". The results were being pulled from a 4chan archive of the board with that name.
I don't think it's related to the "VK" web results problem though. If you search "VK," or "VK!" or VK with any punctuation, all the web results come up as expected.
I'm a bit conflicted on this. I don't want content like this appearing in search results. But I don't think a clumsy keyword filter will make it go away. And it might push pedophiles looking for a quick fix to places with harder content or echo chambers.
DuckDuckGo isn't the best search engine. I really want it to be. I use it a by default and I use it often. However, especially when I'm searching for products/things to buy I end up on google search. DuckDuckGo is more like old school search, think Altavista - tons of results but not much useful.
Maybe it's because Google has very little data on me, but I find Google to be horrible in 2020. It's degraded considerably over the years to the point that I don't even try Google if DDG doesn't return anything useful. It's especially useless if I'm trying to buy something. If I search for a service business using my hometown, it'll return results from businesses everywhere other than my hometown.
Google search has gotten a lot worse for me now that I use it as a backup for DDG. There's some categories it still does well, but other things are like DDG doesn't find it, but neither will Google. I'm not sure how much of that is Google going downhill (which was a trend before I switched), and how much is the loss of personalization data.
I've also noticed google has developed an extreme recency bias in search results. If you're searching for a term that is somehow related even tangentially to any hot or trending topic over the past year, it is virtually impossible to find relevant, yet dated sources on that topic. I know I can do the timeline filter and all that, but sometimes it's not obvious what window of time you should be searching, etc.
Heh, I don't even know how google feels, much like I don't know how ransomware and heroin feels. We don't really have to try everything in life, do we?
I have the same experience. I started to notice about 6 months ago. It's happening a lot more often that i'm frustrated with results even when looking for trivial stuff. My last search frustration was: 'bitmitigate'. I had to make a benchmark of CDN services and their company site isn't on the first page like other products i'm searching for. I suspect it's because this company had business with alt-right imageboards, but it scares the shit of me if this kind of political controversies are incorporated now in google's algorythm.
By contrast, searching 'bitmitigate' on DDG produces the expected outcome: the first result is the official .com site of the company.
(Edit: added the last paragraph comparing to duckduckgo)
>My last search frustration was: 'bitmitigate'. I had to make a benchmark of CDN services and their company site isn't on the first page like other products i'm searching for.
Seems to be working for me? SERP for me (not private browsing, but I'm not signed in and regularly clear cookies)
1. BitMitigate - CDN, WAF, DDoS Protection, Load Balancing, VPN <--- this is the main site, as expected.
2. BitMitigate: What we know about 8chan's new web protection ...
3. BitMitigate (@bitmitigate) | Twitter
4. Epik reverses course, says BitMitigate will not support 8chan ...
5. BitMitigate becomes new security provider for 8chan ...
Tbh I don’t really generically search that much. I search within specific sites (e.g. Wikipedia, YouTube) but my first reach out for tool is rarely if ever generic web search. Lots of bangs on ddg
In my experience google is pretty good for fresh and current data. If you try to find things from some years back, results are overwhelmed by irrelevant current results —even setting custom dates doesn’t help much. Seems it’s biased for the present as well as content from big players (obviously), but to the detriment of smaller outfits.
I've set it as my default and I have to say I didn't realize hw often I search for silly stuff it can nail. Like the url of a blog I know the exact name of. If I start to get into detailed programming issues or deeper searches I revert to google. It only seems to be 2-3x a week for me.
I agree, somewhat. While the results are clearly worse my feeling is that they are covering more of my use cases as time passes. The really cool thing (and a trick Google themselves used in the beginning) is that a Google search is only a !g away, so I can easily have it as my default search engine.
One thing that rocked me to the core and changed my stance on privacy recently is how child porn/abuse/trafficking/grooming/etc is simply swept under the rug by the Tech Giants. Safe Harbor laws have been used to push the problem elsewhere, and the issue is accruing exponentially. These images are absolutely everywhere; Facebook removes millions every year from their platform. And these are just the ones posted in plain sight.
No idea what the solution might be, or if it's even possible to be solved. It's a depressing topic, and you have to have really good opsec to do any journalism on the matter without landing in serious legal trouble.
I really think Sam had the wool pulled over his eyes by Gabriel Dance. Around 17 minutes into the podcast, Dance says that he's been investigating tech companies for years and goes into issues with ad targeting, Twitter bots, etc. Then when he's relating the story about getting a tip, he effectively admits that he's pursuing this because he wants to make tech companies look bad. This explains why he always quotes absolute numbers (instead of percentages of all images/video shared), and why he never says what fraction of that content is teens sexting. It also explains why he avoids suggesting concrete solutions to the problem: there aren't any good ones.
The only solution to the problem of child pornography is backdoors in all encryption. Facebook must be able to scan the messages you write. Apple must be able to scan the photos you take. AWS must be able to scan your servers. The US Government must be able to intercept your communications and decrypt your devices. If there is any consumer device with real encryption, it will be used by perverts, sadists, revolutionaries, psychopaths, and a few weird principled nerds. And the problem of child pornography will still exist, except now the government will be able to spy on everyone's private communications and use that to exert much more control over them.
It's been a few weeks since I listened to the podcast, but I thought Dance explicitly mentioned how sad it was for Facebook to get a bad rep for releasing those numbers when such transparency is a positive thing. It shows that they're at least trying to get it off their platform. And if more people knew the numbers, this issue would finally become too large of an elephant to ignore.
And he did offer one concrete solution: platforms with easy discoverability (in other words, easy to masquerade as and target teens) may not need end-to-end encryption. Let FB and such run their algorithms on those messages. And it may be safe to assume three letter agencies are listening in. End-to-end encryption still serves its purpose for whistleblowing, privacy, and fair democracy; it'll just be found elsewhere, like Signal and such.
This still isn't an ideal solution. I personally think it's an unwinnable battle. Easily duplicated instantaneous secure multimedia communication is ridiculously revolutionary.
> Then when he's relating the story about getting a tip, he effectively admits that he's pursuing this because he wants to make tech companies look bad.
Just because he's a gadfly doesn't automatically make him wrong.
However, I just don't understand how these images are so common. If I ever stumbled on an image like that I would assume that it was obviously a honeypot.
If there was ever a discussion that should be heard by more people working in tech, that’s that right there.
DDG’s results are (perhaps decreasingly) heavily based on Bing, which as you will learn from that discussion had had significant issues with CP discovered (and reported to FBI) in its search results effortlessly—even while the same images were flagged by Microsoft’s own photo DNA algorithm.
Did they ever confirm it was CP? Or did they report it without looking at it because it was flagged by the algorithm and didn't want to be held liable?
I had an associate that was tasked with writing a crawler to find this sort of stuff.
There were two problems: Burn out from seeing what the crawler found. And only scratching the surface and getting more hits than any law department could ever hope to track down.
>We also of course have more traditional links in the search results, which we also source from multiple partners, though most commonly from Bing (and none from Google).
That crawler is only used for Instant Answers for specific queries, not the standard 10 search result links. Yahoo has also been powered by Bing for a long time now as well.
Well that's weird. Searching vkontakte gives a lot of results (in the realm of what you would expect). Maybe it's a command or something else is happening with that query?
That's not correct. You can show the proper results without setting the region. Do this by adding an empty token that doesn't get stripped, "https://duckduckgo.com/?q=vk+%22%22". That also shows the results list.
The single result it gives is a link to VKontakte. Given that it's largely a walled garden, I think that's a reasonable result to give, with no other input.
Also in NZ and as you say lots of results but almost entirely related to the Russian social network 'vk'. Using mojeek, as suggested upthread, gives a much more diverse set of results. Which leaves the question why does NZ get a different set of results than some other places ?
The images returned on DuckDuckGo are, in general, really dodgy. While debugging a web-server of mine, I recently found that various constant strings of file servers return similarly `dodgy' results, like for example "Index of /" [0].
So this is a complete tangent, but one of the images was some photograph off a geocities archive site. The woman had red eyes. I realized I haven't seen that in years. Cameras sure have improved.
It is (used to be?) a pretty common google "hack" to put an intitle:"index.of" on a search to find sites serving files, usually mp3s. Of course once this was widespread enough, SEO spam sites starting including it in the generated clickbait sites.
yeah there is some absolutely questionable stuff there, I wonder why searching for VK brings them all up, some of these seem like random direct-to-youtube uploads but there's so many of kids that it almost seems like someone's collection, but why does a search for VK return them all? seems like a strange coincidence that there'd be troubling image results and empty normal results
There's definitely some suggestive underage content in there. Some of it looks like random photos gathered from the web but there's also some pedo / ephebo shit that was purposely made.
It seems to have been fixed, I refreshed the tab I had open when this first popped up on hn (that had empty results) and the results look normal now (US).
My main problem with ddg is that it's much slower than Bing to load the results. Bing is just so much snappier than even Google for me. I wonder if there's something wrong with my internet.
I've been trying to use duckduckgo for almost a year, but I can't really justify it. I've even gotten used to just adding !g at the end of each search. It has terrible results for anything non english and pretty bad latency where in asia it seems.
Does anyone have a recommendation for another search engine that does decently well in terms of privacy and also supports chinese results?
Another search engine I recommend is Ecosia. They’re slightly better than average with privacy, and focused on the environment. I think they just use straight bing results so I’m not sure how well it handles Chinese searches.
Startpage.com is the one I use day to day (I work at the company that owns it). They use straight Google results through a proxy in their own data centers, it's anonymized before it hits anything
I'm also very interested in the details. According to their website they are paying Google to provide them with the results. But the revenue streams to startpage are not provided on their homepage (or at least I haven't found them yet).
Edit: https://support.startpage.com/index.php?/Knowledgebase/Artic... This gives additional information. Apparently they are showing ads that are only related to the search terms. I guess they are showing more ads than Google? Otherwise this wouldn't really make sense. Or maybe individualized advertisements in the search results aren't that much of a big money-maker for Google (!)
It's the same amount of ads. The hyper targeting Google does is incredibly profitable because of the scale they operate at. If you make the click through rate of an ad even 0.1% better over billions of searches it more than pays for the R&D of an engineering team.
Really the only way to compete against Google from a feature perspective is the privacy angle. You're sacrificing better click through rates to target a market that cares about privacy. Making something private requires a lot less engineering and product power than hyper optimizing tracking. By definition if you're not doing any tracking you don't need to employee people to set up tracking systems.
The deal is 15+ years old. To my knowledge it was made back when Google was still a very young company. There's a surprising amount of legacy contracts from Google/Bing/Yahoo floating around that power all of these search engines (Ecosia, DDG, etc).
Ive been using DDG as a default search for a while now. My conclusion is that the results are just NOT as good as Google's. I find myself using Google searches for time sensitive or mission critical searches, which totally defeats the purpose of a search engine.
As an example, I was playing around with Phaser and found that DDG just does not surface the entire Phaser documentation or forums correctly, and had to switch my default browser search for when I was putting time there.
I want to want to support DDG, but the lack of quality around searches makes it difficult.
I actually like the results on DDG better than Google most of the time:
1. It's more likely to respond to the actual query, rather than guessing what I want based on the query.
2. The results are not stuffed with ads and Google service substitutions.
However it's not as good at searching every last corner of the web, so occasionally I need to g!. It's pretty rare that I use that these days, and rarer that it actually gives me a better result.
Same. DDG often requires 1-2 additional search terms to give it
enough context, but in return I get results that are actually
relevant instead of half a page full of pinterest links and
other spam content.
Actually, by searching with !g, you’re leaking your info to both DuckDuckGo and Google, instead of just Google.
If you want to do Google searches without leaking your data to Google, you could use Startpage.com, but personally I don’t trust Startpage more than Google.
I'm not sure about the "without leaking your info to Google" part, there. All it does is forward you search query to Google as if you'd entered it on google.com. There's not really anything DuckDuckGo can do about that.
I got in the habit of just appending this to many of my queries in time sensitive situations, as if by reflex, negating the purpose (I default my search engine to DDG so !g is the fastest way to Google for me). I am having a hard time breaking the habit.
If it runs the query through DDG's server side, then Google would just see at as an anonymous query along with a million other anonymous queries coming from DuckDuckGo's servers.
Unfortunately, that's NOT what they do. Instead, they just forward you to the google search result, which is pointless and stupid.
Not so. The point is that you can set DDG to be your default search engine, and put !g anywhere in the search string to search google in case you need it.
I agree, sadly, and in (literally) every discussion on DDG that I've seen, people cite as one of DDG's plus points that you can use !g to search Google instead.
Just out of curiosity, what do you (anyone reading) see when you search "curry" on DDG images?
Plenty of sites have terrible search and discovery stories. I routinely search in a general purpose search engine and filter to the site whose results I want because it's faster and more reliable than trying to navigate the site itself.
It's been a decade since the majority of sites had a usable search of their own. Many that still do are just hosting a site-specific Google search at this point.
I think you’re implying that Reddit’s search is subpar, which I agree, although it seems it has improved slightly, recently. When I want firsthand perspective on something I usually append “reddit” to my google searches instead of searching reddit itself, even if I have a reddit tab open already.
On relatively sparse, self-post (fow or no link) subreddits, Reddit search is fairly useful for searchin posts only.
Comments are not indexed.
At volume and with links, search breaks down from both insufficient context (esppecially with editorialised, sensationalised, or vague titles) and excess low-yield content.
In other weird search results if you search for 4chan with tor you get wikipedia and news results. If you search without tor you get the real website and in general more relevant results.
DDG's raison d'etre is to position itself as the Gutmensch of search engines but when it comes to providing peace of mind that I'm getting the best and most relevant results, there's a reason why my grandmother hasn't heard of it.
As others are pointing out, this is actually a popular site not just for russian content. Owing to the jurisdiction, it's actually a popular place for hosting various ripped/pirated PDFs.
If you actually click through Google's result pages, you will find that there are actually only about 87 results. Even if you "repeat the search with the omitted results included" Google returns only about 226 results. Google's estimated result count is routinely off by so much as to make it virtually useless.
I found that going to the last page showed a more realistic number for results. This would only be shown on that "last page" of results though. However, it does seem strange that there are only 447 results for a google search of "apple"[0] down from a whopping 4,070,000,000 results shown before that page. And that is with show omitted results. Seems like max result limiting for the 0.1% of users that want to look past page 10 of search results
It is actually plausible that there could be billions of pages with apple on the internet.
The 447 results you paged through are just the most relevant docs for your particular query ("apple" + country) and Google doesn't bother retrieving any further. Your use case of finding all the pages with the term "apple" is simply too rare and too expensive to support and they don't optimize for it.
If you try different query variants apple + something with different locales (&hl=) you'd get very different top NNN results.
Google like all other web search engines never scan their whole index when searching, it's way too expensive. All sorts of tricks are used to aggressively prune matching results at every opportunity during retrieval until maybe a few hundred best scoring docs are left and that's what you're paging through.
But yeah, estimated result count is still a big lie either way
I am confident that the amount of data that needs to be indexed plays a huge factor too. It just isn’t as easy as it used to be.