Kind of insane how a severely limited company founded 1 year ago competes with the infinite budget of Open AI
Their parent hedge fund company isn't huge either, just 160 employees and $7b AUM according to Wikipedia. If that was a US hedge fund it would be the #180 largest in terms of AUM, so not small but nothing crazy either
The nature of software that has not moat built into it. Which is fantastic for the world, as long as some companies are willing to pay the premium involved in paving the way. But man, what a daunting prospect for developers and investors.
That dystopia will come from an autocratic one party government with deeply entrenched interests in the tech oligarchy, not from really slick AI models.
Even a well intended non autocratic democratically elected multi party system could accidentally pull off a dystopic opening of pandora’s box when it comes to AI. In the grand scheme of things I’m not sure we’re any safer if we live in a democracy.
Good. As much as I don't like some things about China, but damn it they're really good at cutting down costs. I look forward to their version of Nvidia GPUs at half the price.
> I look forward to their version of Nvidia GPUs at half the price.
Arguably China doesn't have the technology required to manufacture 30-series GPUs with the yield or unit cost Nvidia did. I wouldn't hold my breath for Chinese silicon to outperform Nvidia's 40 or 50 series cards any time soon.
I wonder if the US will end the restrictions if China pulls ahead in LLM ability, considering they serve no purpose if China's already ahead? Although given they seem to want to ban Chinese drones without any competitive local alternative, maybe not.
Makes me suspect if the primary plateau is data, and we're now seeing a place where all the AI labs who are actually having a crack at this seem to have similar levels of quality data to train on. Layering in chain of thought and minor architectural changes doesn't seem to be giving anyone a truly groundbreaking lead.
That’s not how system prompts work. You’re simply asking it to role-play a user-assistant chat where the user tries to circumvent the system prompt and asks who the assistant is. Unsurprisingly, the majority of such chat scripts on the web will have been created with ChatGPT. Hence the answer you are seeing.
This is clearly what is happening. Deepseek can train on o1 generated synthetic data and generate a very capable and small model. This requires that somebody build an o1 and make it available via API first.
I might be just being a bitter sceptic (although I'm probably not bitter because I'm very excited by their results), but some of the spending stats feel slightly too good to be true to me. But I can't really claim to have an insider-quality intuition.
It's pretty clear, because OpenAI has no clue what they are doing. If I was the CEO of OpenAI, I would have invested significantly in catastrophic forgetting mitigations and built a model capable of continual learning.
If you have a model that can learn as you go, then the concept of accuracy on a static benchmark would become meaningless, since a perfect continual learning model would memorize all the answers within a few passes and always achieve a 100% score on every question. The only relevant metrics would be sample efficiency and time to convergence. i.e. how quickly does the system learn?
It's actually great if the end result is that the incumbent with infinite money that has unrealistic aspirations of capturing a huge section of the sector lights all the money on fire. It's what happened with Magic Leap - and I think everyone can agree that the house of Saud tossing their money into a brilliant blaze like that is probably better than anything else they would have wanted to do with that money. And if we get some modest movements forward in that technical space because of that, all the better. Sometimes capitalism can be great, because it funnels all the greed into some hubris project like this and all the people that are purely motivated by greed can go spin their wheels off in the corner and minimize the damage they do. And then some little startup like Deepseek can come along and do 90% of the job for 1% of the money
tangential but kind of curious to see models and more generally tech get dragged into geopolitical baron feuds second time seeing that the house of saud & their tech not popular on HN lol
Well, it’s not exactly new news. Saudi Arabia has a long and storied record of being rich, investing in tech, and human rights abuses. That conversation has been going on for a very long time.
DeepSeek is a Chinese AI company and we're talking about military technology. The next world war will be fought by AI, so the Chinese government won't leave China's AI development to chance. The might of the entire Chinese government is backing DeepSeek.
Open source is about standards. If for example USA uses chinese algorithms or whatever, if China discovered some drawbacks in it and did not disclose it originally - it can be used as a weapon.
To my understanding, most people, even in tech, disregard and look down on Chinese software. For some reason they also have a picture of 10 CCP employees sitting on each dev team, reviewing code before it gets released on GitHub.
There was a conversation with some western dev how they kept saying Chinese devs don’t work with scale like Meta/Google do, so they don’t have experience in it either. That was also an interesting thread to read, because without thinking about anything else, WeChat itself has more than 1B users. I’m not sure if it’s pure ignorance, or just people want to feel better about themselves.
I agree that a good chunk of Chinese apps’ UX is trash though.
It is trash because you're thinking with the mind of a Westerner. These apps are created and optimized for Chinese audiences, and they interact in a different way.
Taobao's shop by image is pretty game changing. Whether or not they were the first to do it, they seem to be the most successful iteration of it.
I feel like Chinese UX flows tend to be more clunky than Western ones but I have a certain liking for high information density apps, and find uncluttered screens sometimes a bit annoying and overly patronising.
I thought bullet chat on Bilibili was a very fun concept that probably doesn't translate quite as well to western media but YouTube has come up with a nifty half way by flashing comments with timestamps under the video
Yeah, totally fair. I guess it’s a very subjective opinion, given I grew up in the west, and was introduced to the iPhone era gradually. Like i went through Internet of 90s, desktop apps, old laptops, PCs and etc., and then eventually landing on daily iPhone usage. I can see how it might be a bit different if you went from most using nothing to Android/iPhone society.
That being said, they still use apps like Chrome, Safari, all the other common apps like ours. So they have both UXs available for them, I guess.
I have not said that Deepseek models are bad. Quite the opposite. I'm impressed by them. I have just questiened that they are just some chinese startup.
No, they absolutely export malware still. All of DJI's apps need to be sideloaded on android because the obfuscated data collection they do is not allowed in Play Store apps[0]. TikTok uses an obfuscated VM to do user tracking[1]. Then there's the malware that the US government has to routinely delete from compromised computers [2][3]
Fair points. I guess, market doesn’t care about software being malware, given both of your examples are the leading products in the world within their own market segments.
Like there are 1.4B people in China, obviously there are bad actors. Writing off an average software as a malware ridden crap is kinda weird. And again, the main users of Chinese software are… mainland Chinese. Whether we like it or not, they have very impressive track record of making it run and scale to humongous users.
Anyways, I think I deviated far from my point and sound like a general China-shill.
tech people are notorious for being ignorant assholes about anything outside of their field of expertise. There are multiple very reputable research showing smart people to be more susceptible to propaganda and brainwashing.
The chinese are great at taking secrets. Chatbots are great places for people to put in secrets. Other people say "we're not going to use your data" - with a Chinese company you're pretty much guaranteed that China mothership is going to have access to it.
The open source model is just the bait to make you think they are sincere and generous - chat.deepseek.com is the real game. Almost no-one is going to run these models - they are just going to post their secrets (https://www.cyberhaven.com/blog/4-2-of-workers-have-pasted-c...)
I am not going pretend to know the specifics, but don't the have mandatory Communist Party Committee? Comming from former eastern block country, I assume that they tend to have the final voice.
Are you talking about State-Owned Enterprise? Because yes, those have government tighter oversight and control, but I don't think this company is a SOE, at least from what I can tell.
From the rest, it works the same as in the US. If the government comes with a lawful order for you to do something, you'll do it or be held responsible for ignoring it.
> but I don't think this company is a SOE, at least from what I can tell.
There's no way to really tell. An authoritarian state like China can decide to control this company at any time, if it chooses to, through more direct or indirect means.
It doesn't need to be an authoritarian government. The US government can proclaim a company to be of "national interest" at any time and thus determine what it can export or not, as it has done repeatedly over the last few years.
> From the rest, it works the same as in the US. If the government comes with a lawful order for you to do something, you'll do it or be held responsible for ignoring it.
I’m always amazed when people ignore this. One day it’ll be stories about the CIA or whatever agency demanding data from a big tech company, with gag orders so they legally can’t even tell anyone. The next it’ll be a story about TikTok or DJI being bad because the Chinese government has influenced over them.
I think slight variations of that happens everywhere. Chinese companies have legally required CCP connections, which sounds ominous, but American companies of substantial scale will have ex-government employees, resources allocated for lobbying, and connections to senators. The difference is whether it's codified and imposed or implicitly required for survival.
(not that I support CCP, the requirement do sound ominous to me)
Exactly, in the US the big companies also enter the government complex through board memberships and collaboration with 3 letter agencies, just like in China.
CPC consists of higher management so yeah they have the final voice, just like every other companies.
The antidote for the CCP stuffs, is to alter your mind and accept that the CCP is no longer an ideological party, but a club of social elites. Whether that's a good thing is of course open to debate.
Except it’s not really a fair comparison, since DeepSeek is able to take advantage of a lot of the research pioneered by those companies with infinite budgets who have been researching this stuff in some cases for decades now.
The key insight is that those building foundational models and original research are always first, and then models like DeepSeek always appear 6 to 12 months later. This latest move towards reasoning models is a perfect example.
Or perhaps DeepSeek is also doing all their own original research and it’s just coincidence they end up with something similar yet always a little bit behind.
This is what many folks said about OpenAI when they appeared on the scene building on foundational work done at Google. But the real point here is not to assign arbitrary credit, it’s to ask how those big companies are going to recoup their infinite budgets when all they’re buying is a 6-12 month head start.
For-profit companies don't have to publish papers on the SOTA they product. In previous generations and other industries, it was common to keep some things locked away as company secrets.
But Google, OpenAI and Meta have chosen to let their teams mostly publish their innovations, because they've decided either to be terribly altruistic or that there's a financial benefit in their researchers getting timely credit for their science.
But that means then that anyone with access can read and adapt. They give up the moat for notariety.
And it's a fine comparison to look at how others have leapfrogged. Anthropic is similarly young—just 3 and a bit years old—but no one is accusing them of riding other companies' coat tails in the success of their current frontier models.
A final note that may not need saying is: it's also very difficult to make big tech small while maintaining capabilities. The engineering work they've done is impressive and a credit to the inginuity of their staff.
Anthropic was founded in part from OpenAI alumni, so to some extent it’s true for them too. And it’s still taken them over 3 years to get to this point.
That was a really good article. I dig the CEO's attitude, i agree with everything he says and I am an American. From a Chinese perspective he must be talking an alien language so I salute him with trying to push past the bounds of acceptable hum drum. If the rest of China takes on this attitude the west will have serious competition.
This article is amazing. It explains not just why DeepSeek is so successful, but really indicates that innovators elsewhere will be too: that extensive opportunities exist for improving transformers. Yet few companies do (not just China, but everywhere): incredible amounts are spent just replicating someone else's work with a fear of trying anything substantially different.
There are some significant innovations behind behind v2 and v3 like multi-headed latent attention, their many MoE improvements and multi-token prediction.
Of course not. But in this context the point was simply that it’s not exactly a fair comparison.
I’m reminded how hard it is to reply to a comment and assume that people will still interpret that in the same context as the existing discussion. Never mind.
Don’t get salty just because people aren't interested in your point. I for one, think it’s an entirely _fair_ comparison because culture is transitive. People are not ignoring the context of your point, they’re disagreeing with the utility of it.
If I best you in a 100m sprint people don’t look at our training budgets and say oh well it wasn’t a fair competition you’ve been sponsored by Nike and training for years with specialized equipment and I just took notes and trained on my own and beat you. It’s quite silly in any normal context.
If someone replies to your comment then I think it’s entirely fair that they take your point in the context in which it was intended. Otherwise, if they are not interested in the point then simply don’t reply to it.
No-one enjoys being taken out context.
But I do accept that given the hostility of replies I didn’t make my point very effectively. In a nutshell, the original comment was that it’s surprising a small team like DeepSeek can compete with OpenAI. Another reply was more succinct than mine: that it’s not surprising since following is a lot easier than doing SOTA work. I’ll add that this is especially true in a field where so much research is being shared.
That doesn’t in itself mean DeepSeek aren’t a very capable bunch since I agree with a better reply that fast following is still hard. But I think most simply took at it as an attack on DeepSeek (and yes, the comment was not very favourable to them and my bias towards original research was evident).
Sure, it’s a point. Nobody would be where they are if not for the shoulders of those that came before. I think there are far more interesting points in the discussion.
Also don’t forget that if you think some of the big names are playing fast and loose with copyright / personal data then DeepSeek is able to operate in a regulatory environment that has even less regard for such things, especially so for foreign copyright.
We all benefit from Libgen training, and generally copyright laws do not forbid reading copyrighted content, but to create derivative works, but in that case, at which point a work is derivative and at which point it is not ?
On the paper all works is derivative from something else, even the copyrighted ones.
Disrespecting copyright and personal data is good for users? I guess I disagree. I would say that it’s likely great for the company’s users, but not so great for everyone else (and ultimately, humankind).
I would extend the same reasoning to Mistral as DeekSeek as to where they sit on the innovation pipeline. That doesn’t have to be a bad thing (when done fairly), only to remain mindful that it’s not a fair comparison (to go back to the original point).
This is one message of the founders of Mistral when they accidentally leaked one work-in-progress version that was a fine-tune of LLaMA, and there are few hints for that.
Like:
> What is the architectural difference between Mistral and Llama? HF Mistral seems the same as Llama except for sliding window attention.
So even their “trained from scratch” models like 7B aren’t that impressive if they just pick the dataset and tweak a few parameter.
Right, so Mistral accidentally released one internal prototype that was fine-tuned LLaMA. How does it follow from there that their other models are the same? Given that the weights are open, we can look, and nope, it's not the same. They don't even use the same vocabulary!
And I have no idea what you mean by "they just pick the dataset". The LLaMA training set is not publicly available - it's open weights, not open source (i.e. not reproducible).
Their parent hedge fund company isn't huge either, just 160 employees and $7b AUM according to Wikipedia. If that was a US hedge fund it would be the #180 largest in terms of AUM, so not small but nothing crazy either