“To people who see the performance of DeepSeek and think: ‘China is surpassing the US in AI.’ You are reading this wrong. The correct reading is: ‘Open source models are surpassing proprietary ones.’ DeepSeek has profited from open research and open source (e.g., PyTorch and Llama from Meta). They came up with new ideas and built them on top of other people’s work. Because their work is published and open source, everyone can profit from it. That is the power of open research and open source.”
As if anyone riding this wave and making billions is not sitting on top of thousands of papers and millions of lines of open source code. And as if releasing llama is one of the main reasons we got here in AI…
I’m almost shocked this spooked the market as much as it did, as if the market was so blind to past technological innovation to not see this coming.
Innovation ALWAYS follows this path. Something is invented in a research capacity. Someone implements it for the ultra rich. The price comes down and it becomes commoditized. It was inevitable that “good enough” models became ultra cheap to run as they were refined and made efficient. Anybody looking at LLMs could see they were a brute forced result wasting untold power because they “worked” despite how much overkill they were to get to the end result. Them becoming lean was the obvious next step, now that they had gotten pretty good to the point of some diminishing returns.
sure, but what nobody expected how QUICKLY the efficiency progress has been - aviation took about 30 years to progress from "the rich" to "everybody", personal computers about 20 years (from 1980s to 2000s), I think the market expected to have at least 10 years of "rich premium" - not 2 years and get taken to the cleaners by the economic archenemy, China
The Google transformer paper was 2017. ChatGPT was the “we can give a version away of this for free.” Llama was “we can afford to give away the whole product for free to even the playing field.” Every tech giant comes out with a comparable product simultaneously. And now a hedge fund, not even a megacap company, can churn out a clone by hiring a small or medium size engineering team.
Really this should be an indictment of corporate bloat, having hundreds of thousand headcount companies distracted by performance reviews, shareholders, marketing, rebuilding the same product they launched two years ago under a new name.
>Really this should be an indictment of corporate bloat, having hundreds of thousand headcount companies distracted by performance reviews, shareholders, marketing, rebuilding the same product they launched two years ago under a new name.
Yeah.
There are some shorter words or acronyms for it though, roughly equivalent to your about 30-word paragraph above:
IBM
DEC
Novell
Oracle
MS
Sun
HP
...
MBA
, all in their worse days or incarnations or ...
The notion I now believe more fully is that the money people - managers, executives, investors and shareholders - like to hear about things in units they understand (so money). They don't understand the science, or the maths and in so much as they might acknowledge it exists it's an ambient concern: those things happen anyway (as far as they can tell), and so they don't know how to value them (or don't value them).
Because we saw, what a week ago the leading indicator that the money people were now feeling happy they were in charge which was that weird not-government US$500 billion investment in AI announcement. And we saw the same being breathlessly reported when Elon Musk founded xAI and had "built the largest AI computer cluster!"...as though that statement actually meant anything?
There was a whole heavily implied analogy going on of "more money (via GPUs) === more powerful AIs!" - ignoring any reality of how those systems worked, their scaling rules or the fact that inferrence tended to run on exactly 1 GPU.
Even the internet activist types bought into this, because people complaining about image generators just could not be convinced that the Stable Diffusion models ran locally on extremely limited hardware (the number of arguments where people would discuss this and imply a gate while I'm sitting their with the web GUI in another window on my 4 year old PC).
I would generally agree, but the market isn't rational about the future prospects of a company. It's rational about "can I make money off this stock" and nothing else matters in the slightest.
Riding hype, and dumping at the first sign of issues, follows that perfectly well.
Sure but it's good to recognize Meta never stopped publishing even after Openai and deepmind most notably stopped sharing the good sauce. From clip to dinov2 and llama series, it's a serious track to be remembered.
But there is a big difference, llama is still way behind chatgpt and one of the key reasons to open source it could have been to use open source community to catch up with chatgpt. Deepseek on contrary is already at par with chatgpt.
R1 distills are still very very good. I've used Llama 405b and I would say dsr1-32b is about the same quality, or maybe a bit worse (subjectively within error) and the 70b distill is better.
Right, so it sounds like it's working then given how much people are starting to care about them in this sphere.
We can laugh at that (like I like to do with everything from Facebook's React to Zuck's MMA training), or you can see how others (like Deepseek and to a lesser extent, Mistral, and to an even lesser extent, Claude) are doing the same thing to help themselves (and each other) catch up. What they're doing now, by opening these models, will be felt for years to come. It's draining OpenAI's moat.
There's no need to read it uncharitably. I'm the last person you can call a FB fan, I think overall they're a strong net negative to society, but their open source DL work is quite nice.
This. Even their less known work is pretty solid[1] ( used it the other day and was frankly kinda amazed at how well it performed under the circumstances ). Facebook/Meta sucks like most social madia does, but, not unlike Elon Musk, they are on the record of having some contributions to society as a whole.
<< And as if releasing llama is one of the main reasons we got here in AI…
Wait.. are you saying it wasn't? Just releasing it in that form was a big deal ( and heavily discussed on HN, when it happened ). Not to mention, a lot of the work that followed on llama partly because it let researches and curious people dig deeper into internals.
Yann LeCun also keeps distorting what open source is. Neither Llama nor DeepSeek are open source, and they never were. Releasing weights is not open source - that’s just releasing the final result. DeepSeek does use a more permissive license than Llama does. But they’re not open source because the community does not have the necessary pieces to reproduce their work from scratch.
Open source means we need to be able to reproduce what they’ve built - which means transparency on the training data, training source code, evaluation suites, etc. For example, what AI2 does with their OLMo model:
Deepseek R1 is the closest thing we have to fully open-source currently. Open enough that Huggingface is recreating R1 completely out in the open. https://github.com/huggingface/open-r1
What they’re recreating is the evidence that some of the techniques work. But they’re starting with R1 as the input into those steps, not starting from scratch. I don’t think their work includes creating a base model.
The fundamental problem is that AI depends on massive amounts of IP theft. I’m not going to argue if that’s right or wrong, but without it we won’t even have open weights models.
I don’t buy this at all. If DeepSeek can surpass proprietary models by “profiting” from open research and open source, why couldn’t the proprietary models do the same? Companies making proprietary models have the advantage of using w/e is out there from the open source community AND the proprietary research they have been working on for years.
> If DeepSeek can surpass proprietary models by “profiting” from open research and open source, why couldn’t the proprietary models do the same?
They can “profit” (benefit in product development) from it.
They just can't profit (return gains to investors) much from it, because that requires a moat rather than a market free for all that devolves into price competition and drives market clearing price down to cost to produce.
Yes but in proprietary research you've got fewer peers to bounce ideas off of, and you've got extra constraints to deal with re: coming up with something that's useful in tandem with whatever other proprietary bits are in your stack.
All that cloak and dagger stuff comes at a cost, so it's only worth paying if you think you can maintain your lead while continuing to pay it. If the open source community is able to move faster because they are more focused on results than you are, you might as well drop the charade and run with them.
It's not clear that that's what will happen here, but it's at least plausible.
> DeepSeek can surpass proprietary models by “profiting” from open research and open source, why couldn’t the proprietary models do the same?
DeepSeek did something legitimately innovative with their addition of Group Relative Policy Optimization. Other firms are certainly free to innovate as well.
But it sounds like, from that quoted statement, that LeCun from Meta thinks “open sourced work” is why China was able to surpass (or at least compete with) American AIs. Which sounds like a lame excuse for Meta.
Putting too much thought into the statement Meta's chief AI scientist made about how the new AI innovation is actually because of Meta is probably not going to be fruitful.
If training runs are now on the $6MM/run for SOTA model scale, I think on the contrary: closed labs are screwed, in the same way that Linux clobbered Windows for server-side deployments. Why couldn't Windows just copy whatever Linux did? Well, the codebases and research directions diverged, and additionally MS had to profit off of licensing, so for wide-scale deployments Linux was cheaper and it was faster to ship a fix for your problem by contributing a patch than it was to beg and wait for MS... Causing a virtuous cycle (or, for Microsoft, a vicious cycle) where high-tech companies with the skills to operate Linux deployments collaborated on improving Linux, and as a result saw much lower costs for their large deployments, while also having improved flexibility, which then incentivized more companies to do the same. The open models are becoming much cheaper, and if you want something different you can just run your own finetune on your own hardware.
Worse for the proprietary labs is how much they've trumpeted safety regulations. They can't just release a model without extensive safety testing, or else their entire regulatory push falls apart. DeepSeek can just post a new model to Hugging Face whenever they feel like it — most of their Tiananmen-style filtering isn't at the model level, it's done manually at their API layer. Ditto for anyone running finetunes. In fact, circumventing filtering is one of the most common reasons to run a finetune... A week after R1's release, there are already uncensored versions of the Llama and Qwen distills published on HF. The open source ecosystem publishes faster.
With massively expensive training runs, you could imagine a world where model development remained very centralized and thus the few big labs would easily fend off open-source competition: after all, who would give away the results of their $100MM investment? Pray that Zuck continues? But if the training runs are cheap... Well, there are lots of players who might be interested in cutting out the legs from the centralized big labs. High Flyer — the quant firm that owns DeepSeek — no longer is dependent on OpenAI for any future trading projects that use LLMs, for the cost of $6MM... Not to mention being immune from any future U.S. export controls around access to LLMs. That seems very worthwhile!
As LeCun says: DeepSeek benefitted from Llama, and the next version of Llama will likely benefit from DeepSeek (i.e. massively reduced training costs). As a result, there's incentive for both companies to continue to publish their results and techniques, and that's bad news for the proprietary labs who need the LLMs themselves to be profitable and not just the application of LLMs to be profitable... Because the open models will continue eating their margins away, at least for large-scale deployments by competent tech companies (i.e. like Linux on servers).
Azure Linux is Linux. Microsoft is one of the biggest contributors to Linux in general, in terms of commits/release, and has been for a lot of years now. That doesn't mean Windows is doing what Linux did - Windows is largely still entirely different from Linux at both the kernel and user's pace level, and improvements in one have little to no bearing on the other.
I'm still not sure why they keep LeCun at Facebook; his single most-cited contribution to the field in 2024 has been with NYU[0], not Facebook. What is his role at Facebook exactly, has he explained it? I recall him making all the wrong predictions in 2023 what's changed? Chollet is similarly a mystery to me; it feels like these guys were busy riffing CNN's when the Transformer came about and since then have been trying to far-out in search of gold.
I'm also a bit unclear on why LeCun is so well regarded. I've nothing against him, and his opinions shared on Twitter seem eminently sensible, but at the end of the day it seems his main accomplishment (and/or credit assignment) was inventing CNNs back in the 80's and using them for reading handwriting on checks.
Looking back at the PDP handbook, it's not even clear that LeCun deserves the credit for CNNs, and he himself gives credit for the core "weight sharing" idea to Rumelhart.
Chollet's claim to fame seems to be more as creator of Keras than researcher, which has certainly been of great use to a lot of people. He has recently left Google and is striking out to pursue his own neuro-symbolic vision for AGI. Good luck to him - seems like a nice and very smart guy, and it's good to see people pursuing their own approaches outside of the LLM echo chamber.
It's not and it hasn't surpassed GPT. A lot of that is headline hype.
They literally used GPT and Llama to help build DeekSeek, it responds thinking that it's GPT in countless queries (which people have been posting screenshots of). They 'cheated' exactly as Musk did to build xAI's model/s. So much of this is laughable scaremongering and it's absolutely not an accomplishment of large consequence.
honestly reads like someone trying to justify his massive salary to his boss who is realizing he can just hire someone for 30x less money.
isn't LeCun basically admitting that he and his team didn't have the creative insight to utilize current research and desperately trying to write off the blindside with exceptionalism?
It's like saying that a diesel engine is 6x more efficient than a steam engine, so the guys who spent time working on steam engines just wasted their time and money.
The thing is that the steam engine guys researched thermodynamics and developed the mechanics and tooling which allowed the diesel engine to be invented and built.
Also, for every breakthrough like DeepSeek which is highly publicized, there are dozens of fizzled attempts to explore new ideas which mostly go unnoticed. Are these wasted resources, too?
Given your take, this is a meaningless question, no?
As you point out, all resource usage that lead up to the creation of the diesel engine were necessary preconditions. While one might be able to imagine a parallel universe where the diesel engine was created in another way without all the things in between that might feel like a waste, that is not this universe. In this one, it took what it took.
Same goes for AI. That AI researcher had to eat that sandwich double wrapped in plastic, subsequently placed in another plastic bag in order to get to where he got. Which might feel like a "waste of resources". I am sure you can easily imagine a parallel universe where he didn't eat something that used up so much plastic. But that was the precondition necessary in this universe.
So, ultimately, either everything is a waste of resources or nothing is. And there is no meaning in trying to find a distinction between those two.
Would this extrapolate to the thousands of lightbulb prototypes it took to arrive at the first working one? Rinse repeat for your preferred innovation.
Resource allocation in this context isn’t at all binary.
LeCun has nothing to do with LLamA ... that was built by Meta's GenAI group.
LeCun is in a different part of the organization - FAIR (FaceBook AI Research), and isn't even the head of that. He doesn't believe that LLMs will lead to AGI, and is pursuing a different line of research.
“To people who see the performance of DeepSeek and think: ‘China is surpassing the US in AI.’ You are reading this wrong. The correct reading is: ‘Open source models are surpassing proprietary ones.’ DeepSeek has profited from open research and open source (e.g., PyTorch and Llama from Meta). They came up with new ideas and built them on top of other people’s work. Because their work is published and open source, everyone can profit from it. That is the power of open research and open source.”
[1] https://www.forbes.com/sites/luisromero/2025/01/27/chatgpt-d...