This is the absolute biggest grift of the century by the groq team. They never shared actual TCO, and I remember a Seminalaysis article about the power consumption being actually insane - this makes sense because they scale the number of chips to fit a single model when they have no dram. They have good inference latency but there was no way the economics were going to work out. Meanwhile Nvidia with every advantage in the world decides they’re worth 20B? It actually doesn’t make sense at all. The only scenarios the groq system would be worth it is in the exact throughput-optimized scenarios Nvidia already thrives in.
If the neo-nazis celebrate it as a Nazi salute, I don’t think it really matters what others think. In addition to that, seeing it with my own eyes is all the confirmation I need.
On his intelligence SpaceX and Tesla were/are revolutionary companies, but seeing him buy twitter and then send the DOGE five things email makes me feel like maybe it was more of a right place right time sort of thing and not management prowess. I’ll give him credit for hiring the right people which is a skill but other than that his blunders are just too difficult to ignore.
It’s because capitalism assumes a free market with competition. If you allow monopolies to thrive, you will not get those benefits. It’s just that some of these types of markets have different dynamics due to their structure. E.g. natural monopolies where the barrier to entry is huge up front costs. Interestingly the AI startup ecosystem is raising enough money to surpass the barrier of needing a ton of data to train AI.
That's true but it's not just monopolies per se. Another big part of it is information flow. The kind of idealized free markets that "work" involve relatively transparent interactions between buyers and sellers. Sort of like, well, a market --- an old-fashioned bazaar-type market with sellers offering their wares. You can walk down the line and see 10 people are selling apples or saddles or whatever and you can compare the prices and the products and make your decision.
Even in that setup, people can try to game the market. They can make something that looks like a good saddle and sell it to you and then it falls apart not too long afterward. They can get you to agree to a price but then tell you the stirrups aren't included even though they're attached to the demo model. They can ask for half payment up front while they custom make your item, then skip town.
And mechanisms sprung up to prevent this: regulation. Some are market-internal (reputation) and some are enforced (people can report you to the authorities for selling fraudulent goods, and you can be jailed or whatever).
The problem is mainly that nowadays companies have turned the majority of their innovation energy towards this kind of market-gaming meta-activity. It's no longer about goods, services, buyers, sellers, or any of those things. It's just about finding new ways to manipulate the market itself.
This is what the article seems to be saying, and I agree. I'm not sure I'd call it "hype", though. It's not that "the hype is the product", it's that the market activity is not oriented towards products at all. Products have become like abstract proxy tokens that are moved around to simulate what we think of as market activity, but all the real activity is happening in the meta-market.
> It’s because capitalism assumes a free market with competition
But even in the cases where we do have a free market, we're often seeing one company fiddle with quality, maybe drop the price a little, then the rest quickly follow and price goes right back up across the board.
not attacking your thinking, but the terms "capitalism" and "free market" aren't consistently well enough defined to capture the nuances of what this type discussion requires. At a minimum, capitalism is the right/ability to own something without others taking it from you, and free markets are markets you can participate in if you would like without asking permission from the govt or belonging to a guild.
capitalism works best for everybody on average when free markets are competitive, but when they are not, markets still work, they just work better for some, worse for others but better than nothing, and also overall worse for everybody because markets are not zero sum. The problem with a lot of what-turns-out-to-be left-wing and or populist thinking on markets is the assumption that markets are zero sum, "if there is a winner, there must be a loser", which while an attractive idea turns out to be false.
same is true of the completely overblown idea that people are not rational. people are not perfectly rational, but when it comes to parting with their money they are much more rational than they are not. If it were not true, people wouldn't be living rationally measurably better lives today than 100, 200, etc. years ago. there are many other sources of noise in measuring that swallow irrationality up with the noise. (yes, selling gambling to gambling addicts is an irrational money printing machine, but civilization has not collapsed)
For centuries as capitalism developed from more primitive systems (like feudalism and mercantilism), goods and services that gain value from social network effects were well beyond anyone's imagination. The printing press survived the church's attempts at suppression, but nobody back then could have conceived of a service that automatically distributed copies of your books to your friends — much less one that could profit from knowing who your friends are, rather than from explicitly charging you for the service.
Yeah where they have every inch of SF mapped, and then still have human interventions. We were promised no more human drivers like 5-7 years ago at this point.
High speed connectivity and off vehicle processing for some tasks.
Density of locations to "idle" at.
There are a lot of things that make all these services work that means they can NOT scale.
These are all solvable but we have a compute problem that needs to be addressed before we get there, and I haven't seen any clues that there is anything in the pipeline to help out.
Everyone always thinks this at least in big tech I’ve never heard a PM or exec say a market is not winner take all. It’s some weird corpo grift lang that nothing is worth doing unless its winner take all.
I’m amazed this is even viewed as a “hot take” tbh most of what he said here is pretty high level of abstraction and standard practice for custom hardware. In essence I feel like he’s saying nothing really controversial other than publicly calling out TT for too many abstraction layers (and tbh it’s just in a readme). This is completely fine, he’s a user and this is his experience.
I’m a dev working on torch.compile at meta (previously I worked on ML focused FPGAs) and the approach I would use is build a static graph compiler, use torch.compile (and probably JAX) as graph extraction front-ends and call it a day. I feel like hardware companies don’t know how to handle the flexibility of PyTorch and as a result develop their own APIs which is mistake #1 and virtually makes it impossible to get any market penetration once you head down that path because nobody will ever ever rewrite their models for your hardware when they don’t even know what perf they will get, the risk is just too high. As a result, hardware companies offer inference APIs which hide all of this behind a REST API to basically paper over the lack of generality of the software/hardware interface. This is convenient because then nobody actually knows the perf/$ and they can burn VC money for as long as they want. Whether this is a viable business model or not, we will have to wait until they go public to actually see what their true inference costs are.
To sum it up, start from PyTorch and work your way down to your hardware, this is the only general way if you want to actually sell chips and not just constantly port the model of the day to your hardware.
The idea that cutting research will make a dent in the budget is a fantasy. NSF has a budget of 10 billion. Stop rationalizing gutting crucial programs because of “the deficit” Medicare, social security and the military are the main costs in the US budget. Sure universities are bloated, tackle that problem separately then.
I used this to onboard to the PyTorch team a few years ago. It’s useful for understanding the key concepts of the framework. Torch.compile isn’t covered but the rest of it is still pretty relevant.
