Memory bandwidth is quite low for the price. $3,000 for 128GB @ 273GB/s.
You can buy a Mac Studio with an M4 Max for $3,500. 128GB unified memory @ 546 GB/s.
You're also getting a much faster CPU and more usable daily computer.
I suppose if you're a CUDA developer, this thing is probably better though I doubt you'd be training anything worthwhile on a computer this weak. Nvidia advertises the DGX Spark as a machine that mimics very large DGX clusters so the environment is the same. But in terms of hardware specs, it's very disappointing for $3,000.
DGX Station is another beast. It's Blackwell Ultra with 288GB HBM3e and 496GB LPDDR5X. I'm guessing $150k - $200k.
I love MacBooks and Mac machines however they aren't cut out for specialised work.
I have a M4 Max with 128 GB RAM. If I play Civ VI (a game released in 2016) on it for a few hours without limiting the FPS it will heat up till it turns itself off.
It's not cut out for heavy loads consistently like games or crypto mining. It's cut out for a heavy loads like Xcode compiling for a few minutes then back to editing text.
My gaming machine which has poorer performance (in fps terms) and is equipped with AMD 3700 and a 2080 Super can play Civ VI indefinitely without breaking a sweat.
Couple of things:
This must be new to the M4? I've played Civ VI for hours and hours on an M1 Max without issue.
If someone is looking at this Nvidia box, they're likely fine with a desktop footprint, in which case they'd be looking at the Mac Studio which should not have any thermal issues whatsoever. I'm guessing you're on a laptop?
If they are insistent on a laptop format, you can alleviate overheating issues pretty easily with some thermal pads and running the laptop on a cooling base when you're running heavy operations:
That's pretty surprising and breaks my mental model of how these chips perform. Do you think that's because they don't have the raw FLOPs, something inefficient in the Apple/Metal rendering pipeline, something about Civ VI and how it was converted (i.e. x86 or DirectX emulation), or something else entirely?
I've had no trouble with a (base) M4 mini for regular dev work, though I compile remotely and haven't played any games on it.
FWIW I also just started seeing comments from other devs on my team today about their M4 mini's thermally shutting down during a compile run in Android Studio frequently.
Did you change the fan settings? Default is for fans to be quiet. If you go to energy management in settings you can choose “more power” or something instead of automatic. Your Mac will be louder but throttle less.
I’m curious if Civilization VII has the same issue. Baldur’s Gate 3 was released around the same timeframe as Civ 6 and has some pathological behavior on higher-core-count machines.
Only looking at memory bandwidth as a measure of performance give you an incomplete picture. You also need to look at how much of that bandwidth your processor (CPU, GPU, NPU, etc.) can actually consume because it can be far less than the memory modules are capable of.
You can also get an Epyc 9115 for $800, motherboard for $640, and 12 16-GiB ddr5-6400 dims for $1400, that gives you 614.4 GiB/sec, for around $2800. You may also want to add in a small GPU to do prompt processing (inference on a CPU is memory bandwidth bound, prompt processing is processing bound).
I was going by the number of memory channels the CPU spec says it supports (12). But apparently I was wrong, as that gets bottlenecked by the number of CCDs on the chip. In which case you would need to go with a much higher end epyc processor, and then there are other limits. So much for napkin math
The M series CPUs have very good memory bandwidth and capacity which lets them load in the billions of weights of a large LLM quickly.
Because the bottleneck to producing a single token is typically the time taken to get the weights into the FPU macs perform very well at producing additional tokens.
Producing the first token means processing the entire prompt first. With the prompt you don't need to process one token before moving on to the next because they are all given to you at once. That means loading the weights into the FPU onlu once for the entire prompt, rather than once for every token. That means the bottleneck isn't the time to get the weights to the FPU, it's the time taken to process the tokens.
Macs have comparatively low compute performance (M4 Max runs at about 1/4 the FP16 speed of the small nvidia box in this article, which itself is roughly 1/4 the speed of a 5090 GPU).
Next token is mostly bandwidth bound, prefill/ingest can process tokens in parallel and starts becoming more compute heavy. Next token(s) with speculative decode/draft model also becomes compute heavy since it processes several in parallel and only rolls back on mispredict.
Why do all articles say "with 784 GB of unified memory" for the Station when on the official spec page it's not listed as unified but as "GPU Memory: Up to 288GB HBM3e | 8 TB/s" and "CPU Memory: Up to 496GB LPDDR5X | Up to 396 GB/s"?
TLDR, AMD machines based on HX395 offer comparable memory bandwidth and size at lower cost ($2000) but lack the high speed networking and compute power (60TF FP16 on AMD and 120TF on NVIDIA)
Apple M4 Max costs 23% more with equivalent memory, has twice the memory bandwidth but again no fast networking and significantly less compute (30TF FP16 on M4 Max).
For inference on large language models the extra memory bandwidth probably means M4 Max is the fastest option unless you use large batches or long contexts. For training large models needing more than 128GB RAM two of these Nvidia boxes is probably fastest.
Another thing lacking on AMD machines is the software support. Not just that a lot of stuff is CUDA-only still, but AMD themselves struggle to support their own hardware for ROCm. According to their docs ROCm for HX395 is supported on Windows[0] but not Linux[1]. Though further reading indicates the Windows version is also just a subset of Linux ROCm that seems to not support PyTorch or anything. I really don't know what they're doing over there but I hate it.
Is it possible to install the DGX stack on other distros besides Ubuntu? Nvidia say it's possible but does anyone actually did it?
>You also have the option to install the NVIDIA DGX Software Stack on a regular Ubuntu 22.04 while still benefiting from the advanced DGX features. This installation method supports more flexibility, such as custom partition schemes.
this reminds me of the (now discontinued) lambda tensorbook that I bought a few years back. AI computers are interesting because they are extremely similar to really good gaming computers but the marketing (and some of the hardware) is different.
I don't fully understood why they are separate from gaming computers other than marketing, but I am very happy with my tensorbook. I got it when I was very young and impulsive I don't know if I would get it again, although that fact alone speaks to the computer's longevity.
Gaming computers usually run Windows and games. AI computers often run Linux and AI software. The differences require different configurations of both hardware and software.
So, they make products preloaded with configurations for each market segment. The customers pay for all the headaches this saves them of custom setups.
Good for training, definitely a bad idea for inference. But if you are spending that much money, why not just buy the equivalent of GPUs? You could buy 10 12GB 3060s for that price.
For LLM developers, is there really no advantage to having a big block of unified memory, rather than a bunch of devices with a small amount of memory each?
As with buying consumer GPUs from Nvidia now, these prices mean nothing, because it will be impossible to buy one of these at MSRP.
The actual price will be 2-3x this from some scalper who has a giant pile of them while people who want to do actual work with them pay through the nose.
That's assuming the connectors don't catch on fire, no cores are missing the way ROPs have been missing with 5090s, etc.
They’ve offered reservations that, when selected, give the buyer a 4 day option to purchase. The reservations are accessible enough I was able to grab one without much effort
So would seem like they’re managing the allocations pretty carefully, or at least trying harder than they have before.
Too bad it won't be available at scale. We had a customer who asked us if he could get 200 of those boxes for his AI scientists. But they're apparently only sold in single quantities on the NVidia store or something, not through the official distribution channels
$3k is already close to the price of just a 5090, so if you can have a reasonably fast but way more (equivalent to) VRAM (128gb) machine sounds great for a hobbyist assuming you can easily run and train llama, Stable Diffusion and similar vram-hungry projects.
Nvidia completely skipped a middle tier. DGX Spark is very underpowered for $3,000. DGX Station is likely $150k - $200k. There is no middle tier at all.
A middle tier could be something like an RTX 5090-like chip connected to 128GB - 256GB of GDDR7 RAM for $15k - $30k.
I would support this if it meant that Nvidia would improve their Linux drivers. As it stands, though, a device running SteamOS on Nvidia hardware sounds painful.
It'll take a long time for them to rebuild their reputation on that front. I threw away a pile of perfectly serviceable NVIDIA desktop boards a while back because they were "supported", but they couldn't do stuff like run modern compositors or even XScreenSaver.
These were high-end boards that worked fine in Linux when I bought them (as long as you were willing to periodically patch + recompile the kernel from a text console).
Anyway, once they have feature parity with Windows + current CUDA with a 100% open source stack going back a decade, then I'll consider them again. Until then, I'll happily buy an AMD that's 90% as fast for the same price.
There are already a ton of high-end gaming consoles you can buy today with nvidia chips. In fact, this is exactly what I did for my own AI machine (dual 3090s).
A console would be a device like an XBox, PS5 or Switch. Of those, only the Switch (current + next gen) are NVIDIA devices. The Steam Deck and the other current generation consoles are all AMD.
Am I the only one who became curious only to become more interested in the larger DGX station workstation model they are offering with an unaccounced price tag at the time?
The comparably priced Mac comes with half the RAM, so if that's a constraint, it's already lost. For applications that aren't memory constrained it would be very interesting to see some benchmarks. Although I suspect it's going to come down to the quality of the relevant CUDA vs Metal libraries as much as anything else.
How are you arriving at double the price? 16-CPU-core Mac Studio with storage upgrade to 1TB and RAM upgrade to 128GB comes out to $3,700. Add $700 if you want to upgrade to the 28-core CPU (there’s nothing directly comparable to the DGX Spark’s 20-core CPU).
None of this speaks to the actual GPU performance. The only spec the DGX Spark webpage mentions is “1,000 FP4 TOPS”.
You can buy a Mac Studio with an M4 Max for $3,500. 128GB unified memory @ 546 GB/s.
You're also getting a much faster CPU and more usable daily computer.
I suppose if you're a CUDA developer, this thing is probably better though I doubt you'd be training anything worthwhile on a computer this weak. Nvidia advertises the DGX Spark as a machine that mimics very large DGX clusters so the environment is the same. But in terms of hardware specs, it's very disappointing for $3,000.
DGX Station is another beast. It's Blackwell Ultra with 288GB HBM3e and 496GB LPDDR5X. I'm guessing $150k - $200k.