Sorry, I have to point out: Tensorflow is not comparable to CUDA. Tensorflow is a (arguably) high level library that links against CUDA to run on NVIDIA GPUs, as does PyTorch (the main competitor).
Comparatively few people have “deep” experience with CUDA (basically Tensorflow/Pytorch maintainers, some of whom are NVIDIA employees, and some working in HPC/supercomputing).
CUDA is indeed sticky, but the reason is probably because CUDA is supported on basically every NVIDIA GPU, whereas AMD’s ROCm was until recently limited to CDNA (datacenter) cards, so you couldn’t run it on your local AMD card. Intel is trying the same strategy with oneAPI, but since no one has managed to see a Habana Gaudi card (let alone a Gaudi2), they’re totally out of the running for now.
Separately, CUDA comes with many necessary extensions like cuSparse, cuDNN, etc. Those exist in other frameworks but there’s no comparison readily available, so no one is going to buy an AMD CDNA card.
AMD and Intel need to publish a public accounting of their incompatibilities with PyTorch (no one cares about Tensorflow anymore), even if the benchmarks show that their cards are worse. If you don’t measure in the public no one will believe your vague claims about how much you’re investing into the AI boom. Certainly I would like to buy an Intel Arc A770 with 16GB of VRAM for $350, but I won’t, because no one will tell me that it works with llama.
With respect to the incompatabilities with PyTorch and TensorFlow - given that the AMD and Intel GPU drivers are more likely to be open sourced - do you believe the open source community or a third party vendors will step in to close the gap for AMD/Intel?
It would seem a great startup idea with the intent to get acqui-hired by AMD or Intel to get into the details of these incompatibilities and/performance differences.
At worst it seems you could pivot into some sort of passive income AI benchmarking website/YT channel similar to the ones that exist for Gaming GPU benchmarks.
Drivers are only the lowest level of the stack. You could (in principle) have a great driver ecosystem and a nonexistent user-level ecosystem. And indeed, the user-level ecosystem on AMD and Intel seems to be suffering.
For example, I recently went looking into Numba for AMD GPUs. The answer was basically, "it doesn't exist". There was a version, it got deprecated (and removed), and the replacement never took off. AMD doesn't appear to be investing in it (as far as anyone can tell from an outsider's perspective). So now I've got a code that won't work on AMD GPUs, even though in principle the abstractions are perfectly suited to this sort of cross-GPU-vendor portability.
NVIDIA is years ahead not just in CUDA, but in terms of all the other libraries built on top. Unless I'm building directly on the lowest levels of abstraction (CUDA/HIP/Kokkos/etc. and BLAS, basically), chances are the things I want will exist for NVIDIA but not for the others. Without a significant and sustained ecosystem push, that's just not going to change quickly.
I think this is what George Hotz is doing with tiny corp, but I have to admit I have little hope. Making asynchronous SIMD code fast is very difficult as a base point, let alone without internal view of decisions like “why does this cause a sync” or even “will this unnecessary copy ever get fixed?”. Unfortunately AMD and especially Intel don’t “develop in the open”, so even if the drivers are open sourced, without context it’ll be an uphill battle.
To give some perspective, see @ngimel’s comments and PRs in Github. That’s what AMD and Intel are competing against, along with confidence that optimizing for ML customers will pay off (clearly NVIDIA can justify the investment already).
This kind of software development is hard and expensive. I do not think that this can enable you to make enough income from benchmark website or YT channel, considering most people are not interested in those low level details.
Comparatively few people have “deep” experience with CUDA (basically Tensorflow/Pytorch maintainers, some of whom are NVIDIA employees, and some working in HPC/supercomputing).
CUDA is indeed sticky, but the reason is probably because CUDA is supported on basically every NVIDIA GPU, whereas AMD’s ROCm was until recently limited to CDNA (datacenter) cards, so you couldn’t run it on your local AMD card. Intel is trying the same strategy with oneAPI, but since no one has managed to see a Habana Gaudi card (let alone a Gaudi2), they’re totally out of the running for now.
Separately, CUDA comes with many necessary extensions like cuSparse, cuDNN, etc. Those exist in other frameworks but there’s no comparison readily available, so no one is going to buy an AMD CDNA card.
AMD and Intel need to publish a public accounting of their incompatibilities with PyTorch (no one cares about Tensorflow anymore), even if the benchmarks show that their cards are worse. If you don’t measure in the public no one will believe your vague claims about how much you’re investing into the AI boom. Certainly I would like to buy an Intel Arc A770 with 16GB of VRAM for $350, but I won’t, because no one will tell me that it works with llama.