My understanding from the more hardware savvy friends I have is that AMD is better for inference (so when you run a model) vs for training Nvidia is still king. Would be interesting if OpenAI does in fact take AMD's offer and how they use it (if they even share openly).
The reason I think its worth noting is that Inference is definitely what I imagine an insane amount of their computer is / will be spent on. I could see a very optimal setup where training is all Nvidia, but inference becomes reliant on AMD.
Larger memory, weaker comms. You can optimize for this by doing things like increasing batch size/data parallelism vs sharding schemes with more comms.
At scale training won’t be able to avoid comms entirely, while many models can fit in a single MI300 for serving.
That fact that this even needs to be pointed out but is normalized in the AI industry. Sadly, in the history of Open AI, they've been anything but open.
