When a new generation like this is released, will a typical AI company replace the current GPUs? Is there a chance to acquire the older versions for private use or is it too early for that?
> will a typical AI company replace the current GPUs?
(Not speaking for everyone of course)
No we keep the current GPUs as well, the one we throw away are several generations old (like K80s), so not very interesting.
For private use, those GPUs are really not worth it. Once their used price reaches an acceptable level, you can be sure that the new general public GPUs will outperform them.
For example a Titan RTX is probably still twice as cheap as V100, while only being ~20% slower.
Edit: Actually a Titan RTX is about the same price as a P100 now, while being much better.
A fun quick of GPU pricing economics is that on Google Cloud platform, the relatively recent T4 (on Turing and has FP16 support) is cheaper than the ancient K80s. https://cloud.google.com/compute/gpus-pricing
That's not a great comparison at all though. The K80 is a general purpose chip while the T4 is explicitly marketed as an inference chip. The K80 has more ram (super important for batch sizes during training), can access that ram faster (480 GB/s versus 320 GB/s), and is an overall more powerful chip than the T4 is.
Those metrics are for both GPUs on a board; the GCP K80 only uses one GPU, so those performance metrics in theory would be halved (notably, the GCP K80 has 12 GB VRAM vs. T4's 16 GB VRAM), and it's still more expensive than a T4.
I don't know any specifics about GCP but in general for datacenter design, heat dissipation and energy costs are really important. When you shrink the process width, you get incredible savings in power (saves $$) -> less heat dissipation per flop (saves $$ because you don't have to remove that heat).
That and I suppose there could be a premium for older setups of any kind if for whatever there are licensing agreements that are negotiated for certain hardware (or for certain numbers of cores).
When a new generation like this is released, will a typical AI company replace the current GPUs? Is there a chance to acquire the older versions for private use or is it too early for that?