> No you are probably overestimating the cost by 1-2 orders of magnitude.
You are right! Wow. Thank you for correcting me.
> GPT-3 probably cost under $5 million,
Is that one training run or includes all the fiddling to find the right hyperparameters? Or there aren't many of those in these training or they are not that sensitive?
I think they probably did a lot of hyperparameter searching to train the smaller models and then extrapolated for the largest model, but I'm just guessing. OpenAI had a finite amount of money when they were training GPT-3, they likely do it differently now that inference costs are significant compared to training costs.
You are right! Wow. Thank you for correcting me.
> GPT-3 probably cost under $5 million,
Is that one training run or includes all the fiddling to find the right hyperparameters? Or there aren't many of those in these training or they are not that sensitive?