AI has created a new standard from which “average” is defined. Rather than a real life analog to a mathematic mean, it’s simply understood to be the level at which AI performs.

It naturally assumes the average of its training data.

I'm not sure this is necessarily the case, especially considering techniques like RLHF that can improve a model's performance.