I'm speaking in generic terms, of course. All models should be at the very least unbiased, though -- that's the least demand you can make of them (and generally an easy one to check, compared to other properties, such as having minimum variance of all possible models and such.)
I sort of agree in principle but I’m not sure what unbiased means for a language model or how you would measure it, so this doesn’t sound all that easy to me.
