How about a model that can detect the semantic shift of "Amazon" from "river" to "company"? https://arxiv.org/abs/1703.00607

They're not even using transformers, just simple word embeddings.