For many specialized tasks you can run BERTs (and simpler models in general) at scale, with lower latency, at lesser cost, with similar or even better results.