Huh. I mean it makes sense to train end-to-end on all the interrelated tasks involved in driving but putting a whole-ass language model in the middle of that seems like a stunt. I wonder if it does better than like, any random transformer not trained on language first? Still, I hadn't heard that so I guess I was wrong about that one
No, you were right, this appears to be just research on how applicable LLMs could be to the space. They talk about the improvements their LLM makes, especially in being multimodal vs training multiple independent models, but also the limitations that appear to prevent it from being useable as it is. Maybe some form if it will be used some day (it does seem like it would be useful to have semantic understanding of the world integrated into the system), but at least as of when this was published, it's not actually used.
