Try getting some functional safety certification on a Linux distribution and you see why Linux is not really gaining much weight in certain areas of the embedded market.
However, it is getting small inroads in the instrument clusters for Automobiles. (http://www.genivi.org/)
VxWorks is a real-time OS. I didn't know whether there was a real-time variant of Linux and what I found was that there was, but Wind River acquired the company that was working on it and essentially killed it.
From what I understand, Linux runs the rover simulator environment - known as RSVP. It consists of a DSL that is used to direct the rover's operations and a visualization environment. This way the ground crew can first plan and test the activities they want the rover to perform in this simulated environment and then upload the action plan to the rover - so overall Linux plays a pretty important part in the the rover tech stack.
Most probably the initial rovers ran VxWorks RTOS, because back then it was the best choice (even now it isn't a bad choice at all) - complex systems are built incrementally.
Aside from what others have mentioned keep in mind that something being sent to another planet usually has 5-10 year old technology at best. For reference in 2003 Linux 2.4 was the latest stable version.