Exactly. So what's the issue with people who don't drive also contributing to road infrastructure?

Yes, a lot of the cost is bundled into taxes. But that isn't unique to roads & cars, that happens everywhere. Again, are you going to ask couples with children to truly internalise the cost of public schooling? Are you going to ask non-travellers to truly internalise the cost of airports? It's just an irrelevant point.