This. I can't look at the passenger ether when I'm driving, even on very familiar roads. I never thought this is awkward - but if the passenger is not a friend/ close acquaintance, I mention it casually that I am in fact listening to them even if I'm not looking at them.
The best I could do is keep nodding or make agreeing sounds.