No, the apolitical don't feel the need to discuss politics at work; they'll get on just fine. It's the political who will leave places they disagree with, accelerating polarization.
Plenty of people (from any point on the spectrum) identify as apolitical because they think their beliefs about the world are the neutral standard. They don't feel the need to discuss politics, but in their mind that doesn't cover things like that crazy story on the news about parents who were letting their daughter pretend to be a boy, or mentioning that their sister's wife is visiting.
Yes, yes, I hear what you're saying. Women who wear pants, black people who insist on getting paid for their work, or casually mention their white spouses, people who wear yarmulkes and turbans? Those apolitical people? Why should they feel the need to go anywhere?