That's what an ERV or HRV is for: https://en.wikipedia.org/wiki/Energy_recovery_ventilation https://en.wikipedia.org/wiki/Heat_recovery_ventilation

Et voilà, now it's no longer terrible for efficiency since you're recovering 70-90% of the energy. For fixed-plate exchangers there's no cross-contamination, either.