On a good account, downvotes for unpopular opinions would still be balanced with upvotes from the popular opinions. I often post stuff that goes against the mainstream in here and my karma is still good despite taking the occasional hit.

I agree that the system isn't perfect (it would fail if you only post unpopular - but otherwise valid and within the rules - opinions), but wouldn't it still be better than the current situation?