A counterpoint is that at least on reddit the downvote button is rarely well used. It's always a "I disagree" button, so both bad and controversial things are pushed down. It's not the case here in general I think, but I actually think this is a good argument in favor of removing dislikes. It filters some bad things but also just controversial things for one reason or another.