This is exactly my experience and I also have a YouTube premium account. I use YouTube religiously and have my own system of improving the algorithm so that the vast majority of content recommended is something I might watch.

If I don't like a video, I hit dislike and will maybe even remove it from my watch history.

I suspect that most who are against this change are not heavy YouTube users.