I use Firefox for most things at home, including a lot of browsing YouTube. I don't have the issues you're describing. Though I am on YT Premium and use uBlock.
They might not show you the ads but I guarantee the code is still in the background keeping track of your habits so they can show you more targeted ads elsewhere.
uBlock, not much but SponsorBlock is very useful.
I am using all of those with YT Premium and I do have issues. I was recently offered their redesign and it made it a lot worse. It's hard to watch a video uninterrupted now.
Note that I also chose not to allow DRM videos, so no Netflix on Firefox for me. This might actually influence the codecs used by Youtube and these issues might only appear on "fallback" codecs when you refuse DRM ones.
