Every single argument that can be made in favor of speech restrictions can be made in favor of voting rights restrictions as well.

Nope.

The idea behind freedom generally is that despite the fact that most people don't behave optimally, it is unjust to curtail their right to behave suboptimally, so long as that behaviour doesn't directly impact others.

So we agree then, because the spread of misinformation causes damage for other people, clearly.

> so long as that behaviour doesn't directly impact others

I'm not sure how one person being misinformed directly harms another. Can you elaborate? Surely the misinformed person would have to act to directly harm another person, which is already covered by existing laws I imagine.