I'm not in the space either but I think the answer is an emphatic yes. Three categories come to mind:
1. Online trolls and pranksters (who already taught several different AIs to be racist in a matter of hours - just for the LOLs).
2. Nation states like China who already require models to conform to state narratives.
3. More broadly, when training on "the internet" as a whole there is a huge amount of wrong, confused information mixed in.
There's also a meta-point to make here. On a lot of culture war topics, one person's "poisonous information" is another person's "reasonable conclusion."
Im looking forwards to protoscience/unconventional science and perhaps even that what is worthy of the fringe or pseudoscience labels. The debunking there usually fails to adress the topic as it is incredibly hard to spend even a single day reading about something you "know" to be nonsense. Who has time for that?
If you take a hundred thousand such topics the odds they should all be dismissed without looking arent very good.
Apparently, you haven't been on that Internet thingie in the last five years or so... :-)
But I do agree with your point. What's interesting is the increasing number of people who act like there's some clearly objective and knowable truth about a much a larger percentage of topics than there actually is. Outside of mathematics, logic, physics and other hard sciences, the range of topics on which informed, reasonable people can disagree, at least on certain significant aspects, is vast.
That's why even the concept of having some army of "Fact Checkers" always struck me as bizarre and doomed at best, and at worst, a transparent attempt to censor and control public discourse. That more people didn't see even the idea of it as being obviously brittle is concerning.
1. Online trolls and pranksters (who already taught several different AIs to be racist in a matter of hours - just for the LOLs).
2. Nation states like China who already require models to conform to state narratives.
3. More broadly, when training on "the internet" as a whole there is a huge amount of wrong, confused information mixed in.
There's also a meta-point to make here. On a lot of culture war topics, one person's "poisonous information" is another person's "reasonable conclusion."