My experience has been that the answers are very convincing, but not necessarily true. I would be careful asking gpt questions about abstract knowledge, less about linguistic structure.
That's exactly it. The bot espouses facts with the same tone of confidence regardless of whether they're true or entirely fictional.
I understand it has no sense of knowledge-of-knowledge, so (apparently) no ability to determine how confident it ought to be about what it's saying — it never qualifies with "I'm not entirely sure about this, but..."
I think this is something that needs to be worked in ASAP. It's a fundamental aspect of how people actually interact. Establishing oneself as factually reliable is fundamental for communication and social cohesion, so we're constantly hedging what we say in various ways to signify our confidence in its truthfulness. The absence of those qualifiers in otherwise human-seeming and authoritative-sounding communication is a recipe for trouble.
This is a particular alignment issue. People are used to people spouting bullshit all the time, as long as it's aligned to what we are used to. Take religion for example. People tend to be very confident around the unknowable there.
It is scary in the sense that people love following confident sounding authoritarians, so maybe AI will be our next world leader.
They weren't true in past iterations. Since the new version is 10x as accurate (if you believe the test score measures, going from bottom 10% score to top 10%), we're going to see a lot less confident falseness as the tech improves.
I don't think ChatGPT should be trusted at all until it can tell you roughly how certain it is about an answer, and that this self-reported confidence roughly correponds to how well it will do on a test in that subject.
I don't mind it giving me a wrong answer. What's really bad is confidently giving the wrong answer. If a human replied, they'd say something like "I'm not sure, but if I remember correctly..", or "I would guess that..."
I think the problem is they've trained ChatGPT to respond condidently as long as it has a rough idea about what the answer could be. The AI doesn't get "rewarded" for saying "I don't know".
I'm sure the data about the confidence is there somewhere in the neural net, so they probably just need to somehow train it to present that data in its response.
