The confidence seems to be an artifact of fine tuning. The first instruction trained models were given data sets with answers to questions but generally omitted non answers to things the model didn't know.
Later research showed that models know that they don't know certain pieces of information, but the fine tuning constraint of providing answers did not give them the ability to express that they didn't know.
Asking the model questions against known information can produce a correct/incorrect map detailing a sample of facts that the model knows and does not know. Fine tuning a model to say "I don't know" in response to the those questions where it was incorrect can allow it to generalise the concept to its internal concept of unknown.
It is good to keep in mind that the models we have been playing with are just the first ones to appear. GPT 3.5 is like the Atari 2600. You can get it provide a limited experience for what you want and its cool that you can do it at all, but it is fundamentally limited and far from an ideal solution. I see the current proliferation of models to be like the Cambrian explosion of early 8 bit home computers. Exciting and interesting technology which can be used for real world purposes, but you still have to operate with the knowledge of the limitations forefront in your mind and tailor tasks to allow them to perform the bits they are good at. I have no-idea of the timeframe, but there is plenty more to come. There have been a lot of advances revealed in papers. A huge number of those advances have not yet coalesced into shipping models. When models cost millions to train you want to be using a set of enhancements that play nicely together. Some features will be mutually exclusive. By the time you have analysed the options to find an optimal combination, a whole lot of new papers will be suggesting more options.
We have not yet got the thing for AI that Unix was for computers. We are just now exposing people to the problems that drives the need to create such a thing.
I believe most confident statements people make, are established the same way. There are some anchor points (inputs and vivid memories) and some part of the brain in some stochastic way dreams up connections. Then we convince ourselves that the connections are correct, just because they match some earlier seen pattern or way of reasoning.
The basis of human irrationality is not tied to the basis of LLM irrationality.
LLMs don't get to make value judgements, because they don't "understand". They predict the subsequent points of a pattern given a starting point.
Humans do that, but they also jade their perception with emotive ethics, desires, optimism and pessimism.
It is impossible to say that two humans with the exact same experience would always come to the same conclusion, because two humans will never have the exact same experience. Inputs include emotional state triggered by hormones, physical or mental stress, and so forth, which are often not immediately relevant to any particular decision, but carried over from prior states and biological processes.
Just because humans have additional sources of irrationality doesn't mean they don't also have irrationality based on the same lack of self-awareness that LLMs exhibit.
I could understand that argument as follows: LLMs fill in the gaps in a creative but predictable way. Humans fill in the gaps in creative but unpredictable ways. The creativeness level is affected by the ad hoc state of the brain.
I understand that you relate judgement, ethics and emotions to 'understanding'. I'm not convinced. Emotions might as well be an effect of pattern matching. You hear a (pattern matched) type of song, you feel a certain way.
Conversely, human beings with varying particular experiences can come to the same conclusions, because human cognition can abstract from particulars, while LLMs are, at best, statistical and imagist. No two of us ever experience the same set of triangles, but abstraction allows us to form concepts like "triangularity", which means we can understand what it means to be a triangle per se (something that is not concrete or particular, and therefore cannot be visualized), while an LLM can only proceed based on the concrete and particular data of input triangles and derivations introduced into the model. It can never go "beyond" the surface features of the training model's images, as it were, and where the appearance of having done so occurs, it is not via abstraction, but by way of product of human abstraction. From the LLM's perspective, there is no triangle, only co-occurrence of features, while abstraction goes beyond features, stripping them away to obtain the bare, unimaginable form.
Different LLMs can also come to the same conclusions, that is, predict the same strings of tokens (in the meaning).
Sure, they're a far way from the capacity of humans in terms of short term training. But there is literally nothing that indicates they can't "think" (understand, reason, abstract, whatever word you wanna put in italics) because nobody can explain what it really means, because: it's all just predicting. Text happens to be super useful for us to evaluate certain aspects of predicting.
