> what specifically would you or I do different, apart from having less training data?
Create things that are not in the training data. I could coin a new phrase that uses "bridge", I doubt an LLM can or would have any motivation to. Because LLMs understand data and patterns, but they don't understand the meaning behind them. They can't be imaginative.
For example, an LLM can recite Shakespeare and can even write new works in Shakespeare's style. But it cannot be a new Shakespeare, the is one level above it. An image generator might be able to make a painting is Picasso's style, but it cannot be a new Picasso - that is, create a style never before seen and explain why they did it and what emotions it conveys.
Me: Hi, can you use the concept "bridge" in a
never-seen-before sentence, where it still makes
sense? Please make the sentence very unusual,
but still coherent. Perhaps a couple of examples?
ChatGPT: Sure! Here are a couple of unusual but coherent
sentences using the concept of "bridge":
The stars whispered their ancient stories across the
sky, building a bridge of forgotten time between my
dreams and the universe’s memories.
Each thought she left unspoken laid a ghostly bridge
between us, invisible but weighty, made of all the
words that could have been.
Both sentences play with abstract ideas, pushing the
concept of a bridge beyond its typical use but
maintaining coherence through metaphor and vivid
imagery.
----
It's impossible for me to know everything in the training set but there are no Google results for "building a bridge of forgotten time" or "laid a ghostly bridge between us". If you told me those and said you made them up and didn't copy them from somewhere, I'd have to either find the source you copied or believe you.
Novelty is not the same as understanding. I think it’s telling that one of the two probably works at least as well with something that’s the opposite of a bridge: gulf, chasm, wall. It just picked a pattern for an analogy and stuck your word in it, mad-libs style. Like, the fact that it’s kinda the opposite of what me would expect makes it a bit poetic in that one’s tempted to pursue what it might mean, but of course the LLM didn’t mean anything (though we still might find meaning).
Its patterns encoded two items and “between” being likely to occur, anyway.
Create things that are not in the training data. I could coin a new phrase that uses "bridge", I doubt an LLM can or would have any motivation to. Because LLMs understand data and patterns, but they don't understand the meaning behind them. They can't be imaginative.
For example, an LLM can recite Shakespeare and can even write new works in Shakespeare's style. But it cannot be a new Shakespeare, the is one level above it. An image generator might be able to make a painting is Picasso's style, but it cannot be a new Picasso - that is, create a style never before seen and explain why they did it and what emotions it conveys.