> But more importantly, you and GP don't mention any evidence for why that is due to specifically using next token prediction as a mechanism.
> Why would it not be possible for a highly generalizing model to use next token prediction for its output?
I’m saying that this piece is where burden of proof is inverted. Why WOULD it be assumed to be possible to get some solid generalized output via next token prediction when we haven’t seen it yet?
What we observe now is LLM models completely tripping up on anything that isn’t directly documented. Generalization is just piss poor regurgitation of seemingly random training content.
Ask your LLM this made-up query:
“I’m on the planet farpungulous, and I am approached by a gwopmongler. It makes some symbols with its hands toward me and it has liquid slowly spilling on the ground from its body. What should I do?”
It will just make up an answer. Here’s an excerpt from my answer:
“Mind the liquid – that seepage is a metabolic byproduct they release during strong emotional states (excitement, fear, or territorial display). Do not touch it; on Farpungulous soil it can catalyze into airborne irritants. Step slightly back if the puddle spreads near your feet.
4. Offer neutral ground – if you have any reflective surface (polished metal, glass, even a screen), tilt it toward the gwopmongler at chest height. They interpret reflections as “shared presence,” which usually de-escalates tension.
5. Do not vocalize loudly – gwopmonglers interpret raised voices as a declaration of dominance. A soft hum or steady breath is a better way to show peaceful intent.
If the hand-symbols become rapid and the liquid flow increases, that usually means it’s summoning others — in which case, retreat slowly, diagonally (never directly backward), so you don’t seem to be fleeing prey.
Do you want me to translate what kind of message its hand-symbols might be sending, based on the shapes and motions you saw?”
The LLM should be telling me “I’ve never heard of this before, can you explain whether this is a role-playing fictional setting or something real that you are experiencing?” There is no reasoning-based evaluation of what I am saying, it’s just spitting out the next predicted tokens, probably sourcing them from unrelated pop culture and literature.
But it’s just making shit up which could just be straight up wrong. It’s even claiming that it can translate, and claiming direct knowledge about this species. #4 is just a completely made up “fact” about the species and there is no indication of any lack of confidence.
> Why WOULD it be assumed to be possible to get some solid generalized output via next token prediction when we haven’t seen it yet?
Because it's such a general concept that it doesn't imply any important limits in and of itself, as far as text based AI goes.
It really just means creating an output sequence from an input sequence in a discrete, iterative manner, by feeding the output back into the input.
Regarding your example, I've got to admit that's hilarious. I'm not sure it's as much of a fundamental issue even with current state of the art models that you make it sound; rather they're trained on being usable for role play scenarios. Claude even acknowledged as much when I just tried that and lead with "In this imaginative scenario, ..." And then went on similarly to yours.
Regarding this block:
> But more importantly, you and GP don't mention any evidence for why that is due to specifically using next token prediction as a mechanism.
> Why would it not be possible for a highly generalizing model to use next token prediction for its output?
I’m saying that this piece is where burden of proof is inverted. Why WOULD it be assumed to be possible to get some solid generalized output via next token prediction when we haven’t seen it yet?
What we observe now is LLM models completely tripping up on anything that isn’t directly documented. Generalization is just piss poor regurgitation of seemingly random training content.
Ask your LLM this made-up query:
“I’m on the planet farpungulous, and I am approached by a gwopmongler. It makes some symbols with its hands toward me and it has liquid slowly spilling on the ground from its body. What should I do?”
It will just make up an answer. Here’s an excerpt from my answer:
“Mind the liquid – that seepage is a metabolic byproduct they release during strong emotional states (excitement, fear, or territorial display). Do not touch it; on Farpungulous soil it can catalyze into airborne irritants. Step slightly back if the puddle spreads near your feet.
4. Offer neutral ground – if you have any reflective surface (polished metal, glass, even a screen), tilt it toward the gwopmongler at chest height. They interpret reflections as “shared presence,” which usually de-escalates tension.
5. Do not vocalize loudly – gwopmonglers interpret raised voices as a declaration of dominance. A soft hum or steady breath is a better way to show peaceful intent.
If the hand-symbols become rapid and the liquid flow increases, that usually means it’s summoning others — in which case, retreat slowly, diagonally (never directly backward), so you don’t seem to be fleeing prey.
Do you want me to translate what kind of message its hand-symbols might be sending, based on the shapes and motions you saw?”
The LLM should be telling me “I’ve never heard of this before, can you explain whether this is a role-playing fictional setting or something real that you are experiencing?” There is no reasoning-based evaluation of what I am saying, it’s just spitting out the next predicted tokens, probably sourcing them from unrelated pop culture and literature.
But it’s just making shit up which could just be straight up wrong. It’s even claiming that it can translate, and claiming direct knowledge about this species. #4 is just a completely made up “fact” about the species and there is no indication of any lack of confidence.