>It illustrates that CoPilot is generating maximum likelihood token strings and has no real understanding of the code.
Funny, today I was just thinking of people's tendencies to dismiss AI advances with this very pattern of reasoning: take a reductive description of the system and then dismiss it as obviously insufficient for understanding or whatever the target is. The assumption is that understanding is fundamentally non-reductive, or that there is insufficient complexity contained within the reductive description. But this is a mistake.
The fallacy is that the reductive description is glossing over the source of the complexity, and hence where the capabilities of the model reside. "Generating maximum likelihood token strings" doesn't capture the complexity of the process that generates the token strings, and so an argument that is premised on this reductive description cannot prove the model deficient. For example, the best way to generate maximum likelihood human text is just to simulate a human mind. Genuine understanding is within the solution-space of the problem definition in terms of maximum likelihood strings, thus you cannot dismiss the model based on this reductive description.
The difference between me and you is that I implement neural nets professionally. Here is one of my (non-professional) open source projects: https://NN-512.com
I'm sure if you understood what the transformer was doing, you would be less impressed.
This is the wrong context to go with an appeal to authority. I know what the transformer is doing, I've also developed neural networks before (though not professionally). Your experience is working against you in developing your intuition. There's another common fallacy that because we're somehow "inside" the system, that we understand exactly what is going on, or in this case what isn't going on. Language models are composed of variations of matrix multiplications, but that isn't a complete description of their behavior. It's like saying because we've looked inside the brain and there's just electrical and chemical signals, the mind must reside somewhere else. It's just a specious argument.
Funny, today I was just thinking of people's tendencies to dismiss AI advances with this very pattern of reasoning: take a reductive description of the system and then dismiss it as obviously insufficient for understanding or whatever the target is. The assumption is that understanding is fundamentally non-reductive, or that there is insufficient complexity contained within the reductive description. But this is a mistake.
The fallacy is that the reductive description is glossing over the source of the complexity, and hence where the capabilities of the model reside. "Generating maximum likelihood token strings" doesn't capture the complexity of the process that generates the token strings, and so an argument that is premised on this reductive description cannot prove the model deficient. For example, the best way to generate maximum likelihood human text is just to simulate a human mind. Genuine understanding is within the solution-space of the problem definition in terms of maximum likelihood strings, thus you cannot dismiss the model based on this reductive description.