Great article, nice to see some actual critical thoughts on the shortcomings of LLMs. They are wrong about programming being a "chess-like domain" though. Even at a basic level hidden state is future requirements, and the adversary is self or any other entity that has to modify the code in the future.
AI is good at producing code for scenarios where the stakes are low, there's no expectation about future requirements, or if the thing is so well defined there is a clear best path of implementation.
I address that in part right there itself. Programming has parts like chess (ie bounded) which is what people assume to be actual work. Understanding future requiremnts / stakeholder incentives is part of the work which LLMs dont do well.
> many domains are chess-like in their technical core but become poker-like in their operational context.
The number of legal possible boards in chess is somewhere around 10^44 based on current calculation. That's with 32 chess pieces and their rules.
The number of possible permutations in an application, especially anything allowing turing completeness is far larger than all possible entropy states in the visible universe.
Bounded domains require scaling reasoning/compute. Two separate scenarios - one where you have hidden information, other where you have high number of combinations. Reasoning works in second case because it narrows the search space. Eg: a doctor trying to diagnose a patient is just looking at number of possibilities. If not today, when we scale it up, a model will be able to arrive at the right answer. Same goes with Math, the variance or branching for any given problem is very high. But LLMs are good at it. and getting better. A negotiation is not a high variance thing, and low number of combinations, but llms would be repeated bad at it.
It's not even always a more efficient form of labour. I've experienced many scenarios with AI where prompting it to do the right thing takes longer and requires writing/reading more text compared to writing the code myself.
Agreed. Programming languages are not ambiguous. Human language is very ambiguous, so if I'm writing something with a moderate level of complexity, it's going to take longer to describe what I want to the AI vs writing it myself. Reviewing what an AI writes also takes much longer than reviewing my own code.
AI is getting better at picking up some important context from other code or documentation in a project, but it's still miles away from what it needs to be, and the needed context isn't always present.
> They can write code better than you or I can, and if you don’t believe me, wait six months.
You can use AI to write all your code, but if you want to be a programmer and can't see that the code is pretty mid then you should work on improving your own programming skills.
People have been saying the 6 month thing for years now, and while I do see it improving in breadth, quality/depth still appears to be plateauing.
It's okay if you don't want to be a programmer though, you can be a manager and let AI do an okay job at being your programmer. You better be driven to be a good at manager though. If you're not... then AI can do an okay job of replacing you there too.
Claude requires many lifetimes worth of data to "learn". Evolution aside humans don't require much data to learn, and our learning happens in real-time in response to our environment.
Train Claude without the programming dataset and give it a dozen of the best programming books, it'll have no chance of writing a compiler. Do the same for a human with an interest in learning to program and there's a good chance.
Here's a potentially more uncomfortable thought, if all people through history with potential for "true intelligence" had a tool that did 99% of everything do you think they would've had motivation to learn enough of that 99% to give insight into the yet discovered.
I wouldn't say I need to invent much that is strictly novel, though I often iterate on what exists and delve into novel-ish territory. That being said I'm definitely in a minority where I have the luxury/opportunity to work outside the monotony of average programming.
The part I find concerning is that I wouldn't be in the place I am today without spending a fair amount of time in that monotony and really delving in to understand it and slowly push outside it's boundary. If I was starting programming today I can confidently say I would've given up.
I highly recommend everyone involved in web development to read at least a small proportion of the horrors that are the HTML parser specification. It will leave you yearning for the return of XHTML.
Or you could also read web proposals where the reason for avoiding the ideal implementation is complication of updating HTML parser rules.
Or attempt to use the web features that are already hindered by the HTML parser (custom element table rows).
Grateful in part, but I can't help to think that if there was refusal to build parsers for an outlandish spec in the first place then we'd have fixed the problem by now.
Using existing parsers only hides the poor design up to a point.
I mostly agree with the sentiment, I'd rather have simple parsers and sensible specs, but I'm also happy they do whatever it takes not to break anything (well, they are breaking XSLT…)
AI is good at producing code for scenarios where the stakes are low, there's no expectation about future requirements, or if the thing is so well defined there is a clear best path of implementation.