I think it really, really depends on the language. I haven't been able to make it work at all for Haskell (it's more likely to generate bullshit tests or remove features than actually solve a problem), but for Python I've been able to have it make a whole (working!) graph database backup program just by giving it an api spec and some instructions like, "only use built in python libraries".
The weirdest part about that is Haskell should be way easier due to the compiler feedback and strong static typing.
What I fear most is that it will have a chilling effect on language diversity: instead of choosing the best language for the job, companies might mandate languages that are known to work well with LLMs. That might mean typescript and python become even more dominant :(.
I share similar feelings. I don't want to shit on Python and JS/TS. Those are languages that get stuff done, but they are a local optimum at best. I don't want the whole field to get stuck with what we have today. There surely is place for a new programming language that will be so much better that we will scratch our heads why we ever stuck with what we have today. But when LLMs work "good enough" why even invent a new programming language? And even if that awesome language exists today, why adopt it then? It's frustrating to think about. Even language tooling like static analyzers and linters might get less love now. Although I'm cautiously optimistic, as these tools can feed into LLMs and thus improve how they work. So at least there is an incentive.
The weirdest part about that is Haskell should be way easier due to the compiler feedback and strong static typing.
What I fear most is that it will have a chilling effect on language diversity: instead of choosing the best language for the job, companies might mandate languages that are known to work well with LLMs. That might mean typescript and python become even more dominant :(.