I see less "painting as a luddite" in response to statements like this, and more... surprise. Mild skepticism, perhaps!

Your experience diverges from that of other experienced devs who have used the same tools, on probably similar projects, and reached different conclusions.

That includes me, for what it's worth. I'm a graybeard whose current work is primarily cloud data pipelines that end in fullstack web. Like most devs who have fully embraced LLMs, I don't think they are a magical panacea. But I've found many cases where they're unquestionably an accelerant -- more than enough to justify the cost.

I don't mean to say your conclusions are wrong. There seems to be a bimodal distribution amongst devs. I suspect there's something about _how_ these tools are used by each dev, and in the specific circumstances/codebases/social contexts, that leads to quite different outcomes. I would love to read a better investigation of this.

I think it also depends on _what_ the domain is, and also to a certain degree the tools / stack you use. LLMs aren’t coherent or correct when working on novel problems, novel domains or using novel tools.

They’re great for doing something that has been done before, but their hallucinations are wildly incorrect when novelty is at play - and I’ll add they’re always very authoritative! I’m glad my languages of choice have a compiler!

My recent example for where its helpful.

Pretty nice at autocomplete. Like writing json tags in go structs. Can just autocomplete that's stuff for me no problem, it saved me seconds per line, seconds I tell you.

It's stupid as well... Autofilled a function, looks correct. Reread it 10 minutes later and well... Minor mistake that would have caused a crash at runtime. It looked correct but in reality it just didn't have enough context ( the context is in an external doc on my second screen ... ) and there was no way it would ever have guessed the correct code.

It took me longer to figure out why the code looked wrong than if I had just typed it myself.

Did it speed up my workflow on code I could have given a junior to write? Not really, but some parts were quicker while other were slower.

And imagine if that code bad crashed in production next week instead of right now while the whole context is still in my head. Maybe that would be hours of debugging time...

Maybe as parent said, for a domain where you are braking new ground, it can generate some interesting ideas you wouldn't have thought about. Like a stupid pair that can get you out if a local manima but in general doesn't help much it can be a significant help.

But then again you could do what has been done for decades and speak to another human about the problem, at least they may have signed the same NDA as you...

Yeah, absolutely.

LLMs work best for code when both (a) there's sufficient relevant training data aka we're not doing something particularly novel and (b) there's sufficient context from the current codebase to pick up expected patterns, the peculiarities of the domain models, etc.

Drop (a) and get comical hallucinations; drop (b) and quickly find that LLMs are deeply mediocre at top-level architectural and framework/library choices.

Perhaps there's also a (c) related to precision. You can write code to issue a SQL query and return JSON from an API endpoint in multiple just-fine ways. Misplace a pthread_mutex_lock, however, and you're in trouble. I certainly don't trust LLMs to get things like this right!

(It's worth mentioning that "novelty" is a tough concept in the context of LLM training data. For instance, maybe nobody has implemented a font rasterizer in Rust before, but plenty of people have written font rasterizers and plenty of others have written Rust; LLMs seem quite good at synthesizing the two.)

Yesterday i wanted to understand what a team was doing in a go project. I have never really touched go before. I do understand software, because I develop for plus 20 years. But chatgpt was perfectly able to give me a summary on how the implementation worked. Gave me examples and suggestions. And within a day fulltime pasting code and asking question i had a good understanding of the codebase. It would have be a lot more difficult with only google.

how often do you get to learn an unfamiliar language? is it something you need to do every day? so this use case, did it save you much time overall?

Totally respect your position, given that you actually tried the tool and found it didn't work for you. That said, one valid explanation is that the tool isn't good for what you're trying to achieve. But an alternative explanation is that you haven't learned how to use the tool effectively.

You seem open to this possibility, since you ask:

> I've never actually seen any LLM-driven developers work in real time. Are there any live coding channels that could convince the skeptics what we're missing out on something revolutionary?

I don't know many yet, but Steve Yegge, a fairly famous developer in his own right, has been talking about this for the last few months, and has walked a few people through his "Chat Oriented Programming" (CHOP) ideas. I believe if you search for that phrase, you'll find a few videos, some from him and some from others. Can't guarantee they're all quality videos, though anything Steve himself does is interesting, IMO.

I have very similar experience. For me LLM are good at explaining someone else's complex code, but for some reason they don't help me write new code well. I would also like to see any LLM-driven developers work in real time.

You're the middle ground I was talking about. You tried it. You know where it works and where it doesn't.

I've used LLM to generate code samples and my IDE (IntelliJ) uses an LLM for auto-suggestions. That's mostly about it for me.

I have, in person. Their code requires a lot of cleanup and then there's the AI pit of death they often are not able to crawl out of due to... Mostly language differences. They don't know enough English to look stuff up and figure out how to fix things. Programming resources in other languages are pretty much non-existent

My experience thus far is that LLMs can be quite good at:

* Information lookup

-- when search engines are enshittified and bogged down by SEO spam and when it's difficult to transform a natural language request into a genuinely unique set of search keywords

-- Search-enabled LLMs have the most up to date reach in these circumstances but even static LLMs can work in a pinch when you're searching for info that's probably well represented in their training set before their knowledge cutoff

* Creatively exploring a vaguely defined problem space

-- Especially when one's own head feels like it's too full of lead to think of anything novel

-- Watch out to make sure the wording of your request doesn't bend the LLM too far into a stale direction. For example naming an example can make them tunnel vision onto that example vs considering alternatives to it.

* Pretending to be Stack Exchange

-- EG, the types of questions one might pose on SE one can pose to an LLM and get instant answers, with less criticism for having asked the question in the first place (though Claude is apparently not above gently checking in if one is encountering an X Y problem) and often the LLM's hallucination rate is no worse than that of other SE users

* Shortcut into documentation for tools with either thin or difficult to navigate docs

-- While one must always fact-check the LLM, doing so is usually quicker in this instance than fishing online for which facts to even check

-- This is most effective for tools where tons of people do seem to already know how the tool works (vs tools nobody has ever heard of) but it's just not clear how they learned that.

* Working examples to ice-break a start of project

* Simple automation scripts with few moving parts, especially when one is particular about the goal and the constraints

-- Online one might find example scripts that almost meet your needs but always fail to meet them in some fashion that's irritating to figure out how to coral back into your problem domain

-- LLMs have deep experience with tools and with short snippets of coherent code, so their success rate on utility scripts are much higher than on "portions of complex larger projects".