It's actually 1.2x productivity. Writing code does not take most of the day. If GPT can be just as good at debugging as it is at writing code, maybe the speedup would increase a bit. The ultimate AI speed == human reading + thinking speed, not AI generation speed.
I’ve found it’s very useful for understanding snippets of code, like a 200-line function. But systems are more than “lots of 200-line functions” - there’s a lot of context hidden in flags, conditional blocks, data, Git histories.
Maybe one day, we’ll be able to run it over all the Git histories, Jira tickets, Confluence documentation, Slack conversations, emails, meeting transcripts, presentations, etc. Until then, the humans will need to stitch it all together as best they can.
I can't think of a reason you couldn't specifically train an AI on your own large code base. After all, current LLMs are trained on effectively the entire internet.
Unless your documentation is 20~100x the size of your codebase and written in a conversational tone, the LLM won't be able to be asked any questions about it using English.
If your only aim is to use it like Copilot, sure, it's useful.
You might be able fine tune a model on pull requests if they have really high quality descriptions, high quality commit messages, and the code is well documented and organized.
I haven't yet seen anything that can scan an entire codebase and build, say, a data lineage to understand how a value in the UI was calculated. I'm sure it's coming, though.
but not all human thinking is worthwhile. I had it do a simple chrome extension to let me reply inline on HN, and it coughed out a manifest.json that worked first try. I didn't have to poke around the Internet to find a reference and then debug that via stack overflows. Easily saved me half an hour and gave me more mental bandwidth for the futzing with the DOM that I did need to do. (to your point tho, I didn't try feeding the html to it to see if it could do that part for me.)
so it's somewhere between 1.2x and 10x for me, depending on what I'm doing that day. Maybe 3x on a good day?
I don't mean to pick on you specifically, but this kind of approach doesn't fit the way I like to work.
For example, just because the manifest.json worked doesn't mean it is correct - is it free of issues (security or otherwise)?
I would argue that every system in production today seemed to "just work" when it was built and initially tested, and yet how many serious issues are in the wild today (security or otherwise)?
I prefer to take a little more time solving problems, gaining an understanding of WHY things are done certain ways, and some insight into potential problems that may arise in the future even if something seems to work at first glance.
Now I get that you are just talking about a small chrome extension that maybe you are only using for yourself... but scaling that up to anything beyond that seems like a ticking time bomb to me.
I feel like you would get more benefit out of GPT. you could ask it if it finds any vulnerabilities, common mistakes, other inconstancies. please provide comments on what each line does. what are some other common ways to write this line of code, etc.
what are some ways to handle this XYZ problem. I see you might have missed sql injection attacks. would that apply here?
Same goes for code you find on the internet.
I got this out put for this line of code what do you think the problem is.
Even OpenAI says clearly: You should not, by any means, ask the AI any questions you don't know the answer already!
> more benefit out of GPT. you could ask it if it finds any vulnerabilities, common mistakes, other inconstancies. please provide comments on what each line does. what are some other common ways to write this line of code, etc.
And than it spits out some completely made up bullshit…
How would you know if you don't understand what you're actually doing?
Every time I've tried chatgpt I've been shocked at the mistakes. It isn't a good tool to use if you care about correctness, and I care about correctness.
It may be able to regurgitate code for simple tasks, but that's all I've seen it get right.
Most managers don't care for people like you. Companies sell their product. Another successful fiscal year. Irregardless of the absolute shit code base, wasteful architecture and gaping security vulnerabilities.
Maybe, but I've always had lucrative jobs and my work has always been appreciated. Maybe you just have to find the right employer. I think longer-term, employers that value high quality work will have the upper-hand.
And to be honest, I don't care for managers like that, so the feeling is mutual.
I think it is worthwhile considering the multiple of the alternatives.
That is, SERPs providing relevant discussion and exacting or largely turnkey solutions.
On “easy” tasks in technical niches I’m not familiar with, I would take gpt over DDG + SO the majority of the time.
I’ve had situations where I’m wanting a tutorial or walk through on something and mixing various sub stack and independent blog posts.
There is no consistency in quality or even correctness from those sources, while you must also deal with format and styling variation.
I block adtech within reason, but the problem of greyhat or SEO-focused filler also isn’t really a thing in gpt. You just get the fat of the land.
The biggest problem w gpt 4 is the cutoff date. The LLM needs to be updated regularly, the way Google initially seemed to crawl all the things and make them available to queries as they appeared.
Data recency and the ability to process excess tokens is going to show OP’s test as but a toy example of what the systems can do.
In the absence of constant updates of a massive model and all that entails, I foresee Companies temporarily dominating attention by providing pretrained LORA-like add-ons to LLMs at key events.
For example, Apple could release a new model trained on all of the updated Swift libraries coming to market at WWDC shortly after the keynote.
It can contain all the docs, example code and allow devs to really, really experiment and have warm and fuzzies on the newest stuff.
It could even include the details on product announcements and largely handle questions of compatibility.
If the companies hosted the topic focused chatgpt-like bots, they could also own all the unexpected questions, and both clarify and retrain on what the most enthusiastic users want to know.
This is going a bit of another direction, but I think all of this is very exciting and will hasten the delivery of software for brand new SDKs.
> The biggest problem w gpt 4 is the cutoff date. The LLM needs to be updated regularly, the way Google initially seemed to crawl all the things and make them available to queries as they appeared.
Have you tried out Phind? It's essentially GPT-4 + internet access. It hasn't been perfect, but it's been a very useful tool for me.
Also chances are great that the AI just spit out some code from that extension… (Of course without attribution. Which would make it a copyright volition.)
Definitely this. I spend about 10-15% of my time writing code so a 20% increase really doesn't save me a lot of time. Also AI generated code requires more reading of code, which is harder and more cognitively expensive than writing code.
Depends on the code. If AI can quickly write even mediocre quality automated tests, that's a tremendous speedup, and, if I'm being totally honest, morale boost.
Throwing away and rewriting tests is work (== negative value).
Testing implementation details is always contra productive. Have you watched the video? (I'm not recommending videos often, as I think writings have more value per time-unit, but this talk is a kind of classic on that topic.)
Sorry, I can't accept that a one hour video is the cost to participate in this conversation. I don't disagree that there is a cost associated with doing this, but the cost is so much smaller than it used to be that it can be economical now.
Implementation details are in the eye of the beholder IMO. I'm open to reasons why that's not the case here.
