Anecdotally, this sounds very close to my own experience. The fact that AI can generate code quickly creates a false sense of speed. While experienced engineers are better at vetting and correcting generated code, that still causes significant overhead. There are also delayed impacts when bugs, vulnerabilities, and oversights slip through. The best case scenario is that you break even, but when you pay attention to the big picture, you realize that you are actually taking longer to reach milestones than you used to. I also hypothesize that this overhead gets worse over time as you are far less familiar with the codebase you leave behind than if you had written it yourself.
Remember that AI is incapable of reasoning, so it can’t actually apply logic to the code it generates, which is a problem because all code is literally a representation of logic and reasoning.
Make sure we read table 2 in the paper. The reality is the people behind this study is urging folks to not draw strong conclusions from this study.
Not a developer, but I do a lot of scripting. I recently needed a yaml file with a whole set of parameters from an excel file for about 600 objects. Copilot generated it for me in about a minute. It took some iterations but it saved me a fuckton of time. Anyway, my point is you need to pick your battles. It’s a tool, so wield it like one.
This is the correct approach of course. But tech giants do everything in their power to convince us that we should use AI for everything, including wiping our asses. And there is a significant number of peoople who believe this. So it’s important to check this and spread the word.
Sounds about right.
I’d like to see numbers for inexperienced devs and devs working on somebody else’s code, though.
EDIT: Oh, this is interesting. The full paper breaks down where the time goes. Turns out coders do in fact spend less time actually working on the code when using AI, but the time spent prompting, waiting on the output and processing the output eats up the difference. They also sit idle for longer with AI. So their forecasts aren’t that crazy, they do work less/faster with AI, but the new extra tasks make them less productive overall.
That makes a lot of sense in retrospect, but it’s not what I was expecting.
Thanks for the quick summary! I would probably forget to read this later as im at work right now, so thanks!
deleted by creator
Yeah, I had to dig a bit further for this figure. They display the same data more prominently in percentage of the time devoted to each bug, which gives them smaller error bars, but also doesn’t really answer the question that matters regarding where the time went.
Worth noting that this is a subset of the data, apparently. They recorded about a third of the bug fixes on video and cut out runs with cheating and other suspicious activity. Assuming each recording contains one bug they end up with a fourth of the data broken down this way.
Which is great, but… it does make you wonder why that data is good enough for the overall over/underestimate plot if it’s not good enough for the task breakdown. Some of the stuff they’re filtering out is outright not following the instructions or self-reporting times that are more than 20% off from what the recording shows. So we know some of those runs are so far off they didn’t get counted for this, but presumably the rest of the data that just had no video would be even worse, since the timings are self-reported and they paid them to participate by the hour.
I’d definitely like to see this with more data, this is only 16 people, even if they are counting several hundred bugs. Better methodology, too. And I’d like to see other coder profiles in there. For now they are showing a very large discrepancy between estimate and results and at least this chart gives you some qualitative understanding of how that happened. I learned something from reading it. Plus, hey, it’s not like AI research is a haven of clean, strict data.
Of course most people will just parrot the headline, because that’s the world we live in.
This tracks pretty much with my conclusions for myself, neat.
It’s crazy, I’m fooled every time again and think “surely I must be faster like this” when after a few days an in-depth reflection/looking at actual commits shows that nope, I wasn’t.
Same. I only use it to write boilerplate now. When it is basically copy pasting from docs, it’s faster at it than me.
Every time I try it to write more complex code, I end up redoing major parts of it.
The main reason why I still think it’s faster even if it’s “slower”: it does its work in the background while I can do other things, like respond to emails, attend meetings, look at other bits of code, etc. I turn on the audio notification to have to ping me when it’s done.
Same here. It’s been trained on so much of that kind of code, you have a much better chance of getting useable code on the first prompt.
Interesting study. Also similar to my own observations. I’ve tried AI coding to some degree. Some people recommend it. And it definitely can do some nice things. Like write boilerplate code fast and do some tech-demos and exploring. And it’s kind of nice to bounce off ideas to someone. I mean the process of speaking out things loud and putting it into words helps me think it through. AI can do that (listen) and it’ll occasionally give back some ideas.
The downside of coding with it in real life is, I end up refactoring a lot of stuff. And that’s just tedious and annoying work. I’ve had this happen to me several times now. And I think the net time balance is negative for me as well. I think I’m better off coming up with an idea how to implement something, and then just type it down. Rather than skipping the step and then moving around stuff, changing it, and making sure it handles the edge-cases correctly and fits into the broader picture. Plus I still end up doing the thinking step, just in a different way.
Related to what you said, I found it actually helpful to just write and discuss ideas with some LLM without letting it code.
