AI Coding Assistants Exposed: The Shocking Truth Behind the Hype

Coding assistants have been promoted as a game-changer in the world of software development, promising to revolutionize the way developers work and drastically improve productivity. However, a recent study by Uplevel has cast some serious doubts on these claims.

The study, which analyzed the work of 800 developers using GitHub Copilot, found that there were no significant improvements in productivity, measured by pull request (PR) cycle time and PR throughput. In fact, the use of GitHub Copilot was found to introduce 41% more bugs into the code.

This may come as a surprise to many, given the widespread adoption of AI-powered coding assistants in the industry. A recent survey by GitHub found that 97% of software engineers and developers reported using AI-powered coding assistants. However, the Uplevel study suggests that these tools may not be living up to their promise.

Matt Hoffman, product manager and data analyst at Uplevel, explained that the study was driven by a desire to test the claims of major productivity gains made by proponents of AI-powered coding assistants. "Our team's hypothesis was that we thought that PR cycle time would decrease," he said. "We thought that they would be able to write more code, and we actually thought that defect rate might go down because you're using these gen AI tools to help you review your code before you even get it out there."

The study found that developers using GitHub Copilot did not experience any significant improvements in productivity. In fact, the study found that the amount of working time spent outside of standard hours decreased more for developers who didn't use GitHub Copilot.

One of the main concerns with AI-powered coding assistants is the quality of the code they produce. Ivan Gekht, CEO of Gehtsoft USA, a custom software development firm, explained that his company has been testing coding assistants in sandbox environments but has not used them with customer projects yet. "It becomes increasingly more challenging to understand and debug the AI-generated code, and troubleshooting becomes so resource-intensive that it is easier to rewrite the code from scratch than fix it," he said.

Gekht also pointed out that there is a fundamental difference between writing code and developing software. "Software development is 90% brain function — understanding the requirements, designing the system, and considering limitations and restrictions," he said. "Converting all this knowledge and understanding into actual code is a simpler part of the job."

However, not everyone shares the same skepticism. Travis Rehl, CTO of Innovative Solutions, a cloud services provider, reported that his company has seen significant productivity gains using coding assistants like Claude Dev and GitHub Copilot. "We've seen developer productivity increase by two to three times, based on the speed of developer tickets completed, the turnaround time on customer deliverables, and the quality of tickets, measured by the number of bugs in code," he said.

Rehl acknowledges that the hype surrounding coding assistants has been overblown in some cases. "Expectations around coding assistants should be tempered because they won't write all the code or even all the correct code on the first attempt," he said. "It is an iterative process that, when used correctly, enables a developer to increase the speed of their coding by two or three times."

In conclusion, the study by Uplevel raises important questions about the effectiveness of AI-powered coding assistants. While some companies have reported significant productivity gains, others have experienced mixed results. As the technology continues to evolve, it's clear that we need to have a more nuanced understanding of what these tools can and can't do.