Ai Coding Tools May Not Speed Up Every Developer, Study Shows

Software technologist workflows person been transformed successful caller years by an influx of AI coding devices for illustration Cursor and GitHub Copilot, which committedness to heighten productivity by automatically penning lines of code, fixing bugs, and testing changes. The devices are powered by AI models from OpenAI, Google DeepMind, Anthropic, and xAI that person rapidly accrued their performance connected a scope of package engineering tests successful caller years.

However, a new study published Thursday by nan non-profit AI investigation group METR calls into mobility nan grade to which today’s AI coding devices heighten productivity for knowledgeable developers.

METR conducted a randomized controlled proceedings for this study by recruiting 16 knowledgeable open-source developers and having them complete 246 existent tasks connected ample codification repositories they regularly lend to. The researchers randomly assigned astir half of those tasks arsenic “AI-allowed,” giving developers support to usage state-of-the-art AI coding devices specified arsenic Cursor Pro, while nan different half of tasks forbade nan usage of AI tools.

Before completing their assigned tasks, nan developers forecasted that utilizing AI coding devices would trim their completion clip by 24%. That wasn’t nan case.

“Surprisingly, we find that allowing AI really increases completion clip by 19%— developers are slower erstwhile utilizing AI tooling,” nan researchers said.

Notably, only 56% of nan developers successful nan study had acquisition utilizing Cursor, nan main AI instrumentality offered successful nan study. While astir each nan developers (94%) had acquisition utilizing immoderate web-based LLMs successful their coding workflows, this study was nan first clip immoderate utilized Cursor specifically. The researchers statement that developers were trained connected utilizing Cursor successful mentation for nan study.

Nevertheless, METR’s findings raise questions astir nan expected cosmopolitan productivity gains promised by AI coding devices successful 2025. Based connected nan study, developers shouldn’t presume that AI coding devices — specifically what’s travel to beryllium known arsenic “vibe coders” — will instantly velocity up their workflows.

METR researchers constituent to a fewer imaginable reasons why AI slowed down developers alternatively than speeding them up.

First, developers walk overmuch much clip prompting AI and waiting for it to respond erstwhile utilizing vibe coders alternatively than really coding. AI besides tends to struggle successful large, analyzable codification bases, which this trial used.

The study’s authors are observant not to tie immoderate beardown conclusions from these findings, explicitly noting they don’t judge AI systems presently neglect to velocity up galore aliases astir package developers. Other large standard studies person shown that AI coding devices do velocity up package technologist workflows.

The authors besides statement that AI advancement has been important successful caller years, and that they wouldn’t expect nan aforesaid results moreover 3 months from now. METR has besides recovered that AI coding devices person importantly improved their expertise to complete complex, long-horizon tasks successful caller years.

The investigation offers yet different logic to beryllium skeptical of nan promised gains of AI coding tools. Other studies person shown that today’s AI coding devices tin introduce mistakes, and successful immoderate cases, information vulnerabilities.