September 17, 2024
Broadly speaking “everyone” is on board with AI – from marketing to sales to engineering. AI is everywhere. But when we think about AI in the context of engineering, it’s generative AI coding assistants like GitHub’s Copilot that are changing the day to day experience for developers, their managers and both engineering and business leaders who are now tasked with measuring how AI adoption impacts headcount, efficiency and other key metrics.
While adoption is broad – GitHub counts more than 50,000 companies including one in three of the Fortune 500 as Copilot customers – it’s been challenging to objectively measure the impact of AI coding assistants. Sure we can point to things like increased efficiency and acceptance rate all day long, but most of this data is coming from surveys – not the engineering systems themselves – making it hard to actually prove impact.
A recurring theme across studies that have sought to measure impact is the focus on productivity metrics, while neglecting a holistic examination of daily engineering workflows, behavioral shifts, and broader engineering dynamics. This together with the lack of data-driven insights from a large number of Copilot users are two primary factors limiting the understanding of the impact AI-driven coding tools have on the engineering workspace.
Until now.
Jellyfish leverages empirical data encompassing GitHub, Jira, and more from real-world engineering settings to quantitatively understand the impacts of AI-driven coding tools on engineers’ workflows and work environments on aspects including:
- Where engineering effort is spent
- How engineering effort is divided across the lifecycle of work
- How work is defined
- How engineers review work and collaborate
So what have we actually learned about AI coding assistant adoption? Let’s take a look 👀
Increased coding efficiency with Copilot
Copilot usage led to a statistically significant reduction in the time spent coding per task compared to non-users, with coding speeds boosted 3% on average and up to 15%. This suggests that Copilot enhances coding efficiency, enabling developers to complete their tasks more quickly than their counterparts in the control group.
Shift towards smaller work units
With Copilot, companies saw a 16% reduction in the average size of tasks, reflecting a trend toward defining work in smaller, more manageable chunks. This approach aligns with CI/CD best practices and coincided with an 8% decrease in cycle times, enabling quicker delivery of work.
Evolution in review practices
The PR review process showed signs of change as Copilot users were observed to make longer and more comprehensive PR comments. This suggests that while developers are completing tasks faster, some of that time savings is going towards more robust reviews of Copilot-assisted code.
Reallocation of effort towards strategic work
Copilot usage led to a noticeable shift in focus from KTLO (Keeping the Lights On) and support work to more strategic product roadmap and growth activities, with some companies seeing up to a 17% reallocation of effort. This indicates that Copilot may help teams prioritize high-value work by freeing up time previously spent on routine tasks.
The early data on genAI coding assistants like Copilot show promising results. By boosting coding efficiency, encouraging smaller work units, improving review practices, and shifting focus to more strategic tasks, genAI assistants are reshaping engineering workflows. While the full impact is still being assessed, the data-driven insights reveal significant potential for AI to enhance productivity and value within engineering teams.
Ready to learn more about the impact of genAI on your own engineering team? Learn more and get started with Copilot Dashboard from Jellyfish here.
You can learn more about Jellyfish’s effort to quantify the impact of AI coding assistants from the Jellyfish Research team here.
