June 05, 2024
GenAI coding tools have been top of mind for engineering organizations for the last 18 months. Yet despite all of the predictions and hot takes, we’ve had little hard data to explain what’s actually happening. With the recent release of GitHub’s Copilot API into public preview, we now have visibility into their impact.
Engineering organizations are asking two questions about GenAI. Executives and board members want to know, “How much value is AI delivering?” The engineers themselves have a different question: “How is AI going to impact my career?”
Jellyfish is delivering answers to both questions. We recently surveyed more than 600 software engineers and engineering leaders about the ramifications of AI on their organizations. The 2024 State of Engineering Management Report (SEMR) uncovered serious differences in the experiences of front-line engineers and their leaders, as well as a range of predictions for how AI will shape engineering organizations in the months and years to come.
The SEMR offers a snapshot of the current environment for engineering teams, but leaders also need to understand the impact of AI on a real-time, ongoing basis. That’s why we’re launching our GitHub Copilot Dashboard to answer key questions around the adoption, utilization, and impact of the most popular GenAI coding tool. This is just the first in a series of dashboards we’re building for whichever coding tools your teams use today.
Why engineering organizations must be able to measure AI efforts
While AI spread like wildfire throughout engineering organizations, engineering metrics have been slow to catch up. Many teams rely on anecdotal surveys to track which members of their teams are using AI tools; in the best case, they might also include a qualitative question around productivity. To quantify the business impact of AI investments, companies need a clear view of GenAI adoption: what percentage of the organization is using AI tools; what percentage of AI suggestions are being accepted; and what programming languages are being used.
In addition to adoption metrics, the most successful organizations should also be able to answer these critical impact questions:
- Are you building software more efficiently? With the right tools, companies can compare cycle time or any other operational metric to quantify the difference between teams using AI and those that aren’t. Jellyfish’s own engineering team found that cycle time for engineers using Copilot was up to 34% faster than those not using Copilot.
- Are your initiatives more predictable? By looking at deliverables in progress and completed, an engineering leader can see whether their team is systematically overdelivering on AI-assisted projects — and delivering customer value sooner.
- Are you focusing more on what matters? Engineering leaders can compare work allocations before and after the adoption of AI to understand how eliminating repetitive tasks increases effort toward value-driving work. In some cases, we’ve seen an effort shift upwards of 17% towards growth work.
- How are engineering operations changing? As developers incorporate AI tools into their workflows and their work shifts from coding to reviewing (up to 15%), managers should work with their teams to identify new sources of bottlenecks. Organizations with visibility into the evolving software development life cycle will be able to leverage the full potential of these AI tools.
Engineering leaders expect AI to make a real difference: the SEMR found that 58% said AI can help alleviate burnout and enhance overall team well-being by automating repetitive tasks. Most engineering leaders expect AI to enhance decision-making processes with data-driven insights and provide predictive insights to prevent bottlenecks and resource overloads.
Business leaders and engineers aren’t on the same page
The SEMR found a real disconnect between business leaders and front-line workers. Three-quarters (76%) of executives believe their team has embraced AI, compared to just 52% of engineers. This lack of alignment shouldn’t be a surprise — there’s currently no effective, widely adopted way to measure and manage AI adoption in engineering. Measuring the impact of AI is even more difficult — how can an organization understand whether its AI investments are paying off?
Engineers are going to continue using Copilot and other AI coding tools, and the technology undoubtedly has the potential to transform the way engineering teams plan and execute their work. But the missing piece to the puzzle is measurement: engineering leaders need to know how AI is being applied in their organization and how their work is changing as a result. Real-time metrics allow engineering leaders to understand the state of their organization, ensure teams are rowing in the same direction, and make corrections when necessary. Plus, objective measurement of the impact of AI coding tools enables leaders to secure additional budget to support their team’s needs, setting engineering up for future success.
Jellyfish already provides engineering leaders with useful, easy-to-understand dashboards for engineering metrics, resource allocations, and the software development life cycle. The addition of our new Github Copilot Dashboard ensures that engineering managers gain the same visibility into GenAI.
Introducing the GitHub Copilot Dashboard
The right metrics ensure that engineers are given impactful tasks and see the value in their work. Companies must be able to objectively measure their team’s adoption of AI and — more importantly — the technology’s impact on building better products and shipping them faster.
Jellyfish’s Copilot Dashboard allows customers to answer:
- Adoption: We’re paying for Copilot — is my team using it?
- Utilization: How effective is Copilot overall for my developers, and does it differ with specific coding languages or editors?
- Allocation: How exactly are our users applying Copilot to their work? Cohorted reporting for Copilot users gives customers a quick and easy way to view and analyze data for the custom cohort of Copilot users on their team.
- Impact: How is Copilot driving change within the organization? Our reporting provides customers with insights, including:
- Understanding and reporting on the ROI of Copilot
- Identifying changes in developer and team workflows
- Finding opportunities to maximize the usage of Copilot
Generative AI is reshaping engineering from sprint planning to the code base. Organizations need to combine effective management with objective metrics to maximize value on their AI investments. Twelve months from now, we expect a completely different picture when we survey engineers and engineering managers about their use of AI. It’s time to move beyond predictions and take a data-driven approach to GenAI.
