September 05, 2024
The sudden arrival of generative AI (genAI) in late 2022 led engineering organizations around the world to experiment with new coding tools — most notably, ChatGPT and GitHub Copilot. According to Stack Overflow’s 2024 Developer Survey, 62% of developers currently use AI coding tools in their development process.
Jellyfish’s 2024 State of Engineering Management Report (SEMR) found a disconnect between business leaders and front-line workers when it came to AI. More than three-quarters of executives stated their team had embraced AI, compared to just 52% of engineers.
Copilot and other genAI coding tools represent a revolution for engineering. Unsurprisingly, this revolution is prompting strong emotions on both sides of the coin: those who believe that genAI is driving innovation and efficiency vs. those who think the technology creates more problems than it solves. We connected with several engineering and AI leaders to gather their reactions to genAI coding tools; we offer their anonymized thoughts here to allow them to offer their unfiltered opinions.
GenAI benefits: Efficiency and savings
Two of genAI’s most outspoken supporters pointed to significant savings in terms of both time and money. However, they cautioned that organizations must take steps to customize the tools to drive real value.
An engineering leader at a SaaS company encourages everyone in his organization to use genAI. The company pays for every employee to have a Copilot subscription, and the leader organizes enablement sessions and Slack channels to ensure everyone gets the most out of the tools. The SaaS company saw broad improvements across the organization — not just in using Copilot for engineering productivity, but also using ChatGPT in other business functions like customer support.
Ready to measure Copilot adoption and impact? Now you can with Jellyfish’s Copilot Dashboard. See it in action here.
A machine learning leader at a global localization company also found efficiency gains through ChatGPT, and those improvements led to noticeable time savings for the engineering team. In particular, the localization company leans on genAI for boilerplate code generation and AI-based code review. This automation frees up the engineering team to do higher-level work.
“We worked to remove the ChatGPT smell, but it ended up being a big time saver,” said the machine learning leader. “The tool is still in its infancy, but it really does help with initial friction.”
The two engineering leaders aren’t alone in their opinions. Stack Overflow’s developer survey found that 81% of engineers agree that increasing productivity is one of the most important benefits of AI tools.
GenAI skepticism: Privacy risks and questions about code quality
For the genAI skeptics, that increased productivity could come at a cost.
An engineering leader at a networking company spent six months experimenting with Copilot. In the end, they turned the tool off due to data privacy concerns. While Copilot generated a large volume of code, the networking company found that the code didn’t always meet their highest quality standards — genAI sometimes struggled to guess what the engineering organization actually wanted to accomplish.
“You can’t use it blindly,” said the engineering leader. “You need someone in the driver’s seat with intelligence to pilot it.”
An engineering leader at a SaaS company in the financial space had similar concerns about genAI. Their industry requires the highest levels of privacy and security; the organization isn’t yet able to trust genAI. This engineering leader also wasn’t convinced by the quality of AI-generated code. “At the end of the day, I was left wondering if the amount of rework required negated the benefit we would get out of it in the first place,” said the SaaS engineering leader.
While genAI coding tools may improve over time, these privacy concerns will be a long-standing barrier to adoption. In the 2024 SEMR, 48% of those who don’t use genAI said they haven’t adopted it due to concerns about security.
Measuring GenAI impact
As genAI adoption spreads, engineering leaders need to determine what value they’re getting out of the tools — and if that value matches or exceeds their overall investment.
Earlier this year, Jellyfish released the GitHub Copilot dashboard to allow engineering leaders to measure the most popular AI coding tool. The Copilot dashboard tracks adoption, utilization, and allocation of Copilot, as well as impact — from increased developer velocity to percentage of suggestions accepted.
Despite the skepticism from some engineers, the growing adoption of genAI coding tools indicates that the industry isn’t going back. Engineering leaders must now focus on moving from experimentation to optimization; measuring AI initiatives will be key to driving productivity and maximizing ROI. While the debate may continue, engineering leaders will continue testing AI to see how it might just give them a competitive edge.
To learn more about how Jellyfish supports visibility into genAI coding tools, request a demo of our Copilot dashboard.
