August 22, 2024
Key Results
- 2X improvement in engineer satisfaction for test automation
- Engineers satisfied with 9 out of 10 key topics
- 15% increase in overall satisfaction and well-being
Measuring developer satisfaction and morale is top of mind for engineering organizations, particularly as these companies try to recruit and retain top talent. But many engineering teams struggle to define developer experience, let alone measure it. Until recently, Jellyfish was no different.
Measuring developer experience provides engineering leaders with unbiased data on their team’s health and performance. They can use this data to inform changes to their tools and processes, leading to happier employees and improved retention.
Jellyfish relies on its engineering team to build tools for other engineering organizations, and the company uses its own tools for everything from scenario planning to DevFinOps. After recognizing gaps in how the company measured engineer satisfaction, Jellyfish created its own DevEx tool to conduct surveys and capture unbiased data.
Before using DevEx, Jellyfish took an anecdotal approach to tracking engineer sentiment. Managers asked their employees in one-on-one conversations about specific topics and challenges, and then compared notes to see if these problems were company-wide or localized to just a few engineers.
“The biggest thing for me is to get some representative data and truth behind how people are doing and what is frustrating them,” said Eli Daniel, Jellyfish’s Head of Engineering. “I want to have that be something that we can look at in aggregate without the recency bias of whoever I happen to have spoken to last.” Daniel worked together with engineering managers and the individual contributors responsible for Jellyfish DevEx to start collecting actionable data on engineer sentiment across the company.
Taking a clear look at developer satisfaction
For members of Jellyfish’s executive team, deploying DevEx meant confronting some difficult truths about how the company plans and organizes its work. Jellyfish’s first DevEx survey revealed widespread dissatisfaction with the company’s testing environment.
“This is an area where, historically, we thought our system was just complicated. I had assumed that the work we were doing was too complex for our testing to ever be simple or easy to use,” said Daniel. “It was an area I had gotten accustomed to, but we got some pretty stark, low numbers around sentiment in those areas of the survey. That was a signal that we needed to spend time improving things.” Jellyfish conducted its first DevEx survey during the second half of 2023, asking engineers for their feedback across 10 topics: test automation, documentation, team processes, satisfaction & well-being, context switching, code review, tools & environments, perceived productivity, clear direction, and release process. Jellyfish DevEx includes industry-leading questionnaires and templates, so engineering managers can quickly and easily create and administer employee surveys.
While the answers are anonymous, managers can filter the data to see whether different results are more pronounced among specific roles or teams. This analysis feature allowed engineering leaders like Daniel to identify problems that were truly present across the entire organization, and team managers to focus on individual team challenges .
Committing to positive change
Jellyfish DevEx generates results automatically, including team segments, explanations of each result, and suggestions on how to make improvements. The results of Jellyfish’s first DevEx survey revealed serious dissatisfaction around test automation. The topic scored just 26 out of 100 in terms of developer sentiment, the lowest of the 10 topics surveyed. Daniel and other engineering leaders at Jellyfish recognized that they could no longer put off improvements to their testing environment.
Following the DevEx survey, Jellyfish prioritized test automation as an area for immediate improvement. One engineer spent two weeks working on test automation. Before beginning that session, they surveyed their fellow engineers on the challenges they were facing and what improvements they felt were necessary. Emboldened by the survey results, his colleagues were forthcoming and collaborative in working towards a new solution. After two weeks of work, the engineer addressed many of the most tedious issues with the test environment.
“I think the engagement of devoting some real energy to the problem changed the perception of how addressable these problems are,” said Daniel. “We know now that we can improve things around here when we set our mind to it.” Jellyfish conducted a second DevEx survey during the first half of 2024. Sentiment around test automation improved dramatically — more than doubling from 26 to 58 — the highest increase among the topics measured. Jellyfish’s engineering managers will continue to make improvements to test automation and follow these results in future DevEx surveys.
Satisfaction at every level
Jellyfish’s new approach to DevEx is popular among employees at every level of the engineering organization.
For Greta Heissenberger, a backend engineer who joined the company in late 2023, DevEx shows that Jellyfish is taking actionable steps to improve their work experience:
I love that people are talking about these things in regular meetings. We go over the results, we talk about what’s bad, and then we talk about what we can do about it. You actually see actionable items come out of these discussions. It’s one thing to sit at your desk and say ‘I wish we had better documentation.’ When you actually have a survey and see that everyone agrees, it’s a lot easier to commit to making that a priority and spending time to make it better.
For Marilyn Cole, an engineering manager at Jellyfish, DevEx has changed the way she approaches coaching and feedback with her team:
We asked engineers how they felt about their growth feedback and whether their manager cares about their growth. We did okay, but not as well as we’d like. When we pulled the data, we could see that people who report to one manager felt really great about it — that allowed other managers to talk to him and understand how he gives feedback, and then apply that to their own teams. The data made me more motivated to ask that question and to make more of a conscientious effort.
And for Daniel, DevEx provides clarity on what’s actually happening in the organization:
I love having factual information about where things stand. Are issues improving or not? What was the result of the testing efforts we took? What was the ROI? It’s great to be able to answer those questions. That’s not subjective opinion — it’s really valuable.
Supporting continuous improvement
Jellyfish continues to use its DevEx tool to improve the engineering experience, tracking changes over time while also introducing new topics and questions. Company leaders are working to find the right cadence for surveys — giving the organization enough time to respond to feedback and implement changes before measuring whether the situation has improved. “DevEx gives us a recipe,” added Daniel. “If we’re seeing some smoke that indicates there might be a problem, we can add different questions to the next survey. It simplifies the execution.”
The results of Jellyfish’s second DevEx survey showed the company was moving in the right direction. Engineers recorded positive sentiments (scores greater than 50) in nine out of 10 topics, as well as a 15% improvement in overall satisfaction and well-being. The survey also revealed new areas to target for improvement — namely, documentation. Following the example set with test automation, Jellyfish organized a team of employees who will focus on improving documentation. Those results will be measured after six months in the next DevEx survey. Going forward, DevEx will be a key driver of continuous improvement at Jellyfish, leading to happier engineers and healthier teams.
