September 18, 2023
There are a lot of “click-baity” articles telling engineering leaders what to measure. Every company in our industry writes about metrics, KPIs, DORA, etc. Admittedly, we’re guilty of this too, having written the 10 KPIs for Engineering Leaders eBook. Don’t get us wrong, we think leaders should consider those ten metrics, and we have opinions on the subject based on our own experiences, but they’re just suggestions. And a successful metrics strategy is mostly not about picking that one right metric.
So let’s take a step back and remind ourselves why we use engineering metrics in the first place, and build some guiding principles that keep us from losing the big picture.
No two leadership experiences are the same
No company in this product category has led your engineering organization, and no engineering team is exactly the same. Your business context, the product you’re building, and the unique dynamic between the people on your team should inform what you measure and how you measure it. There isn’t a one-size-fits-all approach to the job. Because of all the variables involved, there is no one magic metric that can measure your team’s success. You can stop searching now; it does not exist.
As a leader, it’s up to you to sift through your knowledge and experience to decide what is best for your teams. That’s part of what makes the job so hard.
So, why rely on engineering metrics at all within this landscape of complexity?
The purpose of engineering metrics
Data-driven insights (and engineering metrics) provide an informed, well-rounded perspective on your team’s work so that you and your team can make better decisions. But in order for this to happen, your engineering metrics must be firmly rooted in your business objectives. The specifics of what you measure should be grounded in a broader metrics strategy and leadership approach that drives the outcomes that matter to your business.
For example, it might not make sense to optimize your team’s ability to experiment rapidly in production if you’re working in a highly regulated industry or making medical devices where the cost of a mistake is extremely high. And it may not matter that you’re shipping features quickly if they are too low-quality to actually work. Or in a more famous example, counting lines of code does not help anyone evaluate the impact of that code (or the developer) on the business.
So think about the big picture of your business, and what that implies about the results you want. And make sure your whole leadership team understands this context too. This will allow you to keep a good data-informed balance rather than falling into the trap of Goodhart’s Law.
Engineering metric guidelines
So how do we make metrics serve the bigger picture? What guiding principles help us move towards the realized vision for our engineering metrics strategy? After much discussion within our team based on our own journey in this area, here are a few guidelines that might serve you well as you navigate the engineering metrics landscape.
1. Outcomes and impact > Targets and goals
Goodhart’s law can be summarized as, “When a measure becomes a target, it ceases to be a good measure.” We discussed this before and you can read more about it here.
But there is a larger point to be made within the application of Goodhart’s law to software engineering. The impact of your engineering team’s work that ultimately matters is the success of your business – there are no bonus points for having a well-oiled engineering machine that doesn’t accomplish anything else. Once you pick your “favorite” metrics, the team will – by human nature – focus on those metrics and achieve any targets you set. But the question becomes, what real-world results are those metrics a proxy for? What are you actually achieving?
Discussions about metrics can force difficult but ultimately productive conversations about what the team should prioritize, and the type of impact you want your team to have. And the best choices are usually made when you consider the big picture, including cascading effects of changes, and tension between goals (such as speed vs quality) that may be independently valuable, but also pull you in conflicting directions.
2. Strive for balance, not a magic metric or a peanut butter spread approach
The SPACE Framework team recommends measuring several metrics across 5 major categories that make up the acronym, SPACE. Pick 3 out of the 5 categories, and then pick metrics that show the status of those categories across three levels of the organization, and that should give you a picture of your engineering organization’s effectiveness.
The intention behind this recommendation is that it’s more important to ensure your organization is holistically aware of its strengths and weaknesses, which requires putting attention across a variety of measurable categories. The goal is to avoid major “blindspots,” without casting your net too wide out of paranoia of missing something. A base-level understanding of your team’s impact is better than a deep understanding of only one aspect, such as delivery management or team health. Measuring everything possible compromises your ability to identify meaningful takeaways (analysis paralysis) while picking very few “magic metrics” leads to gaming the system (Goodhart’s Law strikes yet again), and blind spots. Knowing when to pivot one way or the other will come with experience.
3. Include the team early and often
We can’t expect engineers (or anyone) to do their best work when they don’t feel psychologically safe, or are under constant pressure to deliver against unachievable expectations. And a harmful narrative exists in the world today, which your engineers have definitely heard. This narrative – that metrics are an inherently dangerous tool that employers use to make their employees toe the line – can make some engineers wary and skeptical of metrics. These team members might not feel heard by their leadership, or metrics may have been used against them in the past. Regardless of the reason, your team needs to be brought along in the metric decision-making process in a way that builds trust and transparency. When metrics are rolled out haphazardly, teams become demoralized and their work gets minimized. Engineering metrics are supposed to help boost the impact of engineering, not trivialize or minimize it.
Team health – and most notably the Satisfaction part of SPACE – should not be sacrificed as a part of your metrics strategy; a balanced metric strategy requires team health. Turnover rates or employee sentiment scores can partially measure a growing problem, but if you’re not listening carefully, or if you’re narrowly focusing on only part of the data, you could mistake a delivery management challenge for a deeper cultural issue.
Tools such as direct feedback through 1:1s and confidential employee feedback surveys can help provide insight into where you need to pull back, double down, or refocus. No metric strategy can replace the important leadership skill of listening.
4. Measurements and stories > Measurement or stories
Objectively measuring the operational excellence of your engineering organization is hard. If you focus too much on managing a few specific metrics, you risk missing the broader outcomes you need. Metrics alone are often insufficient to let you answer the all-important question, “But why did that happen?”, but tracking sentiment and emotional well-being by itself will signal that you’re not prioritizing business outcomes.
So: combine quantitative and qualitative data to create a holistic understanding of your engineering organization.
Engineering metrics and anecdotal feedback each require purpose and effective implementation to be successful. These two insight types together are quite powerful, and each helps pick up the story where the other leaves off. Metrics can communicate team impact and lead to further questions, while anecdotal feedback completes “the story” and provides perspective, eliminating the risk of missing important context that explains the phenomenon in the data.
Coming back to the big picture
We’ll leave you with this: don’t lose the big picture. Build creative features that delight customers and move the business forward. Those are usually the ones that developers want to work on anyway. Enable teams to work autonomously, addressing your ability to move issues and epics through the SDLC. And never forget that good culture takes time, effort, sacrifice, and genuine commitment to your people.
In the era of Accelerate, the SPACE Framework, and the countless vendors with “magic metrics” and their own reports, who do you listen to?
Our response might shock you: read them all, but be wary of those that claim to have it all figured out. No one has this down to an exact science yet, and the best we can do is update our guidance as we learn together.
