February 08, 2024
Engineering Leaders Should Focus on Moving the Needle Instead of Finding It
Want to get a group of software engineers into an argument? Ask them how to measure their work.
Last summer, McKinsey released an article that drove a lot of controversy and debate in the engineering community. The consulting firm argues that it’s possible to measure developer productivity, and they recommend a framework that combines DevOps Research and Assessment (DORA) metrics with the SPACE metrics developed by GitHub and Microsoft Research.
The reactions to that article were heated. Many engineers believed that McKinsey’s argument was too simple, even naive. They argued that DORA and SPACE metrics fail to capture the dynamics of truly high-performing engineering teams.
But that article was written for a reason. CEOs, CFOs and board members are increasingly frustrated by their inability to measure and understand the organization’s engineering work. A recent report from BCG found that the average software company spends around 20% of revenues on R&D, while some commit as much as 40%. Other areas of the business are expected to answer for where their money is going. It’s no longer acceptable for a CTO or an engineering leader to throw up their hands and say that engineering is too nuanced to measure. McKinsey might not have offered the right solution, but it’s clear that a new solution is necessary.
Engineering has a metrics problem. Engineering leaders can address it by understanding their audiences and by reframing “productivity” through the lens of business objectives.
Understanding the Metrics Problem
Why has it been so difficult to come to a consensus on how to measure engineering and developer productivity? We’ve been focusing our attention on the wrong aspect of measurement — aiming to quantify productivity as precisely as possible without considering the context and why it matters. To put it another way, engineers have been obsessed with finding the needle without ever thinking about how to move it.
It’s easy to understand why engineering leaders have struggled to put metrics into a broader context. Most engineering leaders come up and advance in a world of individual contributors — they’re the best engineer in their team or the best engineer in their organization, and they’re quickly promoted to manage others even if they’ve never been trained to lead. Engineering managers have learned how to be killer developers, but they’ve never had to think about the broader ecosystem and the business as a whole.
Effective engineering measurements need to center on three overarching themes:
- Strategic alignment: How does engineering work map against the organization’s specific objectives? What effect is engineering having on customer satisfaction?
- Insight into deliverables: What features and other deliverables were recently completed, which are in process, and which are on the horizon?
- R&D effectiveness: What is the cost and ROI for each feature and project type? What does each project demand from the team in terms of resources?
Once an engineering leader understands these three themes, the next step is to define key audiences and how to communicate these themes to each audience. In most software organizations, the engineering leader is communicating with three audiences:
- The company’s board of directors
- The engineering leader’s executive peers
- The engineering teams themselves
Each of these audiences has dramatically different needs and expectations when it comes to measuring engineering.
Speaking the Language of Your Audience
Metrics mean different things to different people. The difference between finding the needle and moving the needle comes down to an engineering leader’s ability to understand the data and put it in relevant context. For the engineering manager’s three audiences, that means knowing what language to speak and how frequently to speak it.
Your board of directors will rarely if ever be interested in the nuts and bolts of your engineering team’s performance. Instead, they want to know what engineering performance means for the business: specifically, in terms of how the engineering organization is optimizing for ROI and delivering value to the customer. They’ll also be interested in overall investment: how much of the roadmap is going towards growth as opposed to keeping the lights on or support? When possible, the board will want to see benchmarks — how does your team’s performance compare to competitors? Where are we outpacing the market and where do we have room for improvement? You should be prepared to present this information on a quarterly basis and speak to the high-level quarter-over-quarter changes.
Your executive peers need to hear from you more frequently, because the pace of your work directly flows into their teams and objectives. An engineering manager should be updating other business leaders on a monthly basis, providing them with up-to-date, easy-to-understand information that will help them plan sales, marketing, and financial strategies. Again, the key here is to tie metrics directly to company objectives and avoid “dev speak.” Story points don’t mean anything to a VP of marketing — you need to overcome the perception that engineering is a black box and figure out how to communicate progress effectively.
The engineers are the ones in the trenches doing the actual work, and you need to keep them updated on a weekly basis regarding how they’re performing against objectives. They want to understand the context of the work and why they’re doing what they’re doing. Once they can see how their work fits into the bigger picture, they’ll be able to provide a more useful gauge of how they’re delivering against timelines. As a result, you’ll be more confident taking those timelines and delivery dates to your peers and the board. It can be easy for engineering leaders to get lost in the minutiae of engineering work — remember, they were recently star engineers themselves. But it’s important to encourage engineers to think on a more macro level and explain how their work will impact the business. Engineering work doesn’t take place in a vacuum; you need to ensure that every member of the engineering team understands their role in the broader organization.
The devil is in the details. Can you effectively measure engineering productivity? Probably not as easily as McKinsey would like you to believe. But the right framework and strategy will allow you to measure the effectiveness of your teams in relation to the business. When every company is an engineering company, we can no longer afford for these operations to stay in a black box. I firmly believe you can measure engineering effectively — It’s time for engineering managers to attack their metrics problem once and for all.
