Why Engineering Leaders Need New Cost Efficiency Metric

Measuring engineering performance is complicated. Measuring the financial performance of an engineering organization? That’s another question entirely.

Engineering organizations often gauge their cost efficiency by a set of imperfect measurements: organizational ratios (e.g. number of devs to each manager; number of devs to each product manager; or number of devs to each QA engineer); benchmarks of spending by type (e.g., headcount, licensing, cloud/infrastructure); and various cost savings levers.

Those metrics can tell you something about the financial performance of your R&D organization, but they’re never going to tell a truly accurate story. Benchmarks or organizational ratios will look different for an early-stage startup than they will for an established enterprise – not to mention nuances for different industries and geographies. You need a deep understanding of both business and technical context to fully understand your cost efficiency.

For engineering leaders to actually measure cost efficiently effectively, they need a new set of measurements – metrics that allow them to think like an investor rather than an engineer.

How investors think about cost efficiency

When investors think about cost efficiency, they’re using the same broad categories regardless of the type of business or industry:

1. Capital: How is the organization using its money, and how much does its actual spending deviate from planned spending?
2. Resources: Does the organization have the right resources in place? If the organization is missing resources – skillsets, structure, costs – how do those missing resources impact performance and productivity? What is the cost of not having the right resources, both in terms of people and tools?

Given these considerations, an engineering leader can measure cost efficiency without holding themselves up to a poor-fitting benchmark. Instead, looking through the lens of an investor means that expectations and performance are right-sized to the current stage and size of the company.

To help, Jellyfish recently released the Software Engineering Intelligence (SEI) Maturity Model, which offers new ways for engineering leaders to measure performance – not only in terms of cost efficiency, but also across areas like R&D impact and roadmap planning.

The SEI Maturity Model suggests three capabilities to measure cost efficiency.

1. Capital efficiency

How can R&D leaders know where their human capital is being deployed, and whether their actual investments are aligned with planned investments? The best approach is to pair roadmap effort allocations across work types (growth work, keeping the lights on (KTLO), and support) with headcount cost allocation.

When those figures are aligned, it demonstrates that the organization is deploying the right engineers to the right work, and it shows that R&D organizational design is aligned with business strategy.

Here’s how the SEI Maturity Model measures maturity in capital efficiency:

2. Regrettable attrition

The cost of attrition is roughly two times the cost of an engineer due to lost productivity, recruitment costs, and ramp time.

Measuring regrettable attrition doesn’t just give an indicator of unnecessary costs, it also shows how effectively the R&D organization is executing, supporting and strategically prioritizing investments to retain engineers. For example, the organization might choose to underpay top performers in order to lower R&D spend. However, that decision may result in increased attrition and poor cost efficiency for the engineering organization as a whole.

The SEI Maturity Model uses regrettable attrition rate to measure cost efficiency, measuring the percent of engineers who wish to remain at the company.

Here’s how the SEI Maturity Model measures maturity in regrettable attrition:

3. Resource fungibility

For resources to be fungible, they need to be interchangeable with other resources of the same type. In the context of an engineering organization, resource fungibility refers to the agility and adaptability of existing resources within the organization. How long does it take an employee to ramp up on a new team, codebase or product? Long internal ramp times point to inefficiencies in the organization, including ineffective knowledge transfer. Short internal ramp times indicate that employees are able to quickly acquire new skills, adapt to new roles, and contribute to new projects.

Resource fungibility is a useful metric for cost efficiency because it suggests reduced (or increased) costs associated with ramping, hiring and resource gaps. To measure internal ramp time, companies should compare the impact of new engineers to the impact of their longer-tenured peers. This can be done by measuring the number of reviews completed, comments on PRs, and diffs per week – within the context of the team the engineer is on.

Here’s how the SEI Maturity Model measures maturity in resource fungibility:

As engineering metrics and tools continue to evolve, R&D leaders need to rethink the way they approach planning, measurement and resource allocation. Download our SEI Maturity Model for a new vision on how to evaluate R&D performance – replacing one-size-fits-all metrics with a framework that applies to organizations of every size, industry and growth stage.