7 Engineering Metrics to Track and Optimize

What are engineering metrics? 

Engineering leaders need to stay on top of many moving parts. They have to be on top of everything from AppSec and SLOs to developer satisfaction, retention and productivity. 

This can be overwhelming, especially when you need to gather data from various tools and systems, making it difficult to see the bigger  picture. 

Engineering metrics can be tied to key performance indicators (KPIs), and can be used in part or as a whole objective within the OKR (objectives and key results) framework. 

They fall within 6 key categories:

1. Code metrics — Analyze code coverage, code quality, and complexity
   
2. Product metrics — Measure the product itself for its scalability, features, design, quality, security, and reliability
3. Process metrics — Analyze the approaches, tools, methods, and deployment cycles of software engineering, helping to unearth potential obstacles and friction points
4. Project metrics — Keep track of progress against an initiative, with a focus on deadlines, input, costs, and resources
5. People metrics — Keep track of developer productivity, experience, and satisfaction. Understanding the churn of developers

Why do engineering metrics matter?

For any engineering leader, everything needs an answer, now:

• Are we on track to meet our goals?
• How can we make developers more productive?
• Which processes slow us down?
• Which issues need my attention now, and which ones can wait?
• Are we compliant with standards? 

With engineering metrics, engineering leaders can quantify and gain an understanding into teams, processes and products to see if there are areas for improvement. They can compare their metrics with your own baselines, historical data, and even benchmarks set by other similar or inspirational organizations. 

Engineering metrics benefits

Engineering metrics help engineering leaders tie working agreements to business results. They do that by enabling teams to:

• Develop and iterate the best possible product
• Optimize processes - reducing resources and cognitive load (reducing time taken to build new features or to resolve tasks)
• Get alignment through visibility - with relevant stakeholders all aware of the metrics and how these link to the team’s overall objectives
• Prioritize  - making it clear which areas require teams or individuals to focus on first. 

Engineering metrics are not a means to an end; they are an important part of a continuous improvement loop, ensuring engineering excellence. 

What are the different types of engineering metrics?

We can separate the different types into two main categories:

1. Classical engineering metrics that are used by engineering leaders to better understand how software is being developed and delivered. These are the metrics that many organizations may already be tracking via software engineering intelligence platforms or in-house tools that analyze raw data from code repositories. They include things like deployment frequency and cycle time, and focus on the health of processes and teams, the effectiveness of teams, efficiency and scalability.
   
   
2. Standards-based engineering metrics - the second set of engineering metrics can be focused around standards, from ownership, documentation, monitoring, testing and even AppSec or cost. 

They appear in many tools used in the SDLC, which means they are not unified nor collected in one place. While various distinct tools provide valuable insights, trying to track engineering metrics in a variety of tools is difficult. It also requires an ongoing effort to consolidate metrics, and attempt to make sense of them, which isn’t trivial. 

Engineering metrics in an internal developer portal

One of the benefits of using an internal developer portal is that it enables you to unify all of these metrics in one place.

The internal developer portal is the ideal fit for where to consolidate your engineering metrics:

• The portal unifies all data and adds context about the SDLC

The portal’s software catalog integrates all data related to the SDLC - about microservices, APIs, cloud resources,PRs, compliance and more. By aggregating data throughout every stage of the life cycle, a portal can show you the connections between different metrics, offering insights that might otherwise go unnoticed.

For example, you might find that a higher deployment frequency correlates with an increase in incidents. This could signal that the team is prioritizing speed over quality, leading to more bugs and errors in production.

The software catalog can also help you assess the impact of discrepancies in your metrics. When a key metric drops, you can identify precisely which product lines or customers are affected, allowing you to take swift targeted actions.

• Identify and investigate non-compliance with standards

Internal developer portals are ideally suited to both defining standards and tracking  them with scorecards, enabling you to detect and investigate any non-compliance with standards. In fact, portals are the ideal place to codify your working agreements and put such a practice into place.

• Drive initiatives

Portals go beyond an analytics interface and fancy reports. They allow managers to drive change by being able to act on the metrics in the portal, driving a cycle of continuous improvement. This applies to both classic engineering metrics and standards related metrics.

What engineering metrics should you measure?

There are many types of engineering metrics. Engineering metrics can be anything, really. You may want more insights into processes or even want to measure time to market for a certain feature, or to closely monitor the health of tenants you’ve provided to your enterprise customers. In some cases, you want to track metrics that provide insight into team collaboration or responsiveness, such as various response times and wait times.

While DORA and SPACE metrics provide insights into engineering processes and team productivity, there are other areas such as understanding code quality, cloud costs, AppSec compliance, production readiness, and SLOs that are equally important.

All these count as engineering metrics, and all of them can be tracked, acted upon, and optimized within an internal developer portal. But how do you agree on what “good” software development looks like? 

Before you enter the portal, it’s important to sit down with your teams and create your own definition of “good” software development using working agreements. Below, we’ll break down how to define working agreements for the two types of engineering metrics, and the different metrics within them.

Software engineering intelligence metrics 

Developer productivity metrics 

The industry-standard DORA metrics provide a solid framework for measuring your team's speed and stability. By tracking Deployment Frequency, DORA lead time, Change Failure Rate, and Mean Time to Recovery (MTTR), you can find the sweet spot between delivering new features quickly while maintaining a reliable system.

But we know that understanding and optimizing your team's performance goes beyond DORA metrics. You should also consider looking at: 

• Pull Request (PR) Cycle Time: How long does it take for code changes to be reviewed, approved, and merged? You might even want to break this down further into time spent in review, time waiting for approval, and time to merge after approval. It will help you spot bottlenecks. 
• Build Time: Are slow builds holding your developers back?
• Developer Onboarding Time: How quickly can new hires become productive contributors?

You may not realize how long a PR languishes in a review stage before it’s picked up until you’ve begun to measure this — and that’s ok. Once you see an issue in the portal, you can raise awareness among your team about your initiative to shorten the PR cycle time and begin creating working agreements that define an appropriate length of time to handle PR review. 

For example, a company with a four-day PR cycle time may set a working agreement that any open PR must be reviewed within one day of submission. Using Insights in an internal developer portal, you can then check and report on your teams’ progress against shortening the overall PR cycle time.

Using the internal developer portal to improve developer productivity

The portal can help you to unearth and fix issues that are holding back the productivity of your team.

• Taking too long to approve PRs? Create an automation to nudge reviewers after 24 hours.
• Slow developer onboarding? Use the internal developer portal to abstract and simplify onboarding
• High CFR (Change Failure Rate)? Make sure quality tools are leveraged and check that all services have minimum code coverage thresholds in the portal.  

Developer experience metrics 

Developer experience (DevEx) metrics measure the quality and efficiency of a developer's interactions with tools, processes, and systems within an organization. These metrics help identify pain points, streamline workflows, and improve overall productivity and satisfaction. It’s also worth establishing a developer experience survey, which can help you to understand your developer pain points, what they need from a developer portal, and then act on those results. 

Here are some of the key DevEx metrics you can track in an internal developer portal:

• Onboarding time -  Ensuring that new developers are quickly brought up to speed, while also making sure they are comfortable and proficient with the tools and services they need to use. This can be measured by metrics like Time To First Hello World (TTFHW) or time to first commit.
• Breadth and quality of documentation - Evaluating whether the documentation for each tool, feature, and action is comprehensive, easy to understand, and complete.
• Platform stability - Monitoring uptime and reliability is crucial to ensure developers can concentrate on coding and problem-solving without interruptions.

Using the internal developer portal to improve developer experience.

• Streamline developer onboarding by having all of the information to get set up in the environment including documentation, details about services and APIs, and the required access, in one place, as well as the relevant self-service actions.
  
•  Evaluate documentation documentation with scorecards, grading each piece of documentation on whether it is easy to find, relevant and up-to-date for developers.
  
• View platform stability metrics with customizable and personalized dashboards. 
  

Further reading: Developer experience metrics: best practices, key components and examples

Standards-based engineering metrics

1. Production readiness metrics

Ensuring the health and compliance of your production assets is critical. But what exactly does "production ready" mean? It can vary from team to team, and that's where internal developer portal Scorecards come in. They enable you to define “production ready” and enforce that standard by asking questions like:

• Ownership: Is every software/cloud asset clearly owned and maintained by a designated team?
• Reliability: Are there sufficient redundancies and fail-safes in place?
• Security: Are all security best practices being followed, and are vulnerabilities promptly addressed?
• Monitoring: Is monitoring in place to detect issues early and provide alerts?
• Quality: Do your software/cloud assets meet your internal standards for code quality and performance?

...and more.

Once you've defined your production readiness criteria, a Scorecard lets you track your assets across different tiers of readiness. Scorecards can be aggregated if needed for reports, and managers can get a birds-eye view using visualizations in the internal developer portal to drill down into those assets that are not compliant. This gives you a clear picture of where you stand in terms of compliance and helps you prioritize improvement efforts.

You can also track trends over time to see how your scorecards are progressing towards production readiness. Are they getting better? Are you slipping in certain areas? 

Using the internal developer portal to improve production readiness

Once you've identified areas for improvement, it's time to drive change. An internal developer portal doesn’t just highlight production readiness issues, but rather adds the ability to resolve them:

• Gamify compliance: You can use portal dashboards to display Scorecard results by team and encourage teams to improve.
  
  
• Drive targeted initiatives: Set clear expectations with initiatives in the internal developer portal, which address specific improvement areas. Define SLAs for completion, add manager approvals, and allow for exception handling when necessary – all within the internal developer portal.

Further reading: Production readiness checklist: ensuring smooth deployments 

2. Incident management metrics

Complying with SLAs is key to maintaining customer trust and satisfaction. 

While MTTR (Mean Time to Recovery) is a key metric, you can get a more complete picture of your incident management process by also tracking:

• Incidents created vs. resolved over time: Are you creating more incidents than you're resolving?
  
• Total open incidents over time: How many incidents are unresolved?

Using the internal developer portal to improve incident management

If reducing MTTR is a top priority, an internal developer portal can help you create an efficient experience for your on-call engineers. This could include:

• A unified incident management interface including ownership details, service dependencies, and relevant dashboards with graphs from several monitoring tools.
  
• Automated access management: grant on-call engineers the necessary production access automatically when their shift begins and revoke it when their shift ends.
  
• Customizable workflows: automating tasks like escalations, notifications, and status updates. 

Further reading: How internal developer portals improve incident management

3. Security (AppSec) metrics

Security is a top concern for any engineering organization. But often, it's hard to differentiate between what's truly critical and what's just noise. Use your portal to track:  

• Critical security issues for tier 1 assets: track the most pressing security vulnerabilities affecting your most important systems.
  
• Mean time to remediate (MTTR) for critical security issues: Understand how long it takes your team to address severe security threats, giving you insight into your exposure windows.
  
• Security initiative tracking (e.g., Log4j): Monitor the progress of specific security initiatives, ensuring that vulnerabilities are patched and risks are mitigated effectively.

Using the internal developer portal to improve your security metrics

Using automations inside the internal developer portal can help you address vulnerabilities proactively.

• When a vulnerability like Log4j surfaces, you need to act fast. Internal developer portals help you identify affected assets instantly, prioritize upgrades based on risk, and create targeted initiatives to patch the issue quickly.
  
• Automations can be configured to notify you only about critical vulnerabilities affecting Tier 1 assets in production and reduce alert fatigue. 

Further reading: Why Appsec Teams Need Internal Developer Portals‍

4. Cloud cost metrics

We've saved the best for last – cloud costs. In a world where these expenses are skyrocketing, you need to ensure your spending is as efficient as possible.

Here are some of the key cost metrics you can track in an internal developer portal:

• Infrastructure costs over time: Usually Finops reports will give you insights of costs per resource type, but you might consider breaking this down per team or per business domains to drive better decision making.
  
• Tagging coverage for FinOps: Ensure that all your cloud resources are properly tagged for financial tracking and optimization.
  
• Ephemeral environment costs: Often, temporary environments are the ones draining your budget. Track their costs separately to identify opportunities for savings.

Using the internal developer portal to improve cost metrics

To tackle runaway cloud costs, consider implementing an automation to terminate ephemeral environments, you can set a TTL for each ephemeral environment. When the TTL expires, the portal will automatically terminate the environment, preventing unnecessary spending on idle resources.

Further reading: Using an internal developer portal for superior FinOps visibility Automations to 

5. Code quality metrics

Code quality metrics help to inform you on whether the value of the codebase is good or bad. This assessment is made in relation to the standards and expectations set by the engineering team.

Here are some of the code quality metrics you can feed in to the portal via plugins to get a centralized view:

• Code coverage, often measured as the percentage of your code executed by automated tests, is a key indicator of the robustness and reliability of your software. The higher the coverage, the more likely you will have bug-free code.
  
• Code duplication can lead to several issues including maintenance overhead, inconsistencies and increased bug surface. 

Using the internal developer portal to improve code quality

A portal can help you to track code quality metrics, but its main benefit is providing additional context to those metrics through a single source of truth. With a portal, you can:

• Unify code quality metrics with other quality metrics
  Use scorecards to combine code quality metrics with other quality-related metrics, to provide you with a quality scorecard that helps you to ensure standards are being met across all fronts.
  
• Ensure your services are leveraging quality tools
  You can create an initiative to ensure that all services are connected to a Sonar project, for example. 

Internal developer portal for engineering metrics

By having a view of your SDLC, an internal developer portal is the ideal place to measure and monitor metrics, and improve on these. 

The key differentiators of an internal developer portal is that it:

• brings together both classical engineering metrics and engineering standards metrics in one, optimized place with insights.
• provides added context via the software catalog so that you know which metrics truly matter.
• enables you to define standards and monitor adherence through scorecards.
• Provides smart notifications to receive alerts whenever key metrics drop below a certain threshold through automations.
• Enables you to drive change through initiatives or self-service actions, and even automations (such as nudges and alerts).

A portal enables engineering leaders to identify the key metrics they need to track, while platform teams can leverage their expertise to build experiences in the portal that drive improvements. This collaboration ensures that the insights gathered from internal developer portals are not just data points, but actionable information that drives positive change within your engineering organization.