How to implement and track DORA metrics in your organization

This post was originally published on The New Stack on 14 November 2024. 

DORA metrics are a key barometer for engineering performance and software stability. But collecting and interpreting DORA metrics isn’t always straightforward: Data needs to be pulled from various sources, such as git providers, CI/CD pipelines, incident management tools, and more. This can be a complex task and is different for each organization.

As an internal developer portal already integrates with an engineering team’s underlying platform and centralizes everything that an engineering organization needs to do and know, it is the ideal platform to provide leaders with all the metrics they need to manage their teams and operations efficiently. 

In Google’s recent DORA report, the company emphasized that internal developer platforms required user-centeredness, developer independence, and a product-oriented approach to get the most out of platform engineering. This is where the portal comes in to abstract away the complexities of the infrastructure and offer direct interaction through a unified interface, this ensures an improved developer experience.

Let’s explore how an internal developer portal can be used to track DORA metrics within your organization.

Before you start

The prerequisite for this guide is to have completed the onboarding process for the internal developer portal, as well as linking the repository of your choice (in this case, we’ll be looking at either GitHub, GitLab, or Azure Repos — but you can bring your own data to the portal too).

How to track your team’s deployments

To track the specific data required for deployment frequency, change failure rate and lead time for changes, you need to create a deployment blueprint. Blueprints are the foundational building blocks of the internal developer portal, holding the schema of the entities that you would want to represent in a centralized repository. They are completely customizable and can support any number of properties the user chooses. In this case, they will be created ensuring the essential information for each deployment is captured. 

Step 1: Set up your data model

The first step is to create a new blueprint, which is as easy as clicking the [+ Blueprint] button on the builder page. Then, name it “deployment” and add the schema below:

– – – – CODE: JSON – – – –
               
{
  "identifier": "deployment",
  "title": "Deployment",
  "icon": "Rocket",
  "schema": {
    "properties": {
      "createdAt": {
        "title": "Deployment Time",
        "type": "string",
        "format": "date-time",
        "description": "The timestamp when the deployment was triggered."
      },
      "environment": {
        "title": "Environment",
        "type": "string",
        "enum": [
          "Production",
          "Staging",
          "Testing"
        ],
        "description": "The environment where the deployment occurred."
      },
      "deploymentStatus": {
        "title": "Deployment Status",
        "type": "string",
        "enum": [
          "Success",
          "Failed"
        ],
        "description": "Indicates whether the deployment was successful or failed."
      }
    },
    "required": []
  },
  "mirrorProperties": {
    "leadTimeHours": {
      "title": "Lead Time (Hours)",
      "path": "pullRequest.leadTimeHours"
    }
  },
  "calculationProperties": {},
  "aggregationProperties": {},
  "relations": {
    "service": {
      "title": "Service",
      "target": "service",
      "required": false,
      "many": false
    },
    "pullRequest": {
      "title": "Pull Request",
      "target": "githubPullRequest",
      "required": false,
      "many": false
    }
  }
}


– – – –ENDS – – – –

For those of you who do not have the lead time configured, you can follow the integration guide for your git provider to add this property. 

Step 2: Pick a deployment tracking strategy

You’ll then have to pick your way of tracking deployments. Here are the options:

• PR/MR Merge

The recommended approach for tracking deployments and calculating lead time is by monitoring when pull requests (PRs) or merge requests (MRs) are merged into a branch. The lead time for these merges is calculated as the difference between when the PR/MR was created and when it was merged.

When a PR/MR is merged, a deployment entity is created within the portal to represent the deployment that took place. The lead time for that PR is calculated and added to the blueprint’s makeup.

• Workflow/job

You can monitor workflow runs in your pipeline to track deployments. This captures all workflow runs from the main branch that match a certain name or label and maps them to deployment entities. The deployment status is set dynamically depending on whether the concluded workflow run was successful. 

For example, when a GitHub Actions workflow, GitLab CI/CD pipeline, Jenkins job, or CircleCI job runs on the main branch with the name “build,” a deployment entity is created in the portal to represent the deployment.

• CI/CD pipelines 

CI/CD pipelines provide a reliable method to track deployments. With Jenkins, for instance, you can create and update entities in your portal dynamically using the portal’s API as part of the pipeline execution. Other CI/CD pipelines such as Octopus Deploy, Circle CI, Azure Pipelines, Codefresh, and GitLab can also be used. 

– – – – CODE: YAML – – – –

pipeline {
  agent any
  environment {
    API_URL = "https://api.getport.io"
  }
  stages {
    stage('Report Deployment to Port') {
      steps {
        withCredentials([
                string(credentialsId: 'port-client-id', variable: 'PORT_CLIENT_ID'),
                string(credentialsId: 'port-client-secret', variable: 'PORT_CLIENT_SECRET')
        ]) {
          script {
            def auth_body = """
              {
                "clientId": "${PORT_CLIENT_ID}",
                "clientSecret": "${PORT_CLIENT_SECRET}"
              }
            """
            def token_response = httpRequest contentType: 'APPLICATION_JSON',
                    httpMode: 'POST',
                    requestBody: auth_body,
                    url: "${API_URL}/v1/auth/access_token"

            def token = new groovy.json.JsonSlurperClassic().parseText(token_response.content).accessToken

            def entity_body = """
              {
                "identifier": "${env.JOB_NAME}-${env.BUILD_NUMBER}",
                "title": "Deployment for ${env.JOB_NAME}",
                "properties": {
                  "environment": "Production",
                  "createdAt": "${env.BUILD_TIMESTAMP}",
                  "deploymentStatus": "${env.BUILD_STATUS == 'SUCCESS' ? 'Success' : 'Failed'}"
                },
                "relations": {
                  "service": {
                    "combinator": "and",
                    "rules": [
                      {
                        "property": "$title",
                        "operator": "=",
                        "value": "${env.JOB_NAME}"
                      }
                    ]
                  }
                }
              }
            """

            httpRequest contentType: "APPLICATION_JSON",
                    httpMode: "POST",
                    url: "${API_URL}/v1/blueprints/deployment/entities?upsert=true&merge=true",
                    requestBody: entity_body,
                    customHeaders: [
                            [name: 'Authorization', value: "Bearer ${token}"]
                    ]
          }
        }
      }
    }
  }
}

– – – –ENDS – – – –

• Releases/Tags

You can track GitHub deployments by linking repository releases and tags directly to deployment entities in Port. These repositories carry essential details like service versions, commits, and release information, making it easier to keep tabs on your deployments.

To implement this, you can add the configuration below to the data sources page in your portal, and select your GitHub integration. 

– – – – CODE: YAML – – – –
               
- kind: release
    selector:
      query: .target_commitish == 'main'
    port:
      entity:
        mappings:
          identifier: .release.name + '-' + .release.tag_name
          title: .release.name + " Deployment on release"
          blueprint: '"deployment"'
          properties:
            environment: '"Production"'
            createdAt: .release.created_at
            deploymentStatus: '"Success"'
          relations:
            service: .repo.name

– – – –ENDS – – – –

• Custom API

For those of you whose tool or workflow is not natively supported in the portal, you can create custom integrations by directly interacting with the portal’s API. This enables you to track deployments from any system that can initiate HTTP API calls. 

Here’s an example of how you can use the API to create a deployment entity in Port:

– – – – CODE: CURL – – – –


curl -X POST https://api.getport.io/v1/blueprints/deployment/entities?upsert=true&merge=true \
-H "Authorization: Bearer $YOUR_PORT_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
  "identifier": "custom-deployment-1234",
  "title": "Custom Deployment for ID 1234",
  "properties": {
    "environment": "Production",
    "createdAt": "2024-09-01T12:00:00Z",
    "deploymentStatus": "Success"
  },
  "relations": {
    "service": {
      "combinator": "and",
      "rules": [
        {
          "property": "$title",
          "operator": "=",
          "value": "Custom-Service-Name"
        }
      ]
    }
  }
}'
– – – –ENDS – – – –

You can also track services or components within a monorepo through custom integrations (API) and advanced mapping.

Incident management

Incidents play a key part in tracking DORA metrics such as change failure rate (CFR) and mean time to recovery (MTTR). By improving the way you track incidents, you’ll have a better handle on how frequently deployments fail and how quickly teams can resolve issues. 

This means having the capability to use incidents to calculate CFR and MTTR, link incidents to specific services to gauge the impact of failures, and aggregate metrics across incidents for more effective monitoring.

Here’s how you can do this:

Step 1: Setting up your data model

First, you have to make sure your PagerDuty incident blueprint is properly configured to map incidents to the correct services. This includes defining the appropriate properties and relations for incidents. You can do this following the guidelines here. 

To add properties to capture incident resolution time and recovery time, use the below:

– – – – CODE: JSON – – – –

 "resolvedAt": {
   "title": "Incident Resolution Time",
   "type": "string",
   "format": "date-time",
   "description": "The timestamp when the incident was resolved"
 },
 "recoveryTime": {
   "title": "Time to Recovery",
   "type": "number",
   "description": "The time (in minutes) between the incident being triggered and resolved"
 }

– – – –ENDS– – – –

Then, you can add the below mapping config to the PagerDuty incident data source:

– – – – CODE: YAML – – – –

   resolvedAt: .resolved_at
   recoveryTime: >-
      (.created_at as $createdAt | .resolved_at as $resolvedAt |
      if $resolvedAt == null then null else 
      ( ($resolvedAt | strptime("%Y-%m-%dT%H:%M:%SZ") | mktime) -
        ($createdAt | strptime("%Y-%m-%dT%H:%M:%SZ") | mktime) ) / 60 end) # Time in minutes and divide by 3600 if you want it calculated in hours

– – – –ENDS– – – –

Ingest incidents from tools

To sync incidents from PagerDuty, simply follow the steps outlined in this PagerDuty guide, which provides a detailed walkthrough for setting up integrations to track deployment-related incidents.

For other incident management tools, refer to these specific guides:

• OpsGenie
• FireHydrant
• ServiceNow
• Statuspage

Relating incidents to services

You can link incidents to your service whether it is in a GitHub, GitLab or Azure DevOps repository.  Here’s an example of the code you would need for GitHub:

– – – – CODE: JSON – – – –

{
  "gitHubRepository": {
    "title": "GitHub Service",
    "target": "service",
    "required": false,
    "many": false
  }
}



– – – –ENDS– – – –

You then update the mapping config to your PagerDuty incident data source.

– – – – CODE: YAML – – – –

   - kind: incidents
     selector:
        query: 'true'
        ...: # Add other selectors as needed
     port:
        entity:
           mappings:
              identifier: .id | tostring
              title: .title
              blueprint: '"pagerdutyIncident"'
              properties:
                 status: .status
                 url: .self
                 resolvedAt: .resolved_at
                 recoveryTime: >-
                    (.created_at as $createdAt | .resolved_at as $resolvedAt |
                    if $resolvedAt == null then null else 
                    ( ($resolvedAt | strptime("%Y-%m-%dT%H:%M:%SZ") | mktime) -
                      ($createdAt | strptime("%Y-%m-%dT%H:%M:%SZ") | mktime) ) / 60 end)
                 ... # Add other properties as needed
              relations:
                 pagerdutyService: .service.id
                 # Add this relation to map the incident to the correct service
                 gitHubRepository:
                    combinator: '"and"'
                    rules:
                       - property: '"$title"'
                         operator: '"="'
                         value: .service.summary


– – – –ENDS– – – –

Aggregating DORA metrics

The next step is to aggregate the DORA metrics at the service level, enabling teams to track deployment frequency, change lead time, change failure rate (CFR) and mean time to recovery (MTTR) for each service. For broader insights, these metrics can be aggregated at higher levels — such as team or domain — by ensuring relevant blueprints are in place and relationships to the service blueprint are defined. Metrics are aggregated monthly by default, but this timeframe can be adjusted to weekly, daily or even hourly to meet your needs.

To set up aggregation and calculation properties in your service blueprint, start by navigating to the Builder in your Port portal. Find and select the service blueprint, then click the options menu (the “...”) in the top-right corner and choose “Edit JSON.” Add the relevant properties under the `aggregationProperties` or `calculationProperties` sections within the JSON schema. Once added, save your changes to activate the new aggregation configuration.

The aggregation properties should look like the following:

Deployment frequency

– – – – CODE: JSON – – – –

"deployment_frequency": {
   "title": "Monthly Deployment Frequency",
   "icon": "DeploymentIcon",
   "type": "number",
   "target": "deployment",
   "query": {
      "combinator": "and",
      "rules": [
         {
            "property": "deploymentStatus",
            "operator": "=",
            "value": "Success"
         }
      ]
   },
   "calculationSpec": {
      "func": "average",
      "averageOf": "month",
      "measureTimeBy": "$createdAt",
      "calculationBy": "entities"
   }
}


– – – –ENDS– – – –

‍Change lead time

– – – – CODE: JSON – – – –



"lead_time_for_change": {
   "title": "Lead Time for Change",
   "icon": "DefaultProperty",
   "type": "number",
   "target": "githubPullRequest",
   "query": {
     "combinator": "and",
     "rules": [
       {
         "property": "status",
         "operator": "=",
         "value": "merged"
       }
     ]
   },
   "calculationSpec": {
     "func": "average",
     "averageOf": "total",
     "property": "leadTimeHours",
     "measureTimeBy": "$createdAt",
     "calculationBy": "property"
   }
 }




– – – –ENDS– – – –

‍CFR

Add the following to the aggregated property in service: 

– – – – CODE: JSON – – – –


  "total_incidents":{
      "title": "Total Incidents",
      "type": "number",
      "target": "pagerdutyIncident",
      "calculationSpec": {
        "calculationBy": "entities",
        "func": "count",
        "averageOf": "month",
        "measureTimeBy": "$createdAt"
      }
  }, 
   "total_deployments": {
      "title": "Total Monthly Deployment Frequency",
      "type": "number",
      "target": "deployment",
      "query": {
         "combinator": "and",
         "rules": [
            {
               "property": "deploymentStatus",
               "operator": "=",
               "value": "Success"
            }
         ]
      },
      "calculationSpec": {
         "func": "average",
         "averageOf": "month",
         "measureTimeBy": "$createdAt",
         "calculationBy": "entities"
      }
   }



– – – –ENDS– – – –

Add this calculation property to calculate the CFR from the aggregated properties:

– – – – CODE: JSON – – – –

      "changeFailureRate": {
        "title": "Change Failure Rate",
        "calculation": "(.properties.total_incidents / .properties.total_deployments) * 100",
        "type": "number"
      }
     


– – – –ENDS– – – –

‍MTTR

– – – – CODE: JSON – – – –



   "mean_time_to_recovery": {
      "title": "Mean Time to Recovery",
      "icon": "DefaultProperty",
      "type": "number",
      "target": "pagerdutyIncident",
      "query": {
         "combinator": "and",
         "rules": [
            {
               "property": "status",
               "operator": "=",
               "value": "resolved"
            }
         ]
      },
      "calculationSpec": {
         "func": "average",
         "averageOf": "month",
         "property": "recoveryTime",
         "measureTimeBy": "$createdAt",
         "calculationBy": "property"
      }
   }



– – – –ENDS– – – –

Visualization

The most powerful way to use these metrics is to present them on custom dashboards and widgets within your portal. 

To set up a dashboard, you:

1. Go to your software catalog.
2. Click on the “+ New” button in the left sidebar.
3. Select “New dashboard.”
4. Name the dashboard (such as  DORA Metrics), choose an icon if desired and click “Create.”

Then you can add the widget of your choice by:

1. Clicking on +Widget and select Number Chart
2. Fill in the respective title 
3. Add an icon that makes the most sense for your team
4. Select `display single property` and choose Service as the Blueprint
5. Select an entity and choose MTTR as the property

‍

Acting on the metrics

Once you’ve put your metrics in place and you’ve set your benchmarks, the next step is to put this information to practical use. You can:

• Use scorecards to evaluate the team’s performance against your chosen benchmarks.
  
  
• Set up alerts for managers when their teams fall below thresholds.
  
  
• Initiate changes to improve your DORA metrics. For example, if code reviews are taking too long, implement an automation that nudges reviewers after a specific period. Or if your MTTR is higher than you’d like, centralize all your incident data in one place or automate your incident response to streamline resolution efforts and reduce recovery times.

Adopting DORA metrics is about giving your team a clearer view of how they’re doing and where they can improve. By using a developer portal to track these metrics, you’re not only centralizing data from various tools but also making it accessible and actionable for everyone. With these insights at your fingertips, it becomes easier to spot trends, set realistic goals and identify areas for improvement. 

Check out Port’s live demo to see what you can do with DORA metrics yourself and visit the docs guide for DORA metrics here.