Using scorecards for standards compliance: a repeatable framework and examples

For software, standards are the definition of done

Standards are important. They are a definition of done, a definition of what good looks like.

It’s pretty obvious software needs standards, to ensure AppSec, cost and production readiness compliance, to define PR review times, DORA metrics and more.

Many organizations struggle to define, establish and enforce engineering standards, not because they don't value them, or understand the benefit of quality, but because they lack the tools to effectively communicate and track standards. The result: production readiness or AppSec are tracked in spreadsheets, and a lot of manual labor is involved, as well as wikis, training sessions and more. 

Internal developer portals present a unique opportunity to make standards compliance simple and easy, so that managers can effectively define and communicate them, and developers know what needs to be done. This is done through the use of scorecards in the internal developer portal. 

As your organization grows, standards become even more crucial. Without them, things can quickly spiral into chaos. But here's the catch: while establishing standards is relatively straightforward, the real challenge is communicating, monitoring, and enforcing them.

This post explains the framework of using scorecards in internal developer portals, and provides examples of what these scorecards can cover. 

^‍Platform engineering, scorecards and standards

Platform engineering introduced the idea of "golden paths," which essentially embed operational excellence into developer workflows. 

For instance, when developers need to create a new service, they can use a self-service action that scaffolds the service with all the necessary standards. This includes, for example, creating a new repository with standardized README and folder structure, generating boilerplate code for tests, configuring the continuous integration (CI) process, and setting up SonarCloud scans on every merge to the "main" branch. This ensures that the organization's standards for quality, reliability, and documentation are consistently kept.

But golden paths alone are not enough. Standards aren't a one-and-done deal; they require ongoing monitoring and proactive management. And this is where scorecards come to place. 

What are scorecards?  

A scorecard is a way to define, measure and track the metrics related to each entity that exists in your internal developer portal. Each scorecard consists of a set of rules, where each rule defines one or more conditions that need to be met, as well as different scorecard levels, such as gold, silver and bronze.

You can use scorecards to establish metrics to grade production readiness, code quality, migration quality, operational performance, and more. You can use them to define DORA metrics, AppSec compliance, and even working agreements, such as PR review times.

Since the software catalog in the internal developer portal is a central system of record, scorecards leverage its power to be able to track and define standards.

How to use scorecards in a portal

Step 1: Discuss quality and standards

First, establish what software catalog assets you would like to monitor by discussing quality and standards with your team. Perhaps there are services, APIs or applications that the team wants more visibility or measurement around - or there are services or applications that haven’t yet been ingested into the software catalog. Discuss the “definition of done/good” and ensure your standards capture it.

Step 2: Set checks

Configure the checks and metrics for your scorecard. For instance, a code quality scorecard may involve checking code coverage. 

Step 3: Set thresholds

Define the thresholds for your different statuses (eg. bronze, silver and gold or best, at risk, and critical state). In the code coverage example, if the percentage fell below a certain threshold it could move from silver to bronze or from ‘at risk’ to ‘critical state’. 

Step 4: Create dashboards

Create dashboards to group scorecard data in a way that lets you see where standards are met and where they aren’t. 

Step 5: Drive actions 

Use automations to make it easy to act. For example: create Jira tickets when a critical asset degrades below a “Fair” score or send Slack notifications to developers when the score of their services degrades. Also set processes for exceptions. Use tiers for the different services.

Step 6: Constantly evaluate

After establishing and using scorecards, its metrics or thresholds should not be considered ‘fixed’. Any changes can impact what is classified as ‘gold’ or ‘needing critical attention’, and these can be adjusted accordingly. You can then use initiatives to drive change.

Which scorecards should you use?

You should have scorecards for everything you want to keep a closer eye on, such as services, running services, deployments, APIs, and even teams. Scorecards are flexible and customizable, allowing you to tailor them to your organization's specific needs. 

Now let’s have a deeper look and see real examples of how you can use scorecards to drive compliance. 

Microservice scorecards

Define service scorecards to monitor documentation, security, reliability, production readiness, health and more.

‍

    For instance, to check the production readiness of a service, we can define a minimum threshold to ensure that:

• the service is tied to a business domain
• it has a defined on-call
• it uses a supported language
• it has been updated in the last year
• there are not too many open incidents, and
• it is scanned through Sonar

We can also define scores, based on the metrics. A higher score can be given when the service is fresher or there are zero incidents. 

Creating service scorecards has some real benefits. As services move through the SDLC, you can ensure services are compliant before they hit production. You can also create automations on top of scorecards, for instance to fail the deployment of a service if it doesn’t meet production readiness standards. 

Security scorecards

A security scorecard can check test coverage, vulnerabilities, branch protection, freshness and more.

‍

Security scorecards have the benefit of consolidating many AppSec tools, effectively creating the ability for managers to centrally manage application security standards management. 

Read more about AppSec and internal developer portals here.

Running service scorecards

It is also important to keep track of the health of running services - especially those in production. One of the nice things about internal developer portals is that they allow you to track a running service entity, in context. 

Read more: why a running service should be part of the data model in your internal developer portal

In this example, we want to ensure that your running services are

• healthy (i.e. high availability)
• do not have active issues
• monitored 
• have a high throughput and 
• minimal error rate.

API scorecards

The last example we’ll discuss is scorecards for APIs.  

APIs are essential assets used both internally and externally. Therefore, you want to quickly provide developers with information about API quality and reliability so they can confidently determine if those APIs should be used.

Here is a scorecard you can build to ensure an API is certified. It needs to comply with discoverability, standardization and security rules. 

More scorecard and standards ideas

Port has the flexibility to link a scorecard to any asset in your catalog and gives you the ability to establish custom checks that align with your organization's specific requirements. That means scorecards can be created for your k8s clusters, deployments, teams, and pretty much any asset that requires close monitoring.

Visualize compliance with scorecards

Once your scorecards are set up, create dashboards to easily identify areas that need attention.

We recommend building a dashboard that provides a bird's eye view of all your scorecards, grouped by teams. This enables you to quickly identify teams that may require support. By using this visual representation,you can monitor compliance effectively and efficiently. You can also group scorecards by business domain or other attributes.

We also recommend monitoring your scorecard distribution.

And adding trend lines to look out for patterns. 

For example, in the below image we monitor the number of services that aren't production ready over time. This shows that there has been a sharp decrease in the services that are not production-ready over consecutive quarters, suggesting that there have been improvements in incorporating production readiness checks throughout the software development lifecycle. 

If you’re launching a specific initiative, we recommend creating an initiative dashboard, like the one below. 

In this example, we want to ensure that all services have implemented Sonarqube. 

It includes a pie chart that indicates the progress of the initiative, as well as important information such as the services that aren't meeting the requirements with their respective owners, the progress made per team, a list of all PRs without a Sonar analysis. It also offers a self-service action to help developers add tags to a SonarQube project directly from the portal, making it easier for them.

Make it actionable

Having this kind of visibility is valuable but to truly make a difference you need your scorecards to be actionable. For example you may want to generate a jira ticket for rules that are not being met.
Port Automations allows you to be granular with ticket creation. This is important because you don't want to create excessive noise and achieve the opposite of what you intended. You can choose to open a ticket only for mission-critical assets and only for specific checks.

Additionally, Port Automations allows you to send Slack/Teams notifications or any other communication you need.

Conclusion

Engineering standards are essential for any organization, but they can be challenging to implement and maintain. By leveraging internal developer portals, you can streamline the process, improve compliance, and empower your developers to build robust, scalable products. 

Ready to see how scorecards can help you maintain your engineering standards? Try out Port today with our live demo!