Blog
/
/
Do DevOps need an Internal developer portal?
Platform Engineering

Do DevOps need an Internal developer portal?

Yonatan Boguslavski
Yonatan Boguslavski
Dec 30, 2022
Sign up
Download PDF
Platform Engineering
Example H2
Example H3

Introduction

‍

Since we started Port, we noticed an interesting phenomenon: the first users of the software catalog weren’t developers. They were platform and devops people. Why? We asked them and this is what one of them said:

“I am in charge of making all systems cloud and platform agnostic. To do this, I need to track everything - which I do today in a huge csv file. I need to know my regions, my dependencies in a multi-cloud environment. An internal developer portal will keep track of that for me.”

‍

‍

Devops need developer portals to track the exact state of distributed applications from the microservice, dependencies, infra, cloud and devtools point of view.

Indeed, the days of tracking it all in a huge CSV file are over. The internal developer portal tracks the state of distributed applications and the systems they run on. With many deploys per day it is almost impossible to know what has changed; the number of tools involved in deploying and monitoring those applications is large and many of them are siloed. This will become a core use case for internal developer platforms in the near future, earlier than other use cases, such as cloud cost tracking.

‍

What are internal developer portals?

Before we dive into the subject, let’s take a minute to define what’s in an internal developer portal.

Let’s use the gartner definition:

“IDPs provide a curated set of tools, capabilities and processes. They are selected by subject matter experts and packaged for easy consumption by development teams. The goal is a frictionless self-service developer experience that offers the right capabilities to enable developers and others to produce valuable software with as little overhead as possible. The platform should increase developer productivity, along with reducing the cognitive load. The platform should include everything development teams need and present it in whatever manner fits best with the team’s preferred workflow."

This quote tells us about developer self-service using the internal developer platform. But another core element is the software catalog. The software catalog within an internal developer portal isn’t just a microservice catalog, since it also covers resources. The software catalog includes the infrastructure and the software deployed over it, and reflects the entire ecosystem surrounding the software development life cycle: dev environments, CI/CD, pipelines, deployments and cloud resources. The software catalog also shows KPIs in context of a certain service, its deployment and the environments it runs on.

‍

{{cta_5}}

‍

Devops was doing fine up until now. Do they need a software catalog?

Here are some signs that the current approach (hopefully not a huge CSV file) isn’t working:

1. You have a manually maintained CSV. The margin of error here may be huge. You probably aren’t tracking everything with the resolution you need

2. You maintain a git file that holds metadata regarding your software and infra.

3. One of your super-engineers is the only person with this information, and they are a bottleneck.

4. It takes more than one minute to answer these questions while you navigate through different tools: 

  • What’s the currently running version in production for a given service?
  • Where are all the pull requests associated with a specific service?
  • What is the general status of a k8s cluster? 
  • What Kubernetes clusters are running across a multi-region, multi-cloud environment?
  • Where is the production changelog.

 

A positive way of making this statement is:

  • Devops need a centralized, single source of truth of the software architecture (microservices, environments, deployments, cloud resources, regions, and more). This overcomes the problem of data about software being siloed all over: in the Git provider, IaC, Cloud, CI/CD, developer tooling, and more.
  • Devops need one interface for change management to keep track of changes that took place and see the history of changes across the entire stack, such as deployments, infrastructure modifications, versions, configurations, etc
  • Devops need visibility for troubleshooting & root cause analysis - Since all metadata is managed in a single source of truth, performing root cause analysis becomes easier. For example, if there is an issue with a specific service in an environment, tracking changes like the version history up to changes within the underlying infrastructure, like Kubernetes configurations, gives you a bird’s eye view of all changes.
  • FinOps & Cost control - seeing assets managed within the developer portal with the associated owner allows them to see cloud expenses from the organizational structure point of view. Identifying orphan resources becomes more accessible. In addition, making automated decisions based on data residing in the catalog can eliminate the number of orphan resources and improve security & cost potential outages.

‍

Additional benefits are eliminating the reliance on tribal knowledge, creating a trusted single source of truth, reducing context switches between different devops tools, and more. 

‍

Where can you find the data for the software catalog?

It’s all over the place, but the API and a couple of exporters usually solve these questions pretty seamlessly. 

Here’s a non-exhaustive list:

1. Kubernetes

  • Existing Kubernetes objects in your cluster;
  • Kubernetes object changes (Create/Update/Delete) in real-time

2. CI/CD  jobs that perform different tasks such as:

  • Builds
  • Deployments
  • Stand alone jobs accessible for developers to run on demand 

3. Cloud - Each cloud account holds data about the provider, region, and all the associated cloud resources within the account

4. IaC - In terraform for example, every apply/destroy taking action holds data about the provisioned infrastructure and the associated metadata

5. GitHub - holds information about git provider components - links, names, pull requests, issues, actions, worklows

6. GitOps - many yamls from kubernetes or terraform, helm values, readme, API documentation, etc

7. DevTools - Jira, Pagerduty, Datadog, Snyk, NewRelic. 

‍

The importance of an API first approach to the software catalog

It’s obvious by now that a software catalog contains metadata about services and resources. Most integrations to software catalogs aim to ingest this metadata into the software catalog. Yet, a robust API can allow the ingestion of much more than metadata, adding live data to the software catalog. Without live data, software catalogs do not contain data about versions, packages, alerts, data that is usually siloed across different pipelines and automations as part of your CI/CD and environments. 

Another requirement here is a nuanced, hierarchical representation of resources. An API lets you show the hierarchy for resources, such as this namespace belongs to this cluster, and this cluster belongs to this cloud environment. The hierarchy provides additional context that can be valuable in several cases associated with the software catalog. This allows you to ask questions like which lambda functions are running in a specific region or which services are deployed on the cluster and not the opposite. This capability isn’t available today in backstage io, but is supported in Port. 

‍

Conclusion

Developer Portals put devops on the golden path to a great setup. This streamlines compliance, audits and security. Getting information about access, change management, and operational readiness becomes easy once all the siloed data is consolidated in a single place. Another benefit is faster onboarding for devOps team members.

{{cta-demo}}

Book a demo right now to check out Port's developer portal yourself

Book a demo
{{jenkins}}

It's a Trap - Jenkins as Self service UI

Read more
{{gitops}}

How do GitOps affect developer experience?

Read more
{{ebook}}

It's a Trap - Jenkins as Self service UI. Click her to download the eBook

Download eBook
{{cyberark}}

Learning from CyberArk - building an internal developer platform in-house

Read more
{{dropdown}}

Example JSON block

{
  "foo": "bar"
}

Order Domain

{
  "properties": {},
  "relations": {},
  "title": "Orders",
  "identifier": "Orders"
}

Cart System

{
  "properties": {},
  "relations": {
    "domain": "Orders"
  },
  "identifier": "Cart",
  "title": "Cart"
}

Products System

{
  "properties": {},
  "relations": {
    "domain": "Orders"
  },
  "identifier": "Products",
  "title": "Products"
}

Cart Resource

{
  "properties": {
    "type": "postgress"
  },
  "relations": {},
  "icon": "GPU",
  "title": "Cart SQL database",
  "identifier": "cart-sql-sb"
}

Cart API

{
 "identifier": "CartAPI",
 "title": "Cart API",
 "blueprint": "API",
 "properties": {
   "type": "Open API"
 },
 "relations": {
   "provider": "CartService"
 },
 "icon": "Link"
}

Core Kafka Library

{
  "properties": {
    "type": "library"
  },
  "relations": {
    "system": "Cart"
  },
  "title": "Core Kafka Library",
  "identifier": "CoreKafkaLibrary"
}

Core Payment Library

{
  "properties": {
    "type": "library"
  },
  "relations": {
    "system": "Cart"
  },
  "title": "Core Payment Library",
  "identifier": "CorePaymentLibrary"
}

Cart Service JSON

{
 "identifier": "CartService",
 "title": "Cart Service",
 "blueprint": "Component",
 "properties": {
   "type": "service"
 },
 "relations": {
   "system": "Cart",
   "resources": [
     "cart-sql-sb"
   ],
   "consumesApi": [],
   "components": [
     "CorePaymentLibrary",
     "CoreKafkaLibrary"
   ]
 },
 "icon": "Cloud"
}

Products Service JSON

{
  "identifier": "ProductsService",
  "title": "Products Service",
  "blueprint": "Component",
  "properties": {
    "type": "service"
  },
  "relations": {
    "system": "Products",
    "consumesApi": [
      "CartAPI"
    ],
    "components": []
  }
}

Component Blueprint

{
 "identifier": "Component",
 "title": "Component",
 "icon": "Cloud",
 "schema": {
   "properties": {
     "type": {
       "enum": [
         "service",
         "library"
       ],
       "icon": "Docs",
       "type": "string",
       "enumColors": {
         "service": "blue",
         "library": "green"
       }
     }
   },
   "required": []
 },
 "mirrorProperties": {},
 "formulaProperties": {},
 "calculationProperties": {},
 "relations": {
   "system": {
     "target": "System",
     "required": false,
     "many": false
   },
   "resources": {
     "target": "Resource",
     "required": false,
     "many": true
   },
   "consumesApi": {
     "target": "API",
     "required": false,
     "many": true
   },
   "components": {
     "target": "Component",
     "required": false,
     "many": true
   },
   "providesApi": {
     "target": "API",
     "required": false,
     "many": false
   }
 }
}

Resource Blueprint

{
 “identifier”: “Resource”,
 “title”: “Resource”,
 “icon”: “DevopsTool”,
 “schema”: {
   “properties”: {
     “type”: {
       “enum”: [
         “postgress”,
         “kafka-topic”,
         “rabbit-queue”,
         “s3-bucket”
       ],
       “icon”: “Docs”,
       “type”: “string”
     }
   },
   “required”: []
 },
 “mirrorProperties”: {},
 “formulaProperties”: {},
 “calculationProperties”: {},
 “relations”: {}
}

API Blueprint

{
 "identifier": "API",
 "title": "API",
 "icon": "Link",
 "schema": {
   "properties": {
     "type": {
       "type": "string",
       "enum": [
         "Open API",
         "grpc"
       ]
     }
   },
   "required": []
 },
 "mirrorProperties": {},
 "formulaProperties": {},
 "calculationProperties": {},
 "relations": {
   "provider": {
     "target": "Component",
     "required": true,
     "many": false
   }
 }
}

Domain Blueprint

{
 "identifier": "Domain",
 "title": "Domain",
 "icon": "Server",
 "schema": {
   "properties": {},
   "required": []
 },
 "mirrorProperties": {},
 "formulaProperties": {},
 "calculationProperties": {},
 "relations": {}
}

System Blueprint

{
 "identifier": "System",
 "title": "System",
 "icon": "DevopsTool",
 "schema": {
   "properties": {},
   "required": []
 },
 "mirrorProperties": {},
 "formulaProperties": {},
 "calculationProperties": {},
 "relations": {
   "domain": {
     "target": "Domain",
     "required": true,
     "many": false
   }
 }
}
{{tabel-1}}

Microservices SDLC

  • Scaffold a new microservice

  • Deploy (canary or blue-green)

  • Feature flagging

  • Revert

  • Lock deployments

  • Add Secret

  • Force merge pull request (skip tests on crises)

  • Add environment variable to service

  • Add IaC to the service

  • Upgrade package version

Development environments

  • Spin up a developer environment for 5 days

  • ETL mock data to environment

  • Invite developer to the environment

  • Extend TTL by 3 days

Cloud resources

  • Provision a cloud resource

  • Modify a cloud resource

  • Get permissions to access cloud resource

SRE actions

  • Update pod count

  • Update auto-scaling group

  • Execute incident response runbook automation

Data Engineering

  • Add / Remove / Update Column to table

  • Run Airflow DAG

  • Duplicate table

Backoffice

  • Change customer configuration

  • Update customer software version

  • Upgrade - Downgrade plan tier

  • Create - Delete customer

Machine learning actions

  • Train model

  • Pre-process dataset

  • Deploy

  • A/B testing traffic route

  • Revert

  • Spin up remote Jupyter notebook

{{tabel-2}}

Engineering tools

  • Observability

  • Tasks management

  • CI/CD

  • On-Call management

  • Troubleshooting tools

  • DevSecOps

  • Runbooks

Infrastructure

  • Cloud Resources

  • K8S

  • Containers & Serverless

  • IaC

  • Databases

  • Environments

  • Regions

Software and more

  • Microservices

  • Docker Images

  • Docs

  • APIs

  • 3rd parties

  • Runbooks

  • Cron jobs

Check out Port's pre-populated demo and see what it's all about.

Check live demo

No email required

Contact sales for a technical product walkthrough

Let’s start

Open a free Port account. No credit card required

Let’s start

Watch Port live coding videos - setting up an internal developer portal & platform

Let’s start

Check out Port's pre-populated demo and see what it's all about.

(no email required)

Let’s start

Contact sales for a technical product walkthrough

Let’s start

Open a free Port account. No credit card required

Let’s start

Watch Port live coding videos - setting up an internal developer portal & platform

Let’s start

Let us walk you through the platform and catalog the assets of your choice.

I’m ready, let’s start