Internal Developer Portals and Kubernetes: What You Need to Know

Introduction

The rise of cloud native and kubernetes created an increase in the complexity that developers need to be aware of in their ordinary flow of work. This presents more tasks and cognitive load, affecting productivity, and also making it difficult for developers to understand where their apps are running, what is deployed where and whether any changes are required for services to run as they should. DevOps lives aren’t necessarily easier either.

The rise of platform engineering seeks to address this. With it comes a movement to create re-usable elements that developers can use to self-serve, with golden paths and guardrails. Platform engineering is especially compelling in the case of Kubernetes, given its complexity. By abstracting Kubernetes complexity away for developers and offering them developer self-service to work independently, internal developer platforms can solve many issues. You can even argue that Kubernetes is one of the core drivers behind the platform engineering movement.

Internal developer portals can help solve Kubernetes for developers. They create a simple one stop shop where developers can go to gain visibility and understanding of what is actually going on in K8s. Developers can see who owns a service, where it’s deployed and also understand quality and standards with regards to K8s. On top of this, developer portals also provide developers with a layer of self-service actions so they can work independently and stop using email or slack messages to request DevOps support. Internal developer portals for Kubernetes need to be “done right” meaning that they should support a flexible data model that represents the organization’s needs, covering:

• Visibility for all Kubernetes objects
• Multi-cluster support
• CRD support; and
• The flexibility to determine the right views of Kubernetes objects for the right users

What are the core elements of an internal developer portal?

‍

‍

The software catalog 

Stores all of the information and visualizes all the software catalog elements, from microservices through workflows, cloud environments, providers, clusters and more, to create a bird’s eye visibility layer that shows everything that's going on, in-context. The software catalog also lets you drill down into that one piece of information you really need, such as information about the current performance of your cluster, or certain CI/CD data, microservice packages, versions, etc.

‍

The self-service layer

Developers need to use engineering assets, and many of those actions, such as setting up an ephemeral environment, scaffolding a microservice etc, are repetitive. In many cases developers don’t have the required permissions or the deep knowledge required to execute these actions themselves. So they open a ticket or email DevOps. DevOps may already have a script to perform that action but they don’t necessarily want to expose it for fear that it’s too complex or that something may break. The idea of the developer self-service layer is to expose those potential reusable self-service actions that were already created by the DevOps or platform engineering teams to developers. Doing this through the internal developer portal is safe, and paves a golden path for developers, providing them with independence when they interact with the infrastructure.

‍

Workflow automation

Workflow automation in the internal developer portal lets machines, such as CI/CD workflows consume or access all the valuable (and up-to-date) information in the software catalog (as well as scorecards, which we will touch on later). They can either query the Port API or subscribe to events coming from the Port software catalog. This information is then used by machines to make decisions. An example is the software catalog showing that a certain service is currently locked for deployments, maybe as a result of the holidays or a code freeze. If someone triggers a new deployment for that service, the workflow automation layer will indicate that and stop the deployment. Another example is implementing TTL and triggering delete for an ephemeral environment once the TTL is over.

‍

Scorecards

Internal developer portal scorecards let you set standards for quality engineering in your services, infrastructure and environment. For instance, you can set a standard that a certain microservice is only mature if it has an X number of replicas or higher. Another example is that you want to enforce certain kubernetes cluster versions. Defining scorecards is a great way to set quality engineering standards. 

Don’t think of scorecards as a Kyverno or conveyor.io alternative. In fact, internal developer portals integrate with these tools and use, for example, Kyverno policies as part of scorecards in the software catalog. This can fit nicely with a Kubernetes initiative, for instance, seeing that kubernetes cluster standards are not up to par and reflect where you are standing with regards to the policies set in Kyverno and similar tools. 

‍

Role-based access control

That defines who can see which information and is authorized to do what in the context of developer self-service actions. One of the main goals of a good internal developer portal is to reduce cognitive load, so we can’t just dump all the information on to the developer. This is especially true for Kubernetes where some of the raw data may be downright confusing for a developer not familiar with the technology. Using RBAC, we show developers what is relevant to them and also control how they use developer self-service. For instance, you may not want to allow all developers to roll-back services, or maybe you want to set a manual approval step for certain actions. 

‍

{{cta_2}}

‍

How internal developer portals interplay with Kubernetes

Before we begin discussing how to abstract Kubernetes in the internal developer portal, let’s examine why developers can benefit from these abstractions and visualizations in the first place. 

Let’s look at some common representations of kubernetes information. Here is an application that’s deployed using ArgoCD. 

‍

‍

If you ask a developer that is not super-versed in kubernetes and ArgoCD whether this view makes sense, they will probably say it’s too much information, too much of a drill down, which probably means cognitive load. 

The case is similar for tools such as Lens, K9S or Rancher. While they may seem simpler than using Kubectl, they still are too complex and difficult for developers encountering Kubernetes for the first time. 

‍

The trick is to expose Kubernetes information that is mapped to the different objects represented in the developer portal.  For example, for a given microservice you can say that you care about the deployment name, the image name and the runtime of the environment, and not access the rest of the information. Or perhaps, you’re looking for data about a running service and need to see how much CPU or memory are used by it, or just see the general service health. This information exists in the Kubernetes API and can be brought to developers to make it easier for them to understand what is going on.

‍

Creating a data model for Kubernetes in your internal developer portal 

‍

‍

It’s important to note that you can (and should!) create a data model of your systems in your internal developer portal. These data models are usually specific to the organization, since every organization cares about different things and different developers have different kubernetes knowledge levels. So your data model doesn’t only need to meet your organization’s needs but also your developer personas. 

In Port, a Blueprint, or custom entity definition, is a data model that allows definition of the metadata associated with software catalog entities. Blueprints are the main building block of Port. Populated blueprints are catalog entities. Blueprints support the representation of any asset in Port, such as microservice, environment, package, cluster, databases etc. Blueprints are not opinionated by definition. It is you, who defines them, that gets to be opinionated about what should best represent your needs (you can also use templates to begin work, there’s even a Kubernetes internal developer portal template you can use).

Using Port blueprints, we're going to set blueprints for clusters, nodes,  namespaces and workloads. Here are what each of those blueprints look like:

‍

‍

‍

Workloads is a generic name that we give for deployments, stateful sets, daemon sets and other workloads in our cluster. When we further explore these data model components we can see that a cluster is a parent entity that can be used to consolidate information, but when we're looking at a namespace we might care about when it was created and its labels. When we look at a node we might care about its readiness, labels and its Kubelet version as well as the total CPU. Again, the idea with a data model as it is represented in Port’s blueprints is that it should fit your organization. A good way to go about this is for the platform engineer to canvas developers and DevOps and ask what information would make their lives easier. 

‍

Ingesting Kubernetes data into the internal developer portal

The best way in Port is to use ETL process and JQ to extract the correct information from kubernetes, using Port’s Kubernetes exporter. Here is an example of a config file that we use in the Kubernetes exporter, which is open source (see link). This is meant to be installed on your kubernetes cluster and with a simple config you can define exactly what information you need (and care about) from the kubernetes cluster. Port’s Kubernetes exporter provides real-time event processing, which results in an accurate real-time representation of your K8s cluster inside Port.

‍

‍

In this example we take replicaset objects from the kubernetes API and we can query by specific fields, so, for example, if I only care about replicasets that are not from the kube default namespace, we can filter all of those out. Then we can define how to map this information. Using the power of JQ you can actually transform those different fields and customize data the way you want to. 

For those of you that are not familiar with it, JQ is a Json processing syntax and it lets you take an existing JSON which outputs from the kubernetes API and transform it, for instance concatenating strings. It also works for numbers and arrays and is a mature syntax that can do pretty much whatever you need. Together with Port’s Kubernetes Exporter it makes it very easy to take the information from the kubernetes API and send it into your developer portal.

‍

Here’s a simple example showing string concatenation:

We start with the following JSON object:

{
    "name": "k8s-webinar",
    "presenter": "Mor Paz"
}

By applying the following JQ filter rule:

"Webinar " + .name + " is presented by " + .presenter

We get the following result: "Webinar k8s-webinar is presented by Mor Paz"

You can play around with the above example by going here.

‍

Here’s a second example that shows how we create a unique identifier for our deployment by combining the deployment name and its namespace, this is a pretty common operation - used to correctly distinguish between the different assets in our K8s when examining them in the software catalog. JQ makes it simple and straightforward:

We start with the resource definition of a K8s deployment (some of the output is omitted for readability):

{
    "apiVersion": "apps/v1",
    "kind": "Deployment",
    "metadata": {
        "annotations": {
            "deployment.kubernetes.io/revision": "1",
            "kubectl.kubernetes.io/last-applied-configuration":
                "{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{},\"name\":\"payment-service\",\"namespace\":\"production\"},\"spec\":{\"replicas\":1,\"revisionHistoryLimit\":3,\"selector\":{\"matchLabels\":{\"app\":\"payment-service\"}},\"template\":{\"metadata\":{\"labels\":{\"app\":\"payment-service\"}},\"spec\":{\"containers\":[{\"image\":\"gcr.io/heptio-images/ks-guestbook-demo:0.2\",\"name\":\"payment-service\",\"ports\":[{\"containerPort\":80}]}]}}}}\n"
         },
         "creationTimestamp": "2023-05-03T07:58:09Z",
         "generation": 1,
         "name": "payment-service",
         "namespace": "production",
         "resourceVersion": "3392",
         "uid": "9f9678f2-ea4e-4681-bacf-833779ae4dd6"
         },
…}

By applying the following JQ filter rule:

.metadata.name + "-Deployment-" + .metadata.namespace

We get the following result: "payment-service-Deployment-production", which can be used as a unique identifier for our deployment of the service in the production environment.

You can play around with the above example by going here

‍

The Kubernetes software catalog

Once we define the Kubernetes data model for our software catalog and ingest the data, we can see what the actual kubernetes software catalog looks like. You can check an example here, in Port’s live demo. Port is special in that its software catalog contains all Kubernetes elements, depending on how you defined them in the blueprints, and is not limited just to services on Kubernetes, as some other solutions are. 

‍

‍

Port’s approach allows you to reflect multiple clusters and map all of them together, even if you have tens or hundreds of kubernetes clusters running thousands of workflows. You can always map those to that one single developer portal and expose all that information in one place, to the correct teams, helping them deal with cognitive overload. You then use RBAC to reflect what each team needs to see, rather than create separate platforms for separate teams.

‍

Next step: Scorecards

‍

‍

The next step is to implement scorecards which let you drive quality initiatives, such as production readiness. When you look at the sample scorecards above you can see that we set tiers. A bronze tier is if the equivalent version is X and it's silver if it's Y (Y>X). Or we can make sure that CPU and memory limits are set or ensure that nothing is being deployed into the default namespace. Scorecards set standards - this is how we expect our infrastructure to look like, and given that we have all the information inside the developer portal we can see if we’re up to those standards and drive a culture of quality.

‍

Kubernetes self-service actions

Now that we have all the information in the portal, let’s see how we drive developer self-service actions. 

Using self-service, developers can provision, terminate and perform day-2 operations on any asset exposed in port’s software catalog. Port connects to existing runners and can trigger GitHub actions, saving labor in setting up. Port self-service actions are loosely coupled with underlying platform engineering assets:

• Triggering webhooks based on a customer provided URL
• Kafka self-service actions
• Webhooks with Port execution agents
• Run GitHub workflow self-service 
• Run Azure Pipelines

‍

‍

Port self-service actions also support asynchronous actions, TTL to allow temporary environments and permissions as well as manual approvals when needed. 

Let’s think of some examples, such as restarting a cluster. This, of course, may be a bit extreme, you may want to set manual approval there (which is supported in Port), but we can take this idea and create a self-service action for modifying the replica count, rolling back a service or changing the deployed image tag. The idea is to give your developers the ability to trigger those actions themselves instead of issuing tickets to DevOps. DevOps don’t need to be bogged down with mundane requests and the developer doesn't need to wait for someone to simply click a button for them.

‍

{{cta_7}}