Using Pulumi With an Internal Developer Portal

Introduction

Infrastructure as code makes life easier both for devops and developers. It gives them a way to create infrastructure that is standardized and reproducible. It shortens time spent creating a new environment and provisioning resources, since it is easy to add new variables or edit files, letting you prototype new infrastructure in a faster, easier and safer way. Combining IaC with GitOps makes it very easy to monitor and manage infrastructure as well as the way it changes and evolves over the environment’s lifetime. 

However, infrastructure as code can be daunting for developers, as it requires them to learn a new framework, syntax or both. Developers also need to understand the underlying infrastructure which is usually something that developers are less versed in.

Since infrastructure as code is directly connected to the environment’s resources, performing the wrong change could easily introduce security vulnerabilities or even cause production downtime. Small changes can have catastrophic effects, making developers less comfortable using IaC alone.

‍

Closing the loop with internal developer portals

Internal developer portals can close the IaC and developer loop. 

• The software catalog that’s in the internal developer portal can show developers concise and abstracted data about the environments they work in. Usually that information already exists in IaC definitions, and the software catalog can show developers a concise form, in the context of their work, to make life easier for them. 
• Developers can also perform self-service actions in the internal developer portal, and quickly and safely provision new infrastructure, by using self-service actions that are golden paths to use actions that have been made reusable by platform engineers. 

• This functionality saves developers the need to understand the underlying IaC mechanism, and simply consume it with a product-like experience that gives them more confidence, unleashing productivity.

‍

Let’s look at a live example of how to use Pulumi and internal developer portals, exemplified by Port. 

1. If you don’t already have a Pulumi account already, can sign up here for free

2. Set up the data model in Port using Pulumi‍

3. Deploy a developer environment and ingest its data using Pulumi 

4. Deploy a new developer environment directly from Port using self-service actions

5. Finally, tie it all together

‍

{{cta_1}}

‍

Setting up the data model

Let’s begin by building a data model that will be the skeleton of Port’s software catalog. In Port, data models are defined through blueprints and their relations. Port is un-opinionated by definition, so you can create the data model that’s best for you.

A Blueprint, or custom entity definition, is a data model that allows definition of the metadata associated with software catalog entities. Blueprints are the main building block of Port. Once data is ingested, populated blueprints become catalog entities. Blueprints support the representation of any asset in Port, such as microservice, environment, package, cluster, databases etc. 

Let’s use a Pulumi file to set up the initial blueprints in our software catalog. Pulumi and Port have a direct integration. This makes it possible to create a class for the Port blueprint resource, which inherits Pulumi’s component resource type, making sure any new blueprint we define in Port is following best practices and is properly documented as part of our Pulumi stack.

After setting up the Port blueprint class, it's only a matter of adding a few lines of code  to create our desired blueprints.

In Port, four blueprints were created:

• Developer environment
• Lambda
• S3 bucket; and 
• SQS Queue

‍

‍

These four blueprints encompass a developer environment which consists of a Lambda function, an S3 bucket and an SQS Queue. Each of these blueprints can contain any set of properties.

The Lambda blueprint, for instance, contains the memory size and the Lambda URL. For the S3 bucket we're going to take the bucket ACL permissions and the bucket URL. The SQS Queue blueprints will contain the queue URL. Our developer environment blueprint will include references to all of the resources that it is using.

Blueprints can actually be much more than this minimal set, and contain any number of properties of any type. 

‍

Ingesting data into Port

A software catalog is made of entities, each of which is defined by the relevant blueprint. Entities are the actual instances of blueprints and they are what makes the software catalog.

In this case, we’ll see our demo Lambda function entity, the S3 entity and the SQS Queue entity. On top of that we have the developer environment entity, which isn’t tied to any particular part of our infrastructure but is related to the actual components that we provisioned using Pulumi.

This developer environment entity ties together the different components that make up the actual developer environment in our infrastructure, and makes it easy to understand which components are part of the environment at a glance, through the portal.

Using the related entities view in Port’s software catalog we can even put this developer environment on a graph and better understand how the developer environment fits in the complete architecture and what it connects to.

‍

‍

Developer self-service actions

To understand how to use self-service actions within a developer portal, let’s examine them from the point of view of a developer. Up until now we had set up the internal developer portal as platform engineers, and now we’ll use the perspective of the portal’s customers: the developers. 

‍

Our developer wants to deploy a new developer environment using the self-service hub in the internal developer portal.Once the user executes the self-service action, Port generates a standardized payload with the user inputs and sends them to the selected backend. Self-service actions are defined in the blueprints created by the user. In this case, we defined two actions in the developer environment blueprint. One is for deploying a stack in an environment and the other one for tearing down that stack.

‍

These actions were defined as part of the initial code that we ran to generate the blueprints. This is another example where we can make use of Port’s integration with Pulumi to configure standardized resources and their respective representation in Port. In this case, we added an invocation method that takes advantage of port's GitHub application. Once I click execute for the action, Port sends that payload to GitHub workflows and triggers it, this means we have a direct integration between our developer portal and our CI/CD pipeline. 

‍

After triggering the action through Port, developers can check a job link to the GitHub workflow that was triggered, as well as view, within the portal, the logs pertaining to the self-service request. This keeps developers in the loop, and let’s them track what is going on as their new developer environment is set up. 

‍

Once the self-service action is complete we can see it in the software catalog, under developer environments.  

‍

‍

When the developer is done using this environment, they need to tear the environment down. Without developer portals, this would possibly require sending a ticket to devops (and in most realistic cases, the developer\DevOps would have forgotten about the environment and it would have just stayed up indefinitely, taking up resources and increasing cloud spend).

‍

‍

In this case, the developer can tear down the environment through the developer portal.  Since the action tears down an existing environment, there is no need for specific user inputs. The teardown action will perform all of the actions we went through with the deployment, but in reverse, the delete action will use Port’s GitHub app to interact with GitHub and trigger a GitHub workflow and run a destroy process.

‍

Keeping the software catalog up to date

The code for the Port and Pulumi integration contains classes and definitions for different sources, for example a Lambda function and an S3 bucket. One of the advantages of the integration is that by creating those classes for the objects and inheriting from Pulumi’s component resource class we can actually tie together the actual resource in our Cloud environment and the respective software catalog entity automatically.  

‍

To see the full live coding session this blog is based on, go here: