The benefits and pitfalls of platform engineering

What is platform engineering?

Platform engineering seeks to connect the tools and capabilities needed to build software with the software developers who focus on creating it. In the world of platform engineering, developers are the ‘customers’, and the goal is to empower them with tooling and resources to promote productivity and a culture of quality.

Platform engineering builds upon the foundation set by DevOps; its chief aims are to improve various elements of the software development lifecycle such as security, costs, and standards compliance through better developer experience and improved developer productivity. 

The practice seeks to reduce the complexity of modern software development by providing an internal developer platform which incorporates standardized workflows and processes (with built-in security and compliance where possible), reusable tools, self-service capabilities, automated workflows and infrastructure operations.  The platform enables developers to build, test and deploy applications.

Building an internal developer platform

To get the most out of platform engineering, platform engineers build a platform using a product management approach (see Platform as a Product). This ensures that the platform is tailored to the specific needs of the developers. Gartner believes that around 80 percent of engineering organizations will have a team dedicated to platform engineering by 2026

The cornerstone of platform engineering is an internal developer portal. 75% of organizations with platform engineering teams are expected to implement portals by 2026, according to Gartner. 

The portal acts as the user interface for the platform, centralizing access to all necessary tools, resources and documentation, providing an improved user experience and ensuring that developers can work productively without needing to understand the intricacies of each tool within the platform (eg. Kubernetes, ArgoCD, etc). The internal developer portal is effectively the developer’s hub, a workspace for everything they need; it brings together a software catalog (a centralized metadata store with information on all services, APIs and documentation), self-service actions (the ability to create a new service or spin up a new environment, with golden paths with guardrails built-in), scorecards (the ability to grade and track standards compliance), automated workflows and dashboards. 

Further reading: The top 10 ways to use an internal developer portal: what we’ve learned from our customers

Platform engineering vs DevOps

DevOps has done a lot for software development. Over the last decade, its practices have aimed to add agility; promoting fast, high-quality production and self-sufficiency within engineering teams. This involved creating scalable, automated development lifecycles with microservices, Kubernetes, and cloud resources, as well as expanding developers' responsibilities to include feature delivery, incident management and interacting with the cloud. At the same time, developers have had to ensure they don’t step on the toes of the security, quality and compliance standards dictated by the organization. 

This has led to the DevOps paradox, where organizations that embraced DevOps have been left with increasing complexity and far too much responsibility on developers, meaning that maintaining quality and efficiency is too challenging. 

As a result, chaos has ensued;

• Quality, security and compliance issues started to degrade - it was too hard to keep track of so many distributed elements
• Onboarding became inefficient
• Managing incidents became incredibly difficult with so many factors that could cause an incident, the different tools and moving parts involved, and the knowledge required

Cutting-edge companies like Spotify, Netflix and Lyft addressed this chaos internally by developing internal developer portals, sparking the current platform engineering revolution.

The platform engineering movement helps organizations to overcome many of the challenges with DevOps, namely cross-company duplication and an increasing cognitive load for developers.

Benefits of platform engineering

The overarching benefits of platform engineering are:

• Faster response time and problem resolution
• Increased scalability
• Acceleration to market
• Reduced operating costs
• Improved developer experience
• Enhanced developer productivity
• Better collaboration between different teams (developers, DevOps, SREs, etc)
• Better management of tasks, standards compliance and resources

These benefits are brought about via the following:

• Developer autonomy via self-service 

Enabling developers to create a new service, rollback a service or gain access to a database can provide them with the autonomy they need to get their work done. Self-service actions provide them with a way to do this, which improves the efficiency of the software development lifecycle, reduces friction points and improves the developer (and DevOps) experience due to the reduction of tickets submitted, questions posed, and time wasted. For instance, developers can test new features or resolve issues more promptly, meaning they can ship faster to production, and this can impact the overall organization as updates, fixes and new features can be delivered to end users promptly. 

• Enhanced visibility 

Developers require visibility into the services, APIs and documentation that they need, swiftly. This can often be a difficulty and can cause major issues for developer onboarding. As a result, a map of all these items, and information about them including ownership and dependencies, can provide developers with the clarity they need to get on with their work. A service catalog, one of the core pillars of an internal developer portal, can provide the discoverability and context that users need, enabling them to find the information they need, reduce the manual effort to update data, grow re-use of services or APIs, and adopt InnerSource practices. 

• Reduction in context-switching

The first step of platform engineering is to reduce tool sprawl by making clear what tools should make up the internal developer platform. The next step is to reduce context-switching between these tools which can result in significant time loss. By using a portal as a hub, users can find all of the information they need in one place without switching tabs constantly. This improves the developer experience and enhances productivity.

• Better managed resources

In terms of scale, platform engineering can help an organization to better understand their services, workloads, traffic and APIs and manage them. This can come through auto-scaling rules, load balancing traffic, using TTL in self-service actions, and an API catalog. The added visibility that platform engineering provides ensures users understand how many ephemeral environments they have in place and remove those that are not in use or barely used - while enabling TTL as part of a self-service to spin up a new dev environment can avoid this happening in the first place. This subsequently helps engineering teams to reduce costs. Another way of reducing costs is by using a portal to maximize investment in cloud cost tracking tools and optimize cloud costs without sacrificing service functionality. Meanwhile, API management can be improved through a catalog which makes APIs discoverable and shows them in context, reducing duplication and making it easier to create APIs. 

• Improved ability to set and comply with standards

While a platform can provide developers with a framework for developers to use that enforces policies and standards around software development, a portal can enable developers (and engineering leads) to define standards, grade services (or APIs, or documentation, or anything else in the service catalog), and track progress of standards compliance through scorecards and initiatives.

Pitfalls of platform engineering

• Focus on tech, not user experience‍

Platform engineering isn’t as simple as an implementation of a tool or a change in job titles; it is a new way of doing things. But to actually yield the benefits, you have to listen to your customers - in this case developers - about their pain points, and then decide what tools, capabilities and use cases you want your platform and portal to enable you to achieve. The best way of getting to know this? Creating a developer experience survey. This means focusing less on the architecture and thinking more about the product experience.

• Focusing too heavily on developer productivity measurement frameworks. 

Our State of Internal Developer Portals report found that in actual fact, surveys and custom reports trump developer productivity measurement frameworks such as SPACE or DORA. While these are important frameworks, they aren’t the only engineering metrics that you should be looking at. There should be a focus on developer experience metrics, software engineering intelligence metrics and standards-based metrics, too.

• Not being able to abstract away the complexity of the platform itself‍

Devs often don’t have the knowledge to interact with certain resources directly (the cloud, Kubernetes, etc) - they need guardrails and guidance to set them up for success - this is where a portal comes in to provide a single user interface, and self-service actions which abstract away the parts of these technologies or resources that they don’t understand. 

• Building a platform on top of older methods‍

Often, as more platform tools are added and as more microservices are introduced - things become difficult to track - and this leads to an increase in deploy failures, longer feature development/discovery times, and general fatigue and developer dissatisfaction because of the unpredictably of bouncing around different platform tools to perform their work. There needs to be a way to track what’s happening throughout the SDLC.

• Adoption - how (and is it possible) to force developers to change the way they work 

In any work environment, there is likely to be pushback for a new way of working from (at least) some of your team. To get developers on board you need to engage with them throughout every part of this, through surveys, meetings, product feedback initiatives and more; there should be an open dialogue of why you want to implement platform engineering in the organization and how it will benefit your developers.

• Self-service isn’t just about creating services from scratch‍

Creating a new service is an important self-service action, but it’s not enough to change the way engineering teams work. There are many more self-service actions developers need such as day-2 operations, delete actions, provisioning ephemeral environments, and more. To be able to follow a golden path, developers need to be able to use self-service actions that meet standards, then be able to measure existing resources based on these standards (using scorecards), and finally bring noncompliant resources back to the golden path - this is where day-2 self-service actions come in, enabling users to rollback deployment, trigger an incident, nudge reviewers about a PR, and more.

Want to see what platform engineering could look like for your organization? Have a play with our live demo.