How Clear Channel Outdoor used Port to reduce developer platform complexity and enable standardization

Use of Internal Developer Portal has led to an increase in efficiency, quality of deployments and improved alert visibility‍

Clear Channel Outdoor (CCO) is one of the US’s largest Out of Home (OOH) advertising companies, that offers 70,000+ outdoor and airports displays reaching 130 million Americans weekly in over 65 markets. Over the last few years, CCO has been bolstering its digital technology and enhancing its Product Development technology stack, as part of the wider technology modernization efforts.

The Product Development group is broken up into teams and is comprising managers, developers, and engineers that work together to help facilitate the technology modernization. One of those teams is called "Engineering Enablement". This team focuses on DevOps, SRE, Platform, and SecOps related work surrounding new Cloud Native frameworks and technologies.

The Product Development group’s modernization efforts involved using the "Strangler Fig" strategy for a gradual transition away from a monolithic application environment to an updated microservice and serverless architecture. This decomposition strategy created an opportunity for the Engineering Enablement team to help redesign how new microservices applications and functions were hosted and deployed. In this effort to leverage cloud native technologies, the team implemented a large array of automation and internal tooling that helped improve operational effectiveness, and flexibility - ultimately allowing for technology agnosticism so that the team could select the best framework/tooling available.

As more platform tools were added, updated, replaced, and as more microservices were introduced; it became increasingly harder and harder to keep track of things from a platform and application perspective. The net result of this ever increasing complexity was an increase in deploy failures, longer feature development/discovery times, and general fatigue and developer dissatisfaction because of the unpredictably bouncing around different platform tools to perform their work. 

According to Serge Yagolnikov, a software engineer that is part of the Engineering Enablement team, it became obvious that there needed to be a consolidation mechanism that unifies and improves the developer experience and sustainably removes the tool sprawl and confusion folks were experiencing.

“We added stop gap tools, automations, and attempted to abstract some of the complexity away, but during the monolith to microservice decomposition work, we noticed that the further we shifted away from the monolithic applications, the more sprawl and complexity was introduced. It was at this point, it was clear we needed a turnkey solution to help better and more sustainably manage things," he said. 

The next order of business was to find and evaluate  a tried and true turnkey solution, which led the Engineering Enablement team to look at a number of dynamic Wiki solutions and Internal Developer Portal tools.

“We tested out and piloted a number of tools, but all of them had noticeable drawbacks from a functionality perspective and we were unable to find a No-code, turnkey, and noise free solution. When we discovered the Port product at the end of 2022, we jumped in feet first as it addressed all our needs  and more” Serge said.

One of the defining moments that confirmed the Engineering Enablement team’s resolve in selecting Port as their Internal Developer Portal, was the realization of the amount of time and effort they saved in maintenance and updating of the portal when contrasted with other solutions. Most importantly, Port’s easy to understand and yet extensible user interface, provided the precise noise free experience that helped improve overall tool adoption and satisfaction amongst all their team members.

How Port’s Internal Developer Portal has made an impact at CCO

Improved efficiency and higher quality deployments

CCO’s engineers and developers had previously used different methods and strategies for deployments as its continuous deployment and release management was somewhat scattered.

“As part of our lean strategy we’ve consolidated and collapsed everything into one central portal, eliminating waste, increasing efficiency, and producing higher quality deployments,” explained Serge.

They did this by unifying and aggregating its CI/CD into two Blueprints. Blueprints are the foundational building blocks of the internal developer portal, holding the schema of the entities that one would want to represent in a centralized repository (software catalog). They are completely customizable and can support any number of properties the user chooses. In this case, the Blueprints are used to define self-service actions for developers, so that developers can act without assistance.

“This means any microservice, bff, proxy, UI deployments, cloud function, and serverless deployment are carried out in the exact same way. Port allowed us to effectively abstract deployment complexity away from the developer. As such today, we no longer have the error prone context switching in how we deploy software,” Serge said.

Developers are also able to use self-service actions through Port for day-one and day-2 jobs such as onboarding microservice apps, UI’s, and cloud functions.

“Port has made it easy to be able to call and trigger Github actions for self-service - all we have to do is click on a button and Port takes care of the rest. Manual work and one-off automation is no longer the norm, things are now self–documented and automated through Port. This has significantly reduced human error,” Serge said.

Enhanced visibility into alerts and incidents

The Engineering Enablement team has 70+ Github Action Workflow automations that perform a variety of tasks - some of them are event-driven, some are CRON-driven, some are developer driven - all of which ensure that the hosting platform functions properly. As the automation runs in the background, the Engineering Enablement team needed a consolidated place to see the order of events that took place and to see if any of the automation tasks have failed or have encountered a problem.

The operational visibility needed to extend beyond the ability of simply managing Github Action runs and automation. More specifically, the broader CCO Product Development team is responsible for taking action on system state events that occur across a variety of Cloud Native tools. Some of these system events include: security, app outages, and general application performance events; each of which have a varying degree of urgency and priority and target specific teams and specific members to take action on.

Today, all these variety of events generated by Cloud Native tools and system events are now sent to Port via API calls, and appropriately funneled into a specific Port Blueprint depending on the type of event. Each event is represented as a Port Entity in the respective Blueprint. For example, the team has an automation Blueprint that aggregates any/all GitHub Action failures with a clear delineation of the urgency for that event. There is also a Blueprint that contains system and application events that is helpful for developers to understand if there are any application error events that have been generated requiring their attention.

The aggregation of events in a single pane of glass organized into logically separated buckets (Blueprints) was a massive win for CCO, however, CCO team members still needed a way to get personalized notifications of events that require action to take without interrupting or exhausting the entire staff for every event.

Engineering Enablement designed a serverless automation script that checks each Port Blueprint for any unassigned event (such as application uptime failure, deployment failure, automation failure, etc). If an event is noticed, that serverless function will locate the on-call engineer or engineer assigned to the specific domain relating to that event and the alert will send a notification directly to them via Slack and potentially via SMS.

By leveraging Port for system information and event system management, coupled with incident response notification automation using serverless scripting, they have an end to end solution that all system and application events are assigned and guaranteed to be resolved with as little cross-noise between teams as possible.

Reduction in Questions

Serge explained that the Engineering Enablement team designed an AI assistant by embedding an off-the-shelf Google product within Port. The product allows the portal’s users to search CCO’s database of information, which includes the necessary knowledge and how-to’s that the team would require to effectively interact with the platform and Port.

“It means if they have a question of ‘how do I deploy something? What do I do if a particular event occurs? Or How do I promote an application to production, etc…’ The AI assistant gives a succinct answer along with more verbose documentation to these questions,” he said.

This reduces the amount of time that Engineering Enablement spends answering reoccurring support related questions. This AI feature also saves the team’s  time as their documentation and broadcasting of documentation changes is simplified - which all translates into an increase of velocity for all team members.

Superior Knowledge Sharing

The Product Engineering team is also leveraging Port for a repository of internal knowledge.

“Each team member acquires information and knowledge, this includes: architectural diagrams, docs, and general nuanced knowledge, which even if shared during a meeting is often lost in the context of the broader team's awareness overtime. Since there are efforts to document and aggregate information inside Port, we can now build a clearer, more detailed, and persistent picture of information that will not get lost overtime. We can do this by leveraging OpenAPI spec, relationship and dependency mapping between microservices, applications and other Port driven blueprints, and by sharing architectural dynamic diagrams using Google Docs inside of Port,” Serge said.

‍Future Plans

The recent modernization efforts with the help of Port has allowed CCO’s product development team to reach their deployment velocity needs as they perform several deployments on a daily basis, which was a stark contrast from the two deployments per month in the previous platform stack.

“Although, it is great that we can now perform 700+ production releases per year, the value gained in moving to a microservice architecture and redesigning of the deployment automation [leveraging Port] is we no longer are bottlenecked or waiting on deployment of applications that used to require a static synchronous deployment process and a rigid schedule with manual tasks requiring multiple people." Serge said.

In addition, the Internal Developer Portal is acting as a single pane of glass for CCO.

“Port provides us with complete transparency - it’s a perfect system of reference and visually represents what’s going on our platform in a very fast, easily digestible manner,” Yagolnikov explained.

The Engineering Enablement team has plans to further enhance and make use of the vast available features offered by Port - in particular, being able to create quick summaries of Port Entities and events using Widget and Pie charts.

“There’s significant value in aggregating operational information into an Ops dashboard and leveraging Widgets to quickly see if there is any action that I need to take. This includes system events, security issues, automation breakage and so on,” Serge explains.