Comcast transforms customer experiences with Elastic Observability

Comcast Corporation is the largest American multinational telecommunications company and the second-largest broadcasting and cable television company in the world by revenue. Its Xfinity brand is the largest pay TV, cable TV, and home internet service provider in the U.S.

Originally founded during the early days of motion pictures, Comcast has always been at the forefront of technology and media. Today, it uses an array of complex networks, hardware, and software to deliver state-of-the-art content via brands including subsidiary NBCUniversal and the streaming application Peacock.

Customers rely on Comcast programming for information and entertainment, and its delivery is mission-critical for Matt Dymek, VP, Software Strategy & Transformation. Dymek leads the Dev Strategy group, part of the Comcast DevOps team responsible for the software and networking that supports 60 million customer relationships, 100 million internet devices, and five million mobile device users.

“Customers expect our systems to always be available with split-second performance. They want our technology to know what they want and to predict what’s next on their watch list. As a result, our environments are growing progressively more complex,” says Dymek.

Besides ensuring always-on availability, Dymek’s team is responsible for innovations that enable Comcast to stand out in a highly competitive industry. Recently it developed a voice activated remote control which enables viewers to switch channels or call up their favorite series without lifting a finger.

For Comcast to maintain its market leading position, ongoing innovations must be delivered quickly. "Building new features themselves isn't the only hard part," says Dymek. "It's nearly impossible to know how everything will come together in production, let alone to understand how things will perform under a production-scale load."

This is where Elastic Observability plays a crucial role. Elastic ingests about 400 terabytes of telemetry data per day, enabling Comcast to aggregate, correlate, and inspect data from about 70 tenants across its IT infrastructure.

"With more than 50,000 software builds a day, getting a single view into everything that's happening with our services is essential. Elastic helps us to see what's occurring at the application layer all the way down through our hardware. We can see how changes are behaving and how they impact the overall system," says Dymek.

Dymek refers to this process as part of the 'shift left' trend in DevOps today. Put simply, it enables developer teams to increase efficiency in the software development process and improve product quality by detecting issues at an earlier stage.

"With insights from Elastic we can undertake small, targeted deployments of our software and send feedback directly to our engineers so that they can iterate quickly and make the changes needed for the service to work in production."

Data consistency is fundamental to Comcast’s Elastic Observability deployment. It has established a logging and metrics standard built on Elastic Common Schema (ECS) that defines a common set of fields, their data types, allowed values, and usage hierarchy for ingested data. ECS also unifies all modes of analysis available in Elastic, including search, drill-down and pivoting, data visualization, machine learning-based anomaly detection, detection rules, and alerting. “The critical aspect of our observability solution is being able to build a Comcast common schema on top of ECS so that our IT team can really understand what all our other tools are doing,” says Dymek.

Elastic also delivers lower total cost of ownership than Comcast’s previous observability solution. “Elastic offers a more strategic approach than other providers where it felt like we were incurring more and more debt,” says Dymek. “There’s also a stronger sense of partnership. We can always turn to the team at Elastic for quick answers if we have any issues.”

Comcast already uses Kibana dashboards to visualize trends and anomalies, and to share insights between teams. Going forward, Dymek sees the potential of Elastic AIOps which uses domain-specific machine learning and anomaly detection to reduce manual troubleshooting. “Being able to automatically understand the signals generated by our operational data can help reduce the burden on our team so that they can focus on developing better software,” says Dymek. Elastic OpenTelemetry is also being considered for analyzing distributed traces and metrics, and to monitor business KPIs and technical components.

Overall, Elastic supports Comcast’s mission to create incredible technology and content that connects millions of people to the moments and experiences that matter most. Today this includes ultra-high-definition broadcasting, access to DVR recordings from multiple networked devices, and a voice-activated remote that handles 15 billion instructions a year. Elastic will also help Dymek and his team prepare for future innovations such as augmented reality and the metaverse.