Elastic contributes Elastic Common Schema (ECS) to OpenTelemetry (OTel), helping accelerate adoption of OTel-based observability and security

Today, data from multiple sources with schemas makes it difficult to pinpoint software issues and understand the root cause of a problem. A common schema helps normalize data to enable improved analysis, visualization, and correlation of data across observability and security solutions, accelerating root cause analysis. To ensure that our customers and the broader community can benefit from standardization, Elastic is committed to developing a common schema for metrics, logs, traces, and security events based on the Elastic Common Schema (ECS) and OpenTelemetry (OTel).  

Elastic is contributing ECS to OTel and committing to joint development of a common schema. OTel is the second highest velocity project in the Cloud Native Computing Foundation (CNCF), and provides a collection of tools, APIs, and SDKs used to generate, collect, process, and export telemetry data (metrics, logs, and traces) for understanding software performance and behavior. Elastic has long been a proponent of open standards, and this contribution continues and expands Elastic’s commitment to the community. By contributing ECS to OpenTelemetry, we are hoping to create a mature and proven common schema in OTel for metrics, logs, traces, and security events based on the widely deployed Elastic Common Schema. Together with OTel, we will continue to develop and support that common schema going forward.

Value to open source community

ECS, an open source specification, was developed with support from the Elastic user community to define a common set of fields to be used when storing event data in Elasticsearch. ECS helps reduce management and storage costs stemming from data duplication, improving operational efficiency.

Similarly, OTel’s Semantic Conventions (SemConv) also specify common names for various kinds of operations and data. The benefit of using OTel SemConv is in following a common naming scheme that can be standardized across a codebase, libraries, and platforms for OTel users.

The merging of ECS and OTel SemConv will help advance OTel’s adoption and the continued evolution and convergence of observability and security domains. 

A commitment to open standards

The goal of the joint development of a common schema is to define vendor-neutral semantic conventions for the most popular types of systems. Supporting vendor-created or open source components (for example, HTTP access logs, network logs, or system access/authentication logs) will extend OTel correlation to these new data types. Contributors to ECS have already done the heavy lifting in defining a unified set of semantic conventions, which can be adopted in OTel.

The contribution of ECS will lead to several advantages for the ecosystem:

• A more standardized and unified structured format for vendor-generated logs along with open source logs
• User benefits from turnkey log integrations that will be fully recognized by OTel-compatible observability products and services
• Expanded OTel support for infrastructure by providing first-class support for Kubernetes application logs, system logs, and application introspection events
• A standardized observability telemetry definition for application and infrastructure, enabling vendor neutrality for observability

As OTel adopts ECS, authors of OTel Collector Logs Receivers will be able to provide richer data definitions in their messages, improving data quality and helping to establish the OTel Collector as the de facto standard log collector in the industry.

ECS has evolved to cover the needs of users gathering many types of telemetry across multiple uses. The adoption of ECS will significantly extend the utility of OTel to DevSecOps.

Elastic currently supports a growing list of CNCF projects such as Kubernetes (K8S), Prometheus, Fluentd, Fluent Bit, and Istio. Elastic’s Application Performance Monitoring (APM) also natively supports OTel ensuring all APM capabilities are available with either Elastic or OTel agents or a combination of the two. In addition to the ECS contribution and ongoing collaboration with OTel SemConv, Elastic has continued to make contributions to other OTel projects including language SDKs (such as OTel Swift, OTel Go, OTel Ruby and others), and participates in several special interest groups (SIGs) to establish OTel as a standard for Observability and Security.

We are excited about our strengthening relationship with OTel and the opportunity to merge schemas in a way that benefits both the Elastic community and the broader OTel community.

Learn more about Elastic’s OpenTelemetry support or contribute to the project. Additional details about this exciting contribution of ECS to OpenTelemetry can be found in our FAQ.