At Elastic, we recently made a decision to fully embrace OpenTelemetry as the premier data collection framework. As an Observability engineer, I firmly believe that vendor agnosticism is essential for delivering the greatest value to our customers. By committing to OpenTelemetry, we are not only staying current with technological advancements but also driving them forward. This investment positions us at the forefront of the industry, championing a more open and flexible approach to observability.
Elastic donated Elastic Common Schema (ECS) to OpenTelemetry and is actively working to converge it with semantic conventions. In the meantime, we are dedicated to support our users by ensuring they don’t have to navigate different standards. Our goal is to provide a seamless end-to-end experience while using OpenTelemetry with our application and infrastructure monitoring solutions. This commitment allows users to benefit from the best of both worlds without any friction.
In this blog, we explore how to use the OpenTelemetry (OTel) collector to capture core system metrics from various sources such as AWS EC2, Google Compute, Kubernetes clusters, and individual systems running Linux or MacOS.
Powering Infrastructure UIs with Two Ingest Paths
Elastic users who wish to have OpenTelemetry as their data collection mechanism can now monitor the health of the hosts where the OpenTelemetry collector is deployed using the Hosts and Inventory UIs available in Elastic Observability.
Elastic offers two distinct ingest paths to power Infrastructure UIs: the ElasticsearchExporter Ingest Path and the OTLP Exporter Ingest Path.
ElasticsearchExporter Ingest Path:
The hostmetrics receiver in OpenTelemetry collects system-level metrics such as CPU, memory, and disk usage from the host machine in OTel Schema. The ElasticsearchExporter ingest path leverages the Hostmetrics Receiver to generate host metrics in the OTel schema. We've developed the ElasticInfraMetricsProcessor, which utilizes the opentelemetry-lib to convert these metrics into a format that Elastic UIs understand.
For example, the
The processor then forwards these metrics to the Elasticsearch Exporter, now enhanced to support exporting metrics in ECS mode. The exporter sends the metrics to an Elasticsearch endpoint, lighting up the Infrastructure UIs with insightful data.
To utilize this path, you can deploy the collector from the Elastic Collector Distro, available here.
An example collector config for this Ingest Path:
receivers:
hostmetrics:
collection_interval: 10s
scrapers:
cpu:
metrics:
system.cpu.utilization:
enabled: true
system.cpu.logical.count:
enabled: true
memory:
metrics:
system.memory.utilization:
enabled: true
process:
metrics:
process.open_file_descriptors:
enabled: true
process.memory.utilization:
enabled: true
process.disk.operations:
enabled: true
network:
processes:
load:
disk:
filesystem:
processors:
resourcedetection/system:
detectors: ["system", "ec2"]
elasticinframetrics:
exporters:
logging:
verbosity: detailed
elasticsearch/metrics:
endpoints: <elasticsearch_endpoint>
api_key: <api_key>
mapping:
mode: ecs
service:
pipelines:
metrics/host:
receivers: [hostmetrics]
processors: [resourcedetection/system, elasticinframetrics]
exporters: [logging, elasticsearch/ metrics]
The Elastic exporter path is ideal for users who would prefer using the custom Elastic Collector Distro. This path includes the ElasticInfraMetricsProcessor, which sends data to Elasticsearch via Elasticsearch exporter.
OTLP Exporter Ingest Path:
In the OTLP Exporter Ingest path, the hostmetrics receiver collects system-level metrics such as CPU, memory, and disk usage from the host machine in OTel Schema. These metrics are sent to the OTLP Exporter, which forwards them to the APM Server endpoint. The APM Server, using the same opentelemetry-lib, converts these metrics into a format compatible with Elastic UIs. Subsequently, the APM Server pushes the metrics to Elasticsearch, powering the Infrastructure UIs.
An example collector configuration for the APM Ingest Path
receivers:
hostmetrics:
collection_interval: 10s
scrapers:
cpu:
metrics:
system.cpu.utilization:
enabled: true
system.cpu.logical.count:
enabled: true
memory:
metrics:
system.memory.utilization:
enabled: true
process:
metrics:
process.open_file_descriptors:
enabled: true
process.memory.utilization:
enabled: true
process.disk.operations:
enabled: true
network:
processes:
load:
disk:
filesystem:
processors:
resourcedetection/system:
detectors: ["system"]
system:
hostname_sources: ["os"]
exporters:
otlphttp:
endpoint: <mis_endpoint>
tls:
insecure: false
headers:
Authorization: <api_key_>
logging:
verbosity: detailed
service:
pipelines:
metrics/host:
receivers: [hostmetrics]
processors: [resourcedetection/system]
exporters: [logging, otlphttp]
The OTLP Exporter Ingest path can help existing users who are already using Elastic APM and want to see the Infrastructure UIs populated as well. These users can use the default OpenTelemetry Collector.
A glimpse of the Infrastructure UIs
The Infrastructure UIs showcase both Host and Kubernetes level views. Below are some of the glimpses of the UIs
The Hosts Overview UI
The Hosts Inventory UI
The Process-related Details of the Host
The Kubernetes Inventory UI
Pod level Metrics
Our next step is to create Infrastructure UIs powered by native OTel data, with dedicated OTel dashboards that run on this native data.
Conclusion
Elastic's integration with OpenTelemetry simplifies the observability landscape and while we are diligently working to align ECS with OpenTelemetry’s semantic conventions, our immediate priority is to support our users by simplifying their experience. With this added support, we aim to deliver a seamless, end-to-end experience for those using OpenTelemetry with our application and infrastructure monitoring solutions. We are excited to see how our users will leverage these capabilities to gain deeper insights into their systems.