- Fleet and Elastic Agent Guide: other versions:
- Fleet and Elastic Agent overview
- Beats and Elastic Agent capabilities
- Quick starts
- Migrate from Beats to Elastic Agent
- Deployment models
- Install Elastic Agents
- Install Fleet-managed Elastic Agents
- Install standalone Elastic Agents
- Install Elastic Agents in a containerized environment
- Run Elastic Agent in a container
- Run Elastic Agent on Kubernetes managed by Fleet
- Install Elastic Agent on Kubernetes using Helm
- Example: Install standalone Elastic Agent on Kubernetes using Helm
- Example: Install Fleet-managed Elastic Agent on Kubernetes using Helm
- Advanced Elastic Agent configuration managed by Fleet
- Configuring Kubernetes metadata enrichment on Elastic Agent
- Run Elastic Agent on GKE managed by Fleet
- Run Elastic Agent on Amazon EKS managed by Fleet
- Run Elastic Agent on Azure AKS managed by Fleet
- Run Elastic Agent Standalone on Kubernetes
- Scaling Elastic Agent on Kubernetes
- Using a custom ingest pipeline with the Kubernetes Integration
- Environment variables
- Run Elastic Agent as an OTel Collector
- Run Elastic Agent without administrative privileges
- Install Elastic Agent from an MSI package
- Installation layout
- Air-gapped environments
- Using a proxy server with Elastic Agent and Fleet
- Uninstall Elastic Agents from edge hosts
- Start and stop Elastic Agents on edge hosts
- Elastic Agent configuration encryption
- Secure connections
- Manage Elastic Agents in Fleet
- Configure standalone Elastic Agents
- Create a standalone Elastic Agent policy
- Structure of a config file
- Inputs
- Providers
- Outputs
- SSL/TLS
- Logging
- Feature flags
- Agent download
- Config file examples
- Grant standalone Elastic Agents access to Elasticsearch
- Example: Use standalone Elastic Agent with Elastic Cloud Serverless to monitor nginx
- Example: Use standalone Elastic Agent with Elasticsearch Service to monitor nginx
- Debug standalone Elastic Agents
- Kubernetes autodiscovery with Elastic Agent
- Monitoring
- Reference YAML
- Manage integrations
- Package signatures
- Add an integration to an Elastic Agent policy
- View integration policies
- Edit or delete an integration policy
- Install and uninstall integration assets
- View integration assets
- Set integration-level outputs
- Upgrade an integration
- Managed integrations content
- Best practices for integrations assets
- Data streams
- Define processors
- Processor syntax
- add_cloud_metadata
- add_cloudfoundry_metadata
- add_docker_metadata
- add_fields
- add_host_metadata
- add_id
- add_kubernetes_metadata
- add_labels
- add_locale
- add_network_direction
- add_nomad_metadata
- add_observer_metadata
- add_process_metadata
- add_tags
- community_id
- convert
- copy_fields
- decode_base64_field
- decode_cef
- decode_csv_fields
- decode_duration
- decode_json_fields
- decode_xml
- decode_xml_wineventlog
- decompress_gzip_field
- detect_mime_type
- dissect
- dns
- drop_event
- drop_fields
- extract_array
- fingerprint
- include_fields
- move_fields
- parse_aws_vpc_flow_log
- rate_limit
- registered_domain
- rename
- replace
- script
- syslog
- timestamp
- translate_sid
- truncate_fields
- urldecode
- Command reference
- Troubleshoot
- Release notes
Using a custom ingest pipeline with the Kubernetes Integration
editUsing a custom ingest pipeline with the Kubernetes Integration
editThis tutorial explains how to add a custom ingest pipeline to a Kubernetes Integration in order to add specific metadata fields for deployments and cronjobs of pods.
Custom pipelines can be used to add custom data processing, like adding fields, obfuscating sensitive information, and more. Find more information in our tutorial about transforming data with custom ingest pipelines.
Metadata enrichment for Kubernetes
editThe Kubernetes Integration is used to collect logs and metrics from Kubernetes clusters with Elastic Agent. During the collection, the integration enhances the collected information with extra useful information that users can correlate with different Kubernetes assets. This additional information added on top of collected data, such as labels, annotations, ancestor names of Kubernetes assets, and others, are called metadata.
The Kubernetes Provider offers the add_resource_metadata
option to configure the metadata enrichment options.
For Elastic Agent versions >[8.10.4], the default configuration for metadata enrichment is add_resource_metadata.deployment=false
and add_resource_metadata.cronjob=false
. This means that pods that are created from replicasets that belong to specific deployments would not be enriched with kubernetes.deployment.name
. Additionally, pods that are created from jobs that belong to specific cronjobs, would not be enriched with kubernetes.cronjob.name
.
Kubernetes Integration Policy > Collect Kubernetes metrics from Kube-state-metrics > Kubernetes Pod Metrics
Example: Enabling the enrichment through add_resource_metadata
in a Managed Elastic Agent Policy
Note: Enabling deployment and cronjob metadata enrichment leads to an increase of Elastic Agent’s memory consumption. Elastic Agent uses a local cache in order to keep records of the Kubernetes assets from being discovered.
Add deployment and cronjob for Kubernetes pods through ingest pipelines
editAs an alternative to keeping the feature enabled and using more memory resources for Elastic Agent, users can make use of ingest pipelines to add the missing fields of kubernetes.deployment.name
and kubernetes.cronjob.name
.
Following the transforming data with custom ingest pipelines tutorial, navigate to state_pod
datastream under: Kubernetes Integration Policy > Collect Kubernetes metrics from Kube-state-metrics > Kubernetes Pod Metrics.
Create the following custom ingest pipeline with two processors:
Processor for deployment
editProcessor for cronjob
editThe final metrics-kubernetes.state_pod@custom
ingest pipeline:
[ { "gsub": { "field": "kubernetes.replicaset.name", "pattern": "(?:.(?!-))+$", "replacement": "", "target_field": "kubernetes.deployment.name", "ignore_missing": true, "ignore_failure": true } }, { "gsub": { "field": "kubernetes.job.name", "pattern": "(?:.(?!-))+$", "replacement": "", "target_field": "kubernetes.cronjob.name", "ignore_missing": true, "ignore_failure": true } } ]
Note: The ingest pipeline does not check for the actual existence of a deployment and cronjob ancestor, it only adds the specific values.
On this page
ElasticON events are back!
Learn about the Elastic Search AI Platform from the experts at our live events.
Register now