- Metricbeat Reference: other versions:
- Metricbeat overview
- Quick start: installation and configuration
- Set up and run
- Upgrade Metricbeat
- How Metricbeat works
- Configure
- Modules
- General settings
- Project paths
- Config file loading
- Output
- Kerberos
- SSL
- Index lifecycle management (ILM)
- Elasticsearch index template
- Kibana endpoint
- Kibana dashboards
- Processors
- Define processors
- add_cloud_metadata
- add_cloudfoundry_metadata
- add_docker_metadata
- add_fields
- add_host_metadata
- add_id
- add_kubernetes_metadata
- add_labels
- add_locale
- add_network_direction
- add_nomad_metadata
- add_observer_metadata
- add_process_metadata
- add_tags
- append
- community_id
- convert
- copy_fields
- decode_base64_field
- decode_duration
- decode_json_fields
- decode_xml
- decode_xml_wineventlog
- decompress_gzip_field
- detect_mime_type
- dissect
- dns
- drop_event
- drop_fields
- extract_array
- fingerprint
- include_fields
- move_fields
- rate_limit
- registered_domain
- rename
- replace
- script
- syslog
- translate_ldap_attribute
- translate_sid
- truncate_fields
- urldecode
- Autodiscover
- Internal queue
- Logging
- HTTP endpoint
- Regular expression support
- Instrumentation
- Feature flags
- metricbeat.reference.yml
- How to guides
- Modules
- ActiveMQ module
- Aerospike module
- Airflow module
- Apache module
- AWS module
- AWS awshealth metricset
- AWS billing metricset
- AWS cloudwatch metricset
- AWS dynamodb metricset
- AWS ebs metricset
- AWS ec2 metricset
- AWS elb metricset
- AWS kinesis metricset
- AWS lambda metricset
- AWS natgateway metricset
- AWS rds metricset
- AWS s3_daily_storage metricset
- AWS s3_request metricset
- AWS sns metricset
- AWS sqs metricset
- AWS transitgateway metricset
- AWS usage metricset
- AWS vpn metricset
- AWS Fargate module
- Azure module
- Azure app_insights metricset
- Azure app_state metricset
- Azure billing metricset
- Azure compute_vm metricset
- Azure compute_vm_scaleset metricset
- Azure container_instance metricset
- Azure container_registry metricset
- Azure container_service metricset
- Azure database_account metricset
- Azure monitor metricset
- Azure storage metricset
- Beat module
- Ceph module
- Ceph cluster_disk metricset
- Ceph cluster_health metricset
- Ceph cluster_status metricset
- Ceph mgr_cluster_disk metricset
- Ceph mgr_cluster_health metricset
- Ceph mgr_osd_perf metricset
- Ceph mgr_osd_pool_stats metricset
- Ceph mgr_osd_tree metricset
- Ceph mgr_pool_disk metricset
- Ceph monitor_health metricset
- Ceph osd_df metricset
- Ceph osd_tree metricset
- Ceph pool_disk metricset
- Cloudfoundry module
- CockroachDB module
- Consul module
- Containerd module
- Coredns module
- Couchbase module
- CouchDB module
- Docker module
- Dropwizard module
- Elasticsearch module
- Elasticsearch ccr metricset
- Elasticsearch cluster_stats metricset
- Elasticsearch enrich metricset
- Elasticsearch index metricset
- Elasticsearch index_recovery metricset
- Elasticsearch index_summary metricset
- Elasticsearch ingest_pipeline metricset
- Elasticsearch ml_job metricset
- Elasticsearch node metricset
- Elasticsearch node_stats metricset
- Elasticsearch pending_tasks metricset
- Elasticsearch shard metricset
- Enterprise Search module
- Envoyproxy module
- Etcd module
- Google Cloud Platform module
- Google Cloud Platform billing metricset
- Google Cloud Platform carbon metricset
- Google Cloud Platform compute metricset
- Google Cloud Platform dataproc metricset
- Google Cloud Platform firestore metricset
- Google Cloud Platform gke metricset
- Google Cloud Platform loadbalancing metricset
- Google Cloud Platform metrics metricset
- Google Cloud Platform pubsub metricset
- Google Cloud Platform storage metricset
- Golang module
- Graphite module
- HAProxy module
- HTTP module
- IBM MQ module
- IIS module
- Istio module
- Jolokia module
- Kafka module
- Kibana module
- Kubernetes module
- Kubernetes apiserver metricset
- Kubernetes container metricset
- Kubernetes controllermanager metricset
- Kubernetes event metricset
- Kubernetes node metricset
- Kubernetes pod metricset
- Kubernetes proxy metricset
- Kubernetes scheduler metricset
- Kubernetes state_container metricset
- Kubernetes state_cronjob metricset
- Kubernetes state_daemonset metricset
- Kubernetes state_deployment metricset
- Kubernetes state_job metricset
- Kubernetes state_node metricset
- Kubernetes state_persistentvolumeclaim metricset
- Kubernetes state_pod metricset
- Kubernetes state_replicaset metricset
- Kubernetes state_resourcequota metricset
- Kubernetes state_service metricset
- Kubernetes state_statefulset metricset
- Kubernetes state_storageclass metricset
- Kubernetes system metricset
- Kubernetes volume metricset
- KVM module
- Linux module
- Logstash module
- Memcached module
- Cisco Meraki module
- MongoDB module
- MSSQL module
- Munin module
- MySQL module
- NATS module
- Nginx module
- Openmetrics module
- Oracle module
- Panw module
- PHP_FPM module
- PostgreSQL module
- Prometheus module
- RabbitMQ module
- Redis module
- Redis Enterprise module
- SQL module
- Stan module
- Statsd module
- SyncGateway module
- System module
- System core metricset
- System cpu metricset
- System diskio metricset
- System entropy metricset
- System filesystem metricset
- System fsstat metricset
- System load metricset
- System memory metricset
- System network metricset
- System network_summary metricset
- System process metricset
- System process_summary metricset
- System raid metricset
- System service metricset
- System socket metricset
- System socket_summary metricset
- System uptime metricset
- System users metricset
- Tomcat module
- Traefik module
- uWSGI module
- vSphere module
- Windows module
- ZooKeeper module
- Exported fields
- ActiveMQ fields
- Aerospike fields
- Airflow fields
- Apache fields
- AWS fields
- AWS Fargate fields
- Azure fields
- Beat fields
- Beat fields
- Ceph fields
- Cloud provider metadata fields
- Cloudfoundry fields
- CockroachDB fields
- Common fields
- Consul fields
- Containerd fields
- Coredns fields
- Couchbase fields
- CouchDB fields
- Docker fields
- Docker fields
- Dropwizard fields
- ECS fields
- Elasticsearch fields
- Enterprise Search fields
- Envoyproxy fields
- Etcd fields
- Google Cloud Platform fields
- Golang fields
- Graphite fields
- HAProxy fields
- Host fields
- HTTP fields
- IBM MQ fields
- IIS fields
- Istio fields
- Jolokia fields
- Jolokia Discovery autodiscover provider fields
- Kafka fields
- Kibana fields
- Kubernetes fields
- Kubernetes fields
- KVM fields
- Linux fields
- Logstash fields
- Memcached fields
- MongoDB fields
- MSSQL fields
- Munin fields
- MySQL fields
- NATS fields
- Nginx fields
- Openmetrics fields
- Oracle fields
- Panw fields
- PHP_FPM fields
- PostgreSQL fields
- Process fields
- Prometheus fields
- Prometheus typed metrics fields
- RabbitMQ fields
- Redis fields
- Redis Enterprise fields
- SQL fields
- Stan fields
- Statsd fields
- SyncGateway fields
- System fields
- Tomcat fields
- Traefik fields
- uWSGI fields
- vSphere fields
- Windows fields
- ZooKeeper fields
- Monitor
- Secure
- Troubleshoot
- Get help
- Debug
- Understand logged metrics
- Common problems
- "open /compat/linux/proc: no such file or directory" error on FreeBSD
- Metricbeat collects system metrics for interfaces you didn’t configure
- Metricbeat uses too much bandwidth
- Error loading config file
- Found unexpected or unknown characters
- Logstash connection doesn’t work
- Publishing to Logstash fails with "connection reset by peer" message
- @metadata is missing in Logstash
- Not sure whether to use Logstash or Beats
- SSL client fails to connect to Logstash
- Monitoring UI shows fewer Beats than expected
- Dashboard could not locate the index-pattern
- High RSS memory usage due to MADV settings
- Contribute to Beats
Add Kubernetes metadata
editAdd Kubernetes metadata
editThe add_kubernetes_metadata
processor annotates each event with relevant
metadata based on which Kubernetes pod the event originated from. This processor only adds metadata to the events that do not have it yet present.
At startup, it detects an in_cluster
environment and caches the
Kubernetes-related metadata. Events are only annotated if a valid configuration
is detected. If it’s not able to detect a valid Kubernetes configuration,
the events are not annotated with Kubernetes-related metadata.
Each event is annotated with:
- Pod Name
- Pod UID
- Namespace
- Labels
In addition, the node and namespace metadata are added to the pod metadata.
The add_kubernetes_metadata
processor has two basic building blocks:
- Indexers
- Matchers
Indexers use pod metadata to create unique identifiers for each one of the
pods. These identifiers help to correlate the metadata of the observed pods with
actual events. For example, the ip_port
indexer can take a Kubernetes pod and
create identifiers for it based on all its pod_ip:container_port
combinations.
Matchers use information in events to construct lookup keys that match the
identifiers created by the indexers. For example, when the fields
matcher takes
["metricset.host"]
as a lookup field, it would construct a lookup key with the
value of the field metricset.host
. When one of these lookup keys matches with one
of the identifiers, the event is enriched with the metadata of the identified
pod.
When add_kubernetes_metadata
is used with Metricbeat, it uses the ip_port
indexer and the fields
matcher with the metricset.host
field. So events that
contain a metricset.host
field are enriched with metadata of the pods that
exposes the same port in the same IP.
This behaviour can be disabled by disabling default indexers and matchers in the configuration:
processors: - add_kubernetes_metadata: default_indexers.enabled: false default_matchers.enabled: false
You can find more information about the available indexers and matchers, and some examples in Indexers and matchers.
The configuration below enables the processor when metricbeat is run as a pod in Kubernetes.
processors: - add_kubernetes_metadata: #labels.dedot: true #annotations.dedot: true
The configuration below enables the processor on a Beat running as a process on the Kubernetes node.
processors: - add_kubernetes_metadata: host: <hostname> # If kube_config is not set, KUBECONFIG environment variable will be checked # and if not present it will fall back to InCluster kube_config: $Metricbeat Reference [8.17]/.kube/config #labels.dedot: true #annotations.dedot: true
The configuration below has the default indexers and matchers disabled and enables ones that the user is interested in.
processors: - add_kubernetes_metadata: host: <hostname> # If kube_config is not set, KUBECONFIG environment variable will be checked # and if not present it will fall back to InCluster kube_config: ~/.kube/config default_indexers.enabled: false default_matchers.enabled: false indexers: - ip_port: matchers: - fields: lookup_fields: ["metricset.host"] #labels.dedot: true #annotations.dedot: true
The add_kubernetes_metadata
processor has the following configuration settings:
-
host
- (Optional) Specify the node to scope metricbeat to in case it cannot be accurately detected, as when running metricbeat in host network mode.
-
scope
-
(Optional) Specify if the processor should have visibility at the node level or at the entire cluster
level. Possible values are
node
andcluster
. Scope isnode
by default. -
namespace
- (Optional) Select the namespace from which to collect the metadata. If it is not set, the processor collects metadata from all namespaces. It is unset by default.
-
add_resource_metadata
-
(Optional) Specify filters and configuration for the extra metadata, that will be added to the event. Configuration parameters:
-
node
ornamespace
: Specify labels and annotations filters for the extra metadata coming from node and namespace. By default all labels are included while annotations are not. To change default behaviourinclude_labels
,exclude_labels
andinclude_annotations
can be defined. Those settings are useful when storing labels and annotations that require special handling to avoid overloading the storage output. Note: wildcards are not supported for those settings. The enrichment ofnode
ornamespace
metadata can be individually disabled by settingenabled: false
. -
deployment
: If resource ispod
and it is created from adeployment
, by default the deployment name is added, this can be disabled by settingdeployment: false
. -
cronjob
: If resource ispod
and it is created from acronjob
, by default the cronjob name is added, this can be disabled by settingcronjob: false
.Example:
-
add_resource_metadata: namespace: include_labels: ["namespacelabel1"] #labels.dedot: true #annotations.dedot: true node: include_labels: ["nodelabel2"] include_annotations: ["nodeannotation1"] #labels.dedot: true #annotations.dedot: true deployment: false cronjob: false
-
kube_config
-
(Optional) Use given config file as configuration for Kubernetes
client. It defaults to
KUBECONFIG
environment variable if present. -
use_kubeadm
- (Optional) Default true. By default requests to kubeadm config map are made in order to enrich cluster name by requesting /api/v1/namespaces/kube-system/configmaps/kubeadm-config API endpoint.
-
kube_client_options
- (Optional) Additional options can be configured for Kubernetes client. Currently client QPS and burst are supported, if not set Kubernetes client’s default QPS and burst will be used. Example:
kube_client_options: qps: 5 burst: 10
-
cleanup_timeout
-
(Optional) Specify the time of inactivity before stopping the
running configuration for a container. This is
60s
by default. -
sync_period
- (Optional) Specify the timeout for listing historical resources.
-
default_indexers.enabled
- (Optional) Enable or disable default pod indexers when you want to specify your own.
-
default_matchers.enabled
- (Optional) Enable or disable default pod matchers when you want to specify your own.
-
labels.dedot
-
(Optional) Default to be true. If set to true, then
.
in labels will be replaced with_
. -
annotations.dedot
-
(Optional) Default to be true. If set to true, then
.
in labels will be replaced with_
.
Indexers and matchers
editIndexers
editIndexers use pods metadata to create unique identifiers for each one of the pods.
Available indexers are:
-
container
- Identifies the pod metadata using the IDs of its containers.
-
ip_port
-
Identifies the pod metadata using combinations of its IP and its exposed ports.
When using this indexer metadata is identified using the IP of the pods, and the
combination if
ip:port
for each one of the ports exposed by its containers. -
pod_name
-
Identifies the pod metadata using its namespace and its name as
namespace/pod_name
. -
pod_uid
- Identifies the pod metadata using the UID of the pod.
Matchers
editMatchers are used to construct the lookup keys that match with the identifiers created by indexes.
field_format
editLooks up pod metadata using a key created with a string format that can include event fields.
This matcher has an option format
to define the string format. This string
format can contain placeholders for any field in the event.
For example, the following configuration uses the ip_port
indexer to identify
the pod metadata by combinations of the pod IP and its exposed ports, and uses
the destination IP and port in events as match keys:
processors: - add_kubernetes_metadata: ... default_indexers.enabled: false default_matchers.enabled: false indexers: - ip_port: matchers: - field_format: format: '%{[destination.ip]}:%{[destination.port]}'
fields
editLooks up pod metadata using as key the value of some specific fields. When multiple fields are defined, the first one included in the event is used.
This matcher has an option lookup_fields
to define the files whose value will
be used for lookup.
For example, the following configuration uses the ip_port
indexer to identify
pods, and defines a matcher that uses the destination IP or the server IP for the
lookup, the first it finds in the event:
processors: - add_kubernetes_metadata: ... default_indexers.enabled: false default_matchers.enabled: false indexers: - ip_port: matchers: - fields: lookup_fields: ['destination.ip', 'server.ip']
It’s also possible to extract the matching key from fields using a regex pattern.
The optional regex_pattern
field can be used to set the pattern. The pattern
must contain a capture group named key
, whose value will be used as the matching key.
For example, the following configuration uses the container
indexer to identify
containers by their id, and extracts the matching key from the cgroup id field added
to system process metrics. This field has the form cri-containerd-<id>.scope
, so
we need a regex pattern to obtain the container id.
processors: - add_kubernetes_metadata: indexers: - container: matchers: - fields: lookup_fields: ['system.process.cgroup.id'] regex_pattern: 'cri-containerd-(?P<key>[0-9a-z]+)\.scope'