Add Kubernetes metadata

The add_kubernetes_metadata processor annotates each event with relevant metadata based on which Kubernetes Pod the event originated from. At startup it detects an in_cluster environment and caches the Kubernetes-related metadata.

For events to be annotated with Kubernetes-related metadata, the Kubernetes configuration must be valid.

Each event is annotated with:

In addition, the node and namespace metadata are added to the Pod metadata.

The add_kubernetes_metadata processor has two basic building blocks:

Indexers use Pod metadata to create unique identifiers for each one of the Pods. These identifiers help to correlate the metadata of the observed Pods with actual events. For example, the ip_port indexer can take a Kubernetes Pod and create identifiers for it based on all its pod_ip:container_port combinations.

Matchers use information in events to construct lookup keys that match the identifiers created by the indexers. For example, when the fields matcher takes ["metricset.host"] as a lookup field, it constructs a lookup key with the value of the field metricset.host. When one of these lookup keys matches with one of the identifiers, the event is enriched with the metadata of the identified Pod.

For more information about available indexers and matchers, plus some examples, refer to Indexers and matchers.

This configuration enables the processor when Elastic Agent is run as a Pod in Kubernetes.

  - add_kubernetes_metadata:
      # Defining indexers and matchers manually is required for {beatname_lc}, for instance:
      #indexers:
      #  - ip_port:
      #matchers:
      #  - fields:
      #      lookup_fields: ["metricset.host"]
      #labels.dedot: true
      #annotations.dedot: true

This configuration enables the processor on an Elastic Agent running as a process on the Kubernetes node:

  - add_kubernetes_metadata:
      host: <hostname>
      # If kube_config is not set, KUBECONFIG environment variable will be checked
      # and if not present it will fall back to InCluster
      kube_config: $Fleet and Elastic Agent Guide [8.15]/.kube/config
      # Defining indexers and matchers manually is required for {beatname_lc}, for instance:
      #indexers:
      #  - ip_port:
      #matchers:
      #  - fields:
      #      lookup_fields: ["metricset.host"]
      #labels.dedot: true
      #annotations.dedot: true

This configuration disables the default indexers and matchers, and then enables different indexers and matchers:

  - add_kubernetes_metadata:
      host: <hostname>
      # If kube_config is not set, KUBECONFIG environment variable will be checked
      # and if not present it will fall back to InCluster
      kube_config: ~/.kube/config
      default_indexers.enabled: false
      default_matchers.enabled: false
      indexers:
        - ip_port:
      matchers:
        - fields:
            lookup_fields: ["metricset.host"]
      #labels.dedot: true
      #annotations.dedot: true

      add_resource_metadata:
        namespace:
          include_labels: ["namespacelabel1"]
          # use_regex_include: false
          # use_regex_exclude: false
          # exclude_labels: ["namespacelabel2"]
          #labels.dedot: true
          #annotations.dedot: true
        node:
          # use_regex_include: false
          include_labels: ["nodelabel2"]
          include_annotations: ["nodeannotation1"]
          # use_regex_exclude: false
          # exclude_annotations: ["nodeannotation2"]
          #labels.dedot: true
          #annotations.dedot: true
        deployment: true
        cronjob: true

      kube_client_options:
        qps: 5
        burst: 10

The add_kubernetes_metadata processor has two basic building blocks:

Indexers use Pod metadata to create unique identifiers for each one of the Pods.

Available indexers are:

Matchers are used to construct the lookup keys that match with the identifiers created by indexes.

Available matchers are: