Autodiscover

When you run applications on containers, they become moving targets to the monitoring system. Autodiscover allows you to track them and adapt settings as changes happen. By defining configuration templates, the autodiscover subsystem can monitor services as they start running.

You define autodiscover settings in the filebeat.autodiscover section of the filebeat.yml config file. To enable autodiscover, you specify a list of providers.

Autodiscover providers work by watching for events on the system and translating those events into internal autodiscover events with a common format. When you configure the provider, you can optionally use fields from the autodiscover event to set conditions that, when met, launch specific configurations.

On start, Filebeat will scan existing containers and launch the proper configs for them. Then it will watch for new start/stop events. This ensures you don’t need to worry about state, but only define your desired configs.

The Docker autodiscover provider watches for Docker containers to start and stop.

It has the following settings:

These are the fields available within config templating. The docker.* fields will be available on each emitted event. event:

For example:

{
  "host": "10.4.15.9",
  "port": 6379,
  "docker": {
    "container": {
      "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51"
      "name": "redis",
      "image": "redis:3.2.11",
      "labels": {
        "io.kubernetes.pod.namespace": "default"
        ...
      }
    }
  }
}

You can define a set of configuration templates to be applied when the condition matches an event. Templates define a condition to match on autodiscover events, together with the list of configurations to launch when this condition happens.

Conditions match events from the provider. Providers use the same format for Conditions that processors use.

Configuration templates can contain variables from the autodiscover event. They can be accessed under the data namespace. For example, with the example event, "${data.port}" resolves to 6379.

Filebeat supports templates for inputs and modules.

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.image: redis
          config:
            - type: container
              paths:
                - /var/lib/docker/containers/${data.docker.container.id}/*.log
              exclude_lines: ["^\\s+[\\-`('.|_]"]  # drop asciiart lines

This configuration launches a docker logs input for all containers running an image with redis in the name. labels.dedot defaults to be true for docker autodiscover, which means dots in docker labels are replaced with _ by default.

If you are using modules, you can override the default input and use the docker input instead.

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.image: redis
          config:
            - module: redis
              log:
                input:
                  type: container
                  paths:
                    - /var/lib/docker/containers/${data.docker.container.id}/*.log

autodiscover.providers:
  - type: docker
    templates:
      - condition.contains:
          docker.container.image: nginx
        config:
          - type: log
            paths:
              - "/mnt/logs/*/*.log"

autodiscover.providers:
  - type: docker
    templates:
      - condition.contains:
          docker.container.image: nginx
        config:
          - type: log
            paths:
              - "/mnt/logs/${data.docker.container.id}/*.log"

The Kubernetes autodiscover provider watches for Kubernetes nodes, pods, services to start, update, and stop.

The kubernetes autodiscover provider has the following configuration settings:

      kube_client_options:
        qps: 5
        burst: 10

      add_resource_metadata:
        namespace:
          include_labels: ["namespacelabel1"]
        node:
          include_labels: ["nodelabel2"]
          include_annotations: ["nodeannotation1"]
        deployment: false
        cronjob: false

The configuration of templates and conditions is similar to that of the Docker provider. Configuration templates can contain variables from the autodiscover event. They can be accessed under data namespace.

These are the fields available within config templating. The kubernetes.* fields will be available on each emitted event.

If the include_annotations config is added to the provider config, then the list of annotations present in the config are added to the event.

If the include_labels config is added to the provider config, then the list of labels present in the config will be added to the event.

If the exclude_labels config is added to the provider config, then the list of labels present in the config will be excluded from the event.

if the labels.dedot config is set to be true in the provider config, then . in labels will be replaced with _. By default it is true.

if the annotations.dedot config is set to be true in the provider config, then . in annotations will be replaced with _. By default it is true.

For example:

{
  "host": "172.17.0.21",
  "port": 9090,
  "kubernetes": {
    "container": {
      "id": "bb3a50625c01b16a88aa224779c39262a9ad14264c3034669a50cd9a90af1527",
      "image": "prom/prometheus",
      "name": "prometheus"
    },
    "labels": {
      "project": "prometheus",
      ...
    },
    "namespace": "default",
    "node": {
      "name": "minikube"
    },
    "pod": {
      "name": "prometheus-2657348378-k1pnh"
    }
  },
}

Filebeat supports templates for inputs and modules.

filebeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            equals:
              kubernetes.namespace: kube-system
          config:
            - type: container
              paths:
                - /var/log/containers/*-${data.kubernetes.container.id}.log
              exclude_lines: ["^\\s+[\\-`('.|_]"]  # drop asciiart lines

This configuration launches a docker logs input for all containers of pods running in the Kubernetes namespace kube-system.

If you are using modules, you can override the default input and use the docker input instead.

filebeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            equals:
              kubernetes.container.image: "redis"
          config:
            - module: redis
              log:
                input:
                  type: container
                  paths:
                    - /var/log/containers/*-${data.kubernetes.container.id}.log

The Jolokia autodiscover provider uses Jolokia Discovery to find agents running in your host or your network.

The configuration of this provider consists in a set of network interfaces, as well as a set of templates as in other providers. The network interfaces will be the ones used for discovery probes, each item of interfaces has these settings:

Jolokia Discovery mechanism is supported by any Jolokia agent since version 1.2.0, it is enabled by default when Jolokia is included in the application as a JVM agent, but disabled in other cases as the OSGI or WAR (Java EE) agents. In any case, this feature is controlled with two properties:

There are multiple ways of setting these properties, and they can vary from application to application, please refer to the documentation of your application to find the more suitable way to set them in your case.

Jolokia Discovery is based on UDP multicast requests. Agents join the multicast group 239.192.48.84, port 24884, and discovery is done by sending queries to this group. You have to take into account that UDP traffic between Filebeat and the Jolokia agents has to be allowed. Also notice that this multicast address is in the 239.0.0.0/8 range, that is reserved for private use within an organization, so it can only be used in private networks.

These are the available fields during within config templating. The jolokia.* fields will be available on each emitted event.

Filebeat supports templates for inputs and modules:

filebeat.autodiscover:
  providers:
    - type: jolokia
      interfaces:
      - name: lo
      templates:
      - condition:
          contains:
            jolokia.server.product: "kafka"
        config:
        - module: kafka
          log:
            enabled: true
            var.paths:
            - /var/log/kafka/*.log

This configuration starts a jolokia module that collects logs of kafka if it is running. Discovery probes are sent using the local interface.

The Nomad autodiscover provider watches for Nomad jobs to start, update, and stop.

The nomad autodiscover provider has the following configuration settings:

namespace "*" {
  policy = "read"
}
node {
  policy = "read"
}
agent {
  policy = "read"
}

The configuration of templates and conditions is similar to that of the Docker provider. Configuration templates can contain variables from the autodiscover event. They can be accessed under data namespace.

These are the available fields during config templating. The nomad.* fields will be available on each emitted event.

If the include_labels config is added to the provider config, then the list of labels present in the config will be added to the event.

If the exclude_labels config is added to the provider config, then the list of labels present in the config will be excluded from the event.

if the labels.dedot config is set to be true in the provider config, then . in labels will be replaced with _.

For example:

{
  ...
  "region": "europe",
  "allocation": {
    "name": "coffeshop.api[0]",
    "id": "35eba07f-e5e4-20ac-6def-85117bee6efb",
    "status": "running"
  },
  "datacenters": [
    "europe-west4"
  ],
  "namespace": "default",
  "job": {
    "type": "service",
    "name": "coffeshop"
  },
  "task": {
    "service": {
      "name": [
        "coffeshop"
      ],
      "tags": [
        "coffeshop",
        "nginx"
      ],
      "canary_tags": [
        "coffeshop"
      ]
    },
    "name": "api"
  },
  ...
}

Filebeat supports templates for inputs and modules.

filebeat.autodiscover:
  providers:
    - type: nomad
      node: nomad1
      scope: local
      hints.enabled: true
      allow_stale: true
      templates:
        - condition:
            equals:
              nomad.namespace: web
          config:
            - type: log
              paths:
                - /var/lib/nomad/alloc/${data.nomad.allocation.id}/alloc/logs/${data.nomad.task.name}.stderr.[0-9]*
              exclude_lines: ["^\\s+[\\-`('.|_]"]  # drop asciiart lines

This configuration launches a log input for all jobs under the web Nomad namespace.

If you are using modules, you can override the default input and customize it to read from the ${data.nomad.task.name}.stdout and/or ${data.nomad.task.name}.stderr files.

filebeat.autodiscover:
  providers:
    - type: nomad
      templates:
        - condition:
            equals:
              nomad.task.service.tags: "redis"
          config:
            - module: redis
              log:
                input:
                  type: log
                  paths:
                    - /var/lib/nomad/alloc/${data.nomad.allocation.id}/alloc/logs/${data.nomad.task.name}.*