Autodiscover
editAutodiscover
editWhen you run applications on containers, they become moving targets to the monitoring system. Autodiscover allows you to track them and adapt settings as changes happen. By defining configuration templates, the autodiscover subsystem can monitor services as they start running.
You define autodiscover settings in the metricbeat.autodiscover
section of the metricbeat.yml
config file. To enable autodiscover, you specify a list of providers.
Providers
editAutodiscover providers work by watching for events on the system and translating those events into internal autodiscover events with a common format. When you configure the provider, you can use fields from the autodiscover event to set conditions that, when met, launch specific configurations.
On start, Metricbeat will scan existing containers and launch the proper configs for them. Then it will watch for new start/stop events. This ensures you don’t need to worry about state, but only define your desired configs.
Docker
editThe Docker autodiscover provider watches for Docker containers to start and stop. These are the available fields on every event:
- host
- port
- docker.container.id
- docker.container.image
- docker.container.name
- docker.container.labels
For example:
{ "host": "10.4.15.9", "port": 6379, "docker": { "container": { "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51" "name": "redis", "image": "redis:3.2.11", "labels": { "io.kubernetes.pod.namespace": "default" ... } } } }
You can define a set of configuration templates to be applied when the condition matches an event. Templates define a condition to match on autodiscover events, together with the list of configurations to launch when this condition happens.
Conditions match events from the provider. Providers use the same format for Conditions that processors use.
Configuration templates can contain variables from the autodiscover event. They can be accessed under the data
namespace.
For example, with the example event, "${data.port}
" resolves to 6379
.
Metricbeat supports templates for modules:
metricbeat.autodiscover: providers: - type: docker templates: - condition: contains: docker.container.image: redis config: - module: redis metricsets: ["info", "keyspace"] hosts: "${data.host}:6379"
This configuration launches a redis
module for all containers running an image with redis
in the name.
Kubernetes
editThe Kubernetes autodiscover provider watches for Kubernetes pods to start, update, and stop. These are the available fields on every event:
- host
- port
- kubernetes.container.id
- kubernetes.container.image
- kubernetes.container.name
- kubernetes.labels
- kubernetes.namespace
- kubernetes.node.name
- kubernetes.pod.name
If the include_annotations
config is added to the provider config, then the list of annotations present in the config
are added to the event.
For example:
{ "host": "172.17.0.21", "port": 9090, "kubernetes": { "container": { "id": "bb3a50625c01b16a88aa224779c39262a9ad14264c3034669a50cd9a90af1527", "image": "prom/prometheus", "name": "prometheus" }, "labels": { "project": "prometheus", ... }, "namespace": "default", "node": { "name": "minikube" }, "pod": { "name": "prometheus-2657348378-k1pnh" } }, }
The configuration of templates and conditions is similar to that of the Docker provider. Configuration templates can contain variables from the autodiscover event. They can be accessed under data namespace.
The kubernetes
autodiscover provider has the following configuration settings:
-
in_cluster
-
(Optional) Use in cluster settings for Kubernetes client,
true
by default. -
host
-
(Optional) In case
in_cluster
is false, use this host to connect to Kubernetes API. -
kube_config
- (Optional) Use given config file as configuration for Kubernetes client.
Metricbeat supports templates for modules:
metricbeat.autodiscover: providers: - type: kubernetes include_annotations: ["prometheus.io.scrape"] templates: - condition: contains: kubernetes.annotations.prometheus.io.scrape: "true" config: - module: prometheus metricsets: ["collector"] hosts: "${data.host}:{data.port}"
This configuration launches a prometheus
module for all containers of pods annotated prometheus.io.scrape=true
.
There are cases where the PodSpec does not expose a port. In such cases the host can be provided as ${data.host}:9090
directly. However, the metadata which is used to enrich the metric would not have information regarding the container since
the discovery mechanism would not have information on which container the port maps to.