Autodiscover

When you run applications on containers, they become moving targets to the monitoring system. Autodiscover allows you to track them and adapt settings as changes happen. By defining configuration templates, the autodiscover subsystem can monitor services as they start running.

You define autodiscover settings in the heartbeat.autodiscover section of the heartbeat.yml config file. To enable autodiscover, you specify a list of providers.

Autodiscover providers work by watching for events on the system and translating those events into internal autodiscover events with a common format. When you configure the provider, you can optionally use fields from the autodiscover event to set conditions that, when met, launch specific configurations.

On start, Heartbeat will scan existing containers and launch the proper configs for them. Then it will watch for new start/stop events. This ensures you don’t need to worry about state, but only define your desired configs.

The Docker autodiscover provider watches for Docker containers to start and stop.

These are the available fields during within config templating. The docker.* fields will be available on each emitted event. event:

For example:

{
  "host": "10.4.15.9",
  "port": 6379,
  "docker": {
    "container": {
      "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51"
      "name": "redis",
      "image": "redis:3.2.11",
      "labels": {
        "io.kubernetes.pod.namespace": "default"
        ...
      }
    }
  }
}

You can define a set of configuration templates to be applied when the condition matches an event. Templates define a condition to match on autodiscover events, together with the list of configurations to launch when this condition happens.

Conditions match events from the provider. Providers use the same format for Conditions that processors use.

Configuration templates can contain variables from the autodiscover event. They can be accessed under the data namespace. For example, with the example event, "${data.port}" resolves to 6379.

Heartbeat supports templates for modules:

heartbeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.image: redis
          config:
            - type: tcp
              hosts: ["${data.host}:${data.port}"]
              schedule: "@every 1s"
              timeout: 1s

This configuration launches a redis monitor for all containers running an image with redis in the name.

The Kubernetes autodiscover provider watches for Kubernetes nodes, pods, services to start, update, and stop.

These are the available fields during within config templating. The kubernetes.* fields will be available on each emitted event.

If the include_annotations config is added to the provider config, then the list of annotations present in the config are added to the event.

If the include_labels config is added to the provider config, then the list of labels present in the config will be added to the event.

If the exclude_labels config is added to the provider config, then the list of labels present in the config will be excluded from the event.

if the labels.dedot config is set to be true in the provider config, then . in labels will be replaced with _.

if the annotations.dedot config is set to be true in the provider config, then . in annotations will be replaced with _.

For example:

{
  "host": "172.17.0.21",
  "port": 9090,
  "kubernetes": {
    "container": {
      "id": "bb3a50625c01b16a88aa224779c39262a9ad14264c3034669a50cd9a90af1527",
      "image": "prom/prometheus",
      "name": "prometheus"
    },
    "labels": {
      "project": "prometheus",
      ...
    },
    "namespace": "default",
    "node": {
      "name": "minikube"
    },
    "pod": {
      "name": "prometheus-2657348378-k1pnh"
    }
  },
}

The configuration of templates and conditions is similar to that of the Docker provider. Configuration templates can contain variables from the autodiscover event. They can be accessed under data namespace.

The kubernetes autodiscover provider has the following configuration settings:

      add_resource_metadata:
        namespace:
          enabled: true
          include_labels: ["namespacelabel1"]
        node:
          enabled: true
          include_labels: ["nodelabel2"]

Heartbeat supports templates for modules:

heartbeat.autodiscover:
  providers:
    - type: kubernetes
      include_annotations: ["prometheus.io.scrape"]
      templates:
        - condition:
            contains:
              kubernetes.annotations.prometheus.io/scrape: "true"
          config:
            - type: http
              hosts: ["${data.host}:${data.port}"]
              schedule: "@every 1s"
              timeout: 1s

This configuration launches an http module for all containers of pods annotated with prometheus.io/scrape=true.

Declare exposed ports in your pod spec if possible. Otherwise, you will need to use multiple templates with complex filtering rules. The {port} variable will not be present, and you will need to hardcode ports. Example: {data.host}:1234

When ports are not declared, Autodiscover generates a config using your provided template once per pod, and once per container. These generated configs are de-duplicated after they are generated. If the generated configs for multiple containers are identical, they will be merged into one config.

Pods share an identical host. If only the {data.host} variable is interpolated, then one config will be generated per host. The configs will be identical. After they are de-duplicated, only one will be used.

Note: This provider is experimental

The Amazon ELB autodiscover provider discovers ELBs and their listeners. This is useful when you don’t want to connect directly to a service, but rather to the ELB fronting a pool of services.

This provider will yield one config block per ELB Listener. So, if you have one ELB exposing both ports 80 and 443, it will generate two configs, one for each port. Keep in mind that the beat will de-duplicate configs. So, if the generated configs are the same only one will actually run.

This provider will load AWS credentials using the standard AWS environment variables and shared credentials files see Best Practices for Managing AWS Access Keys for more information. If you do not wish to use these, you may explicitly set the access_key_id and secret_access_key variables.

These are the available fields during within config templating. The elb_listener.* fields will be available on each emitted event.

Heartbeat supports templates for modules:

heartbeat.autodiscover:
  providers:
  - type: aws_elb
    period: 1m
    regions: ["us-east-1", "us-east-2"]
    access_key_id: my-access-key
    secret_access_key: my-secret-access-key
    templates:
    - condition:
        equals.port: 8080
      config:
      - type: tcp
        hosts: ["${data.host}:${data.port}"]
        schedule: "@every 5s"
        timeout: 1s

This configuration launches a tcp monitor for all ELBs that have a declared port.

This autodiscover provider takes our standard AWS credentials options.

To configure AWS credentials, either put the credentials into the Heartbeat configuration, or use a shared credentials file, as shown in the following examples.

Users can either put the credentials into metricbeat module configuration or use environment variable AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and/or AWS_SESSION_TOKEN instead.

heartbeat.autodiscover:
  providers:
  - type: aws_elb
    period: 1m
    regions: ["us-east-1", "us-east-2"]
    access_key_id: '<access_key_id>'
    secret_access_key: '<secret_access_key>'
    session_token: '<session_token>'
    templates:
    - type: tcp
      hosts: ["${data.host}:${data.port}"]
      schedule: "@every 5s"
      timeout: 1s

or

heartbeat.autodiscover:
  providers:
  - type: aws_elb
    period: 1m
    regions: ["us-east-1", "us-east-2"]
    access_key_id: '${AWS_ACCESS_KEY_ID:""}'
    secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
    session_token: '${AWS_SESSION_TOKEN:""}'
    templates:
    - type: tcp
      hosts: ["${data.host}:${data.port}"]
      schedule: "@every 5s"
      timeout: 1s

heartbeat.autodiscover:
  providers:
  - type: aws_elb
    period: 1m
    regions: ["us-east-1", "us-east-2"]
    credential_profile_name: test-hb
    templates:
    - type: tcp
      hosts: ["${data.host}:${data.port}"]
      schedule: "@every 5s"
      timeout: 1s

credential_profile_name is optional. If you use different credentials for different tools or applications, you can use profiles to configure multiple access keys in the same configuration file. If there is no credential_profile_name given, the default profile will be used.

shared_credential_file is optional to specify the directory of your shared credentials file. If it’s empty, the default directory will be used. In Windows, shared credentials file is at C:\Users\<yourUserName>\.aws\credentials. For Linux, macOS or Unix, the file is located at ~/.aws/credentials. Please see Create Shared Credentials File for more details.

There are two different types of AWS credentials can be used: access keys and temporary security credentials.

AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are the two parts of access keys. They are long-term credentials for an IAM user or the AWS account root user. Please see AWS Access Keys and Secret Access Keys for more details.

temporary security credentials has a limited lifetime and consists of an access key ID, a secret access key, and a security token which typically returned from GetSessionToken. MFA-enabled IAM users would need to submit an MFA code while calling GetSessionToken. default_region identifies the AWS Region whose servers you want to send your first API request to by default. This is typically the Region closest to you, but it can be any Region. Please see Temporary Security Credentials for more details. sts get-session-token AWS CLI can be used to generate temporary credentials. For example. with MFA-enabled:

aws> sts get-session-token --serial-number arn:aws:iam::1234:mfa/your-email@example.com --token-code 456789 --duration-seconds 129600

Because temporary security credentials are short term, after they expire, the user needs to generate new ones and modify the aws.yml config file with the new credentials. Unless live reloading feature is enabled for Metricbeat, the user needs to manually restart Metricbeat after updating the config file in order to continue collecting Cloudwatch metrics. This will cause data loss if the config file is not updated with new credentials before the old ones expire. For Metricbeat, we recommend users to use access keys in config file to enable aws module making AWS api calls without have to generate new temporary credentials and update the config frequently.

IAM policy is an entity that defines permissions to an object within your AWS environment. Specific permissions needs to be added into the IAM user’s policy to authorize Metricbeat to collect AWS monitoring metrics. Please see documentation under each metricset for required permissions.