Prometheus collector metricset

The Prometheus collector metricset scrapes data from prometheus exporters.

To scrape metrics from a Prometheus exporter, configure the hosts field to it. The path to retrieve the metrics from (/metrics by default) can be configured with metrics_path.

- module: prometheus
  period: 10s
  hosts: ["node:9100"]
  metrics_path: /metrics

  #username: "user"
  #password: "secret"

  # This can be used for service account based authorization:
  #bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  #ssl.certificate_authorities:
  #  - /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt

metricbeat.modules:
- module: prometheus
  period: 10s
  hosts: ["localhost:9090"]
  use_types: true
  rate_counters: false

use_types parameter (default: false) enables a different layout for metrics storage, leveraging Elasticsearch types, including histograms.

rate_counters parameter (default: false) enables calculating a rate out of Prometheus counters. When enabled, Metricbeat stores the counter increment since the last collection. This metric should make some aggregations easier and with better performance. This parameter can only be enabled in combination with use_types.

When use_types and rate_counters are enabled, metrics are stored like this:

{
    "prometheus": {
        "labels": {
            "instance": "172.27.0.2:9090",
            "job": "prometheus"
        },
        "prometheus_target_interval_length_seconds_count": {
            "counter": 1,
            "rate": 0
        },
        "prometheus_target_interval_length_seconds_sum": {
            "counter": 15.000401344,
            "rate": 0
        }
        "prometheus_tsdb_compaction_chunk_range_seconds_bucket": {
            "histogram": {
                "values": [50, 300, 1000, 4000, 16000],
                "counts": [10, 2, 34, 7]
            }
        }
    },
}

This module can scrape all metrics stored in a Prometheus server, by using the federation API. By pointing this config to the Prometheus server:

metricbeat.modules:
- module: prometheus
  period: 10s
  hosts: ["localhost:9090"]
  metrics_path: '/federate'
  query:
    'match[]': '{__name__!=""}'

In order to filter out/in metrics one can make use of metrics_filters.include metrics_filters.exclude settings:

- module: prometheus
  period: 10s
  hosts: ["localhost:9090"]
  metrics_path: /metrics
  metrics_filters:
    include: ["node_filesystem_*"]
    exclude: ["node_filesystem_device_*"]

The configuration above will include only metrics that match node_filesystem_* pattern and do not match node_filesystem_device_*.

To keep only specific metrics, anchor the start and the end of the regexp of each metric:

- module: prometheus
  period: 10s
  hosts: ["localhost:9090"]
  metrics_path: /metrics
  metrics_filters:
    include: ["^node_network_net_dev_group$", "^node_network_up$"]

This is a default metricset. If the host module is unconfigured, this metricset is enabled by default.

{
    "@timestamp": "2019-03-01T08:05:34.853Z",
    "event": {
        "dataset": "prometheus.collector",
        "duration": 115000,
        "module": "prometheus"
    },
    "metricset": {
        "name": "collector",
        "period": 10000
    },
    "prometheus": {
        "labels": {
            "job": "prometheus"
        },
        "metrics": {
            "up": 1
        }
    },
    "service": {
        "address": "127.0.0.1:55555",
        "type": "prometheus"
    }
}