Prometheus collector metricset

The Prometheus collector metricset scrapes data from prometheus exporters.

Scraping from a Prometheus exporter

To scrape metrics from a Prometheus exporter, configure the hosts field to it. The path to retrieve the metrics from (/metrics by default) can be configured with metrics_path.

- module: prometheus
  period: 10s
  hosts: ["node:9100"]
  metrics_path: /metrics

  #username: "user"
  #password: "secret"

  # This can be used for service account based authorization:
  #bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  #ssl.certificate_authorities:
  #  - /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt

Histograms and types

edit

This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.

metricbeat.modules:
- module: prometheus
  period: 10s
  hosts: ["localhost:9090"]
  use_types: true
  rate_counters: false

use_types parameter (default: false) enables a different layout for metrics storage, leveraging Elasticsearch types, including histograms.

rate_counters parameter (default: false) enables calculating a rate out of Prometheus counters. When enabled, Metricbeat stores the counter increment since the last collection. This metric should make some aggregations easier and with better performance. This parameter can only be enabled in combination with use_types.

When use_types and rate_counters are enabled, metrics are stored like this:

{
    "prometheus": {
        "labels": {
            "instance": "172.27.0.2:9090",
            "job": "prometheus"
        },
        "prometheus_target_interval_length_seconds_count": {
            "counter": 1,
            "rate": 0
        },
        "prometheus_target_interval_length_seconds_sum": {
            "counter": 15.000401344,
            "rate": 0
        }
        "prometheus_tsdb_compaction_chunk_range_seconds_bucket": {
            "histogram": {
                "values": [50, 300, 1000, 4000, 16000],
                "counts": [10, 2, 34, 7]
            }
        }
    },
}

Scraping all metrics from a Prometheus server

edit

Depending on your scale this method may not be suitable. We recommend using the remote_write metricset for this, and make Prometheus push metrics to Metricbeat.

This module can scrape all metrics stored in a Prometheus server, by using the federation API. By pointing this config to the Prometheus server:

metricbeat.modules:
- module: prometheus
  period: 10s
  hosts: ["localhost:9090"]
  metrics_path: '/federate'
  query:
    'match[]': '{__name__!=""}'

federation API returns all metrics as untyped, as a result even in case use_types and rate_counters parameters are enabled, rate metrics will NOT be calculated out of Prometheus counters. To get rate metrics calculated should be used remote_write metricset instead.

Filtering metrics

edit

In order to filter out/in metrics one can make use of metrics_filters.include metrics_filters.exclude settings:

- module: prometheus
  period: 10s
  hosts: ["localhost:9090"]
  metrics_path: /metrics
  metrics_filters:
    include: ["node_filesystem_*"]
    exclude: ["node_filesystem_device_*"]

The configuration above will include only metrics that match node_filesystem_* pattern and do not match node_filesystem_device_*.

To keep only specific metrics, anchor the start and the end of the regexp of each metric:

the caret ^ matches the beginning of a text or line,
the dollar sign $ matches the end of a text.

- module: prometheus
  period: 10s
  hosts: ["localhost:9090"]
  metrics_path: /metrics
  metrics_filters:
    include: ["^node_network_net_dev_group$", "^node_network_up$"]

This is a default metricset. If the host module is unconfigured, this metricset is enabled by default.

Fields

For a description of each field in the metricset, see the exported fields section.

Here is an example document generated by this metricset:

{
    "@timestamp": "2019-03-01T08:05:34.853Z",
    "event": {
        "dataset": "prometheus.collector",
        "duration": 115000,
        "module": "prometheus"
    },
    "metrics_count": 2,
    "metricset": {
        "name": "collector",
        "period": 10000
    },
    "prometheus": {
        "labels": {
            "job": "prometheus",
            "listener_name": "http"
        },
        "metrics": {
            "net_conntrack_listener_conn_accepted_total": 3,
            "net_conntrack_listener_conn_closed_total": 0
        }
    },
    "service": {
        "address": "127.0.0.1:55555",
        "type": "prometheus"
    }
}

« Prometheus module Prometheus query metricset »