Collecting Elasticsearch monitoring data with Metricbeat
editCollecting Elasticsearch monitoring data with Metricbeat
editIn 6.5 and later, you can use Metricbeat to collect data about Elasticsearch and ship it to the monitoring cluster, rather than routing it through exporters as described in Legacy collection methods.
-
Enable the collection of monitoring data.
Set
xpack.monitoring.collection.enabled
totrue
on the production cluster. By default, it is is disabled (false
).You can use the following APIs to review and change this setting:
GET _cluster/settings PUT _cluster/settings { "persistent": { "xpack.monitoring.collection.enabled": true } }
If Elasticsearch security features are enabled, you must have
monitor
cluster privileges to view the cluster settings andmanage
cluster privileges to change them.For more information, see Monitoring settings and Cluster update settings.
- Install Metricbeat on each Elasticsearch node in the production cluster. Failure to install on each node may result in incomplete or missing results.
-
Enable the Elasticsearch X-Pack module in Metricbeat on each Elasticsearch node.
For example, to enable the default configuration in the
modules.d
directory, run the following command:metricbeat modules enable elasticsearch-xpack
Alternatively, you can use the Elasticsearch module, as described in the Elasticsearch module usage for Elastic Stack monitoring.
-
Configure the Elasticsearch X-Pack module in Metricbeat on each Elasticsearch node.
The
modules.d/elasticsearch-xpack.yml
file contains the following settings:- module: elasticsearch xpack.enabled: true period: 10s hosts: ["http://localhost:9200"] #scope: node #username: "user" #password: "secret" #ssl.enabled: true #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] #ssl.certificate: "/etc/pki/client/cert.pem" #ssl.key: "/etc/pki/client/cert.key" #ssl.verification_mode: "full"
By default, the module collects Elasticsearch monitoring metrics from
http://localhost:9200
. If that host and port number are not correct, you must update thehosts
setting. If you configured Elasticsearch to use encrypted communications, you must access it via HTTPS. For example, use ahosts
setting likehttps://localhost:9200
.By default,
scope
is set tonode
and each entry in thehosts
list indicates a distinct node in an Elasticsearch cluster. If you setscope
tocluster
, each entry in thehosts
list indicates a single endpoint for a distinct Elasticsearch cluster (for example, a load-balancing proxy fronting the cluster).If Elastic security features are enabled, you must also provide a user ID and password so that Metricbeat can collect metrics successfully:
-
Create a user on the production cluster that has the
remote_monitoring_collector
built-in role. Alternatively, use theremote_monitoring_user
built-in user. -
Add the
username
andpassword
settings to the Elasticsearch module configuration file. -
If TLS is enabled on the HTTP layer of your Elasticsearch cluster, you must either use https as the URL scheme in the
hosts
setting or add thessl.enabled: true
setting. Depending on the TLS configuration of your Elasticsearch cluster, you might also need to specify additional ssl.* settings.
-
Create a user on the production cluster that has the
-
Optional: Disable the system module in Metricbeat.
By default, the system module is enabled. The information it collects, however, is not shown on the Monitoring page in Kibana. Unless you want to use that information for other purposes, run the following command:
metricbeat modules disable system
-
Identify where to send the monitoring data.
In production environments, we strongly recommend using a separate cluster (referred to as the monitoring cluster) to store the data. Using a separate monitoring cluster prevents production cluster outages from impacting your ability to access your monitoring data. It also prevents monitoring activities from impacting the performance of your production cluster.
For example, specify the Elasticsearch output information in the Metricbeat configuration file (
metricbeat.yml
):output.elasticsearch: # Array of hosts to connect to. hosts: ["http://es-mon-1:9200", "http://es-mon2:9200"] # Optional protocol and basic auth credentials. #protocol: "https" #username: "elastic" #password: "changeme"
If you configured the monitoring cluster to use encrypted communications, you must access it via HTTPS. For example, use a
hosts
setting likehttps://es-mon-1:9200
.The Elasticsearch monitoring features use ingest pipelines, therefore the cluster that stores the monitoring data must have at least one ingest node.
If Elasticsearch security features are enabled on the monitoring cluster, you must provide a valid user ID and password so that Metricbeat can send metrics successfully:
-
Create a user on the monitoring cluster that has the
remote_monitoring_agent
built-in role. Alternatively, use theremote_monitoring_user
built-in user. -
Add the
username
andpassword
settings to the Elasticsearch output information in the Metricbeat configuration file.
For more information about these configuration options, see Configure the Elasticsearch output.
-
Create a user on the monitoring cluster that has the
- Start Metricbeat on each node.
-
Disable the default collection of Elasticsearch monitoring metrics.
Set
xpack.monitoring.elasticsearch.collection.enabled
tofalse
on the production cluster.You can use the following API to change this setting:
PUT _cluster/settings { "persistent": { "xpack.monitoring.elasticsearch.collection.enabled": false } }
If Elasticsearch security features are enabled, you must have
monitor
cluster privileges to view the cluster settings andmanage
cluster privileges to change them. - View the monitoring data in Kibana.