Monitoring in a production environment
editMonitoring in a production environment
editIn production, you should send monitoring data to a separate monitoring cluster so that historical data is available even when the nodes you are monitoring are not. For example, you can use Metricbeat to ship monitoring data about Kibana, Elasticsearch, Logstash, and Beats to the monitoring cluster. If you are sending your data to the ESMS, see Elastic Stack Monitoring Service.
Metricbeat is the recommended method for collecting and shipping monitoring data to a monitoring cluster.
If you have previously configured legacy collection methods, you should migrate to using Metricbeat collection. Use either Metricbeat collection or legacy collection methods; do not use both.
Learn more about Collecting monitoring data with Metricbeat.
If you have at least a Gold Subscription, using a dedicated monitoring cluster also enables you to monitor multiple clusters from a central location.
To store monitoring data in a separate cluster:
-
Set up the Elasticsearch cluster you want to use as the monitoring cluster. For example, you might set up a two host cluster with the nodes
es-mon-1
andes-mon-2
.- To monitor an Elasticsearch 7.x cluster, you must run Elasticsearch 7.x on the monitoring cluster.
- There must be at least one ingest node in the monitoring cluster; it does not need to be a dedicated ingest node.
-
(Optional) Verify that the collection of monitoring data is disabled on the monitoring cluster. By default, the
xpack.monitoring.collection.enabled
setting isfalse
.For example, you can use the following APIs to review and change this setting:
GET _cluster/settings PUT _cluster/settings { "persistent": { "xpack.monitoring.collection.enabled": false } }
-
If the Elasticsearch security features are enabled on the monitoring cluster, create users that can send and retrieve monitoring data.
If you plan to use Kibana to view monitoring data, username and password credentials must be valid on both the Kibana server and the monitoring cluster.
-
If you plan to use Metricbeat to collect data about Elasticsearch or Kibana,
create a user that has the
remote_monitoring_collector
built-in role and a user that has theremote_monitoring_agent
built-in role. Alternatively, use theremote_monitoring_user
built-in user. -
If you plan to use HTTP exporters to route data through your production cluster, create a user that has the
remote_monitoring_agent
built-in role.For example, the following request creates a
remote_monitor
user that has theremote_monitoring_agent
role:POST /_security/user/remote_monitor { "password" : "changeme", "roles" : [ "remote_monitoring_agent"], "full_name" : "Internal Agent For Remote Monitoring" }
Alternatively, use the
remote_monitoring_user
built-in user.
-
If you plan to use Metricbeat to collect data about Elasticsearch or Kibana,
create a user that has the
-
Configure your production cluster to collect data and send it to the monitoring cluster:
- (Optional) Configure Logstash to collect data and send it to the monitoring cluster.
-
(Optional) Configure the Beats to collect data and send it to the monitoring cluster.
-
(Optional) Configure Kibana to collect data and send it to the monitoring cluster:
-
(Optional) Create a dedicated Kibana instance for monitoring, rather than using a single Kibana instance to access both your production cluster and monitoring cluster.
-
(Optional) Disable the collection of monitoring data in this Kibana instance.
Set the
xpack.monitoring.kibana.collection.enabled
setting tofalse
in thekibana.yml
file. For more information about this setting, see Monitoring settings in Kibana.
-
(Optional) Disable the collection of monitoring data in this Kibana instance.
Set the
- Configure Kibana to retrieve and display the monitoring data.