IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« Monitoring in a production environment Collecting monitoring data »

› ›

Elastic Stack Monitoring Service

edit

Elastic Stack Monitoring Service

edit

We have stopped adding new customers to our Elastic Stack Monitoring Service.

If you are interested in similar capabilities, contact Elastic Support to discuss available options.

The Elastic Stack Monitoring Service (ESMS) is a monitoring cluster on Elastic Cloud. Elastic provides and maintains ESMS for self-managed commercial customers. If you send your monitoring data to ESMS, it can also be used by Elastic support to provide better and faster incident resolution.

Collecting monitoring data about Elasticsearch

edit

There are two methods for collecting and sending data about the health of your production cluster to ESMS:

Metricbeat
collectors and exporters

If you want to monitor Logstash, you must use collectors and exporters to route data from the production cluster to ESMS. Otherwise, it is simplest to use Metricbeat.

To use Metricbeat:

Enable the collection of monitoring data on your cluster.

Set xpack.monitoring.collection.enabled to true on the production cluster. By default, it is is disabled (false).

You can use the following APIs to review and change this setting:
```
GET _cluster/settings

PUT _cluster/settings
{
  "persistent": {
    "xpack.monitoring.collection.enabled": true
  }
}
```
If Elasticsearch security features are enabled, you must have monitor cluster privileges to view the cluster settings and manage cluster privileges to change them.

For more information about these settings, see Monitoring settings.
Disable the default collection of Elasticsearch monitoring metrics.

Set xpack.monitoring.elasticsearch.collection.enabled to false on the production cluster.

You can use the following API to change this setting:
```
PUT _cluster/settings
{
  "persistent": {
    "xpack.monitoring.elasticsearch.collection.enabled": false
  }
}
```
If Elasticsearch security features are enabled, you must have monitor cluster privileges to view the cluster settings and manage cluster privileges to change them.
Install Metricbeat on each node.
Enable the Elasticsearch module in Metricbeat on each node.

For example, to enable the default configuration in the modules.d directory, run the following command:
```
metricbeat modules enable elasticsearch
```
For more information, see Specify which modules to run and Elasticsearch module.
Configure the Elasticsearch module in Metricbeat on each node.

For example, specify the following settings in the modules.d/elasticsearch.yml file:
```
- module: elasticsearch
  metricsets:
    - ccr
    - cluster_stats
    - index
    - index_recovery
    - index_summary
    - ml_job
    - node_stats
    - shard
  period: 10s
  hosts: ["http://localhost:9200"]
  #username: "user"
  #password: "secret"
  xpack.enabled: true
```
By default, the module collects Elasticsearch monitoring metrics from http://localhost:9200. If that host and port number are not correct, you must update the hosts setting. If you configured Elasticsearch to use encrypted communications, you must access it via HTTPS. For example, use a hosts setting like https://localhost:9200.

If Elastic security features are enabled, you must also provide a user ID and password so that Metricbeat can collect metrics successfully:
1. Create a user on the production cluster that has the remote_monitoring_collector built-in role. Alternatively, use the remote_monitoring_user built-in user.
2. Add the username and password settings to the Elasticsearch module configuration file.
Optional: Disable the system module in Metricbeat.

By default, the system module is enabled. The information it collects, however, is not shown on the Monitoring page in Kibana. Unless you want to use that information for other purposes, run the following command:
```
metricbeat modules disable system
```

Identify where to send the Elasticsearch monitoring data and supply the necessary security information. Add the following settings in the Metricbeat configuration file (metricbeat.yml):

output.elasticsearch:
  hosts: ["MONITORING_ELASTICSEARCH_URL"] 
  username: cloud_monitoring_agent 
  password: MONITORING_AGENT_PASSWORD

	Replace `MONITORING_ELASTICSEARCH_URL` with the appropriate URL for ESMS.
	The Elastic support team creates this user in ESMS and grants it the `remote_monitoring_agent` built-in role.
	Replace `MONITORING_AGENT_PASSWORD` with the value provided to you by the Elastic support team.

Start Metricbeat.
Verify that your monitoring data exists in ESMS.

Open Kibana in your web browser. Use the Kibana URL and the administrator user ID that was provided to you by the Elastic support team. View the Elasticsearch metrics on the Monitoring page.

If you do not see your metrics yet, see Troubleshooting monitoring features.

Collecting monitoring data about Kibana

edit

There are two methods for sending monitoring data about Kibana to ESMS. You can send it directly to ESMS by using Metricbeat or you can route it through exporters on the production cluster.

It is simplest to use Metricbeat.

To use Metricbeat:

Disable the default collection of Kibana monitoring metrics.

Add the following setting in the Kibana configuration file (kibana.yml):
```
xpack.monitoring.kibana.collection.enabled: false
```
Leave the xpack.monitoring.enabled set to its default value (true).

For more information, see Monitoring settings in Kibana.
Start Kibana.
Ensure that the xpack.monitoring.collection.enabled setting is true on each node in the production cluster.
Install Metricbeat on the same server as Kibana.
Enable the Kibana module in Metricbeat.

For example, to enable the default configuration in the modules.d directory, run the following command:
```
metricbeat modules enable kibana
```
For more information, see Specify which modules to run and Kibana module.
Configure the Kibana module in Metricbeat.

For example, specify the following settings in the modules.d/kibana.yml file:
```
- module: kibana
  metricsets:
    - stats
  period: 10s
  hosts: ["http://localhost:5601"]
  xpack.enabled: true
  #username: "user"
  #password: "secret"
```
By default, the module collects Kibana monitoring metrics from localhost:5601. If the host and port number are not correct, you must update the hosts setting. . If you configured Kibana to use encrypted communications, you must access it via HTTPS. For example, use a hosts setting like https://localhost:5601.

If the Elastic security features are enabled, you must also provide a user ID and password so that Metricbeat can collect metrics successfully.
1. Create a user on the production cluster that has the remote_monitoring_collector built-in role. Alternatively, use the remote_monitoring_user built-in user.
2. Add the username and password settings to the Kibana module configuration file.
Optional: Disable the system module in Metricbeat.

By default, the system module is enabled. The information it collects, however, is not shown on the Monitoring page in Kibana. Unless you want to use that information for other purposes, run the following command:
```
metricbeat modules disable system
```

Identify where to send the Kibana monitoring data and supply the necessary security information. Add the following settings in the Metricbeat configuration file (metricbeat.yml):

  output.elasticsearch:
    hosts: ["MONITORING_ELASTICSEARCH_URL"] 
    username: cloud_monitoring_agent 
    password: MONITORING_AGENT_PASSWORD

	Replace `MONITORING_ELASTICSEARCH_URL` with the appropriate URL for ESMS, which was provided by the Elastic support team.
	The Elastic support team creates this user in ESMS and grants it the `remote_monitoring_agent` built-in role.
	Replace `MONITORING_AGENT_PASSWORD` with the value provided to you by the Elastic support team.

Start Metricbeat.
Verify that your monitoring data exists in ESMS.

Open Kibana in your web browser. Use the Kibana URL and the administrator user ID that was provided to you by the Elastic support team. View the Kibana metrics on the Monitoring page.

If you do not see your metrics yet, see Troubleshooting monitoring features.

« Monitoring in a production environment Collecting monitoring data »