Elastic Stack Monitoring Service
editElastic Stack Monitoring Service
editWe have stopped adding new customers to our Elastic Stack Monitoring Service.
If you are interested in similar capabilities, contact Elastic Support to discuss available options.
The Elastic Stack Monitoring Service (ESMS) is a monitoring cluster on Elastic Cloud. Elastic provides and maintains ESMS for self-managed commercial customers. If you send your monitoring data to ESMS, it can also be used by Elastic support to provide better and faster incident resolution.
Collecting monitoring data about Elasticsearch
editThere are two methods for collecting and sending data about the health of your production cluster to ESMS:
- Metricbeat
- collectors and exporters
To use Metricbeat:
-
Enable the collection of monitoring data on your cluster.
Set
xpack.monitoring.collection.enabled
totrue
on the production cluster. By default, it is is disabled (false
).You can use the following APIs to review and change this setting:
GET _cluster/settings PUT _cluster/settings { "persistent": { "xpack.monitoring.collection.enabled": true } }
If Elasticsearch security features are enabled, you must have
monitor
cluster privileges to view the cluster settings andmanage
cluster privileges to change them.For more information about these settings, see Monitoring settings.
-
Disable the default collection of Elasticsearch monitoring metrics.
Set
xpack.monitoring.elasticsearch.collection.enabled
tofalse
on the production cluster.You can use the following API to change this setting:
PUT _cluster/settings { "persistent": { "xpack.monitoring.elasticsearch.collection.enabled": false } }
If Elasticsearch security features are enabled, you must have
monitor
cluster privileges to view the cluster settings andmanage
cluster privileges to change them. - Install Metricbeat on each node.
-
Enable the Elasticsearch X-Pack module in Metricbeat on each node.
For example, to enable the default configuration in the
modules.d
directory, run the following command:metricbeat modules enable elasticsearch-xpack
For more information, see Specify which modules to run and Elasticsearch module.
-
Configure the Elasticsearch X-Pack module in Metricbeat on each node.
The
modules.d/elasticsearch-xpack.yml
file contains the following settings:- module: elasticsearch metricsets: - ccr - cluster_stats - index - index_recovery - index_summary - ml_job - node_stats - shard - enrich period: 10s hosts: ["http://localhost:9200"] #username: "user" #password: "secret" xpack.enabled: true
By default, the module collects Elasticsearch monitoring metrics from
http://localhost:9200
. If that host and port number are not correct, you must update thehosts
setting. If you configured Elasticsearch to use encrypted communications, you must access it via HTTPS. For example, use ahosts
setting likehttps://localhost:9200
.If Elastic security features are enabled, you must also provide a user ID and password so that Metricbeat can collect metrics successfully:
-
Create a user on the production cluster that has the
remote_monitoring_collector
built-in role. Alternatively, use theremote_monitoring_user
built-in user. -
Add the
username
andpassword
settings to the Elasticsearch module configuration file.
-
Create a user on the production cluster that has the
-
Optional: Disable the system module in Metricbeat.
By default, the system module is enabled. The information it collects, however, is not shown on the Monitoring page in Kibana. Unless you want to use that information for other purposes, run the following command:
metricbeat modules disable system
-
Identify where to send the Elasticsearch monitoring data and supply the necessary security information.
Add the following settings in the Metricbeat configuration file (
metricbeat.yml
):output.elasticsearch: hosts: ["MONITORING_ELASTICSEARCH_URL"] username: cloud_monitoring_agent password: MONITORING_AGENT_PASSWORD
Replace
MONITORING_ELASTICSEARCH_URL
with the appropriate URL for ESMS, which was provided by the Elastic support team.The Elastic support team creates this user in ESMS and grants it the
remote_monitoring_agent
built-in role.Replace
MONITORING_AGENT_PASSWORD
with the value provided to you by the Elastic support team. - Start Metricbeat.
-
Verify that your monitoring data exists in ESMS.
Open Kibana in your web browser. Use the Kibana URL and the administrator user ID that was provided to you by the Elastic support team. View the Elasticsearch metrics on the Stack Monitoring page.
If you do not see your metrics yet, see Troubleshooting monitoring features.
Collecting monitoring data about Kibana
editThere are two methods for sending monitoring data about Kibana to ESMS. You can send it directly to ESMS by using Metricbeat or you can route it through exporters on the production cluster.
To use Metricbeat:
-
Disable the default collection of Kibana monitoring metrics.
Add the following setting in the Kibana configuration file (
kibana.yml
):xpack.monitoring.kibana.collection.enabled: false
Leave the
xpack.monitoring.enabled
set to its default value (true
).For more information, see Monitoring settings in Kibana.
- Start Kibana.
-
Ensure that the
xpack.monitoring.collection.enabled
setting istrue
on each node in the production cluster. - Install Metricbeat on the same server as Kibana.
-
Enable the Kibana X-Pack module in Metricbeat.
For example, to enable the default configuration in the
modules.d
directory, run the following command:metricbeat modules enable kibana-xpack
For more information, see Specify which modules to run and Kibana module.
-
Configure the Kibana X-Pack module in Metricbeat.
The
modules.d/kibana-xpack.yml
file contains the following settings:- module: kibana metricsets: - stats period: 10s hosts: ["localhost:5601"] #basepath: "" #username: "user" #password: "secret" xpack.enabled: true
By default, the module collects Kibana monitoring metrics from
localhost:5601
. If that host and port number are not correct, you must update thehosts
setting. If you configured Kibana to use encrypted communications, you must access it via HTTPS. For example, use ahosts
setting likehttps://localhost:5601
.If the Elastic security features are enabled, you must also provide a user ID and password so that Metricbeat can collect metrics successfully:
-
Create a user on the production cluster that has the
remote_monitoring_collector
built-in role. Alternatively, use theremote_monitoring_user
built-in user. -
Add the
username
andpassword
settings to the Kibana module configuration file.
-
Create a user on the production cluster that has the
-
Optional: Disable the system module in Metricbeat.
By default, the system module is enabled. The information it collects, however, is not shown on the Monitoring page in Kibana. Unless you want to use that information for other purposes, run the following command:
metricbeat modules disable system
-
Identify where to send the Kibana monitoring data and supply the necessary security information.
Add the following settings in the Metricbeat configuration file (
metricbeat.yml
):output.elasticsearch: hosts: ["MONITORING_ELASTICSEARCH_URL"] username: cloud_monitoring_agent password: MONITORING_AGENT_PASSWORD
Replace
MONITORING_ELASTICSEARCH_URL
with the appropriate URL for ESMS, which was provided by the Elastic support team.The Elastic support team creates this user in ESMS and grants it the
remote_monitoring_agent
built-in role.Replace
MONITORING_AGENT_PASSWORD
with the value provided to you by the Elastic support team. - Start Metricbeat.
-
Verify that your monitoring data exists in ESMS.
Open Kibana in your web browser. Use the Kibana URL and the administrator user ID that was provided to you by the Elastic support team. View the Kibana metrics on the Stack Monitoring page.
If you do not see your metrics yet, see Troubleshooting monitoring features.
Collecting monitoring data about Beats
editThere are two methods for sending monitoring data about Beats to ESMS. You can send it directly to ESMS by using Metricbeat or you can route it through exporters on the production cluster.
For example, to use Metricbeat to monitor Packetbeat:
-
Enable the HTTP endpoint to allow external collection of monitoring data:
Add the following setting in the Packetbeat configuration file (
packetbeat.yml
):http.enabled: true
By default, metrics are exposed on port 5066. If you need to monitor multiple Beats shippers running on the same server, set
http.port
to expose metrics for each shipper on a different port number:http.port: 5067
-
Disable the default collection of Packetbeat monitoring metrics.
Add the following setting in the Packetbeat configuration file (
packetbeat.yml
):monitoring.enabled: false
- Start Packetbeat.
- Install Metricbeat on the same server as Packetbeat. If you already have Metricbeat installed on the server, skip this step.
-
Enable the
beat-xpack
module in Metricbeat.For example, to enable the default configuration in the
modules.d
directory, run the following command, using the correct command syntax for your OS:metricbeat modules enable beat-xpack
For more information, see Specify which modules to run and beat module.
-
Configure the
beat-xpack
module in Metricbeat.The
modules.d/beat-xpack.yml
file contains the following settings:- module: beat metricsets: - stats - state period: 10s hosts: ["http://localhost:5066"] #username: "user" #password: "secret" xpack.enabled: true
Set the
hosts
,username
, andpassword
settings as required by your environment. For other module settings, it’s recommended that you accept the defaults.By default, the module collects Packetbeat monitoring data from
localhost:5066
. If you exposed the metrics on a different host or port when you enabled the HTTP endpoint, update thehosts
setting.To monitor multiple Beats agents or APM Server instances, specify a list of hosts, for example:
hosts: ["http://localhost:5066","http://localhost:5067","http://localhost:5068"]
If you configured Packetbeat to use encrypted communications, you must access it via HTTPS. For example, use a
hosts
setting likehttps://localhost:5066
.If the Elastic security features are enabled, you must also provide a user ID and password so that Metricbeat can collect metrics successfully:
-
Create a user on the production cluster that has the
remote_monitoring_collector
built-in role. Alternatively, if it’s available in your environment, use theremote_monitoring_user
built-in user. -
Add the
username
andpassword
settings to the beat module configuration file.
-
Create a user on the production cluster that has the
-
Optional: Disable the system module in the Metricbeat.
By default, the system module is enabled. The information it collects, however, is not shown on the Stack Monitoring page in Kibana. Unless you want to use that information for other purposes, run the following command:
metricbeat modules disable system
-
Identify where to send the monitoring data and supply the necessary security information.
Add the following settings in the Metricbeat configuration file (
metricbeat.yml
):output.elasticsearch: hosts: ["MONITORING_ELASTICSEARCH_URL"] username: cloud_monitoring_agent password: MONITORING_AGENT_PASSWORD
Replace
MONITORING_ELASTICSEARCH_URL
with the appropriate URL for ESMS, which was provided by the Elastic support team.The Elastic support team creates this user in ESMS and grants it the
remote_monitoring_agent
built-in role.Replace
MONITORING_AGENT_PASSWORD
with the value provided to you by the Elastic support team. - Start Metricbeat.
-
Verify that your monitoring data exists in ESMS.
Open Kibana in your web browser. Use the Kibana URL and the administrator user ID that was provided to you by the Elastic support team. View the Beats metrics on the Stack Monitoring page.
If you do not see your metrics yet, see Troubleshooting monitoring features.
For more information about monitoring Beats, see: