IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« Machine learning settings in Elasticsearch Security settings in Elasticsearch »

› › ›

Monitoring settings in Elasticsearch

edit

Monitoring settings in Elasticsearch

edit

By default, monitoring is enabled but data collection is disabled. To enable data collection, use the xpack.monitoring.collection.enabled setting.

You can configure these monitoring settings in the elasticsearch.yml file. You can also dynamically set some of these settings using the cluster update settings API.

Cluster settings take precedence over settings in the elasticsearch.yml file.

To adjust how monitoring data is displayed in the monitoring UI, configure xpack.monitoring settings in kibana.yml. To control how monitoring data is collected from Logstash, configure xpack.monitoring settings in logstash.yml.

For more information, see Monitor a cluster.

General Monitoring Settings

edit

xpack.monitoring.enabled: Set to true (default) to enable Elasticsearch X-Pack monitoring for Elasticsearch on the node.

To enable data collection, you must also set xpack.monitoring.collection.enabled to true. Its default value is false.

Monitoring Collection Settings

edit

The xpack.monitoring.collection settings control how data is collected from your Elasticsearch nodes. You can dynamically change all monitoring collection settings using the cluster update settings API.

xpack.monitoring.collection.enabled (Dynamic)

[6.3.0] Added in 6.3.0. Set to true to enable the collection of monitoring data. When this setting is false (default), Elasticsearch monitoring data is not collected and all monitoring data from other sources such as Kibana, Beats, and Logstash is ignored.

xpack.monitoring.collection.interval (Dynamic)

Setting to -1 to disable data collection is no longer supported beginning with 7.0.0. [6.3.0] Deprecated in 6.3.0. Use xpack.monitoring.collection.enabled set to false instead.

Controls how often data samples are collected. Defaults to 10s. If you modify the collection interval, set the xpack.monitoring.min_interval_seconds option in kibana.yml to the same value.

xpack.monitoring.elasticsearch.collection.enabled (Dynamic)

Controls whether statistics about your Elasticsearch cluster should be collected. Defaults to true. This is different from xpack.monitoring.collection.enabled, which allows you to enable or disable all monitoring collection. However, this setting simply disables the collection of Elasticsearch data while still allowing other data (e.g., Kibana, Logstash, Beats, or APM Server monitoring data) to pass through this cluster.

xpack.monitoring.collection.cluster.stats.timeout (Dynamic)

(time value) Timeout for collecting the cluster statistics. Defaults to 10s.

xpack.monitoring.collection.node.stats.timeout (Dynamic)

(time value) Timeout for collecting the node statistics. Defaults to 10s.

xpack.monitoring.collection.indices (Dynamic)

Controls which indices Monitoring collects data from. Defaults to all indices. Specify the index names as a comma-separated list, for example test1,test2,test3. Names can include wildcards, for example test*. You can explicitly exclude indices by prepending -. For example test*,-test3 will monitor all indexes that start with test except for test3. System indices like .security* or .kibana* always start with a ., and generally should be monitored. Consider adding .* to the list of indices ensure monitoring of system indices. For example .*,test*,-test3

xpack.monitoring.collection.index.stats.timeout (Dynamic)

(time value) Timeout for collecting index statistics. Defaults to 10s.

xpack.monitoring.collection.index.recovery.active_only (Dynamic)

Controls whether or not all recoveries are collected. Set to true to collect only active recoveries. Defaults to false.

xpack.monitoring.collection.index.recovery.timeout (Dynamic)

(time value) Timeout for collecting the recovery information. Defaults to 10s.

xpack.monitoring.history.duration (Dynamic)

(time value) Retention duration beyond which the indices created by a Monitoring exporter are automatically deleted. Defaults to 7d (7 days).

This setting has a minimum value of 1d (1 day) to ensure that something is being monitored, and it cannot be disabled.

This setting currently only impacts local-type exporters. Indices created using the http exporter will not be deleted automatically.

xpack.monitoring.exporters

Configures where the agent stores monitoring data. By default, the agent uses a local exporter that indexes monitoring data on the cluster where it is installed. Use an HTTP exporter to send data to a separate monitoring cluster. For more information, see local exporter settings, HTTP exporter settings, and How it works.

Local Exporter Settings

edit

The local exporter is the default exporter used by Monitoring. As the name is meant to imply, it exports data to the local cluster, which means that there is not much needed to be configured.

If you do not supply any exporters, then Monitoring will automatically create one for you. If any exporter is provided, then no default is added.

xpack.monitoring.exporters.my_local:
  type: local

type: The value for a Local exporter must always be local and it is required.
use_ingest: Whether to supply a placeholder pipeline to the cluster and a pipeline processor with every bulk request. The default value is true. If disabled, then it means that it will not use pipelines, which means that a future release cannot automatically upgrade bulk requests to future-proof them.
cluster_alerts.management.enabled: Whether to create cluster alerts for this cluster. The default value is true. To use this feature, Watcher must be enabled. If you have a basic license, cluster alerts are not displayed.

HTTP Exporter Settings

edit

The following lists settings that can be supplied with the http exporter. All settings are shown as what follows the name you select for your exporter:

xpack.monitoring.exporters.my_remote:
  type: http
  host: ["host:port", ...]

type

The value for an HTTP exporter must always be http and it is required.

host

Host supports multiple formats, both as an array or as a single value. Supported formats include hostname, hostname:port, http://hostname http://hostname:port, https://hostname, and https://hostname:port. Hosts cannot be assumed. The default scheme is always http and the default port is always 9200 if not supplied as part of the host string.

xpack.monitoring.exporters:
  example1:
    type: http
    host: "10.1.2.3"
  example2:
    type: http
    host: ["http://10.1.2.4"]
  example3:
    type: http
    host: ["10.1.2.5", "10.1.2.6"]
  example4:
    type: http
    host: ["https://10.1.2.3:9200"]

auth.username

The username is required if a auth.password is supplied.

auth.password

The password for the auth.username.

connection.timeout

(time value) Amount of time that the HTTP connection is supposed to wait for a socket to open for the request. The default value is 6s.

connection.read_timeout

(time value) Amount of time that the HTTP connection is supposed to wait for a socket to send back a response. The default value is 10 * connection.timeout (60s if neither are set).

ssl

Each HTTP exporter can define its own TLS / SSL settings or inherit them. See the TLS / SSL section below.

proxy.base_path

The base path to prefix any outgoing request, such as /base/path (e.g., bulk requests would then be sent as /base/path/_bulk). There is no default value.

headers

Optional headers that are added to every request, which can assist with routing requests through proxies.

xpack.monitoring.exporters.my_remote:
  headers:
    X-My-Array: [abc, def, xyz]
    X-My-Header: abc123

Array-based headers are sent n times where n is the size of the array. Content-Type and Content-Length cannot be set. Any headers created by the Monitoring agent will override anything defined here.

index.name.time_format

A mechanism for changing the default date suffix for the, by default, daily Monitoring indices. The default value is yyyy.MM.dd, which is why the indices are created daily.

use_ingest

Whether to supply a placeholder pipeline to the monitoring cluster and a pipeline processor with every bulk request. The default value is true. If disabled, then it means that it will not use pipelines, which means that a future release cannot automatically upgrade bulk requests to future-proof them.

cluster_alerts.management.enabled

Whether to create cluster alerts for this cluster. The default value is true. To use this feature, Watcher must be enabled. If you have a basic license, cluster alerts are not displayed.

cluster_alerts.management.blacklist

Prevents the creation of specific cluster alerts. It also removes any applicable watches that already exist in the current cluster.

You can add any of the following watch identifiers to the blacklist:

elasticsearch_cluster_status
elasticsearch_version_mismatch
elasticsearch_nodes
kibana_version_mismatch
logstash_version_mismatch
xpack_license_expiration

For example: ["elasticsearch_version_mismatch","xpack_license_expiration"].

X-Pack monitoring TLS/SSL Settings

edit

You can configure the following TLS/SSL settings. If the settings are not configured, the Default TLS/SSL Settings are used.

xpack.monitoring.exporters.$NAME.ssl.supported_protocols: Supported protocols with versions. Valid protocols: SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2, TLSv1.3. Defaults to TLSv1.3,TLSv1.2,TLSv1.1 if the JVM supports TLSv1.3, otherwise TLSv1.2,TLSv1.1.
xpack.monitoring.exporters.$NAME.ssl.verification_mode: Controls the verification of certificates. Valid values are none, certificate, and full. Defaults to full.
xpack.monitoring.exporters.$NAME.ssl.cipher_suites: Supported cipher suites can be found in Oracle’s Java Cryptography Architecture documentation. Defaults to ``.

X-Pack monitoring TLS/SSL Key and Trusted Certificate Settings

edit

The following settings are used to specify a private key, certificate, and the trusted certificates that should be used when communicating over an SSL/TLS connection. A private key and certificate are optional and would be used if the server requires client authentication for PKI authentication. If none of the settings below are specified, the Default TLS/SSL Settings are used.

PEM Encoded Files

edit

When using PEM encoded files, use the following settings:

xpack.monitoring.exporters.$NAME.ssl.key: Path to a PEM encoded file containing the private key.
xpack.monitoring.exporters.$NAME.ssl.key_passphrase: The passphrase that is used to decrypt the private key. This value is optional as the key might not be encrypted.
xpack.monitoring.exporters.$NAME.ssl.secure_key_passphrase (Secure): The passphrase that is used to decrypt the private key. This value is optional as the key might not be encrypted.
xpack.monitoring.exporters.$NAME.ssl.certificate: Path to a PEM encoded file containing the certificate (or certificate chain) that will be presented when requested.
xpack.monitoring.exporters.$NAME.ssl.certificate_authorities: List of paths to the PEM encoded certificate files that should be trusted.

Java Keystore Files

edit

When using Java keystore files (JKS), which contain the private key, certificate and certificates that should be trusted, use the following settings:

xpack.monitoring.exporters.$NAME.ssl.keystore.path: Path to the keystore that holds the private key and certificate.
xpack.monitoring.exporters.$NAME.ssl.keystore.password: Password to the keystore.
xpack.monitoring.exporters.$NAME.ssl.keystore.secure_password (Secure): Password to the keystore.
xpack.monitoring.exporters.$NAME.ssl.keystore.key_password: Password for the private key in the keystore. Defaults to the same value as xpack.monitoring.exporters.$NAME.ssl.keystore.password.
xpack.monitoring.exporters.$NAME.ssl.keystore.secure_key_password (Secure): Password for the private key in the keystore.
xpack.monitoring.exporters.$NAME.ssl.truststore.path: Path to the truststore file.
xpack.monitoring.exporters.$NAME.ssl.truststore.password: Password to the truststore.
xpack.monitoring.exporters.$NAME.ssl.truststore.secure_password (Secure): Password to the truststore.

PKCS#12 Files

edit

Elasticsearch can be configured to use PKCS#12 container files (.p12 or .pfx files) that contain the private key, certificate and certificates that should be trusted.

PKCS#12 files are configured in the same way as Java Keystore Files:

xpack.monitoring.exporters.$NAME.ssl.keystore.path: Path to the PKCS#12 file that holds the private key and certificate.
xpack.monitoring.exporters.$NAME.ssl.keystore.type: Set this to PKCS12 to indicate that the keystore is a PKCS#12 file.
xpack.monitoring.exporters.$NAME.ssl.keystore.password: Password to the PKCS#12 file.
xpack.monitoring.exporters.$NAME.ssl.keystore.secure_password (Secure): Password to the PKCS#12 file.
xpack.monitoring.exporters.$NAME.ssl.keystore.key_password: Password for the private key stored in the PKCS#12 file. Defaults to the same value as xpack.monitoring.exporters.$NAME.ssl.keystore.password.
xpack.monitoring.exporters.$NAME.ssl.keystore.secure_key_password (Secure): Password for the private key stored in the PKCS#12 file.
xpack.monitoring.exporters.$NAME.ssl.truststore.path: Path to the PKCS#12 file that holds the certificates to be trusted.
xpack.monitoring.exporters.$NAME.ssl.truststore.type: Set this to PKCS12 to indicate that the truststore is a PKCS#12 file.
xpack.monitoring.exporters.$NAME.ssl.truststore.password: Password to the PKCS#12 file.
xpack.monitoring.exporters.$NAME.ssl.truststore.secure_password (Secure): Password to the PKCS#12 file.

PKCS#11 Tokens

edit

Elasticsearch can be configured to use a PKCS#11 token that contains the private key, certificate and certificates that should be trusted.

PKCS#11 token require additional configuration on the JVM level and can be enabled via the following settings:

xpack.monitoring.exporters.$NAME.keystore.type: Set this to PKCS11 to indicate that the PKCS#11 token should be used as a keystore.
xpack.monitoring.exporters.$NAME.truststore.type: Set this to PKCS11 to indicate that the PKCS#11 token should be used as a truststore.

When configuring the PKCS#11 token that your JVM is configured to use as a keystore or a truststore for Elasticsearch, the PIN for the token can be configured by setting the appropriate value to ssl.truststore.password or ssl.truststore.secure_password in the context that you are configuring. Since there can only be one PKCS#11 token configured, only one keystore and truststore will be usable for configuration in Elasticsearch. This in turn means that only one certificate can be used for TLS both in the transport and the http layer.

« Machine learning settings in Elasticsearch Security settings in Elasticsearch »