It is time to say goodbye: This version of Elastic Cloud Enterprise has reached end-of-life (EOL) and is no longer supported.

The documentation for this version is no longer being maintained. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.

« Manage your APM Server Enable Graph (versions before 5.0) »
Elastic Docs ›Elastic Cloud Enterprise Reference [2.3] ›Administering deployments

Enable Monitoring (formerly Marvel)

edit

Enable Monitoring (formerly Marvel)

edit

The X-Pack monitoring features let you monitor Elasticsearch through Kibana. You can view your cluster’s health and performance in real time and analyze past cluster, index, and node metrics. In Elasticsearch versions before 5.0, Marvel provides similar monitoring functionality.

In Elasticsearch 5.0, the monitoring features of Marvel became part of X-Pack. If you are using an Elasticsearch version before 5.0, think Marvel whenever you read about the X-Pack monitoring features.

Monitoring consists of two components:

  • A Monitoring agent that is installed on each node in your cluster. The Monitoring agent collects and indexes metrics from Elasticsearch, either on the same cluster or by sending metrics to an external monitoring cluster. Elastic Cloud Enterprise manages the installation and configuration of the monitoring agent for you, and you should not modify any of the settings.
  • The Monitoring (formerly Marvel) application plugin in Kibana that visualizes the monitoring metrics through a dashboard.

The steps in this section cover only the enablement of monitoring. For more information on how to use monitoring itself, see Monitor a cluster (for 6.3 and later), Monitoring the Elastic Stack (for 5.0 to 6.2), or the Marvel documentation if you are using an Elasticsearch version before 5.0.

Monitoring for production use

edit

For production use, you should log metrics for clusters to a dedicated monitoring deployment (but never to the logging-and-metrics cluster that is used by ECE). Monitoring indexes metrics into Elasticsearch and these indexes consume storage, memory, and CPU cycles like any other index. By using a separate monitoring cluster, you avoid affecting your production clusters.

You should also create a dedicated user for the clusters sending metrics and the monitoring cluster receiving them. For more information on creating a user with the right privileges, see Monitoring and security (for version 6.3 and later), Monitoring and security (for version 5.0 to 6.2), and Using Marvel with Shield (for versions before 5.0).

How many monitoring clusters you use depends on your requirements:

  • You can ship metrics for many clusters to a single monitoring cluster if your business requirements permit it.
  • While monitoring will work with a cluster running a single node, you need a minimum of three monitoring nodes to make monitoring highly available.

  • You might need to create dedicated monitoring clusters for isolation purposes in some cases. For example:

    • If you have many clusters and some of them are much larger than others, creating separate monitoring clusters prevents a large cluster from potentially affecting monitoring performance for smaller clusters.
    • If you need to silo Elasticsearch data for different business departments. Clusters that have been configured to ship metrics to a target monitoring cluster have access to indexing data and can manage monitoring index templates, which is addressed by creating separate monitoring clusters.

Monitoring indices that get sent to a monitoring cluster are not cleaned up automatically. You can use Curator to clean up these monitoring indices, like any other time-based index.

To avoid compatibility issues between versions, the cluster sending monitoring metrics and the monitoring cluster receiving them should be at the same Elasticsearch version. If using the same version is not feasible, check for breaking changes in the X-Pack Release Notes or the Marvel Release Notes to make sure that your versions are compatible.

Configure where monitoring data is sent

edit

When you enable monitoring on a cluster, you are configuring where the monitoring agent for your current cluster should send its metrics.

There are some prerequisites to keep in mind:

  • Both the cluster that is sending monitoring metrics and the monitoring cluster that receives the metrics must be configured to use Security (formerly Shield).

  • Only monitoring clusters that are at a compatible version are shown in the Cloud UI. The following versions are compatible:

    • 2.0 - 2.2
    • 2.3 - 2.4

    • 5.x - 5.x

      For example: An Elasticsearch cluster on version 2.1.3 can be configured to send metrics to a monitoring cluster on version 2.2.0, but it cannot send metrics to a monitoring cluster on version 2.3.4. The search results will show only the compatible cluster on version 2.2.0.

To make sure that monitoring data continues to be sent, we recommend that you keep the monitoring cluster at a version level that is equal to or ahead of the production cluster.

To enable monitoring, you need to:

  1. Log into the Cloud UI.

  2. From the Deployments page, select your deployment where you want to enable monitoring.

    Narrow the list by name, ID, or choose from several other filters. To further define the list, use a combination of filters.

  3. From the Elasticsearch page, click Enable monitoring.

  4. Enter the name of the cluster where you would like the monitoring data sent and save your changes.

    If a cluster is not listed, make sure that it is running a compatible version and is configured to use Security.

    Remember to send metrics for production clusters to a dedicated monitoring cluster (but never to the logging-and-metrics cluster that is used by ECE).

To work with the monitoring metrics, access the Monitoring application in Kibana.

« Manage your APM Server Enable Graph (versions before 5.0) »