IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« View and query logs Troubleshooting »
Elastic Docs ›

Monitoring

edit

Monitoring

edit

When deploying an Enterprise Search cluster into production, we highly recommend setting up some kind of monitoring for it to help you more effectively troubleshoot any issues with the deployment, properly scale the system, etc.

In this guide we will cover different options you have for setting up monitoring and for troubleshooting any issues with your systems.

Overview

edit

There are multiple features available in Enterprise Search and the Elastic Stack to help you monitor and troubleshoot your Enterprise Search deployments.

  • Monitoring APIs – Enterprise Search provides a number of APIs that can be used to get access to internal system metrics, product usage information and other data that may be useful for understanding the health of your deployment. These APIs can be accessed via our official API clients or with any HTTP-enabled metrics collection agent.
  • Enterprise Search Self-Monitoring – you can enable a self-monitoring feature within Enterprise Search, which will automatically ship metrics information to the underlying Elasticsearch deployment, allowing you to access it via Kibana dashboards.
  • External Metricbeat Monitoring – you can use the official Metricbeat module for Enterprise Search to collect monitoring information from your deployments and index them into any Elasticsearch cluster for future access via Kibana dashboards.
  • Elastic APM Integration – you can enable Elastic APM agent built into Enterprise Search and ship a lot of instrumentation data to a given Elastic APM deployment for future analysis via the APM application in Kibana.

Enterprise Search self-monitoring

edit

Starting with Enterprise search 7.16, you can enable monitoring information reporting by setting this Enterprise Search configuration setting and restarting your deployment. This feature works for both self-managed and for Elastic Cloud deployments.

+

monitoring.reporting_enabled: true

After self-monitoring is enabled, Enterprise Search will automatically provision a Metricbeat instance with the enterprisesearch module and will start reporting a detailed set of monitoring metrics to the underlying Elasticsearch cluster every 10 seconds to be stored in a set of indexes called .monitoring-ent-search-*.

Metricbeat indexes will have ILM enabled on them so the data will be stored in Elasticsearch for up to 30 days and indexes will be rotated every 100Mb.

You cannot control the ent-search-metricbeat ILM policy at the moment. Even if you use the Kibana UI or Elasticsearch API to directly update the policy, it will get overwritten by Enterprise Search.

If you need full control of the metricbeat ILM policies for your Enterprise Search monitoring data, please deploy a standalone Metricbeat instance.

The metrics in Elasticsearch can be then accessed via Kibana dashboards and visualizations. See the dedicated section below for more information on Kibana dashboards available for you.

For more information on the settings that can be used in your deployment configuration to control the frequency of monitoring data reporting, index names used and other aspects of the self-monitoring feature within Enterprise Search, please check out the monitoring section of our configuration guide.

Enterprise Search Metricbeat module

edit

Starting with version 7.16.0, Elastic Metricbeat contains an official Enterprise Search monitoring module that can be used for ingest monitoring data from your Enterprise Search deployments into any Elasticsearch cluster.

The data produced by the Metricbeat module can then be accessed via Kibana dashboards, including the Enterprise Search monitoring dashboard described below.

See the Metricbeat module documentation for more details on how to use the new module.

To use Metricbeat with Enterprise Search on Elastic Cloud, you need to deploy Metricbeat on your own infrastructure. If you would like to run the new module directly on Elastic Cloud, you can use the self-monitoring feature described above.

If deploying Metricbeat on your own infrastructure, Metricbeat version 7.16 must be used in order to generate Enterprise Search metrics. We are currently working on enabling Enterprise Search metrics for Metricbeat 8.x versions.

Enterprise Search Kibana monitoring dashboards (preferred)

edit

To help our users visualize the metrics ingested into Elasticsearch via the self-monitoring feature or via a dedicated Metricbeat deployment, we have created a dedicated dashboard, that can be imported into any Kibana deployment.

We have provided a Kibana dashboard export file that can be imported into Kibana using the Stack Management > Saved Objects UI.

You can find the dashboard export file and additional information on how to import it in a dedicated Github repository elastic/ent-search-monitoring.

Monitoring Enterprise Search using Kibana stack monitoring (experimental)

edit

Visualize Enterprise Search metrics by using Kibana’s stack monitoring. In Kibana, use Management → Stack Monitoring.

To use Kibana stack monitoring with Enterprise Search on Elastic Cloud, you need to enable Logs and Metrics in the Cloud deployment. Currently, there is a known issue in which you must log metrics to the current deployment (not recommended).

Elastic APM integration

edit

Starting with version 7.16.0, all Enterprise Search packages come with an Elastic APM agent built in. The APM agent is disabled by default due to a potential performance impact of APM tracing and an increased resource usage caused by APM data reporting on production deployments.

To enable APM instrumentation and control the APM agent configuration, you will need to change the Enterprise Search configuration and restart your deployment. See below for details.

Deploying APM

edit

Before you can use APM with Enterprise Search, you need an APM deployment. The easiest way is to deploy an APM on Elastic Cloud.

Here are some key steps you need to take after you have provisioned an APM deployment:

  1. Open Kibana for your Elastic APM deployment
  2. Click on Observability → APM in the sidebar
  3. Scroll down and click on the Setup instructions button
  4. Select the Ruby on Rails tab
  5. The config/elastic_apm.yml example at the end of the page contains all the data you will need going forward to configure Enterprise Search APM integration. Take a note of that information somewhere.

Configuring APM in a self-managed Enterprise Search deployment

edit

If you want to enable APM for your self-managed Enterprise search deployment, here is what you need to do:

  1. Open your configuration (depending on your deployment method).
  2. Uncomment apm.enabled: true to enable the APM agent.
  3. Uncomment and change apm.server_url and apm.secret_token based on the information you got from Kibana before (the config/elastic_apm.yml example).
  4. If you want to run multiple copies of Enterprise Search with the same APM, it may be helpful to configure apm.service_name as well and set it to something uniquely identifying your particular deployment.
  5. Start Enterprise Search as usual and make a few HTTP calls to it (open the dashboard or make some API calls).
  6. Open Kibana for your APM deployment and navigate to Observability → APM to see APM data for your Enterprise Search deployment.

Configuring APM in an Elastic Cloud Enterprise Search deployment

edit

If you want to enable the Enterprise Search APM integration on a deployment running on Elastic Cloud, here is what you need to do:

  1. Open the Elastic Cloud console for your deployment
  2. Click "Edit" in the sidebar to open the page for changing the details of your deployment
  3. Scroll down to "Enterprise Search" and click on the "Edit enterprise-search.yml" link on the "User settings" line.

  4. Add the following information to the text area:

    apm.enabled: true
apm.server_url: 'https://your-apm-name.apm.europe-west1.gcp.cloud.es.io:443'
apm.secret_token: 'your-token-here'
  5. Scroll down and click "Save"
  6. Confirm your changes in the "Save configuration settings?" pop-up window by clicking "Confirm".
  7. Wait for the deployment to finish a rolling restart and you should have APM integration enabled and data flowing into your APM deployment.
« View and query logs Troubleshooting »