IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Log retention Release notes »
Elastic Docs ›

Monitoring

edit

Monitoring

edit

When deploying an Enterprise Search cluster into production, we highly recommend setting up some kind of monitoring for it to help you more effectively troubleshoot any issues with the deployment, properly scale the system, etc.

In this guide we will cover different options you have for setting up monitoring and for troubleshooting any issues with your systems.

Overview

edit

There are multiple features available in Enterprise Search and the Elastic Stack to help you monitor and troubleshoot your Enterprise Search deployments.

  • Monitoring APIs – Enterprise Search provides a number of APIs that can be used to get access to internal system metrics, product usage information and other data that may be useful for understanding the health of your deployment. These APIs can be accessed via our official API clients or with any HTTP-enabled metrics collection agent.
  • Enterprise Search Self-Monitoring – you can enable a self-monitoring feature within Enterprise Search, which will automatically ship metrics information to the underlying Elasticsearch deployment, allowing you to access it via Kibana dashboards.
  • External Metricbeat Monitoring – you can use the official Metricbeat module for Enterprise Search to collect monitoring information from your deployments and index them into any Elasticsearch cluster for future access via Kibana dashboards.
  • Elastic APM Integration – you can enable Elastic APM agent built into Enterprise Search and ship a lot of instrumentation data to a given Elastic APM deployment for future analysis via the APM application in Kibana.

Enterprise Search self-monitoring

edit

Starting with Enterprise search 7.16, you can enable monitoring information reporting by setting the monitoring.reporting_enabled configuration setting to true and restarting your deployment. This feature works for both self-managed and for Elastic Cloud deployments.

After self-monitoring is enabled, Enterprise Search is going to automatically provision a Metricbeat instance with the enterprisesearch module and will start reporting a detailed set of monitoring metrics to the underlying Elasticsearch cluster every 10 seconds to be stored in a set of indexes called metricbeat-ent-search-*.

Metricbeat indexes will have ILM enabled on them so the data will be stored in Elasticsearch for up to 30 days and indexes will be rotated every 100Mb.

You cannot control the ent-search-metricbeat ILM policy at the moment. Even if you use the Kibana UI or Elasticsearch API to directly update the policy, it will get overwritten by Enterprise Search.

If you need full control of the metricbeat ILM policies for your Enterprise Search monitoring data, please deploy a standalone Metricbeat instance.

The metrics in Elasticsearch can be then accessed via Kibana dashboards and visualizations. See the dedicated section below for more information on Kibana dashboards available for you.

For more information on the settings that can be used in your deployment configuration to control the frequency of monitoring data reporting, index names used and other aspects of the self-monitoring feature within Enterprise Search, please check out the monitoring section of our configuration guide.

Enterprise Search Metricbeat module

edit

Starting with version 7.16.0, Elastic Metricbeat contains an official Enterprise Search monitoring module that can be used for ingest monitoring data from your Enterprise Search deployments into any Elasticsearch cluster.

The data produced by the Metricbeat module can then be accessed via Kibana dashboards, including the Enterprise Search monitoring dashboard described below.

See the Metricbeat module documentation for more details on how to use the new module.

To use Metricbeat with Enterprise Search on Elastic Cloud, you need to deploy Metricbeat on your own infrastructure. If you would like to run the new module directly on Elastic Cloud, you can use the self-monitoring feature described above.

Enterprise Search Kibana monitoring dashboards

edit

To help our users visualize the metrics ingested into Elasticsearch via the self-monitoring feature or via a dedicated Metricbeat deployment, we have created a dedicated dashboard, that can be imported into any Kibana deployment.

There are two ways to get the Enterprise Search Monitoring Kibana dashboard into your deployment. What one you will use depends on the specifics of your own deployment configuration (self-managed or cloud, OS packages or Kubernetes, etc) and on your level of access to the underlying system.

Importing Kibana dashboards via Metricbeat

edit

When you deploy the Enterprise Search Metricbeat module, it comes with a monitoring dashboard that can be imported into Kibana.

This feature requires you to run a command in your console providing it with access details to your Kibana deployment:

./metricbeat setup \
    --dashboards \
    -E setup.kibana.host=http://your-kibana.server.url:5601 \
    -E setup.kibana.username=your_kibana_user \
    -E setup.kibana.password=your_kibana_password

See the Metricbeat documentation for more details on how to use this feature.

Importing Kibana dashboards directly using Kibana

edit

For those users who do not want to run a Metricbeat setup command or do not have the appropriate access to do so (for example, on Elastic Cloud), we have provided a Kibana dashboard export file that can be imported into Kibana using the Stack Management > Saved Objects UI.

You can find the dashboard export file and additional information on how to import it in a dedicated Github repository elastic/ent-search-monitoring.

Elastic APM integration

edit

Starting with version 7.16.0, all Enterprise Search packages come with an Elastic APM agent built in. The APM agent is disabled by default due to a potential performance impact of APM tracing and an increased resource usage caused by APM data reporting on production deployments.

To enable APM instrumentation and control the APM agent configuration, you will need to change the Enterprise Search configuration and restart your deployment. See below for details.

Deploying APM

edit

Before you can use APM with Enterprise Search, you need an APM deployment. The easiest way is to deploy an APM on Elastic Cloud.

Here are some key steps you need to take after you have provisioned an APM deployment:

  1. Open Kibana for your Elastic APM deployment
  2. Click on Observability → APM in the sidebar
  3. Scroll down and click on the Setup instructions button
  4. Select the Ruby on Rails tab
  5. The config/elastic_apm.yml example at the end of the page contains all the data you will need going forward to configure Enterprise Search APM integration. Take a note of that information somewhere.

Configuring APM in a self-managed Enterprise Search deployment

edit

If you want to enable APM for your self-managed Enterprise search deployment, here is what you need to do:

  1. Open your configuration (depending on your deployment method).
  2. Uncomment apm.enabled: true to enable the APM agent.
  3. Uncomment and change apm.server_url and apm.secret_token based on the information you got from Kibana before (the config/elastic_apm.yml example).
  4. If you want to run multiple copies of Enterprise Search with the same APM, it may be helpful to configure apm.service_name as well and set it to something uniquely identifying your particular deployment.
  5. Start Enterprise Search as usual and make a few HTTP calls to it (open the dashboard or make some API calls).
  6. Open Kibana for your APM deployment and navigate to Observability → APM to see APM data for your Enterprise Search deployment.

Configuring APM in an Elastic Cloud Enterprise Search deployment

edit

If you want to enable the Enterprise Search APM integration on a deployment running on Elastic Cloud, here is what you need to do:

  1. Open the Elastic Cloud console for your deployment
  2. Click "Edit" in the sidebar to open the page for changing the details of your deployment
  3. Scroll down to "Enterprise Search" and click on the "Edit enterprise-search.yml" link on the "User settings" line.

  4. Add the following information to the text area:

    apm.enabled: true
apm.server_url: 'https://your-apm-name.apm.europe-west1.gcp.cloud.es.io:443'
apm.secret_token: 'your-token-here'
  5. Scroll down and click "Save"
  6. Confirm your changes in the "Save configuration settings?" pop-up window by clicking "Confirm".
  7. Wait for the deployment to finish a rolling restart and you should have APM integration enabled and data flowing into your APM deployment.
« Log retention Release notes »