Troubleshooting

If you’ve lost access to Enterprise Search, you can recover specific users or reset all access to Enterprise Search.

Refer to the following sections in the Enterprise Search documentation:

Enterprise Search creates several log indices. You can create dashboards for these indices within Kibana, or you can query these indices using Elasticsearch.

For a list of these indices, see Log retention in the Enterprise Search documentation.

Running the binary with the --diagnostic-report flag to capture a diagnostic bundle:

bin/enterprise-search --diagnostic-report

The bundle generates a zipped folder, with the following structure:

enterprise-search-report-yyyy-mm-dd-ss-mm-hh
- app_info.json
- config.json
- elasticsearch_info.json
- logs
  - stats.log
  - system.log
- recent_jobs_info.json
- system_info.json

When you submit feedback or issue a support request, attach this bundle to provide our team with:

The richness of the bundle will change depending on what you have set for log_level in your config/enterprise-search.yml.

You can select one of: debug, info, warn, error, fatal, or unknown.

If you are sending the bundle into support, the debug setting will render the most thorough information:

...
log_level: debug
...

Look through the logs before sending to ensure that you are comfortable sharing the information.

You will need to use the correct Java version if you receive an error such as:

Found java executable in PATH
Java version: 1.7.0_80
Elastic Enterprise Search requires Java version 1.8 or higher, current version is 1.7.0_80

On Linux or MacOS, you can manage your Java environment using something like jEnv.

Ensure you have 1.8 or higher installed; just installing a new Java version often is not enough, ensure it is in your PATH.

Successful search API responses return a 200 response but may contain either alerts or warnings.

This page will help you identify and troubleshoot alerts.

For warnings, consult the search API Reference for instructions on how better format your query based on warning contents.

The response object is rich.

Errors will appear within the alerts or warnings array of the meta object.

{
  "meta": {
    "alerts": [
      {
        "code": integer,
        "message": string,
        "link": string // URL to additional documentation about this alert
      },
      {
        "code": integer,
        "message": string,
        "link": string // URL to additional documentation about this alert
      }
    ], // Indicates issues with the Enterprise Search service.
    "warnings": [  // This section returns warnings either as an array of objects or a concrete warning string
      {
       "someField": string
      }
    ], // Indicates issues with query formation.
    "page": {
      "current": integer,
      "total_pages": integer,
      "total_results": integer,
      "size": integer
    },
    "request_id": "9c4c527ba0f57e6bafb1f318167eaa04"
  },
  "results": [

  # ... Documents.

  ]
}

It is helpful to know whether a query has timed out, or whether the query returned no results.

A 5001 error indicates that the query could not be completed in full.

Use the 5001 error to initiate a retry of the query, perhaps after assessing whether the given parameters are bloating the query.

If the issue persists, please contact support.

App Search will not wait indefinitely for Elasticsearch to respond. Sometimes, for particularly compute-heavy requests, and/or if Elasticsearch is overwhelmed, you may see this error message in your logs, and may receive 500 responses.

Faraday::TimeoutError: Read timed out

This indicates that the 30-second timeout that App Search sets for Elasticsearch traffic was exceeded.

Check your Elasticsearch logs and metrics to make sure that it is healthy and has sufficient resources. Often times, this issue can be resolved by scaling up your Elasticsearch deployment. Another important thing to check is if you can optimize your queries to be more efficient. Fewer aggregations, fewer terms, fewer clauses, fewer wildcards, etc, can all improve performance and reduce risk of timeouts.

You can debug Enterprise Search queries by simply adding a header to your Search API requests. This allows you to analyze the queries Enterprise Search is sending to Elasticsearch and the responses returned. A common use case is to find the root cause of slow queries.

This is a fast and lightweight alternative to enabling monitoring and logging for your entire deployment.

If the X-Enterprise-Search-Debug header is present and set to true in your Search API request, this enables request-level debug logging. The response body now returns all the raw Elasticsearch queries associated with your Enterprise Search request, along with detailed timing data.

Here’s how the request looks using curl:

curl -X POST '<ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/test/search' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer search-REDACTED' \
  -H 'X-Enterprise-Search-Debug: true' \

You can add another header to make the queries easy to track. The Search API automatically generates a unique ID for requests. However, if you add the X-Request-ID header, you can set this ID to an arbitrary unique value of your choice. This helps you find the request and all its associated Elasticsearch requests in Enterprise Search application logs. The following example sets the Request ID to pancakes.

Here is an example curl request with the header X-Enterprise-Search-Debug: true and the X-Request-ID header set to pancakes:

curl -X POST '<ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/test/search' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer search-REDACTED' \
  -H 'X-Enterprise-Search-Debug: true' \
  -H 'X-Request-ID: pancakes' \
  -d '{"query": "great"}'

A simple way to see how the response has changed is to run the same search API request with and without the X-Enterprise-Search-Debug: true header and the X-Request-ID header. When you add the headers, the request response body will now have additional fields:

Under elasticsearch you will find:

You can search for pancakes in the Enterprise Search application logs to find all activity associated with this request. An example of a response body with this additional information looks like this:

{
  "meta": {
    "alerts": [],
    "warnings": [],
    "precision": 2,
    "page": { ...
    },
    "engine": {
      "name": "longevity",
      "type": "default"
    },
    "elasticsearch": {
      "request_header": {
        "index": "enterprise-search-engine-test",
        "preference": "de2f15d014d40b93578d255e6221fd60"
      },
      "request_body": {
        "query": { ...

        // The raw Elasticsearch query appears in this object...

        },
        "sort": [ ...
        ],
        "highlight": { ...
        },
        "size": 10,
        "from": 0,
        "timeout": "30000ms",
        "_source": [
          "title",
          "id"
        ]
      },
      "response": {

        // The amount of time Elasticsearch spent processing the raw query, in milliseconds:
        "took": 17,

        "timed_out": false,
        "_shards": {
          "total": 2,
          "successful": 2,
          "skipped": 0,
          "failed": 0
        },
        "hits": {
          "total": {
            "value": 4,
            "relation": "eq"
          },
          "max_score": null,
          "hits": [
          ]
        },
        "status": 200
      }
    },

    // The original X-Request-ID
    "request_id": "pancakes"
  },

You can run the raw Elasticsearch query directly against your Elasticsearch instance using the Elasticsearch search API. The Kibana dev console provides a UI where you can make search API requests against Elasticsearch.

You can also use the Elasticsearch search profile API parameter, profile: true, when querying Elasticsearch directly, to see more Elasticsearch performance profiling metrics. Paste the profiled Elasticsearch response into the Kibana Search Profiler UI to visualize it. Add the 'X-Enterprise-Search-Profile:true' header to your request to return detailed profiling information.

The debug information is also written into the Enterprise Search logs. The logs will include all Elasticsearch queries Enterprise Search performed up to and including the final raw search. For example, for App Search, the logs will show all queries to Elasticsearch to retrieve the engine’s saved synonyms and curations.

You will need to be comfortable analyzing raw Elasticsearch queries to debug using these tools. For difficult to diagnose issues, contact Elastic support for assistance.

If you need to dig deeper into your search query performance, you can enable request-level Elasticsearch profiling. Adding the X-Enterprise-Search-Profile:true header lets you profile Enterprise Search search queries for specific requests.

This returns detailed timing information about the execution of individual components in a search request, by leveraging the Elasticsearch Profile API:

"The Profile API gives the user insight into how search requests are executed at a low level so that the user can understand why certain requests are slow, and take steps to improve them."

The profiling information is returned directly in the response body and is also written into the Enterprise Search logs. This is an advanced tool and the output of the response is verbose. You will need to pretty-print the response and be comfortable analyzing Elasticsearch queries.

Use the X-Enterprise-Search-Profile: true header together with the X-Enterprise-Search-Debug: true header to view profiling data and the underlying Elasticsearch request.

Optionally, add the X-Request-ID header to make it easier to find profiling information for the request in application log files.

Here is an example curl request with the header X-Enterprise-Search-Debug: true, the header X-Enterprise-Search-Profile:true, and the X-Request-ID header set to pancakes:

curl -X POST '<ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/test/search' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer search-REDACTED' \
  -H 'X-Enterprise-Search-Debug: true' \
  -H 'X-Enterprise-Search-Profile: true' \
  -H 'X-Request-ID: pancakes' \
  -d '{"query": "great"}'

Here is what the response might look like. Note this is a truncated example of the meta.profile portion of the response body:

{
  "meta": {
    "alerts": [],
    "warnings": [],
    "precision": 2,
    "page": {
    "current": 1,
    "total_pages": 1,
    "total_results": 9,
    "size": 10
  },
  "engine": {
    "name": "test",
    "type": "default"
    },
  "profile": {
    "shards": [
      {
        "id": "[pancakes][.enterprise-search-engine-test-1][0]",
        "searches": [
          {
            "query": [
              {
                "type": "MatchAllDocsQuery",
                "description": "*:*",
                "time_in_nanos": 4228,
                "breakdown": {
                  "set_min_competitive_score_count": 0,
                  "match_count": 0,
                  "shallow_advance_count": 0,
                  "set_min_competitive_score": 0,
                  "next_doc": 270,
                  "match": 0,
                  "next_doc_count": 1,
                  "score_count": 1,
                  "compute_max_score_count": 0,
                  "compute_max_score": 0,
                  "advance": 663,
                  "advance_count": 1,
                  "score": 620,
                  "build_scorer_count": 2,
                  "create_weight": 512,
                  "shallow_advance": 0,
                  "create_weight_count": 1,
                  "build_scorer": 2163
               }
             }
           ],
           "rewrite_time": 1596,
           "collector": [
             {
              "name": "SimpleFieldCollector",
              "reason": "search_top_hits",
              "time_in_nanos": 15307
             }
           ]
         }
       ],
          "aggregations": []
      ...

You can visualize the profiling information using the Kibana Search Profiler UI. Search Profiler is automatically enabled in Kibana. Open the main menu, click Dev Tools, and then click Search Profiler to get started. Paste in the full JSON object surrounding the profile section, e.g.:

{
   "profile": {
      ...
   }
}

For difficult to diagnose issues, contact Elastic support for assistance.