Troubleshooting Enterprise Search setup
editTroubleshooting Enterprise Search setup
editWhen navigating to Search within Kibana, you may see "Unable to connect". To troubleshoot this, first enable and view logs for Enterprise Search and Kibana. Then review common setup issues to help identify and solve your problem.
Enable and view logs
editEnterprise Search and Kibana each log information that may help identify and solve your problem. If necessary, first enable logging. Then view the logs for each service.
Refer to the following documentation for your deployment type:
Review common setup issues
editReview the following common issues to help identify and solve your problem:
-
Enterprise Search’s
kibana.external_url
setting is incorrect or unreachable -
Kibana’s
enterpriseSearch.host
setting is incorrect or unreachable -
Kibana’s
enterpriseSearch.host
is set tohttp
while Enterprise Search is running onhttps
- Kibana’s connection to Enterprise Search is timing out
- Kibana cannot find or does not trust the SSL certificates for Enterprise Search
- Enterprise Search has SSL enabled but hostname verification fails
- Enterprise Search and Kibana versions are incompatible
- Enterprise Search in Kibana is "forbidden" when traffic filters are set on Elastic Cloud
Enterprise Search’s kibana.external_url
setting is incorrect or unreachable
editEnterprise Search will not start and logs the following:
[2021-09-28T17:57:33.438+00:00][8][2002][app-server][INFO]: Performing pre-flight checks for Kibana running on http://kibana1:5601... Unexpected exception while running Enterprise Search: Error: kibana1: Name or service not known at /usr/share/enterprise-search/lib/war/gems/gems/manticore-0.7.0-java/lib/faraday/adapter/manticore.rb:88:in `block in call' /usr/share/enterprise-search/lib/war/gems/gems/manticore-0.7.0-java/lib/manticore/response.rb:79:in `call' /usr/share/enterprise-search/lib/war/gems/gems/manticore-0.7.0-java/lib/faraday/adapter/manticore.rb:92:in `call' /usr/share/enterprise-search/lib/war/lib/middleware/request_logging_middleware.class:25:in `call' /usr/share/enterprise-search/lib/war/gems/gems/faraday-1.3.0/lib/faraday/rack_builder.rb:154:in `build_response' /usr/share/enterprise-search/lib/war/gems/gems/faraday-1.3.0/lib/faraday/connection.rb:492:in `run_request' /usr/share/enterprise-search/lib/war/gems/gems/faraday-1.3.0/lib/faraday/connection.rb:198:in `get' /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/kibana_checks.class:89:in `check_kibana_connection_with_retries!' /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/kibana_checks.class:36:in `check_kibana_connection!' /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/kibana_checks.class:20:in `block in run!' /usr/share/enterprise-search/lib/war/gems/gems/activesupport-5.1.7/lib/active_support/tagged_logging.rb:69:in `block in tagged' /usr/share/enterprise-search/lib/war/gems/gems/activesupport-5.1.7/lib/active_support/tagged_logging.rb:26:in `tagged' /usr/share/enterprise-search/lib/war/gems/gems/activesupport-5.1.7/lib/active_support/tagged_logging.rb:69:in `tagged' /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/kibana_checks.class:19:in `run!' /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/kibana_checks.class:13:in `run!' /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo.class:290:in `configure_kibana!' /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo.class:248:in `configure!' /usr/share/enterprise-search/lib/war/config/application.class:19:in `<main>' org/jruby/RubyKernel.java:1016:in `load' /usr/share/enterprise-search/lib/war/gems/gems/activesupport-5.1.7/lib/active_support/dependencies.rb:286:in `block in load' /usr/share/enterprise-search/lib/war/gems/gems/activesupport-5.1.7/lib/active_support/dependencies.rb:258:in `load_dependency' /usr/share/enterprise-search/lib/war/gems/gems/activesupport-5.1.7/lib/active_support/dependencies.rb:286:in `load' /usr/share/enterprise-search/lib/war/config/application.rb:1:in `<main>' org/jruby/RubyKernel.java:974:in `require' /usr/share/enterprise-search/lib/war/gems/gems/activesupport-5.1.7/lib/active_support/dependencies.rb:292:in `block in require' /usr/share/enterprise-search/lib/war/gems/gems/activesupport-5.1.7/lib/active_support/dependencies.rb:258:in `load_dependency' /usr/share/enterprise-search/lib/war/gems/gems/activesupport-5.1.7/lib/active_support/dependencies.rb:292:in `require' /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/command.class:36:in `initialize' /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit' /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:143:in `run_supported_command' /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:125:in `run_command' /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:112:in `run!' bin/enterprise-search-internal:15:in `<main>'
Solution
editSet kibana.external_url
to an accessible Kibana address.
Kibana’s enterpriseSearch.host
setting is incorrect or unreachable
editKibana logs the following:
{"type":"log","@timestamp":"2021-09-28T18:05:43+00:00","tags":["error","plugins","enterpriseSearch"],"pid":1221,"message":"Could not perform access check to Enterprise Search: FetchError: request to https://enterprise-search1:3002/api/ent/v2/internal/client_config failed, reason: getaddrinfo ENOTFOUND enterprise-search1"}
Solution
editSet enterpriseSearch.host
to an accessible Enterprise Search address.
See Configure Kibana.
Kibana’s enterpriseSearch.host
is set to http
while Enterprise Search is running on https
editKibana logs the following:
{"type":"log","@timestamp":"2021-09-28T17:39:05+00:00","tags":["error","plugins","enterpriseSearch"],"pid":1219,"message":"Could not perform access check to Enterprise Search: FetchError: request to http://enterprise-search:3002/api/ent/v2/internal/client_config failed, reason: Parse Error: Expected HTTP/"}
Solution
editSet enterpriseSearch.host
to the correct protocol within Kibana.
See Configure Kibana.
Kibana’s connection to Enterprise Search is timing out
editKibana logs the following:
{"type":"log","@timestamp":"2021-09-28T17:52:31+00:00","tags":["warning","plugins","enterpriseSearch"],"pid":1219,"message":"Enterprise Search access check took over 300ms. Please ensure your Enterprise Search server is responding normally and not adversely impacting Kibana load speeds."} {"type":"log","@timestamp":"2021-09-28T17:52:35+00:00","tags":["warning","plugins","enterpriseSearch"],"pid":1219,"message":"Exceeded 5000ms timeout while checking https://enterprise-search:3002. Please consider increasing your enterpriseSearch.accessCheckTimeout value so that users aren't prevented from accessing Enterprise Search plugins due to slow responses."}
Solution
editTroubleshoot performance issues in Enterprise Search. To get started, see Monitoring APIs.
If necessary, contact Elastic Support.
Kibana cannot find or does not trust the SSL certificates for Enterprise Search
editKibana logs the following:
{"type":"log","@timestamp":"2021-09-28T17:42:11+00:00","tags":["error","plugins","enterpriseSearch"],"pid":1220,"message":"Could not perform access check to Enterprise Search: FetchError: request to https://enterprise-search:3002/api/ent/v2/internal/client_config failed, reason: unable to verify the first certificate"}
Solution
editConfigure enterpriseSearch.ssl.certificateAuthorities
to reference the correct certificate or certificate chain.
See Configure Kibana to Trust Your SSL Certificate Authority.
Enterprise Search does not support requests from the Enterprise Search Server to Kibana using custom CAs.
Enterprise Search has SSL enabled but hostname verification fails
editWhen enterpriseSearch.ssl.verificationMode
is configured to perform hostname verification but hostname verification fails.
Kibana logs the following:
{"type":"log","@timestamp":"2021-09-28T18:39:04+00:00","tags":["error","plugins","enterpriseSearch"],"pid":1220,"message":"Could not perform access check to Enterprise Search: FetchError: request to https://enterprise-search:3002/api/ent/v2/internal/client_config failed, reason: Hostname/IP does not match certificate's altnames: Host: enterprise-search. is not in the cert's altnames: DNS:localhost, IP Address:127.0.0.1, DNS:node1"}
Solution
editIn production or other environments that must be secure, regenerate the certificates with the correct Subject Alternative Names matching enterpriseSearch.host
in Kibana.
In development and some other environments, reducing security may be acceptable.
In these cases, set enterpriseSearch.ssl.verificationMode
to a lower setting.
For example, change full
to certificate
or none
.
See Configure Kibana to Trust Your SSL Certificate Authority.
Reduce the security of your deployment only as a last resort. Ideally, hostname verification should be enabled in all environments.
Enterprise Search and Kibana versions are incompatible
editThe Kibana UI displays "Incompatible version error".
Kibana logs the following:
{"type":"log","@timestamp":"2021-09-28T19:03:00+00:00","tags":["warning","plugins","enterpriseSearch"],"pid":1225,"message":"Your Kibana instance (v7.15.0) is not the same version as your Enterprise Search instance (vundefined), which may cause unexpected behavior. Use matching versions for the best experience."}
Solution
editRun Enterprise Search and Kibana services of the same major and minor versions.
Enterprise Search in Kibana is "forbidden" when traffic filters are set on Elastic Cloud
editThe Kibana UI displays "Forbidden".
Kibana logs the following:
ResponseError: security_exception: [security_exception] Reason: action [indices:data/read/search] is unauthorized for user [cloud-internal-enterprise_search-server] with roles [cloud-internal-enterprise_search-server] on indices [.kibana_7.14.0_001,.kibana], this action is granted by the index privileges [read,all] at onBody (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:337:23) at IncomingMessage.onEnd (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:264:11) at IncomingMessage.emit (events.js:387:35) at endReadableNT (internal/streams/readable.js:1317:12) at processTicksAndRejections (internal/process/task_queues.js:82:21) { meta: { body: { error: [Object], status: 403 }, statusCode: 403, headers: { 'content-length': '625', 'content-type': 'application/json; charset=UTF-8', 'x-cloud-request-id': 'O_v__Y_hRf-I528DechRKQ', 'x-elastic-product': 'Elasticsearch', 'x-found-handling-cluster': '<deployment_id>', 'x-found-handling-instance': 'instance-0000000000', 'x-opaque-id': 'd77d5906-8317-40c1-86e4-d78a408908b5', date: 'Mon, 30 Aug 2021 12:14:08 GMT' }, meta: { context: null, request: [Object], name: 'elasticsearch-js', connection: [Object], attempts: 0, aborted: false } } }
Solution
editKnown issues were resolved via the 8.3.3 release. Please contact Elastic Support if this error is encountered on versions on or after 8.3.3.