Logs and logging

edit

Logs and logging

edit

Your Enterprise Search deployment produces a range of logs, including crawler, API, and audit logs. What gets logged, how it is logged, and where you can find these logs, differs between Elastic Cloud and self-managed deployment options.

This guide will give you a high-level overview of Enterprise Search logging:

  • Manage your logs: Learn how to manage your logs, to ensure you have the information you need when you need it.
  • Logs quick reference: Jump here for a handy overview of the various Enterprise Search log types, including the data views (or index patterns) you need to find them.

Read this guide to learn how to create data views, and use two Kibana tools, Logs and Discover, to view and query your logs.

Manage your logs

edit

Your deployment writes some logs directly to disk, and others to Elasticsearch. For the full list, see the Logs quick reference.

Managing logs on Elastic Cloud deployments

edit

Elastic Cloud users can set up logs and monitoring in their Cloud environment, and redirect logs and metrics to the same or a different cluster.

Follow these steps to enable logs and metrics:

  1. Navigate to DeploymentsDeploymentLogs and metricsEnable.
  2. Create a new monitoring deployment, or select an existing one.

Learn more about enabling logging and monitoring in Elastic Cloud.

Managing logs on self-managed deployments using Elasticsearch

edit

A subset of logs are indexed into Elasticsearch automatically, using Filebeat, as part of the Enterprise Search installation process. This does not include general logs such as app-server.log.

We encourage self-managed users to index these additional logs into Elasticsearch using a monitoring deployment. Apply the Elastic Enterprise Search Filebeat configuration to ingest Enterprise Search log files into Elasticsearch.

When logs are written to Elasticsearch you can manage log retention policies using index lifecycle management (ILM). Otherwise, you will have to manage the rollover and rotation of your logs yourself. The sooner you automate these policies, the better!

View self-managed deployment logs on-disk

edit

Self-managed deployments have access to additional logs that are otherwise not available to Elastic Cloud users. You’ll find these in the Logs quick reference below, where the Cloud Elasticsearch index pattern is marked N/A.

To view your log files stored on disk, the process varies by deployment type.

Tar, deb, and rpm package installs:

  • Find the logs in /log/dir.

Docker:

  • Use the command docker exec -it <container name> /bin/bash to launch a bash shell in the container. Find the logs in /log/dir.
  • Alternatively, the command docker logs <container-name> will show the stdout of Enterprise Search.

Elastic Cloud Kubernetes

  • Use the command kubectl exec --stdin --tty <pod-name> -- /bin/bash to launch a bash shell to a running Enterprise Search pod. Find the logs in the /var/log/enterprise-search directory in an Enterprise Search pod.
  • Alternatively, the command kubectl logs <pod-name> will show the stdout of an Enterprise Search pod.

Logs quick reference

edit

The following table contains key information about Enterprise Search logs, including log types, index patterns, and filenames:

  • The event.dataset value makes it easier to filter by events when querying your logs.
  • Filter event.dataset further by finding the Elasticsearch index patterns for your logs of interest in the table below. Use these index patterns when creating data views in Kibana.
  • This table specifies which Elastic Cloud index patterns are intended to be used on your Enterprise Search deployment, versus your monitoring deployment, and whether a given index pattern matches hidden data streams and indices.
  • Read this guide to learn how to create data views, and use the Kibana tools, Logs and Discover, to view and query your logs.
Log type Elasticsearch index pattern (Self-managed) Elasticsearch index pattern (Cloud) Local filename event.dataset Description

API logs

logs-enterprise_search.api-default

logs-enterprise_search.api-default (Enterprise Search deployment)

filebeat.log

api

Enterprise Search API logs, including queries and inserts.

App Search analytics

logs-app_search.analytics-default

logs-app_search.analytics-default (Enterprise Search deployment)

filebeat.log

app-search-analytics

App Search Analytics logs, including click-through and query metrics.

Workplace Search analytics

logs-workplace_search.analytics-default

logs-workplace_search.analytics-default (Enterprise Search deployment)

filebeat.log

workplace-search-analytics

Workplace Search Analytics logs, including click-through and query metrics.

Application logs

N/A

.ds-elastic-cloud-logs* (monitoring deployment, hidden index)

app-server.log

enterprise_search.server

Main application logs, including API calls made to the server, errors and corresponding stack traces.

When troubleshooting an issue, you will typically start here. Elasticsearch debug logs also appear here.

Audit logs

logs-enterprise_search.audit-default

logs-enterprise_search.audit-default (Enterprise Search deployment)

audit.log

enterprise-search-audit

Log of events across Enterprise Search, useful for security-related auditing.

Content sources events

logs-workplace_search.content_events-default

logs-workplace_search.content_events-default (Enterprise Search deployment)

filebeat.log

workplace_search.content_events

Workplace Search content sources events.

Web crawler logs

logs-crawler-default

logs-crawler-default (Enterprise Search deployment)

ds-elastic-cloud-logs* (monitoring deployment, hidden index)

crawler.log

enterprise_search.crawler

Detailed App Search web crawler logs.

Filebeat process

N/A

N/A

filebeat

N/A

Use these logs to confirm that Filebeat is correctly pushing events to Elasticsearch.

JVM GC statistics

N/A

N/A

gc.log

N/A

Java Virtual Machine (JVM) garbage collection statistics.

Internal statistics

N/A

N/A

stats.log

N/A

Timing statistics for various internal processes.

System requests

N/A

.ds-elastic-cloud-logs* (monitoring deployment, hidden index)

system.log

enterprise_search.system

Contains web requests, including method, status code, and parameters used.

Internal worker jobs

N/A

.ds-elastic-cloud-logs* (monitoring deployment, hidden index)

worker.log

enterprise_search.worker

High-level worker job queue logs.

Search relevance suggestions events

logs-app_search.search_relevance_suggestions-default

logs-app_search.search_relevance_suggestions-default (Enterprise Search deployment)

filebeat.log

search-relevance-suggestions, search-relevance-suggestions

Created when Curations powered by Adaptive Relevance is enabled.

Logs all adaptive relevance curation suggestion events.

Search relevance suggestions system logs

N/A

N/A

search-relevance-suggestions.log

N/A

Created when Curations powered by Adaptive Relevance is enabled.

Logs all adaptive relevance curation suggestion details.

Apply the Elastic Enterprise Search Filebeat configuration to ingest Enterprise Search log files into Elasticsearch.

Learn more about logging in Enterprise Search

edit

For production use, you should send your deployment logs and metrics to a dedicated monitoring deployment. Learn more in this guide.

App Search exposes several event logging settings through its log settings API and a corresponding section of the Settings dashboard. Read the Log settings guide for more details.

Learn about log retention and how to disable logging in App Search.

Read this guide to learn how to create data views, and use two Kibana tools, Logs and Discover, to view and query your logs.