Logs and logging
editLogs and logging
editYour Enterprise Search deployment produces a range of logs, including crawler, API, and audit logs. What gets logged, how it is logged, and where you can find these logs, differs between Elastic Cloud and self-managed deployment options.
This guide will give you a high-level overview of Enterprise Search logging:
- Manage your logs: Learn how to manage your logs, to ensure you have the information you need when you need it.
- Logs quick reference: Jump here for a handy overview of the various Enterprise Search log types, including the data views (or index patterns) you need to find them.
Read this guide to learn how to create data views, and use two Kibana tools, Logs and Discover, to view and query your logs.
Manage your logs
editYour deployment writes some logs directly to disk, and others to Elasticsearch. For the full list, see the Logs quick reference.
Managing logs on Elastic Cloud deployments
editElastic Cloud users can set up logs and monitoring in their Cloud environment, and redirect logs and metrics to the same or a different cluster.
Follow these steps to enable logs and metrics:
- Navigate to Deployments → Deployment → Logs and metrics → Enable.
- Create a new monitoring deployment, or select an existing one.
Learn more about enabling logging and monitoring in Elastic Cloud.
Managing logs on self-managed deployments using Elasticsearch
editA subset of logs are indexed into Elasticsearch automatically, using Filebeat, as part of the Enterprise Search installation process.
This does not include general logs such as app-server.log
.
We encourage self-managed users to index these additional logs into Elasticsearch using a monitoring deployment. Apply the Elastic Enterprise Search Filebeat configuration to ingest Enterprise Search log files into Elasticsearch.
When logs are written to Elasticsearch you can manage log retention policies using index lifecycle management (ILM). Otherwise, you will have to manage the rollover and rotation of your logs yourself. The sooner you automate these policies, the better!
View self-managed deployment logs on-disk
editSelf-managed deployments have access to additional logs that are otherwise not available to Elastic Cloud users.
You’ll find these in the Logs quick reference below, where the Cloud Elasticsearch index pattern
is marked N/A
.
To view your log files stored on disk, the process varies by deployment type.
Tar, deb, and rpm package installs:
-
Find the logs in
/log/dir
.
Docker:
-
Use the command
docker exec -it <container name> /bin/bash
to launch a bash shell in the container. Find the logs in/log/dir
. -
Alternatively, the command
docker logs <container-name>
will show thestdout
of Enterprise Search.
Elastic Cloud Kubernetes
-
Use the command
kubectl exec --stdin --tty <pod-name> -- /bin/bash
to launch a bash shell to a running Enterprise Search pod. Find the logs in the/var/log/enterprise-search
directory in an Enterprise Search pod. -
Alternatively, the command
kubectl logs <pod-name>
will show thestdout
of an Enterprise Search pod.
Logs quick reference
editThe following table contains key information about Enterprise Search logs, including log types, index patterns, and filenames:
-
The
event.dataset
value makes it easier to filter by events when querying your logs. -
Filter
event.dataset
further by finding the Elasticsearch index patterns for your logs of interest in the table below. Use these index patterns when creating data views in Kibana. - This table specifies which Elastic Cloud index patterns are intended to be used on your Enterprise Search deployment, versus your monitoring deployment, and whether a given index pattern matches hidden data streams and indices.
- Read this guide to learn how to create data views, and use the Kibana tools, Logs and Discover, to view and query your logs.
Log type | Elasticsearch index pattern (Self-managed) | Elasticsearch index pattern (Cloud) | Local filename | event.dataset | Description |
---|---|---|---|---|---|
API logs |
|
|
|
|
Enterprise Search API logs, including queries and inserts. |
App Search analytics |
|
|
|
|
App Search Analytics logs, including click-through and query metrics. |
Workplace Search analytics |
|
|
|
|
Workplace Search Analytics logs, including click-through and query metrics. |
Application logs |
N/A |
|
|
|
Main application logs, including API calls made to the server, errors and corresponding stack traces. When troubleshooting an issue, you will typically start here. Elasticsearch debug logs also appear here. |
Audit logs |
|
|
|
|
Log of events across Enterprise Search, useful for security-related auditing. |
Content sources events |
|
|
|
|
Workplace Search content sources events. |
Web crawler logs |
|
|
|
|
Detailed App Search web crawler logs. |
Filebeat process |
N/A |
N/A |
|
N/A |
Use these logs to confirm that Filebeat is correctly pushing events to Elasticsearch. |
JVM GC statistics |
N/A |
N/A |
|
N/A |
Java Virtual Machine (JVM) garbage collection statistics. |
Internal statistics |
N/A |
N/A |
|
N/A |
Timing statistics for various internal processes. |
System requests |
N/A |
|
|
|
Contains web requests, including method, status code, and parameters used. |
Internal worker jobs |
N/A |
|
|
|
High-level worker job queue logs. |
Search relevance suggestions events |
|
|
|
|
Created when Curations powered by Adaptive Relevance is enabled. Logs all adaptive relevance curation suggestion events. |
Search relevance suggestions system logs |
N/A |
N/A |
|
N/A |
Created when Curations powered by Adaptive Relevance is enabled. Logs all adaptive relevance curation suggestion details. |
Apply the Elastic Enterprise Search Filebeat configuration to ingest Enterprise Search log files into Elasticsearch.
Learn more about logging in Enterprise Search
editFor production use, you should send your deployment logs and metrics to a dedicated monitoring deployment. Learn more in this guide.
App Search exposes several event logging settings through its log settings API and a corresponding section of the Settings dashboard. Read the Log settings guide for more details.
Learn about log retention and how to disable logging in App Search.
Read this guide to learn how to create data views, and use two Kibana tools, Logs and Discover, to view and query your logs.