Using Kibana server logs

Kibana Logs is a great way to see what’s going on in your application and to debug performance issues. Navigating through a large number of generated logs can be overwhelming, and following are some techniques that you can use to optimize the process.

Start by defining a problem area that you are interested in. For example, you might be interested in seeing how a particular Kibana Plugin is performing, so no need to gather logs for all of Kibana. Or you might want to focus on a particular feature, such as requests from the Kibana server to the Elasticsearch server. Depending on your needs, you can configure Kibana to generate logs for a specific feature.

logging:
  appenders:
    file:
      type: file
      fileName: ./kibana.log
      layout:
        type: json

### gather all the Kibana logs into a file
logging.root:
    appenders: [file]
    level: all

### or gather a subset of the logs
logging.loggers:
  ### responses to an HTTP request
  - name: http.server.response
    level: debug
    appenders: [file]
  ### result of a query to the Elasticsearch server
  - name: elasticsearch.query
    level: debug
    appenders: [file]
  ### logs generated by my plugin
  - name: plugins.myPlugin
    level: debug
    appenders: [file]

Kibana’s file appender is configured to produce logs in ECS JSON format. It’s the only format that includes the meta information necessary for log correlation out-of-the-box.

The next step is to define what observability tools are available. For a better experience, set up an Observability integration provided by Elastic to debug your application with the APM UI. To debug something quickly without setting up additional tooling, you can work with the plain Kibana logs.

APM UI

edit

Prerequisites Kibana logs are configured to be in ECS JSON format to include tracing identifiers.

To debug Kibana with the APM UI, you must set up the APM infrastructure. You can find instructions for the setup process on the Observability integrations page.

Once you set up the APM infrastructure, you can enable the APM agent and put Kibana under load to collect APM events. To analyze the collected metrics and logs, use the APM UI as demonstrated in the docs.

Plain Kibana logs

edit

Prerequisites Kibana logs are configured to be in ECS JSON format to include tracing identifiers.

Open Kibana Logs and search for an operation you are interested in. For example, suppose you want to investigate the response times for queries to the /internal/telemetry/clusters/_stats Kibana endpoint. Open Kibana Logs and search for the HTTP server response for the endpoint. It looks similar to the following (some fields are omitted for brevity).

{
  "message":"POST /internal/telemetry/clusters/_stats 200 1014ms - 43.2KB",
  "log":{"level":"DEBUG","logger":"http.server.response"},
  "trace":{"id":"9b99131a6f66587971ef085ef97dfd07"},
  "transaction":{"id":"d0c5bbf14f5febca"}
}

You are interested in the trace.id field, which is a unique identifier of a trace. The trace.id provides a way to group multiple events, like transactions, which belong together. You can search for "trace":{"id":"9b99131a6f66587971ef085ef97dfd07"} to get all the logs that belong to the same trace. This enables you to see how many Elasticsearch requests were triggered during the 9b99131a6f66587971ef085ef97dfd07 trace, what they looked like, what Elasticsearch endpoints were hit, and so on.

« Troubleshooting Trace Elasticsearch query to the origin in Kibana »