Logging service

With pattern layout it’s possible to define a string pattern with special placeholders %conversion_pattern that will be replaced with data from the actual log message. By default the following pattern is used: [%date][%level][%logger] %message.

The conversions that are provided out of the box are:

level Outputs the level of the logging event. Example of %level output: TRACE, DEBUG, INFO.

logger Outputs the name of the logger that published the logging event. Example of %logger output: server, server.http, server.http.kibana.

message Outputs the application supplied message associated with the logging event.

meta Outputs the entries of meta object data in json format, if one is present in the event. Example of %meta output:

// Meta{from: 'v7', to: 'v8'}
'{"from":"v7","to":"v8"}'
// Meta empty object
'{}'
// no Meta provided
''

date Outputs the date of the logging event. The date conversion specifier may be followed by a set of braces containing a name of predefined date format and canonical timezone name. Timezone name is expected to be one from TZ database name. Timezone defaults to the host timezone when not explicitly specified. Example of %date output:

pid Outputs the process ID.

The pattern layout also offers a highlight option that allows you to highlight some parts of the log message with different colors. Highlighting is quite handy if log messages are forwarded to a terminal with color support.

With json layout log messages will be formatted as JSON strings in ECS format that includes a timestamp, log level, logger, message text and any other metadata that may be associated with the log message itself.

Similar to Log4j’s RollingFileAppender, this appender will log into a file, and rotate it following a rolling strategy when the configured policy triggers.

logging:
  appenders:
    rolling-file:
      type: rolling-file
      fileName: /var/logs/kibana.log
      policy:
        type: size-limit
        size: 50mb
      strategy:
        //...
      layout:
        type: pattern

logging:
  appenders:
    rolling-file:
      type: rolling-file
      fileName: /var/logs/kibana.log
      policy:
        type: time-interval
        interval: 10s
        modulate: true
      strategy:
        //...
      layout:
        type: pattern

logging:
  appenders:
    rolling-file:
      type: rolling-file
      fileName: /var/logs/kibana.log
      policy:
        // ...
      strategy:
        type: numeric
        pattern: '-%i'
        max: 2
      layout:
        type: pattern

Similar to log4j’s RewriteAppender, this appender serves as a sort of middleware, modifying the provided log events before passing them along to another appender.

logging:
  appenders:
    my-rewrite-appender:
      type: rewrite
      appenders: [console, file] # name of "destination" appender(s)
      policy:
        # ...

The most common use case for the RewriteAppender is when you want to filter or censor sensitive data that may be contained in a log entry. In fact, with a default configuration, Kibana will automatically redact any authorization, cookie, or set-cookie headers when logging http requests & responses.

To configure additional rewrite rules, you’ll need to specify a RewritePolicy.

logging:
  appenders:
    my-rewrite-appender:
      type: rewrite
      appenders: [console]
      policy:
        type: meta # indicates that we want to rewrite the LogMeta
        mode: update # will update an existing property only
        properties:
          - path: "http.request.headers.cookie" # path to property
            value: "[REDACTED]" # value to replace at path

logging:
  appenders:
    remove-request-headers:
      type: rewrite
      appenders: [censor-response-headers] # redirect to the next rewrite appender
      policy:
        type: meta
        mode: remove
        properties:
          - path: "http.request.headers" # remove all request headers
    censor-response-headers:
      type: rewrite
      appenders: [console] # output to console
      policy:
        type: meta
        mode: update
        properties:
          - path: "http.response.headers.set-cookie"
            value: "[REDACTED]"

logging:
  appenders:
    custom_console:
      type: console
      layout:
        type: pattern
        highlight: true
        pattern: "[%date][%level][%logger] %message %meta"
    file:
      type: file
      fileName: ./kibana.log
      layout:
        type: json
    censor:
      type: rewrite
      appenders: [custom_console, file]
      policy:
        type: meta
        mode: update
        properties:
          - path: "http.request.headers.cookie"
            value: "[REDACTED]"
  loggers:
    - name: http.server.response
      appenders: [censor] # pass these logs to our rewrite appender
      level: debug

Root

The root logger has a dedicated configuration node since this logger is special and should always exist. By default root is configured with info level and default appender that is also always available. This is the configuration that all custom loggers will use unless they’re re-configured explicitly.

For example to see all log messages that fall back on the root logger configuration, just add one line to the configuration:

logging.root.level: all

Or disable logging entirely with off:

logging.root.level: off

Metrics Logs

The metrics.ops logger is configured with debug level and will automatically output sample system and process information at a regular interval. The metrics that are logged are a subset of the data collected and are formatted in the log message as follows:

The log interval is the same as the interval at which system and process information is refreshed and is configurable under ops.interval:

ops.interval: 5000

The minimum interval is 100ms and defaults to 5000ms.

Request and Response Logs

The http.server.response logger is configured with debug level and will automatically output data about http requests and responses occurring on the Kibana server. The message contains some high-level information, and the corresponding log meta contains the following: