Troubleshooting Logstash

Symptoms of persistent queue problems include Logstash or one or more pipelines not starting successfully, accompanied by an error message similar to this one.

message=>"java.io.IOException: Page file size is too small to hold elements"

See the troubleshooting information in the persistent queue section for more information on remediating problems with persistent queues.

For general performance tuning tips and guidelines, see Performance tuning.

Pipelines, by definition, are unique. Here are some guidelines to help you get started.

For basic pipelines, this configuration could be enough to make the problem show itself.

input {stdin{}} output {stdout{}}

Logstash can separate logs by pipeline. This feature can help you identify the offending pipeline. Set pipeline.separate_logs: true in your logstash.yml to enable the log per pipeline feature.

For more complex pipelines, the problem could be caused by a series of plugins in a specific order. Troubleshooting these pipelines usually requires trial and error. Start by systematically removing input and output plugins until you’re left with the minimum set that manifest the issue.

We want to expand this section to make it more helpful. If you have troubleshooting tips to share, please:

Symptoms

Simple filters such as mutate or json filter can take several milliseconds per event to execute. Inputs and outputs might be affected, too.

Background

The different plugins running on Logstash can be quite verbose if the logging level is set to debug or trace. As the logging library used in Logstash is synchronous, heavy logging can affect performances.

Solution

Reset the logging level to info.

Symptoms

When log format is json and certain log events (for example errors from JSON codec plugin) contains two instances of the message field.

Without setting this flag, json log would contain objects like:

{
   "level":"WARN",
   "loggerName":"logstash.codecs.jsonlines",
   "timeMillis":1712937761955,
   "thread":"[main]<stdin",
   "logEvent":{
      "message":"JSON parse error, original data now in message field",
      "message":"Unexpected close marker '}': expected ']' (for Array starting at [Source: (String)\"{\"name\": [}\"; line: 1, column: 10])\n at [Source: (String)\"{\"name\": [}\"; line: 1, column: 12]",
      "exception":"LogStash::Json::ParserError",
      "data":"{\"name\": [}"
   }
}

Please note the duplication of message field, while being technically valid json, it is not always parsed correctly.

Solution In config/logstash.yml enable the strict json flag:

log.format.json.fix_duplicate_message_fields: true

or pass the command line switch

bin/logstash --log.format.json.fix_duplicate_message_fields true

With log.format.json.fix_duplicate_message_fields enabled the duplication of message field is removed, adding to the field name a _1 suffix:

{
   "level":"WARN",
   "loggerName":"logstash.codecs.jsonlines",
   "timeMillis":1712937629789,
   "thread":"[main]<stdin",
   "logEvent":{
      "message":"JSON parse error, original data now in message field",
      "message_1":"Unexpected close marker '}': expected ']' (for Array starting at [Source: (String)\"{\"name\": [}\"; line: 1, column: 10])\n at [Source: (String)\"{\"name\": [}\"; line: 1, column: 12]",
      "exception":"LogStash::Json::ParserError",
      "data":"{\"name\": [}"
   }
}