WARNING: Version 6.1 of Filebeat has passed its EOL date.

This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.

« Specify multiple prospectors Examples of multiline configuration »
Elastic Docs ›Filebeat Reference [6.1] ›Configuring Filebeat

Manage multiline messages

edit

Manage multiline messages

edit

The files harvested by Filebeat may contain messages that span multiple lines of text. For example, multiline messages are common in files that contain Java stack traces. In order to correctly handle these multiline events, you need to configure multiline settings in the filebeat.yml file to specify which lines are part of a single event.

If you are sending multiline events to Logstash, use the options described here to handle multiline events before sending the event data to Logstash. Trying to implement multiline event handling in Logstash (for example, by using the Logstash multiline codec) may result in the mixing of streams and corrupted data.

Also read YAML tips and gotchas and Regular expression support to avoid common mistakes.

Configuration options

edit

You can specify the following options in the filebeat.prospectors section of the filebeat.yml config file to control how Filebeat deals with messages that span multiple lines.

The following example shows how to configure Filebeat to handle a multiline message where the first line of the message begins with a bracket ([).

multiline.pattern: '^\['
multiline.negate: true
multiline.match: after

Filebeat takes all the lines that do not start with [ and combines them with the previous line that does. For example, you could use this configuration to join the following lines of a multiline message into a single event:

[beat-logstash-some-name-832-2015.11.28] IndexNotFoundException[no such index]
    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:566)
    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:133)
    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:77)
    at org.elasticsearch.action.admin.indices.delete.TransportDeleteIndexAction.checkBlock(TransportDeleteIndexAction.java:75)
multiline.pattern
Specifies the regular expression pattern to match. Note that the regexp patterns supported by Filebeat differ somewhat from the patterns supported by Logstash. See Regular expression support for a list of supported regexp patterns. Depending on how you configure other multiline options, lines that match the specified regular expression are considered either continuations of a previous line or the start of a new multiline event. You can set the negate option to negate the pattern.
multiline.negate
Defines whether the pattern is negated. The default is false.
multiline.match

Specifies how Filebeat combines matching lines into an event. The settings are after or before. The behavior of these settings depends on what you specify for negate:

Setting for negate Setting for match Result Example pattern: ^b

false

after

Consecutive lines that match the pattern are appended to the previous line that doesn’t match.

Lines a b b c b b become "abb" and "cbb"

false

before

Consecutive lines that match the pattern are prepended to the next line that doesn’t match.

Lines b b a b b c become "bba" and "bbc"

true

after

Consecutive lines that don’t match the pattern are appended to the previous line that does match.

Lines b a c b d e become "bac" and "bde"

true

before

Consecutive lines that don’t match the pattern are prepended to the next line that does match.

Lines a c b d e b become "acb" and "deb"

The after setting is equivalent to previous in Logstash, and before is equivalent to next.

multiline.flush_pattern
Specifies a regular expression, in which the current multiline will be flushed from memory, ending the multline-message.
multiline.max_lines
The maximum number of lines that can be combined into one event. If the multiline message contains more than max_lines, any additional lines are discarded. The default is 500.
multiline.timeout
After the specified timeout, Filebeat sends the multiline event even if no new pattern is found to start a new event. The default is 5s.
« Specify multiple prospectors Examples of multiline configuration »