Manage multiline messages
editManage multiline messages
editThe files harvested by Filebeat may contain messages that span multiple
lines of text. For example, multiline messages are common in files that contain
Java stack traces. In order to correctly handle these multiline events, you need
to configure multiline
settings in the filebeat.yml
file to specify
which lines are part of a single event.
If you are sending multiline events to Logstash, use the options described here to handle multiline events before sending the event data to Logstash. Trying to implement multiline event handling in Logstash (for example, by using the Logstash multiline codec) may result in the mixing of streams and corrupted data.
Also read YAML tips and gotchas and Regular expression support to avoid common mistakes.
Configuration options
editYou can specify the following options in the filebeat.inputs
section of
the filebeat.yml
config file to control how Filebeat deals with messages
that span multiple lines.
The following example shows how to configure Filebeat to handle a multiline message where the first line of the message begins with a bracket ([
).
multiline.pattern: '^\[' multiline.negate: true multiline.match: after
Filebeat takes all the lines that do not start with [
and combines them with the previous line that does. For example, you could use this configuration to join the following lines of a multiline message into a single event:
[beat-logstash-some-name-832-2015.11.28] IndexNotFoundException[no such index] at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:566) at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:133) at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:77) at org.elasticsearch.action.admin.indices.delete.TransportDeleteIndexAction.checkBlock(TransportDeleteIndexAction.java:75)
-
multiline.pattern
-
Specifies the regular expression pattern to match. Note that the regexp patterns supported by Filebeat
differ somewhat from the patterns supported by Logstash. See Regular expression support for a list of supported regexp patterns.
Depending on how you configure other multiline options, lines that match the specified regular expression are considered
either continuations of a previous line or the start of a new multiline event. You can set the
negate
option to negate the pattern. -
multiline.negate
-
Defines whether the pattern is negated. The default is
false
. -
multiline.match
-
Specifies how Filebeat combines matching lines into an event. The settings are
after
orbefore
. The behavior of these settings depends on what you specify fornegate
:Setting for negate
Setting for match
Result Example pattern: ^b
false
after
Consecutive lines that match the pattern are appended to the previous line that doesn’t match.
false
before
Consecutive lines that match the pattern are prepended to the next line that doesn’t match.
true
after
Consecutive lines that don’t match the pattern are appended to the previous line that does match.
true
before
Consecutive lines that don’t match the pattern are prepended to the next line that does match.
The
after
setting is equivalent toprevious
in Logstash, andbefore
is equivalent tonext
. -
multiline.flush_pattern
- Specifies a regular expression, in which the current multiline will be flushed from memory, ending the multline-message.
-
multiline.max_lines
-
The maximum number of lines that can be combined into one event. If
the multiline message contains more than
max_lines
, any additional lines are discarded. The default is 500. -
multiline.timeout
- After the specified timeout, Filebeat sends the multiline event even if no new pattern is found to start a new event. The default is 5s.