Grok filter plugin

The dissect filter plugin is another way to extract unstructured event data into fields using delimiters.

Dissect differs from Grok in that it does not use regular expressions and is faster. Dissect works well when data is reliably repeated. Grok is a better choice when the structure of your text varies from line to line.

You can use both Dissect and Grok for a hybrid use case when a section of the line is reliably repeated, but the entire line is not. The Dissect filter can deconstruct the section of the line that is repeated. The Grok filter can process the remaining field values with more regex predictability.

Break on first match. The first successful match by grok will result in the filter being finished. If you want grok to try all patterns (maybe you are parsing different things), then set this to false.

Controls this plugin’s compatibility with the Elastic Common Schema (ECS). The value of this setting affects extracted event field names when a composite pattern (such as HTTPD_COMMONLOG) is matched.

If true, keep empty captures as event fields.

A hash that defines the mapping of where to look, and with which patterns.

For example, the following will match an existing value in the message field for the given pattern, and if a match is found will add the field duration to the event with the captured value:

    filter {
      grok {
        match => {
          "message" => "Duration: %{NUMBER:duration}"
        }
      }
    }

If you need to match multiple patterns against a single field, the value can be an array of patterns:

    filter {
      grok {
        match => {
          "message" => [
            "Duration: %{NUMBER:duration}",
            "Speed: %{NUMBER:speed}"
          ]
        }
      }
    }

To perform matches on multiple fields just use multiple entries in the match hash:

    filter {
      grok {
        match => {
          "speed" => "Speed: %{NUMBER:speed}"
          "duration => "Duration: %{NUMBER:duration}"
        }
      }
    }

However, if one pattern depends on a field created by a previous pattern, separate these into two separate grok filters:

    filter {
      grok {
        match => {
          "message" => "Hi, the rest of the message is: %{GREEDYDATA:rest}"
        }
      }
      grok {
        match => {
          "rest => "a number %{NUMBER"number}, and a word %{WORD:word}"
        }
      }
    }

If true, only store named captures from grok.

The fields to overwrite.

This allows you to overwrite a value in a field that already exists.

For example, if you have a syslog line in the message field, you can overwrite the message field with part of the match like so:

    filter {
      grok {
        match => { "message" => "%{SYSLOGBASE} %{DATA:message}" }
        overwrite => [ "message" ]
      }
    }

In this case, a line like May 29 16:37:11 sadness logger: hello world will be parsed and hello world will overwrite the original message.

If you are using a field reference in overwrite, you must use the field reference in the pattern. Example:

    filter {
      grok {
        match => { "somefield" => "%{NUMBER} %{GREEDYDATA:[nested][field][test]}" }
        overwrite => [ "[nested][field][test]" ]
      }
    }

A hash of pattern-name and pattern tuples defining custom patterns to be used by the current filter. Patterns matching existing names will override the pre-existing definition. Think of this as inline patterns available just for this definition of grok

Logstash ships by default with a bunch of patterns, so you don’t necessarily need to define this yourself unless you are adding additional patterns. You can point to multiple pattern directories using this setting. Note that Grok will read all files in the directory matching the patterns_files_glob and assume it’s a pattern file (including any tilde backup files).

    patterns_dir => ["/opt/logstash/patterns", "/opt/logstash/extra_patterns"]

Pattern files are plain text with format:

    NAME PATTERN

For example:

    NUMBER \d+

The patterns are loaded when the pipeline is created.

Glob pattern, used to select the pattern files in the directories specified by patterns_dir

Append values to the tags field when there has been no successful match

Tag to apply if a grok regexp times out.

Define target namespace for placing matches.

Attempt to terminate regexps after this amount of time. This applies per pattern if multiple patterns are applied This will never timeout early, but may take a little longer to timeout. Actual timeout is approximate based on a 250ms quantization. Set to 0 to disable timeouts

When multiple patterns are provided to match, the timeout has historically applied to each pattern, incurring overhead for each and every pattern that is attempted; when the grok filter is configured with timeout_scope => event, the plugin instead enforces a single timeout across all attempted matches on the event, so it can achieve similar safeguard against runaway matchers with significantly less overhead.

It’s usually better to scope the timeout for the whole event.

If this filter is successful, add any arbitrary fields to this event. Field names can be dynamic and include parts of the event using the %{field}.

Example:

    filter {
      grok {
        add_field => { "foo_%{somefield}" => "Hello world, from %{host}" }
      }
    }

    # You can also add multiple fields at once:
    filter {
      grok {
        add_field => {
          "foo_%{somefield}" => "Hello world, from %{host}"
          "new_field" => "new_static_value"
        }
      }
    }

If the event has field "somefield" == "hello" this filter, on success, would add field foo_hello if it is present, with the value above and the %{host} piece replaced with that value from the event. The second example would also add a hardcoded field.

If this filter is successful, add arbitrary tags to the event. Tags can be dynamic and include parts of the event using the %{field} syntax.

Example:

    filter {
      grok {
        add_tag => [ "foo_%{somefield}" ]
      }
    }

    # You can also add multiple tags at once:
    filter {
      grok {
        add_tag => [ "foo_%{somefield}", "taggedy_tag"]
      }
    }

If the event has field "somefield" == "hello" this filter, on success, would add a tag foo_hello (and the second example would of course add a taggedy_tag tag).

Disable or enable metric logging for this specific plugin instance. By default we record all the metrics we can, but you can disable metrics collection for a specific plugin.

Add a unique ID to the plugin configuration. If no ID is specified, Logstash will generate one. It is strongly recommended to set this ID in your configuration. This is particularly useful when you have two or more plugins of the same type, for example, if you have 2 grok filters. Adding a named ID in this case will help in monitoring Logstash when using the monitoring APIs.

    filter {
      grok {
        id => "ABC"
      }
    }

Call the filter flush method at regular interval. Optional.

If this filter is successful, remove arbitrary fields from this event. Fields names can be dynamic and include parts of the event using the %{field} Example:

    filter {
      grok {
        remove_field => [ "foo_%{somefield}" ]
      }
    }

    # You can also remove multiple fields at once:
    filter {
      grok {
        remove_field => [ "foo_%{somefield}", "my_extraneous_field" ]
      }
    }

If the event has field "somefield" == "hello" this filter, on success, would remove the field with name foo_hello if it is present. The second example would remove an additional, non-dynamic field.

If this filter is successful, remove arbitrary tags from the event. Tags can be dynamic and include parts of the event using the %{field} syntax.

Example:

    filter {
      grok {
        remove_tag => [ "foo_%{somefield}" ]
      }
    }

    # You can also remove multiple tags at once:
    filter {
      grok {
        remove_tag => [ "foo_%{somefield}", "sad_unwanted_tag"]
      }
    }

If the event has field "somefield" == "hello" this filter, on success, would remove the tag foo_hello if it is present. The second example would remove a sad, unwanted tag as well.