Data processing with DISSECT and GROK

A dissect pattern is defined by the parts of the string that will be discarded. In the previous example, the first part to be discarded is a single space. Dissect finds this space, then assigns the value of clientip everything up until that space. Next, dissect matches the [ and then ] and then assigns @timestamp to everything in-between [ and ]. Paying special attention to the parts of the string to discard will help build successful dissect patterns.

An empty key (%{}) or named skip key can be used to match values, but exclude the value from the output.

All matched values are output as keyword string data types. Use the Type conversion functions to convert to another data type.

Dissect also supports key modifiers that can change dissect’s default behavior. For example, you can instruct dissect to ignore certain fields, append fields, skip over padding, etc.

The following example parses a string that contains a timestamp, some text, and an IP address:

ROW a = "2023-01-23T12:15:00.000Z - some text - 127.0.0.1"
| DISSECT a """%{date} - %{msg} - %{ip}"""
| KEEP date, msg, ip

By default, DISSECT outputs keyword string columns. To convert to another type, use Type conversion functions:

ROW a = "2023-01-23T12:15:00.000Z - some text - 127.0.0.1"
| DISSECT a """%{date} - %{msg} - %{ip}"""
| KEEP date, msg, ip
| EVAL date = TO_DATETIME(date)

Key modifiers can change the default behavior for dissection. Key modifiers may be found on the left or right of the %{keyname} always inside the %{ and }. For example %{+keyname ->} has the append and right padding modifiers.

ROW message="1998-08-10T17:15:42          WARN"
| DISSECT message """%{ts->} %{level}"""

ROW message="[1998-08-10T17:15:42]          [WARN]"
| DISSECT message """[%{ts}]%{->}[%{level}]"""

ROW message="john jacob jingleheimer schmidt"
| DISSECT message """%{+name} %{+name} %{+name} %{+name}""" APPEND_SEPARATOR=" "

ROW message="john jacob jingleheimer schmidt"
| DISSECT message """%{+name/2} %{+name/4} %{+name/3} %{+name/1}""" APPEND_SEPARATOR=","

ROW message="1.2.3.4 - - 30/Apr/1998:22:00:52 +0000"
| DISSECT message """%{clientip} %{?ident} %{?auth} %{@timestamp}"""

The DISSECT command does not support reference keys.

The syntax for a grok pattern is %{SYNTAX:SEMANTIC}

The SYNTAX is the name of the pattern that matches your text. For example, 3.44 is matched by the NUMBER pattern and 55.3.244.1 is matched by the IP pattern. The syntax is how you match.

The SEMANTIC is the identifier you give to the piece of text being matched. For example, 3.44 could be the duration of an event, so you could call it simply duration. Further, a string 55.3.244.1 might identify the client making a request.

By default, matched values are output as keyword string data types. To convert a semantic’s data type, suffix it with the target data type. For example %{NUMBER:num:int}, which converts the num semantic from a string to an integer. Currently the only supported conversions are int and float. For other types, use the Type conversion functions.

For an overview of the available patterns, refer to GitHub. You can also retrieve a list of all patterns using a REST API.

Grok is based on regular expressions. Any regular expressions are valid in grok as well. Grok uses the Oniguruma regular expression library. Refer to the Oniguruma GitHub repository for the full supported regexp syntax.

If grok doesn’t have a pattern you need, you can use the Oniguruma syntax for named capture which lets you match a piece of text and save it as a column:

(?<field_name>the pattern here)

For example, postfix logs have a queue id that is a 10 or 11-character hexadecimal value. This can be captured to a column named queue_id with:

(?<queue_id>[0-9A-F]{10,11})

The following example parses a string that contains a timestamp, an IP address, an email address, and a number:

ROW a = "2023-01-23T12:15:00.000Z 127.0.0.1 some.email@foo.com 42"
| GROK a """%{TIMESTAMP_ISO8601:date} %{IP:ip} %{EMAILADDRESS:email} %{NUMBER:num}"""
| KEEP date, ip, email, num

By default, GROK outputs keyword string columns. int and float types can be converted by appending :type to the semantics in the pattern. For example {NUMBER:num:int}:

ROW a = "2023-01-23T12:15:00.000Z 127.0.0.1 some.email@foo.com 42"
| GROK a """%{TIMESTAMP_ISO8601:date} %{IP:ip} %{EMAILADDRESS:email} %{NUMBER:num:int}"""
| KEEP date, ip, email, num

For other type conversions, use Type conversion functions:

ROW a = "2023-01-23T12:15:00.000Z 127.0.0.1 some.email@foo.com 42"
| GROK a """%{TIMESTAMP_ISO8601:date} %{IP:ip} %{EMAILADDRESS:email} %{NUMBER:num:int}"""
| KEEP date, ip, email, num
| EVAL date = TO_DATETIME(date)

If a field name is used more than once, GROK creates a multi-valued column:

FROM addresses
| KEEP city.name, zip_code
| GROK zip_code """%{WORD:zip_parts} %{WORD:zip_parts}"""

To write and debug grok patterns, you can use the Grok Debugger. It provides a UI for testing patterns against sample data. Under the covers, it uses the same engine as the GROK command.

The GROK command does not support configuring custom patterns, or multiple patterns. The GROK command is not subject to Grok watchdog settings.