IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Date Index Name Processor Dot Expander Processor »
Elastic Docs ›Elasticsearch Guide [6.7] ›Ingest Node ›Processors

Dissect Processor

edit

Dissect Processor

edit

Similar to the Grok Processor, dissect also extracts structured fields out of a single text field within a document. However unlike the Grok Processor, dissect does not use Regular Expressions. This allows dissect’s syntax to be simple and for some cases faster than the Grok Processor.

Dissect matches a single text field against a defined pattern.

For example the following pattern:

%{clientip} %{ident} %{auth} [%{@timestamp}] \"%{verb} %{request} HTTP/%{httpversion}\" %{status} %{size}

will match a log line of this format:

1.2.3.4 - - [30/Apr/1998:22:00:52 +0000] \"GET /english/venues/cities/images/montpellier/18.gif HTTP/1.0\" 200 3171

and result in a document with the following fields:

"doc": {
  "_index": "_index",
  "_type": "_type",
  "_id": "_id",
  "_source": {
    "request": "/english/venues/cities/images/montpellier/18.gif",
    "auth": "-",
    "ident": "-",
    "verb": "GET",
    "@timestamp": "30/Apr/1998:22:00:52 +0000",
    "size": "3171",
    "clientip": "1.2.3.4",
    "httpversion": "1.0",
    "status": "200"
  }
}

A dissect pattern is defined by the parts of the string that will be discarded. In the example above the first part to be discarded is a single space. Dissect finds this space, then assigns the value of clientip is everything up until that space. Later dissect matches the [ and then ] and then assigns @timestamp to everything in-between [ and ]. Paying special attention the parts of the string to discard will help build successful dissect patterns.

Successful matches require all keys in a pattern to have a value. If any of the %{keyname} defined in the pattern do not have a value, then an exception is thrown and may be handled by the on_failure directive. An empty key %{} or a named skip key can be used to match values, but exclude the value from the final document. All matched values are represented as string data types. The convert processor may be used to convert to expected data type.

Dissect also supports key modifiers that can change dissect’s default behavior. For example you can instruct dissect to ignore certain fields, append fields, skip over padding, etc. See below for more information.

Table 34. Dissect Options

Name Required Default Description

field

yes

-

The field to dissect

pattern

yes

-

The pattern to apply to the field

append_separator

no

"" (empty string)

The character(s) that separate the appended fields.

ignore_missing

no

false

If true and field does not exist or is null, the processor quietly exits without modifying the document

if

no

-

Conditionally execute this processor.

on_failure

no

-

Handle failures for this processor. See Handling Failures in Pipelines.

ignore_failure

no

false

Ignore failures for this processor. See Handling Failures in Pipelines.

tag

no

-

An identifier for this processor. Useful for debugging and metrics.

 
{
  "dissect": {
    "field": "message",
    "pattern" : "%{clientip} %{ident} %{auth} [%{@timestamp}] \"%{verb} %{request} HTTP/%{httpversion}\" %{status} %{size}"
   }
}

Dissect key modifiers

edit

Key modifiers can change the default behavior for dissection. Key modifiers may be found on the left or right of the %{keyname} always inside the %{ and }. For example %{+keyname ->} has the append and right padding modifiers.

Table 35. Dissect Key Modifiers

Modifier Name Position Example Description Details

->

Skip right padding

(far) right

%{keyname1->}

Skips any repeated characters to the right

link

+

Append

left

%{+keyname} %{+keyname}

Appends two or more fields together

link

+ with /n

Append with order

left and right

%{+keyname/2} %{+keyname/1}

Appends two or more fields together in the order specified

link

?

Named skip key

left

%{?ignoreme}

Skips the matched value in the output. Same behavior as %{}

link

* and &

Reference keys

left

%{*r1} %{&r1}

Sets the output key as value of * and output value of &

link

Right padding modifier (->)

edit

The algorithm that performs the dissection is very strict in that it requires all characters in the pattern to match the source string. For example, the pattern %{fookey} %{barkey} (1 space), will match the string "foo bar" (1 space), but will not match the string "foo  bar" (2 spaces) since the pattern has only 1 space and the source string has 2 spaces.

The right padding modifier helps with this case. Adding the right padding modifier to the pattern %{fookey->} %{barkey}, It will now will match "foo bar" (1 space) and "foo  bar" (2 spaces) and even "foo          bar" (10 spaces).

Use the right padding modifier to allow for repetition of the characters after a %{keyname->}.

The right padding modifier may be placed on any key with any other modifiers. It should always be the furthest right modifier. For example: %{+keyname/1->} and %{->}

Right padding modifier example

Pattern

%{ts->} %{level}

Input

1998-08-10T17:15:42,466          WARN

Result
  • ts = 1998-08-10T17:15:42,466
  • level = WARN

The right padding modifier may be used with an empty key to help skip unwanted data. For example, the same input string, but wrapped with brackets requires the use of an empty right padded key to achieve the same result.

Right padding modifier with empty key example

Pattern

[%{ts}]%{->}[%{level}]

Input

[1998-08-10T17:15:42,466]            [WARN]

Result
  • ts = 1998-08-10T17:15:42,466
  • level = WARN

Append modifier (+)

edit

Dissect supports appending two or more results together for the output. Values are appended left to right. An append separator can be specified. In this example the append_separator is defined as a space.

Append modifier example

Pattern

%{+name} %{+name} %{+name} %{+name}

Input

john jacob jingleheimer schmidt

Result
  • name = john jacob jingleheimer schmidt

Append with order modifier (+ and /n)

edit

Dissect supports appending two or more results together for the output. Values are appended based on the order defined (/n). An append separator can be specified. In this example the append_separator is defined as a comma.

Append with order modifier example

Pattern

%{+name/2} %{+name/4} %{+name/3} %{+name/1}

Input

john jacob jingleheimer schmidt

Result
  • name = schmidt,john,jingleheimer,jacob

Named skip key (?)

edit

Dissect supports ignoring matches in the final result. This can be done with an empty key %{}, but for readability it may be desired to give that empty key a name.

Named skip key modifier example

Pattern

%{clientip} %{?ident} %{?auth} [%{@timestamp}]

Input

1.2.3.4 - - [30/Apr/1998:22:00:52 +0000]

Result
  • clientip = 1.2.3.4
  • @timestamp = 30/Apr/1998:22:00:52 +0000

Reference keys (* and &)

edit

Dissect support using parsed values as the key/value pairings for the structured content. Imagine a system that partially logs in key/value pairs. Reference keys allow you to maintain that key/value relationship.

Reference key modifier example

Pattern

[%{ts}] [%{level}] %{*p1}:%{&p1} %{*p2}:%{&p2}

Input

[2018-08-10T17:15:42,466] [ERR] ip:1.2.3.4 error:REFUSED

Result
  • ts = 1998-08-10T17:15:42,466
  • level = ERR
  • ip = 1.2.3.4
  • error = REFUSED
« Date Index Name Processor Dot Expander Processor »