Field References Deep Dive

It is often useful to be able to refer to a field or collection of fields by name. To do this, you can use the Logstash field reference syntax.

The syntax to access a field specifies the entire path to the field, with each fragment wrapped in square brackets.

Field References can be expressed literally within Conditional statements in your pipeline configurations, as string arguments to your pipeline plugins, or within sprintf statements that will be used by your pipeline plugins:

filter {
  #  +----literal----+     +----literal----+
  #  |               |     |               |
  if [@metadata][date] and [@metadata][time] {
    mutate {
      add_field {
        "[@metadata][timestamp]" => "%{[@metadata][date]} %{[@metadata][time]}"
      # |                      |    |  |               |    |               | |
      # +----string-argument---+    |  +--field-ref----+    +--field-ref----+ |
      #                             +-------- sprintf format string ----------+
      }
    }
  }
}

Formal Grammar

edit

Below is the formal grammar of the Field Reference, with notes and examples.

Field Reference Literal

edit

A Field Reference Literal is a sequence of one or more Path Fragments that can be used directly in Logstash pipeline conditionals without any additional quoting (e.g. [request], [response][status]).

fieldReferenceLiteral
  : ( pathFragment )+
  ;

In Logstash 7.x and earlier, a quoted value (such as ["foo"]) is considered a field reference and isn’t treated as a single element array. This behavior might cause confusion in conditionals, such as [message] in ["foo", "bar"] compared to [message] in ["foo"]. We discourage using names with quotes, such as "\"foo\"", as this behavior might change in the future.

Field Reference (Event APIs)

edit

The Event API’s methods for manipulating the fields of an event or using the sprintf syntax are more flexible than the pipeline grammar in what they accept as a Field Reference. Top-level fields can be referenced directly by their Field Name without the square brackets, and there is some support for Composite Field References, simplifying use of programmatically-generated Field References.

A Field Reference for use with the Event API is therefore one of:

a single Field Reference Literal; OR
a single Field Name (referencing a top-level field); OR
a single Composite Field Reference.

eventApiFieldReference
  : fieldReferenceLiteral
  | fieldName
  | compositeFieldReference
  ;

Path Fragment

edit

A Path Fragment is a Field Name wrapped in square brackets (e.g., [request]).

pathFragment
  : '[' fieldName ']'
  ;

Field Name

edit

A Field Name is a sequence of characters that are not square brackets ([ or ]).

fieldName
  : ( ~( '[' | ']' ) )+
  ;

Composite Field Reference

edit

In some cases, it may be necessary to programmatically compose a Field Reference from one or more Field References, such as when manipulating fields in a plugin or while using the Ruby Filter plugin and the Event API.

    fieldReference = "[path][to][deep nested field]"
    compositeFieldReference = "[@metadata][#{fieldReference}][size]"
    # => "[@metadata][[path][to][deep nested field]][size]"

Canonical Representations of Composite Field References

edit

Acceptable Composite Field Reference	Canonical Field Reference Representation
`+[[deep][nesting]][field]+`	`+[deep][nesting][field]+`
`+[foo][[bar]][bingo]+`	`+[foo][bar][bingo]+`
`+[[ok]]+`	`+[ok]+`

A Composite Field Reference is a sequence of one or more Path Fragments or Embedded Field References.

compositeFieldReference
  : ( pathFragment | embeddedFieldReference )+
  ;

Composite Field References are supported by the Event API, but are not supported as literals in the Pipeline Configuration.

Embedded Field Reference

edit

embeddedFieldReference
  : '[' fieldReference ']'
  ;

An Embedded Field Reference is a Field Reference that is itself wrapped in square brackets ([ and ]), and can be a component of a Composite Field Reference.

« Configuring Centralized Pipeline Management Managing Logstash »