Dot expander processor

edit

Expands a field with dots into an object field. This processor allows fields with dots in the name to be accessible by other processors in the pipeline. Otherwise these fields can’t be accessed by any processor.

Table 15. Dot Expand Options

Name Required Default Description

field

yes

-

The field to expand into an object field. If set to *, all top-level fields will be expanded.

path

no

-

The field that contains the field to expand. Only required if the field to expand is part another object field, because the field option can only understand leaf fields.

override

no

false

Controls the behavior when there is already an existing nested object that conflicts with the expanded field. When false, the processor will merge conflicts by combining the old and the new values into an array. When true, the value from the expanded field will overwrite the existing value.

description

no

-

Description of the processor. Useful for describing the purpose of the processor or its configuration.

if

no

-

Conditionally execute the processor. See Conditionally run a processor.

ignore_failure

no

false

Ignore failures for the processor. See Handling pipeline failures.

on_failure

no

-

Handle failures for the processor. See Handling pipeline failures.

tag

no

-

Identifier for the processor. Useful for debugging and metrics.

{
  "dot_expander": {
    "field": "foo.bar"
  }
}

For example the dot expand processor would turn this document:

{
  "foo.bar" : "value"
}

into:

{
  "foo" : {
    "bar" : "value"
  }
}

If there is already a bar field nested under foo then this processor merges the foo.bar field into it. If the field is a scalar value then it will turn that field into an array field.

For example, the following document:

{
  "foo.bar" : "value2",
  "foo" : {
    "bar" : "value1"
  }
}

is transformed by the dot_expander processor into:

{
  "foo" : {
    "bar" : ["value1", "value2"]
  }
}

Contrast that with when the override option is set to true.

{
  "dot_expander": {
    "field": "foo.bar",
    "override": true
  }
}

In that case, the value of the expanded field overrides the value of the nested object.

{
  "foo" : {
    "bar" : "value2"
  }
}

The value of field can also be set to a * to expand all top-level dotted field names:

{
  "dot_expander": {
    "field": "*"
  }
}

The dot expand processor would turn this document:

{
  "foo.bar" : "value",
  "baz.qux" : "value"
}

into:

{
  "foo" : {
    "bar" : "value"
  },
  "baz" : {
    "qux" : "value"
  }
}

If the dotted field is nested within a non-dotted structure, then use the path option to navigate the non-dotted structure:

{
  "dot_expander": {
    "path": "foo"
    "field": "*"
  }
}

The dot expand processor would turn this document:

{
  "foo" : {
    "bar.one" : "value",
    "bar.two" : "value"
  }
}

into:

{
  "foo" : {
    "bar" : {
      "one" : "value",
      "two" : "value"
    }
  }
}

If any field outside of the leaf field conflicts with a pre-existing field of the same name, then that field needs to be renamed first.

Consider the following document:

{
  "foo": "value1",
  "foo.bar": "value2"
}

Then the foo needs to be renamed first before the dot_expander processor is applied. So in order for the foo.bar field to properly be expanded into the bar field under the foo field the following pipeline should be used:

{
  "processors" : [
    {
      "rename" : {
        "field" : "foo",
        "target_field" : "foo.bar"
      }
    },
    {
      "dot_expander": {
        "field": "foo.bar"
      }
    }
  ]
}

The reason for this is that Ingest doesn’t know how to automatically cast a scalar field to an object field.