Foreach processor

Runs an ingest processor on each element of an array or object.

All ingest processors can run on array or object elements. However, if the number of elements is unknown, it can be cumbersome to process each one in the same way.

The foreach processor lets you specify a field containing array or object values and a processor to run on each element in the field.

When iterating through an array or object, the foreach processor stores the current element’s value in the _ingest._value ingest metadata field. _ingest._value contains the entire element value, including any child fields. You can access child field values using dot notation on the _ingest._value field.

When iterating through an object, the foreach processor also stores the current element’s key as a string in _ingest._key.

You can access and change _ingest._key and _ingest._value in the processor. For an example, see the object example.

If the foreach processor fails to process an element and no on_failure processor is specified, the foreach processor silently exits. This leaves the entire array or object value unchanged.

The following examples show how you can use the foreach processor with different data types and options:

Assume the following document:

{
  "values" : ["foo", "bar", "baz"]
}

When this foreach processor operates on this sample document:

{
  "foreach" : {
    "field" : "values",
    "processor" : {
      "uppercase" : {
        "field" : "_ingest._value"
      }
    }
  }
}

Then the document will look like this after processing:

{
  "values" : ["FOO", "BAR", "BAZ"]
}

Assume the following document:

{
  "persons" : [
    {
      "id" : "1",
      "name" : "John Doe"
    },
    {
      "id" : "2",
      "name" : "Jane Doe"
    }
  ]
}

In this case, the id field needs to be removed, so the following foreach processor is used:

{
  "foreach" : {
    "field" : "persons",
    "processor" : {
      "remove" : {
        "field" : "_ingest._value.id"
      }
    }
  }
}

After processing the result is:

{
  "persons" : [
    {
      "name" : "John Doe"
    },
    {
      "name" : "Jane Doe"
    }
  ]
}

For another array of objects example, refer to the attachment processor documentation.

You can also use the foreach processor on object fields. For example, the following document contains a products field with object values.

{
  "products" : {
    "widgets" : {
      "total_sales" : 50,
      "unit_price": 1.99,
      "display_name": ""
    },
    "sprockets" : {
      "total_sales" : 100,
      "unit_price": 9.99,
      "display_name": "Super Sprockets"
    },
    "whizbangs" : {
      "total_sales" : 200,
      "unit_price": 19.99,
      "display_name": "Wonderful Whizbangs"
    }
  }
}

The following foreach processor changes the value of products.display_name to uppercase.

{
  "foreach": {
    "field": "products",
    "processor": {
      "uppercase": {
        "field": "_ingest._value.display_name"
      }
    }
  }
}

When run on the document, the foreach processor returns:

{
  "products" : {
    "widgets" : {
      "total_sales" : 50,
      "unit_price" : 1.99,
      "display_name" : ""
    },
    "sprockets" : {
      "total_sales" : 100,
      "unit_price" : 9.99,
      "display_name" : "SUPER SPROCKETS"
    },
    "whizbangs" : {
      "total_sales" : 200,
      "unit_price" : 19.99,
      "display_name" : "WONDERFUL WHIZBANGS"
    }
  }
}

The following foreach processor sets each element’s key to the value of products.display_name. If products.display_name contains an empty string, the processor deletes the element.

{
  "foreach": {
    "field": "products",
    "processor": {
      "set": {
        "field": "_ingest._key",
        "value": "{{_ingest._value.display_name}}"
      }
    }
  }
}

When run on the previous document, the foreach processor returns:

{
  "products" : {
    "Wonderful Whizbangs" : {
      "total_sales" : 200,
      "unit_price" : 19.99,
      "display_name" : "Wonderful Whizbangs"
    },
    "Super Sprockets" : {
      "total_sales" : 100,
      "unit_price" : 9.99,
      "display_name" : "Super Sprockets"
    }
  }
}

The wrapped processor can have a on_failure definition. For example, the id field may not exist on all person objects. Instead of failing the index request, you can use an on_failure block to send the document to the failure_index index for later inspection:

{
  "foreach" : {
    "field" : "persons",
    "processor" : {
      "remove" : {
        "field" : "_value.id",
        "on_failure" : [
          {
            "set" : {
              "field": "_index",
              "value": "failure_index"
            }
          }
        ]
      }
    }
  }
}

In this example, if the remove processor does fail, then the array elements that have been processed thus far will be updated.