Foreach Processor
editForeach Processor
editProcesses elements in an array of unknown length.
All processors can operate on elements inside an array, but if all elements of an array need to
be processed in the same way, defining a processor for each element becomes cumbersome and tricky
because it is likely that the number of elements in an array is unknown. For this reason the foreach
processor exists. By specifying the field holding array elements and a processor that
defines what should happen to each element, array fields can easily be preprocessed.
A processor inside the foreach processor works in the array element context and puts that in the ingest metadata
under the _ingest._value
key. If the array element is a json object it holds all immediate fields of that json object.
and if the nested object is a value is _ingest._value
just holds that value. Note that if a processor prior to the
foreach
processor used _ingest._value
key then the specified value will not be available to the processor inside
the foreach
processor. The foreach
processor does restore the original value, so that value is available to processors
after the foreach
processor.
Note that any other field from the document are accessible and modifiable like with all other processors. This processor
just puts the current array element being read into _ingest._value
ingest metadata attribute, so that it may be
pre-processed.
If the foreach
processor fails to process an element inside the array, and no on_failure
processor has been specified,
then it aborts the execution and leaves the array unmodified.
Table 41. Foreach Options
Name | Required | Default | Description |
---|---|---|---|
|
yes |
- |
The array field |
|
yes |
- |
The processor to execute against each field |
|
no |
false |
If |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
Assume the following document:
{ "values" : ["foo", "bar", "baz"] }
When this foreach
processor operates on this sample document:
{ "foreach" : { "field" : "values", "processor" : { "uppercase" : { "field" : "_ingest._value" } } } }
Then the document will look like this after preprocessing:
{ "values" : ["FOO", "BAR", "BAZ"] }
Let’s take a look at another example:
{ "persons" : [ { "id" : "1", "name" : "John Doe" }, { "id" : "2", "name" : "Jane Doe" } ] }
In this case, the id
field needs to be removed,
so the following foreach
processor is used:
{ "foreach" : { "field" : "persons", "processor" : { "remove" : { "field" : "_ingest._value.id" } } } }
After preprocessing the result is:
{ "persons" : [ { "name" : "John Doe" }, { "name" : "Jane Doe" } ] }
The wrapped processor can have a on_failure
definition.
For example, the id
field may not exist on all person objects.
Instead of failing the index request, you can use an on_failure
block to send the document to the failure_index index for later inspection:
{ "foreach" : { "field" : "persons", "processor" : { "remove" : { "field" : "_value.id", "on_failure" : [ { "set" : { "field", "_index", "value", "failure_index" } } ] } } } }
In this example, if the remove
processor does fail, then
the array elements that have been processed thus far will
be updated.
Another advanced example can be found in the attachment processor documentation.