Processors

All processors are defined in the following way within a pipeline definition:

{
  "PROCESSOR_NAME" : {
    ... processor configuration options ...
  }
}

Each processor defines its own configuration parameters, but all processors have the ability to declare tag, on_failure and if fields. These fields are optional.

A tag is simply a string identifier of the specific instantiation of a certain processor in a pipeline. The tag field does not affect the processor’s behavior, but is very useful for bookkeeping and tracing errors to specific processors.

The if field must contain a script that returns a boolean value. If the script evaluates to true then the processor will be executed for the given document otherwise it will be skipped. The if field takes an object with the script fields defined in script-options and accesses a read only version of the document via the same ctx variable used by scripts in the Script Processor.

{
  "set": {
    "if": "ctx.foo == 'someValue'",
    "field": "found",
    "value": true
  }
}

See Conditional Execution in Pipelines to learn more about the if field and conditional execution.

See Handling Failures in Pipelines to learn more about the on_failure field and error handling in pipelines.

The node info API can be used to figure out what processors are available in a cluster. The node info API will provide a per node list of what processors are available.

Custom processors must be installed on all nodes. The put pipeline API will fail if a processor specified in a pipeline doesn’t exist on all nodes. If you rely on custom processor plugins make sure to mark these plugins as mandatory by adding plugin.mandatory setting to the config/elasticsearch.yml file, for example:

plugin.mandatory: ingest-attachment,ingest-geoip

A node will not start if either of these plugins are not available.

The node stats API can be used to fetch ingest usage statistics, globally and on a per pipeline basis. Useful to find out which pipelines are used the most or spent the most time on preprocessing.

Additional ingest processors can be implemented and installed as Elasticsearch plugins. See Ingest plugins for information about the available ingest plugins.