Conditionals with the Pipeline Processor

The combination of the if conditional and the Pipeline Processor can result in a simple, yet powerful means to process heterogeneous input. For example, you can define a single pipeline that delegates to other pipelines based on some criteria.

PUT _ingest/pipeline/logs_pipeline
{
  "description": "A pipeline of pipelines for log files",
  "version": 1,
  "processors": [
    {
      "pipeline": {
        "if": "ctx.service?.name == 'apache_httpd'",
        "name": "httpd_pipeline"
      }
    },
    {
      "pipeline": {
        "if": "ctx.service?.name == 'syslog'",
        "name": "syslog_pipeline"
      }
    },
    {
      "fail": {
        "message": "This pipeline requires service.name to be either `syslog` or `apache_httpd`"
      }
    }
  ]
}

The above example allows consumers to point to a single pipeline for all log based index requests. Based on the conditional, the correct pipeline will be called to process that type of data.

This pattern works well with a default pipeline defined in an index mapping template for all indexes that hold data that needs pre-index processing.