Complex Conditionals

The if condition can be more complex than a simple equality check. The full power of the Painless Scripting Language is available and running in the ingest processor context.

A more complex if condition that drops the document (i.e. not index it) unless it has a multi-valued tag field with at least one value that contains the characters prod (case insensitive).

PUT _ingest/pipeline/not_prod_dropper
{
  "processors": [
    {
      "drop": {
        "if": "Collection tags = ctx.tags;if(tags != null){for (String tag : tags) {if (tag.toLowerCase().contains('prod')) { return false;}}} return true;"
      }
    }
  ]
}

The conditional needs to be all on one line since JSON does not support new line characters. However, Kibana’s console supports a triple quote syntax to help with writing and debugging scripts like these.

PUT _ingest/pipeline/not_prod_dropper
{
  "processors": [
    {
      "drop": {
        "if": """
            Collection tags = ctx.tags;
            if(tags != null){
              for (String tag : tags) {
                  if (tag.toLowerCase().contains('prod')) {
                      return false;
                  }
              }
            }
            return true;
        """
      }
    }
  ]
}

POST test/_doc/1?pipeline=not_prod_dropper
{
  "tags": ["application:myapp", "env:Stage"]
}

The document is dropped since prod (case insensitive) is not found in the tags.

The following document is indexed (i.e. not dropped) since prod (case insensitive) is found in the tags.

POST test/_doc/2?pipeline=not_prod_dropper
{
  "tags": ["application:myapp", "env:Production"]
}

The Simulate pipeline with verbose can be used to help build out complex conditionals. If the conditional evaluates to false it will be omitted from the verbose results of the simulation since the document will not change.

Care should be taken to avoid overly complex or expensive conditional checks since the condition needs to be checked for each and every document.