From version 5.0 onward, Watcher is part of X-Pack. For more information, see Alerting on cluster and index events.

« Actions Java API »

› ›

Transform

edit

Transform

edit

A transform processes the payload in the watch execution context to prepare the payload for watch actions.

If no transforms are defined, the actions have access to the payload as loaded by the watch input.

You can define transforms in two places:

As a top level construct in the watch definition. In this case, the payload is transformed before any of the watch actions are executed.
As part of the definition of a particular action. In this case, the payload is transformed before that action is executed. The transformation is only applied to the payload for that specific action.

If all actions require the same view of the payload, define a transform as part of the watch definition. If each action requires a different view of the payload, define different transforms as part of the action definitions so each action has the payload prepared by its own dedicated transform.

The following example defines two transforms, one at the watch level and one as part of the definition of the my_webhook action.

Watch Transform Constructs.

{
  "trigger" : { ...}
  "input" : { ... },
  "condition" : { ... },
  "transform" : { 
    "search" : {
      "body" : { "query" : { "match_all" : {} } }
    }
  }
  "actions" : {
    "my_webhook": {
      "transform" : { 
      	"script" : "return ctx.payload.hits"
      }
      "webhook" : {
      	"host" : "host.domain",
      	"port" : 8089,
      	"path" : "/notify/{{ctx.watch_id}}"
      }
    }
  ]
  ...
}

	A watch level `transform`
	An action level `transform`

Watcher supports three types of transforms: search, script and chain.

Search Transform

edit

A Transform that executes a search on the cluster and replaces the current payload in the watch execution context with the returned search results. The following snippet shows how a simple search transform can be defined on the watch level:

Simple Search Transform.

{
  ...

  "transform" : {
    "search" : {
      "request" : {
        "body" : { "query" : { "match_all" : {} }}
      }
    }
  }
  ...
}

Like every other search based construct, one can make use of elasticsearch’s full search API by providing additional parameters:

Simple Search Transform.

{
  "transform" : {
    "search" : {
      "request" : {
        "indices" : [ "logstash-*" ],
        "body" : {
          "size" : 0,
          "query" : {
            "match" : { "priority" : "error"}
          }
        }
      }
    }
  }
}

The above example executes a count search over all the logstash indices, matching all the events with error priority.

The following table lists all available settings for the search transform:

Table 15. Search Transform Settings

Name	Required	Default	Description
`request.search_type`	no	query_then_fetch	The search search type
`request.indices`	no	all indices	One or more indices to search on (may be a comma-delimited string or an array of indices names).
`request.types`	no	all types	One or more document types to search on (may be a comma-delimited string or an array of document types names)
`request.body`	no	`match_all` query	The body of the request. The request body follows the same structure you normally send in the body of a REST `_search` request. The body can be static text or include `mustache` templates.
`request.indices_options.expand_wildcards`	no	`open`	Determines how to expand indices wildcards. Can be one of `open`, `closed`, `none` or `all` (see multi-index support)
`request.indices_options.ignore_unavailable`	no	`true`	A boolean value that determines whether the search should leniently ignore unavailable indices ((see multi-index support)
`request.indices_options.allow_no_indices`	no	`true`	A boolean value that determines whether the search should leniently return no results when no indices are resolved ((see multi-index support)
`request.template`	no	-	The body of the search template. See configure templates for more information.
`timeout`	no	30s	The timeout for waiting for the search api call to return. If no response is returned within this time, the search transform times out and fails. This setting overrides the default internal search operations timeouts.

Template Support

edit

As can be seen in the table above, the search transform support mustache templates. This can either be as part of the body definition, or alternatively, point to a pre defined/registered template (either defined in a file or registered as a script in elasticsearch). The following snippet shows an example of a search that refers to the scheduled time of the watch:

Simple Search Transform using body template support.

{
  "transform" : {
    "search" : {
      "index" : [ "logstash-*" ],
      "type" : "event",
      "body" : {
        "size" : 0,
        "query" : {
          "filtered" : {
            "filter" : {
              "bool" : {
                "must" : [
                  {
                    "range" : {
                      "@timestamp" : {
                        "from" : "{{ctx.trigger.scheduled_time}}||-30s",
                        "to" : "{{ctx.trigger.triggered_time}}"
                      }
                    }
                  },
                  {
                    "query" : {
                      "match" : { "priority" : "error"}
                    }
                  }
                ]
              }
            }
          }
        }
      }
    }
  }
}

The model of the template (based on which the mustache template will be evaluated) is a union between the provided template.params settings and the standard watch execution context model.

Simple Search Transform using an inline template.

{
  "transform" : {
    "search" : {
      "index" : [ "logstash-*" ],
      "type" : "event",
      "body" : {
        "size" : 0,
        "template" {
          "inline" : {
            "query" : {
              "filtered" : {
                "filter" : {
                  "bool" : {
                    "must" : [
                      {
                        "range" : {
                          "@timestamp" : {
                            "from" : "{{ctx.trigger.scheduled_time}}||-30s",
                            "to" : "{{ctx.trigger.triggered_time}}"
                          }
                        }
                      },
                      {
                        "query" : {
                          "match" : { "priority" : "{{priority}}"}
                        }
                      }
                    ]
                  }
                }
              }
            },
            "params" : {
              "priority" : "error"
            }
          }
        }
      }
    }
  }
}

Script Transform

edit

A Transform that executes a script on the current payload in the watch execution context and replaces it with a newly generated one. The following snippet shows how a simple script transform can be defined on the watch level:

Simple Script Transform.

{
  ...

  "transform" : {
    "script" : "return [ time : ctx.trigger.scheduled_time ]" 
  }
  ...
}

A simple groovy script that creates a new payload with a single time field holding the scheduled time.

The executed script may either return a valid model that is the equivalent of a Java™ Map or a JSON object (you will need to consult the documentation of the specific scripting language to find out what this construct is). Any other value that is returned will be assigned and accessible to/via the _value variable.

As seen above, the script may hold a string value in which case it will be treated as the script itself and the default elasticsearch script languages will be assumed (as described here). It is possible to have more control over the scripting languages and also utilize pre-registered/pre-configured scripts in elasticsearch. For this, the script field will be defined as an object, and the following table lists the possible settings that can be configured:

Table 16. Script Transform Settings

Name	Required	Default	Description
`inline`	yes*	-	When using an inline script, this field holds the script itself.
`file`	yes*	-	When refering to a script file, this field holds the name of the file.
`id`	yes*	-	When refering to an indexed script, this field holds the id of the script.
`lang`	no	`groovy`	The script language
`params`	no	-	Additional parameters/variables that are accessible by the script

When using the object notation of the script, one (and only one) of inline, file or id fields must be defined

In addition to the provided params, the scripts also have access to the Standard Watch Execution Context Parameters

Script Type

edit

When suing inline scripts, if you’re running Elasticsearch 1.3.8 or above, or 1.4.3 or above, you will need to explicitly enable dynamic scripts in elasticsearch.yml.

As indicated by the table above, it is possible to utilize the full scripting support in elasticsearch and to base the script on pre-registered indexed scripts or pre-defined scripts in file. Please note, for security reasons, starting from elasticsearch v1.4.3, inline groovy scripts are disabled by default. Furthermore, it is considered a best practice to pre-define the script in stored files. To read more about elasticsearch search scripting support and possible related vulnerabilities, please see here.

The script transform is often useful when used in combination with the search transform, where the script can extract only the significant data from a search result, and by that, keep the payload minimal. This can be achieved with the chain transform.

Chain Transform

edit

A Transform that executes an ordereed list of configured transforms in a chain, where the output of one transform serves as the input of the next transform in the chain. The payload that is accepted by this transform serves as the input of the first transform in the chain and the output of the last transform in the chain is the output of the chain transform as a whole.

You can use chain transforms to build more complex transforms out of the other available transforms. For example, you can combine a search transform and a script transform, as shown in the following snippet:

"transform" : {
  "chain" : [ 
    {
      "search" : {  
        "indices" : [ "logstash-*" ],
        "body" : {
          "size" : 0,
          "query" : {
            "match" : { "priority" : "error" }
          }
        }
      }
    },
    {
      "script" : "return [ error_count : ctx.payload.hits.total ]"  
    }
  ]
}

	The `chain` transform definition
	The first transform in the chain (in this case, a `search` transform)
	The second and final transform in the chain (in this case, a `script` transform)

This example executes a count search on the cluster to look for error events. The search results are then passed to the second script transform. The script transform extracts the total hit count and assigns it to the error_count field in a newly-generated payload. This newly-generated payload is the output of the chain transform and replaces the payload in the watch execution context.

« Actions Java API »