From version 5.0 onward, Watcher is part of X-Pack. For more information, see Alerting on cluster and index events.

› ›

Actions

The actions associated with a watch are executed whenever the watch is executed, its condition is met, and the watch is not throttled. The actions are executed one at a time and each action executes independently from the others. Any failures encountered while executing an action are recorded in the action result and in the watch history.

If no actions are defined for a watch, no actions are executed. However, a watch_record is still written to the Watch History.

Actions have access to the payload in the execution context. They can use it to support their execution in any way they need. For example, the payload might serve as a model for a templated email body.

Acknowledgement and Throttling

edit

During the watch execution, once the condition is met, a decision is made per configured action as to whether it should be throttled. The main purpose of action throttling is to prevent too many executions of the same action for the same watch.

For example, suppose you have a watch that detects errors in an application’s log entries. The watch is triggered every five minutes and searches for errors during the last hour. In this case, if there are errors, there is a period of time where the watch is checked and its actions are executed multiple times based on the same errors. As a result, the system administrator receives multiple notifications about the same issue, which can be annoying.

To address this issue, Watcher supports time-based throttling. You can define a throttling period as part of the action configuration to limit how often the action is executed. When you set a throttling period, Watcher prevents repeated execution of the action if it has already executed within the throttling period time frame (now - throttling period).

The following snippet shows a watch for the scenario described above - associating a throttle period with the email_administrator action:

Watch Definition Example.

PUT _watcher/watch/log_event_watch
{
  "metadata" : {
    "color" : "red"
  },
  "trigger" : {
    "schedule" : {
      "interval" : "5m"
    }
  },
  "input" : {
    "search" : {
      "request" : {
        "search_type" : "count",
        "indices" : "log-events",
        "body" : {
          "query" : { "match" : { "status" : "error" } }
        }
      }
    }
  },
  "condition" : {
    "script" : "return ctx.payload.hits.total > 5"
  },
  "actions" : {
    "email_administrator" : {
      "throttle_period": "15m", 
      "email" : { 
        "to" : "sys.admino@host.domain",
        "subject" : "Encountered {{ctx.payload.hits.total}} errors",
        "body" : "Too many error in the system, see attached data",
        "attach_data" : true,
        "priority" : "high"
      }
    }
  }
}

	There will be at least 15 minutes between subsequent `email_administrator` action executions.
	See Email Action for more information.

You can also define a throttle period at the watch level. The watch-level throttle period serves as the default throttle period for all of the actions defined in the watch:

Watch Definition Example.

PUT _watcher/watch/log_event_watch
{
  "trigger" : {
    ...
  },
  "input" : {
    ...
  },
  "condition" : {
    ...
  },
  "throttle_period" : "15m", 
  "actions" : {
    "email_administrator" : {
      "email" : {
        "to" : "sys.admino@host.domain",
        "subject" : "Encountered {{ctx.payload.hits.total}} errors",
        "body" : "Too many error in the system, see attached data",
        "attach_data" : true,
        "priority" : "high"
      }
    },
    "notify_pager" : {
      "webhook" : {
        "method" : "POST",
        "host" : "pager.service.domain",
        "port" : 1234,
        "path" : "/{{watch_id}}",
        "body" : "Encountered {{ctx.payload.hits.total}} errors"
      }
    }
  }
}

There will be at least 15 minutes between subsequent action executions (applies to both email_administrator and notify_pager actions)

If you do not define a throttle period at the action or watch level, the global default throttle period is applied. Initially, this is set to 5 seconds. To change the global default, configure the watcher.execution.default_throttle_period setting in elasticsearch.yml:

watcher.execution.default_throttle_period: 15m

Watcher also supports acknowledgement-based throttling. You can acknowledge a watch using the Ack Watch API to prevent the watch actions from being executed again while the watch condition remains true. This essentially tells Watcher "I received the notification and I’m handling it, please do not notify me about this error again". An acknowledged watch action remains in the acked state until the watch’s condition evaluates to false. When that happens, the action’s state changes to awaits_successful_execution.

To acknowledge an action, you use the ack API:

PUT _watcher/watch/<id>/_ack?actions=<action_ids>

Where <id> is the id of the watch and <action_ids> is a comma-separated list of the action ids you want to acknowledge. To acknowledge all actions, omit the actions parameter.

The following diagram illustrates the throttling decisions made for each action of a watch during its execution:

Watcher supports four action types: Email, Webhook, Index and Logging.

Email Action

edit

A watch action that sends email notifications. To use the email action, you must configure at least one email account. For instructions, see Configuring Email Accounts.

See Table 11, “Email Action Attributes” for the supported attributes. Any attributes that are missing from the email action definition are looked up in the configuration of the account from which the email is being sent. The required attributes must either be set in the email action definition or the account’s email_defaults.

Configuring Email Actions

edit

You configure email actions in a watch’s actions array. Action-specific attributes are specified using the email keyword.

The following snippet shows a basic email action definition:

"actions" : {
  "email_admin" : { 
    "transform" : { ... }, 
    "email": {
      "to": "'John Doe <john.doe@example.com>'", 
      "subject": "{{ctx.watch_id}} executed", 
      "body": "{{ctx.watch_id}} executed with {{ctx.payload.hits.total}} hits" 
    }
  }
}

	The id of the action.
	An optional transform to transform the payload before processing the email.
	One or more addresses to send the email to. If not specified, the `to` address is read from the account’s `email_defaults`.
	The subject of the email (static text or a Mustache template).
	The body of the email (static text or a Mustache template).

Table 11. Email Action Attributes

Name	Required	Default	Description
`account`	no	the default account	The account to use to send the email.
`from`	no	-	The email address from which the email will be sent. The `from` field can contain Mustache templates as long as it resolves to a valid email address.
`to`	yes	-	The email addresses of the `to` recipients. The `to` field can contain Mustache templates as long as it resolves to a valid email address.
`cc`	no	-	The email addresses of the `cc` recipients. The `cc` field can contain Mustache templates as long as it resolves to a valid email address.
`bcc`	no	-	The email addresses of the `bcc` recipients. The `bcc` field can contain Mustache templates as long as it resolves to a valid email address.
`reply_to`	no	-	The email addresses that will be set on the message’s `Reply-To` header. The `reply_to` field can contain Mustache templates as long as it resolves to a valid email address.
`subject`	no	-	The subject of the email. The subject can be static text or contain Mustache templates.
`body`	no	-	The body of the email. When this field holds a string, it will default to the text body of the email. Set as an object to specify either the text or the html body or both (using the fields bellow)
`body.text`	yes*	-	The plain text body of the email. The body can be static text or contain Mustache templates.
`body.html`	yes*	-	The html body of the email. The body can be static text or contain Mustache templates. This body will be sanitized to remove dangerous content such as scripts. This behavior can be disabled by setting `watcher.actions.email.sanitize_html: false` in elasticsearch.yaml.
`priority`	no	-	The priority of this email. Valid values are: `lowest`, `low`, `normal`, `high` and `highest`. The priority can contain a Mustache template as long as it resolves to one of the valid values.
`attach_data`	no	false	Indicates whether the watch execution data should be attached to the email. You can specify a Boolean value or an object. If `attach_data` is set to `true`, the data is attached as a YAML file called `data.yml`. If it’s set to `false`, no data is attached. To control the format of the attached data, specify an object that contains a `format` field`.
`attach_data.format`	no	yaml	When `attach_data` is specified as an object, this field controls the format of the attached data. The supported formats are `json` and `yaml`.

When setting the body object, at least one of its text or html fields must be defined.

Email Address: An email address can contain two possible parts—the address itself and an optional personal name as described in RFC 822. The address can be represented either as a string of the form user@host.domain or Personal Name <user@host.domain>. You can also specify an email address as an object that contains name and address fields.

Address List: A list of addresses can either be specified as a comma-delimited string or as an array: 'Personal Name <user1@host.domain>, user2@host.domain' or [ 'Personal Name <user1@host.domain>', 'user2@host.domain' ]

Webhook Action

edit

A watch action that connects to a web server and listens on a specific port. The webhook action supports both HTTP and HTTPS connections. See Table 12, “Webhook Action Attributes” for the supported attributes.

Configuring Webhook Actions

edit

You configure webhook actions in a watch’s actions array. Action-specific attributes are specified using the webhook keyword.

The following snippet shows a simple webhook action definition:

"actions" : {
  "my_webhook" : { 
    "transform" : { ... }, 
    "throttle_period" : "5m", 
    "webhook" : {
      "method" : "POST", 
	    "host" : "mylisteningserver", 
	    "port" : 9200, 
      "path": ":/{{ctx.watch_id}", 
	    "body" : "{{ctx.watch_id}}:{{ctx.payload.hits.total}}" 
    }
  }
}

	The id of the action
	An optional transform to transform the payload before executing the `webhook` action
	An optional throttle period for the action (5 minutes in this example)
	The HTTP method to use when connecting to the host
	The host to connect to
	The port to connect to
	The path (URI) to use in the HTTP request
	The body to send with the request

You can use basic authentication when sending a request to a secured webservice. For example:

"actions" : {
  "my_webhook" : {
    "webhook" : {
      "auth" : {
        "basic" : {
          "username" : "<username>", 
          "password" : "<password>" 
        }
      }
      "method" : "POST",
      "host" : "mylisteningserver",
      "port" : 9200,
      "path": ":/{{ctx.watch_id}",
      "body" : "{{ctx.watch_id}}:{{ctx.payload.hits.total}}"
    }
  }
}

	The username
	The corresponding password

By default, both the username and the password are stored in the .watches index in plain text. When Shield is installed, Watcher can be configured to encrypt the password before storing it.

Query Parameters

edit

You can specify query parameters to send with the request with the params field. This field simply holds an object where the keys serve as the parameter names and the values serve as the parameter values:

"actions" : {
  "my_webhook" : {
    "webhook" : {
      "method" : "POST",
      "host" : "mylisteningserver",
      "port" : 9200,
      "path": ":/alert",
      "params" : {
        "watch_id" : "{{ctx.watch_id}}" 
      }
    }
  }
}

The parameter values can contain templated strings.

Custom Request Headers

edit

You can specify request headers to send with the request with the headers field. This field simply holds an object where the keys serve as the header names and the values serve as the header values:

"actions" : {
  "my_webhook" : {
    "webhook" : {
      "method" : "POST",
      "host" : "mylisteningserver",
      "port" : 9200,
      "path": ":/alert/{{ctx.watch_id}}",
      "headers" : {
        "Content-Type" : "application/yaml" 
      },
      "body" : "count: {{ctx.payload.hits.total}}"
    }
  }
}

The header values can contain templated strings.

Table 12. Webhook Action Attributes

Name	Required	Default	Description
`scheme`	no	http	The connection scheme. Valid values are: `http` or `https`.
`host`	yes	-	The host to connect to.
`port`	yes	-	The port the HTTP service is listening on.
`path`	no	-	The URL path. The path can be static text or include Mustache templates. URL query string parameters must be specified via the `request.params` attribute.
`method`	no	get	The HTTP method. Valid values are: `head`, `get`, `post`, `put` and `delete`.
`headers`	no	-	The HTTP request headers. The header values can be static text or include Mustache templates.
`params`	no	-	The URL query string parameters. The parameter values can be static text or include Mustache templates.
`auth`	no	-	Authentication related HTTP headers. Currently, only basic authentication is supported.
`body`	no	-	The HTTP request body. The body can be static text or include Mustache templates. When not specified, an empty body is sent.
`connection_timeout`	no	10s	The timeout for setting up the http connection. If the connection could not be set up within this time, the action will timeout and fail. It is also possible to configure the default connection timeout for all http connection timeouts.

Index Action

edit

A watch Action that enable you to index data in Elasticsearch. See Table 13, “Index Action Attributes” for the supported attributes.

Configuring Index Actions

edit

The following snippet shows a simple index action definition:

"actions" : {
  "index_payload" : { 
    "transform": { ... },
    "index" : {
      "index" : "my-index", 
      "doc_type" : "my-type" 
    }
  }
}

	The id of the action
	An optional transform to transform the payload and prepare the data that should be indexed
	The elasticsearch index to store the data to
	The document type to store the data as

Table 13. Index Action Attributes

Name	Required	Default	Description
`index`	yes	-	The Elasticsearch index to index into. Dynamic index names are supported
`doc_type`	yes	-	The type of the document the data will be indexed as.
`execution_time_field`	no	_timestamp	The field that will store/index the watch execution time. When not set or when set to `_timestamp`, the execution time will serve as the document’s timestamp.
`timeout`	no	60s	The timeout for waiting for the index api call to return. If no response is returned within this time, the index action times out and fails. This setting overrides the default internal index/bulk operations timeouts.
`dynamic_name_timezone`	no	-	The time zone to use for resolving the index name based on Dynamic Index Names. The default time zone also can be configured globally.

Multi-Document Support

edit

Like with all other actions, you can use a transform to replace the current execution context payload with another and by that change the document that will end up indexed.

The index action plays well with transforms with its support for the special _doc payload field.

When resolving the document to be indexed, the index action first looks up for a _doc field in the payload. When not found, the payload is indexed as a single document.

When a _doc field exists, if the field holds an object, it is extracted and indexed as a single document. If the field holds an array of objects, each object is treated as a document and the index aciton indexes all of them in a bulk.

Logging Action

edit

A watch action that simply logs text to the standard Elasticsearch logs. See Table 14, “Logging Action Attributes” for the supported attributes.

This action is primarily used during development and debugging.

Configuring Logging Actions

edit

You configure logging actions in a watch’s actions array. Action-specific attributes are specified using the logging keyword.

The following snippet shows a simple logging action definition:

"actions" : {
  "log" : { 
    "transform" : { ... }, 
    "logging" : {
      "text" : "executed at {{ctx.execution_time}}" 
    }
  }
}

	The id of the action.
	An optional transform to transform the payload before executing the `logging` action.
	The text to be logged.

Table 14. Logging Action Attributes

Name	Required	Default	Description
`text`	yes	-	The text that should be logged. Can be static text or include Mustache templates.
`category`	no	watcher.actions.logging	The category under which the text will be logged.
`level`	no	info	The logging level. Valid values are: `error`, `warn`, `info`, `debug` and `trace`.

« Condition Transform »