Actions
editActions
editThe actions associated with a watch are executed whenever the watch is executed, its condition is met, and the watch is not throttled. The actions are executed one at a time and each action executes independently from the others. Any failures encountered while executing an action are recorded in the action result and in the watch history.
If no actions are defined for a watch, no actions are executed. However, a watch_record
is still written to the Watch History.
Actions have access to the payload in the execution context. They can use it to support their execution in any way they need. For example, the payload might serve as a model for a templated email body.
Acknowledgement and Throttling
editDuring the watch execution, once the condition is met, a decision is made per configured action as to whether it should be throttled. The main purpose of action throttling is to prevent too many executions of the same action for the same watch.
For example, suppose you have a watch that detects errors in an application’s log entries. The watch is triggered every five minutes and searches for errors during the last hour. In this case, if there are errors, there is a period of time where the watch is checked and its actions are executed multiple times based on the same errors. As a result, the system administrator receives multiple notifications about the same issue, which can be annoying.
To address this issue, Watcher supports time-based throttling. You can define a throttling
period as part of the action configuration to limit how often the action is executed. When you
set a throttling period, Watcher prevents repeated execution of the action if it has already
executed within the throttling period time frame (now - throttling period
).
The following snippet shows a watch for the scenario described above - associating a throttle
period with the email_administrator
action:
Watch Definition Example.
PUT _watcher/watch/log_event_watch { "metadata" : { "color" : "red" }, "trigger" : { "schedule" : { "interval" : "5m" } }, "input" : { "search" : { "request" : { "indices" : "log-events", "body" : { "size" : 0, "query" : { "match" : { "status" : "error" } } } } } }, "condition" : { "script" : "return ctx.payload.hits.total > 5" }, "actions" : { "email_administrator" : { "throttle_period": "15m", "email" : { "to" : "sys.admino@host.domain", "subject" : "Encountered {{ctx.payload.hits.total}} errors", "body" : "Too many error in the system, see attached data", "attachments" : { "attached_data" : { "data" : { "format" : "json" } } }, "priority" : "high" } } } }
There will be at least 15 minutes between subsequent |
|
See Email Action for more information. |
You can also define a throttle period at the watch level. The watch-level throttle period serves as the default throttle period for all of the actions defined in the watch:
Watch Definition Example.
PUT _watcher/watch/log_event_watch { "trigger" : { ... }, "input" : { ... }, "condition" : { ... }, "throttle_period" : "15m", "actions" : { "email_administrator" : { "email" : { "to" : "sys.admino@host.domain", "subject" : "Encountered {{ctx.payload.hits.total}} errors", "body" : "Too many error in the system, see attached data", "attachments" : { "attached_data" : { "data" : { "format" : "json" } } }, "priority" : "high" } }, "notify_pager" : { "webhook" : { "method" : "POST", "host" : "pager.service.domain", "port" : 1234, "path" : "/{{watch_id}}", "body" : "Encountered {{ctx.payload.hits.total}} errors" } } } }
There will be at least 15 minutes between subsequent action executions (applies to both
|
If you do not define a throttle period at the action or watch level, the global default
throttle period is applied. Initially, this is set to 5 seconds. To change the global default,
configure the watcher.execution.default_throttle_period
setting in elasticsearch.yml
:
watcher.execution.default_throttle_period: 15m
Watcher also supports acknowledgement-based throttling. You can acknowledge a watch using the
Ack Watch API to prevent the watch actions from being executed again
while the watch condition remains true
. This essentially tells Watcher "I received the
notification and I’m handling it, please do not notify me about this error again".
An acknowledged watch action remains in the acked
state until the watch’s condition evaluates
to false
. When that happens, the action’s state changes to awaits_successful_execution
.
To acknowledge an action, you use the ack
API:
PUT _watcher/watch/<id>/_ack?actions=<action_ids>
Where <id>
is the id of the watch and <action_ids>
is a comma-separated list of the action
ids you want to acknowledge. To acknowledge all actions, omit the actions
parameter.
The following diagram illustrates the throttling decisions made for each action of a watch during its execution:
Watcher supports six action types: Email, Webhook, Index, Logging, HipChat and Slack.
Email Action
editA watch action that sends email notifications. To use the email
action, you must configure at least one email account. For instructions, see Configuring Email Accounts.
See Table 13, “Email Action Attributes” for the supported attributes. Any attributes that are missing from the email action definition are looked up in the configuration of the account from which the email is being sent. The required attributes must either be set in the email action definition or the account’s email_defaults
.
Configuring Email Actions
editYou configure email actions in a watch’s actions
array. Action-specific attributes are
specified using the email
keyword.
The following snippet shows a basic email action definition:
"actions" : { "email_admin" : { "transform" : { ... }, "email": { "to": "'John Doe <john.doe@example.com>'", "subject": "{{ctx.watch_id}} executed", "body": "{{ctx.watch_id}} executed with {{ctx.payload.hits.total}} hits" } } }
The id of the action. |
|
An optional transform to transform the payload before processing the email. |
|
One or more addresses to send the email to. If not specified, the |
|
The subject of the email (static text or a Mustache template). |
|
The body of the email (static text or a Mustache template). |
Configuring Email Attachments
editYou can attach the watch payload or data from an any HTTP service to the email notification. There is no limit on the number of attachments you can configure.
To configure attachments, specify a name for the attached file and the type of attachment: data
or http
.
The data
attachment type attaches the watch payload to the email message. The http
attachment type enables
you to issue an HTTP request and attach the response to the email message. When configuring the http
attachment
type, you must specify the request URL.
"actions" : { "email_admin" : { "email": { "to": "'John Doe <john.doe@example.com>'", "attachments" : { "my_report.pdf" : { "http" : { "content_type" : "application/pdf", "request" : { "url": "http://example.org/foo/my-report" } } }, "data.yml" : { "data" : { "format" : "yaml" } } } } } }
The ID of the attachment, which is used as the file name in the email attachment. |
|
The type of the attachment and its specific configuration. |
|
The URL from which to retrieve the attachment. |
|
Data attachments default to JSON if you don’t specify the format. |
Table 11. Attributes for the http
type
Name | Description |
---|---|
|
Sets the content type for the email attachment. By default, the content type is extracted from the response sent by the HTTP service. You can explicitly specify the content type to ensure that the type is set correctly in the email in case the response does not specify the content type or it’s specified incorrectly. Optional. |
|
Contains the HTTP request attributes. At a minimum, you must specify the |
Table 12. Attributes for the data
type
Name | Description |
---|---|
|
Attaches the watch data, equivalent to specifying |
Table 13. Email Action Attributes
Name | Required | Default | Description |
---|---|---|---|
|
no |
the default account |
The account to use to send the email. |
|
no |
- |
The email address from which the email will be sent. The |
|
yes |
- |
The email addresses of the |
|
no |
- |
The email addresses of the |
|
no |
- |
The email addresses of the |
|
no |
- |
The email addresses that will be set on the message’s |
|
no |
- |
The subject of the email. The subject can be static text or contain Mustache templates. |
|
no |
- |
The body of the email. When this field holds a string, it will default to the text body of the email. Set as an object to specify either the text or the html body or both (using the fields below) |
|
yes |
- |
The plain text body of the email. The body can be static text or contain Mustache templates. The email |
|
yes |
- |
The html body of the email. The body can be static text or contain Mustache templates. This body will be sanitized to remove dangerous content such as scripts. This behavior can be disabled by setting |
|
no |
- |
The priority of this email. Valid values are: |
|
no |
- |
Attaches the watch payload ( |
|
no |
false |
Indicates whether the watch execution data should be attached to the email. You can specify a Boolean value or an object. If |
|
no |
yaml |
When |
- Email Address
-
An email address can contain two possible parts—the address itself and an optional personal name as described in RFC 822. The address can be represented either as a string of the form
user@host.domain
orPersonal Name <user@host.domain>
. You can also specify an email address as an object that containsname
andaddress
fields.
Webhook Action
editA watch action that connects to a web server and listens on a specific port. The webhook action supports both HTTP and HTTPS connections. See Table 14, “Webhook Action Attributes” for the supported attributes.
Configuring Webhook Actions
editYou configure webhook actions in a watch’s actions
array. Action-specific attributes are
specified using the webhook
keyword.
The following snippet shows a simple webhook action definition:
"actions" : { "my_webhook" : { "transform" : { ... }, "throttle_period" : "5m", "webhook" : { "method" : "POST", "host" : "mylisteningserver", "port" : 9200, "path": ":/{{ctx.watch_id}", "body" : "{{ctx.watch_id}}:{{ctx.payload.hits.total}}" } } }
The id of the action |
|
An optional transform to transform the payload before executing the |
|
An optional throttle period for the action (5 minutes in this example) |
|
The HTTP method to use when connecting to the host |
|
The host to connect to |
|
The port to connect to |
|
The path (URI) to use in the HTTP request |
|
The body to send with the request |
You can use basic authentication when sending a request to a secured webservice. For example:
"actions" : { "my_webhook" : { "webhook" : { "auth" : { "basic" : { "username" : "<username>", "password" : "<password>" } } "method" : "POST", "host" : "mylisteningserver", "port" : 9200, "path": ":/{{ctx.watch_id}", "body" : "{{ctx.watch_id}}:{{ctx.payload.hits.total}}" } } }
By default, both the username and the password are stored in the .watches
index in plain text. When
Shield is installed, Watcher can be configured to encrypt the password before
storing it.
You can also use PKI-based authentication when submitting requests to a cluster secured with Shield.
When you use PKI-based authentication instead of HTTP basic auth, you don’t need to store any
authentication information in the watch itself. To use PKI-based authentication, you configure
the SSL keystore path and password for Watcher in elasticsearch.yml
. If the
watcher.http.ssl.keystore.path
and watcher.http.ssl.keystore
password are not set, the Watcher
HTTP client falls back to the Shield settings, shield.ssl.keystore.path
and shield.ssl.keystore.password
.
Query Parameters
editYou can specify query parameters to send with the request with the params
field. This field simply
holds an object where the keys serve as the parameter names and the values serve as the parameter values:
Custom Request Headers
editYou can specify request headers to send with the request with the headers
field. This field simply
holds an object where the keys serve as the header names and the values serve as the header values:
"actions" : { "my_webhook" : { "webhook" : { "method" : "POST", "host" : "mylisteningserver", "port" : 9200, "path": ":/alert/{{ctx.watch_id}}", "headers" : { "Content-Type" : "application/yaml" }, "body" : "count: {{ctx.payload.hits.total}}" } } }
Table 14. Webhook Action Attributes
Name | Required | Default | Description |
---|---|---|---|
|
no |
http |
The connection scheme. Valid values are: |
|
yes |
- |
The host to connect to. |
|
yes |
- |
The port the HTTP service is listening on. |
|
no |
- |
The URL path. The path can be static text or include Mustache templates. URL query string parameters must be specified via the |
|
no |
get |
The HTTP method. Valid values are: |
|
no |
- |
The HTTP request headers. The header values can be static text or include Mustache templates. |
|
no |
- |
The URL query string parameters. The parameter values can be static text or include Mustache templates. |
|
no |
- |
Authentication related HTTP headers. Currently, only basic authentication is supported. |
|
no |
- |
The HTTP request body. The body can be static text or include Mustache templates. When not specified, an empty body is sent. |
|
no |
- |
The proxy host to use when connecting to the host. |
|
no |
- |
The proxy port to use when connecting to the host. |
|
no |
10s |
The timeout for setting up the http connection. If the connection could not be set up within this time, the action will timeout and fail. It is also possible to configure the default connection timeout for all http connection timeouts. |
|
no |
10s |
The timeout for reading data from http connection. If no response was received within this time, the action will timeout and fail. It is also possible to configure the default read timeout for all http connection timeouts. |
|
no |
- |
A shortcut for specifying the request scheme, host, port, and path as a single string. For example, |
Index Action
editA watch Action that enable you to index data in Elasticsearch. See Table 15, “Index Action Attributes” for the supported attributes.
Configuring Index Actions
editThe following snippet shows a simple index
action definition:
"actions" : { "index_payload" : { "transform": { ... }, "index" : { "index" : "my-index", "doc_type" : "my-type" } } }
The id of the action |
|
An optional transform to transform the payload and prepare the data that should be indexed |
|
The elasticsearch index to store the data to |
|
The document type to store the data as |
Table 15. Index Action Attributes
Name | Required | Default | Description |
---|---|---|---|
|
yes |
- |
The Elasticsearch index to index into. are supported |
|
yes |
- |
The type of the document the data will be indexed as. |
|
no |
_timestamp |
The field that will store/index
the watch execution time. When
not set or when set to |
|
no |
60s |
The timeout for waiting for the index api call to return. If no response is returned within this time, the index action times out and fails. This setting overrides the default internal index/bulk operations timeouts. |
Multi-Document Support
editLike with all other actions, you can use a transform to replace the current execution context payload with another and by that change the document that will end up indexed.
The index action plays well with transforms with its support for the special _doc
payload field.
When resolving the document to be indexed, the index action first looks up for a
_doc
field in the payload. When not found, the payload is indexed as a single
document.
When a _doc
field exists, if the field holds an object, it is extracted and indexed
as a single document. If the field holds an array of objects, each object is treated as
a document and the index aciton indexes all of them in a bulk.
Logging Action
editA watch action that simply logs text to the standard Elasticsearch logs. See Table 16, “Logging Action Attributes” for the supported attributes.
This action is primarily used during development and debugging.
Configuring Logging Actions
editYou configure logging actions in a watch’s actions
array. Action-specific attributes are
specified using the logging
keyword.
The following snippet shows a simple logging action definition:
"actions" : { "log" : { "transform" : { ... }, "logging" : { "text" : "executed at {{ctx.execution_time}}" } } }
The id of the action. |
|
An optional transform to transform the payload before executing the |
|
The text to be logged. |
Table 16. Logging Action Attributes
Name | Required | Default | Description |
---|---|---|---|
|
yes |
- |
The text that should be logged. Can be static text or include Mustache templates. |
|
no |
watcher.actions.logging |
The category under which the text will be logged. |
|
no |
info |
The logging level. Valid values are: |
HipChat Action
editA watch action that sends messages to HipChat rooms or users. To use the HipChat action, you need to configure at least one HipChat account in Watcher. For information about configuring accounts, see Configuring Watcher to Send Notifications to HipChat.
Configuring HipChat Actions
editYou configure HipChat actions in a watch’s actions
array. Action-specific attributes are
specified using the hipchat
keyword. You must specify the message
attribute for all
hipchat
actions. If you omit the account
attribute, the message is sent
using the default HipChat account configured in elasticsearch.yml
.
For example, the following action is configured to send messages using a HipChat account that uses the integration profile. Because this type of account can only send messages to a particular room, the only required attribute is the message itself.
"actions" : { "notify-hipchat" : { "transform" : { ... }, "throttle_period" : "5m", "hipchat" : { "account" : "integration-account", "message" : { "body" : "Encountered {{ctx.payload.hits.total}} errors in the last 5 minutes (facepalm)", "format" : "text", "color" : "red", "notify" : true } } } }
The name of a HipChat account configured in |
|
The message you want to send to HipChat. |
To send messages with a HipChat account that uses the user profile, you need
to specify what rooms and users you want to send the message to. For example, the following action
is configured to send messages to the mission-control
and devops
rooms
as well as the user website-admin@example.com
. (To send to multiple users or rooms, specify an
array of strings.)
"actions" : { "notify-hipchat" : { "transform" : { ... }, "throttle_period" : "5m", "hipchat" : { "account" : "user-account", "message" : { "room" : [ "mission-control", "devops" ], "user" : "website-admin@example.com", "body" : "Encountered {{ctx.payload.hits.total}} errors in the last 5 minutes (facepalm)", "format" : "text", "color" : "red", "notify" : true } } } }
To send messages with a HipChat account that uses the v1 profile, you need
to specify what room or rooms you want to send the message to. For example, the following action
is configured to send messages to the server-status
room. (To send to multiple
rooms, specify an array of strings.)
"actions" : { "notify-hipchat" : { "transform" : { ... }, "throttle_period" : "5m", "hipchat" : { "account" : "v1-account", "message" : { "from" : "Watcher", "room" : [ "server-status", "infra-team" ], "body" : "Encountered {{ctx.payload.hits.total}} errors in the last 5 minutes (facepalm)", "format" : "text", "color" : "red", "notify" : true } } } }
HipChat Action Attributes
editThe following table lists the attributes you can set for hipchat
actions.
Name |
Required |
Default |
Description |
|
no |
Default account |
The HipChat account to use to send the message. |
`message.body ` |
yes |
- |
The message content. Can contain up to 1000 characters. |
|
no |
html |
The format of the message: |
|
no |
yellow |
The background color of the notification in the
room: |
|
no |
false |
Indicates whether people in the room should be actively notified |
|
no |
the watch ID |
The name that appears as the notification sender. Only valid for accounts that use the v1 profile. |
|
no |
- |
The rooms that the notification should go to.
Accepts a string value or an array of string
values. Must be specified when using the
v1 profile. At least one room or user must
be specified when using the |
|
no |
- |
The users that the notification should go to.
Accepts a string value or an array of string
values. At least one room or user must
be specified when using the |
Slack Action
editA watch action that sends messages to a Slack team’s channels or users. To use the Slack action, you need to configure at least one Slack account in Watcher. For information about configuring accounts, see Configuring Watcher to Send Notifications to Slack.
Configuring Slack Actions
editYou configure Slack actions in a watch’s actions
array. Action-specific attributes are
specified using the slack
keyword.
The following snippet shows a simple slack action definition:
Using Attachments to Format Slack Messages
editIn addition to sending simple text-based messages, you can use the Slack attachment mechanism to send formatted messages. Watcher leverages Slack attachments to enable you to dynamically populate templated messages from the watch payload.
The following snippet shows a standard message attachment.
"actions" : { "notify-slack" : { "throttle_period" : "5m", "slack" : { "account" : "team1", "message" : { "from" : "watcher", "to" : [ "#admins", "@chief-admin" ], "text" : "System X Monitoring", "attachments" : [ { "title" : "Errors Found", "text" : "Encountered {{ctx.payload.hits.total}} errors in the last 5 minutes (facepalm)", "color" : "danger" } ] } } } }
To define an attachment template that is dynamically populated from the watch payload, you specify
dynamic_attachments
in the watch action. For example, a dynamic attachment could reference
histogram buckets in the payload and build an attachment per bucket.
In the following example, the watch input executes a search with a date histogram aggregation and the Slack action:
- Transforms the payload to a list where each item in the list holds the month, the user count for that month, and the color that represents the sentiment associated with that count (good or danger).
- Defines an attachment template that references items in the list generated by the transform.
"input" : { "search" : { "request" : { "body" : { "aggs" : { "users_per_month" : { "date_histogram" : { "field" : "@timestamp", "interval" : "1m" } } } } } } }, ... "actions" : { "notify-slack" : { "throttle_period" : "5m", "transform" : { "script" : "return [ items : ctx.payload.hits.aggs.users_per_month.buckets.collect { [ month : it.key_as_string, count : it.doc_count, color : it.doc_count < 100 ? 'danger' : 'good' ] }]" }, "slack" : { "account" : "team1", "message" : { "from" : "watcher", "to" : [ "#admins", "@chief-admin" ], "text" : "System X Monitoring", "dynamic_attachments" : { "list_path" : "ctx.payload.items" "attachment_template" : { "title" : "{{month}}", "text" : "Users Count: {{count}}", "color" : "{{color}}" } } } } } }
Slack Action Attributes
editName | Required | Description |
---|---|---|
|
no |
The sender name to display in the Slack message. Overrides the incoming webhook’s configured name. |
|
yes |
The channels and users you want to send
the message to. Channel names must start
with |
|
no |
The icon to display in the Slack messages. Overrides the incoming webhook’s configured icon. Accepts a public URL to an image. |
|
yes |
The message content. |
|
no |
Slack message attachments. Message attachments enable you to create more richly-formatted messages. Specified as as array as defined in the Slack attachments documentation. |
|
no |
Slack message attachments that can be populated dynamically based on the current watch payload. For more information, see Using Attachments to Format Slack Messages. |
PagerDuty Action
editA watch action that creates events in PagerDuty. To use the PagerDuty action, you need to configure at least one PagerDuty account in Watcher.
Configuring PagerDuty Actions
editYou configure PagerDuty actions in a watch’s actions
array. Action-specific attributes are
specified using the pagerduty
keyword.
The following snippet shows a simple slack action definition:
Adding meta information to a PagerDuty Incident
editIn order to give the pagerduty incident some more context, you can attach the payload as well as an array of contexts to the action.
"actions" : { "notify-pagerduty" : { "throttle_period" : "5m", "pagerduty" : { "account" : "team1", "description" : "Main system down, please check! Happened at {{ctx.execution_time}}" "attach_payload" : true, "client" : "/foo/bar/{{ctx.watch_id}}", "client_url" : "http://www.example.org/", "context" : [ { "type": "link", "href": "http://acme.pagerduty.com" },{ "type": "link", "href": "http://acme.pagerduty.com", "text": "View the incident on {{ctx.payload.link}}" } ] } } }
Pagerduty Action Attributes
editName | Required | Description |
---|---|---|
|
no |
The account to use, falls back to the default one. The account needs a |
Trigger Incident Attributes
editName | Required | Description |
---|---|---|
|
yes |
A quick description for this event |
|
no |
The event type to sent. Must be one of |
|
no |
The incident key on the pagerduty side, also used for deduplication and allows to resolve or ackknowledge incidents. |
|
no |
Name of the client triggering the incident, i.e. |
|
no |
A client URL to visit to get more detailed information. |
|
no |
If set to |
|
no |
An array of objects, that allow you to provide additional links or images in order to provide more context to the trigger. |
You can configure defaults for the above values for the whole service using the watcher.actions.pagerduty.service.event_defaults.*
properties as well as per account using watcher.actions.pagerduty.service.account.your_account_name.event_defaults.*
Note: All of those objects have templating support, so you can use data from the context and the payload as part of all the fields.
Context Attributes
editA context can be a link or an image.
Name | Required | Description |
---|---|---|
|
yes |
One of |
|
yes/no |
A link to include more information. Must be there if the type is |
|
no |
A src attribute for the |