From version 5.0 onward, Watcher is part of X-Pack. For more information, see Alerting on cluster and index events.
« Condition Transform »
Elastic Docs ›Elasticsearch Watcher [2.2] ›Reference

Actions

edit

Actions

edit

The actions associated with a watch are executed whenever the watch is executed, its condition is met, and the watch is not throttled. The actions are executed one at a time and each action executes independently from the others. Any failures encountered while executing an action are recorded in the action result and in the watch history.

If no actions are defined for a watch, no actions are executed. However, a watch_record is still written to the Watch History.

Actions have access to the payload in the execution context. They can use it to support their execution in any way they need. For example, the payload might serve as a model for a templated email body.

Acknowledgement and Throttling

edit

During the watch execution, once the condition is met, a decision is made per configured action as to whether it should be throttled. The main purpose of action throttling is to prevent too many executions of the same action for the same watch.

For example, suppose you have a watch that detects errors in an application’s log entries. The watch is triggered every five minutes and searches for errors during the last hour. In this case, if there are errors, there is a period of time where the watch is checked and its actions are executed multiple times based on the same errors. As a result, the system administrator receives multiple notifications about the same issue, which can be annoying.

To address this issue, Watcher supports time-based throttling. You can define a throttling period as part of the action configuration to limit how often the action is executed. When you set a throttling period, Watcher prevents repeated execution of the action if it has already executed within the throttling period time frame (now - throttling period).

The following snippet shows a watch for the scenario described above - associating a throttle period with the email_administrator action:

Watch Definition Example.

PUT _watcher/watch/log_event_watch
{
  "metadata" : {
    "color" : "red"
  },
  "trigger" : {
    "schedule" : {
      "interval" : "5m"
    }
  },
  "input" : {
    "search" : {
      "request" : {
        "indices" : "log-events",
        "body" : {
          "size" : 0,
          "query" : { "match" : { "status" : "error" } }
        }
      }
    }
  },
  "condition" : {
    "script" : "return ctx.payload.hits.total > 5"
  },
  "actions" : {
    "email_administrator" : {
      "throttle_period": "15m", 
      "email" : { 
        "to" : "sys.admino@host.domain",
        "subject" : "Encountered {{ctx.payload.hits.total}} errors",
        "body" : "Too many error in the system, see attached data",
        "attach_data" : true,
        "priority" : "high"
      }
    }
  }
}

There will be at least 15 minutes between subsequent email_administrator action executions.

See Email Action for more information.

You can also define a throttle period at the watch level. The watch-level throttle period serves as the default throttle period for all of the actions defined in the watch:

Watch Definition Example.

PUT _watcher/watch/log_event_watch
{
  "trigger" : {
    ...
  },
  "input" : {
    ...
  },
  "condition" : {
    ...
  },
  "throttle_period" : "15m", 
  "actions" : {
    "email_administrator" : {
      "email" : {
        "to" : "sys.admino@host.domain",
        "subject" : "Encountered {{ctx.payload.hits.total}} errors",
        "body" : "Too many error in the system, see attached data",
        "attach_data" : true,
        "priority" : "high"
      }
    },
    "notify_pager" : {
      "webhook" : {
        "method" : "POST",
        "host" : "pager.service.domain",
        "port" : 1234,
        "path" : "/{{watch_id}}",
        "body" : "Encountered {{ctx.payload.hits.total}} errors"
      }
    }
  }
}

There will be at least 15 minutes between subsequent action executions (applies to both email_administrator and notify_pager actions)

If you do not define a throttle period at the action or watch level, the global default throttle period is applied. Initially, this is set to 5 seconds. To change the global default, configure the watcher.execution.default_throttle_period setting in elasticsearch.yml:

watcher.execution.default_throttle_period: 15m

Watcher also supports acknowledgement-based throttling. You can acknowledge a watch using the Ack Watch API to prevent the watch actions from being executed again while the watch condition remains true. This essentially tells Watcher "I received the notification and I’m handling it, please do not notify me about this error again". An acknowledged watch action remains in the acked state until the watch’s condition evaluates to false. When that happens, the action’s state changes to awaits_successful_execution.

To acknowledge an action, you use the ack API:

PUT _watcher/watch/<id>/_ack?actions=<action_ids>

Where <id> is the id of the watch and <action_ids> is a comma-separated list of the action ids you want to acknowledge. To acknowledge all actions, omit the actions parameter.

The following diagram illustrates the throttling decisions made for each action of a watch during its execution:

action throttling

Watcher supports six action types: Email, Webhook, Index, Logging, HipChat and Slack.

Email Action

edit

A watch action that sends email notifications. To use the email action, you must configure at least one email account. For instructions, see Configuring Email Accounts.

See Table 11, “Email Action Attributes” for the supported attributes. Any attributes that are missing from the email action definition are looked up in the configuration of the account from which the email is being sent. The required attributes must either be set in the email action definition or the account’s email_defaults.

Configuring Email Actions

edit

You configure email actions in a watch’s actions array. Action-specific attributes are specified using the email keyword.

The following snippet shows a basic email action definition:

"actions" : {
  "email_admin" : { 
    "transform" : { ... }, 
    "email": {
      "to": "'John Doe <john.doe@example.com>'", 
      "subject": "{{ctx.watch_id}} executed", 
      "body": "{{ctx.watch_id}} executed with {{ctx.payload.hits.total}} hits" 
    }
  }
}

The id of the action.

An optional transform to transform the payload before processing the email.

One or more addresses to send the email to. If not specified, the to address is read from the account’s email_defaults.

The subject of the email (static text or a Mustache template).

The body of the email (static text or a Mustache template).

Table 11. Email Action Attributes

Name Required Default Description

account

no

the default account

The account to use to send the email.

from

no

-

The email address from which the email will be sent. The from field can contain Mustache templates as long as it resolves to a valid email address.

to

yes

-

The email addresses of the to recipients. The to field can contain Mustache templates as long as it resolves to a valid email address.

cc

no

-

The email addresses of the cc recipients. The cc field can contain Mustache templates as long as it resolves to a valid email address.

bcc

no

-

The email addresses of the bcc recipients. The bcc field can contain Mustache templates as long as it resolves to a valid email address.

reply_to

no

-

The email addresses that will be set on the message’s Reply-To header. The reply_to field can contain Mustache templates as long as it resolves to a valid email address.

subject

no

-

The subject of the email. The subject can be static text or contain Mustache templates.

body

no

-

The body of the email. When this field holds a string, it will default to the text body of the email. Set as an object to specify either the text or the html body or both (using the fields below)

body.text

yes*

-

The plain text body of the email. The body can be static text or contain Mustache templates.

body.html

yes*

-

The html body of the email. The body can be static text or contain Mustache templates. This body will be sanitized to remove dangerous content such as scripts. This behavior can be disabled by setting watcher.actions.email.sanitize_html: false in elasticsearch.yaml.

priority

no

-

The priority of this email. Valid values are: lowest, low, normal, high and highest. The priority can contain a Mustache template as long as it resolves to one of the valid values.

attach_data

no

false

Indicates whether the watch execution data should be attached to the email. You can specify a Boolean value or an object. If attach_data is set to true, the data is attached as a YAML file called data.yml. If it’s set to false, no data is attached. To control the format of the attached data, specify an object that contains a format field`.

attach_data.format

no

yaml

When attach_data is specified as an object, this field controls the format of the attached data. The supported formats are json and yaml.
  • When setting the body object, at least one of its text or html fields must be defined.
Email Address
An email address can contain two possible parts—​the address itself and an optional personal name as described in RFC 822. The address can be represented either as a string of the form user@host.domain or Personal Name <user@host.domain>. You can also specify an email address as an object that contains name and address fields.
Address List
A list of addresses can either be specified as a comma-delimited string or as an array: 'Personal Name <user1@host.domain>, user2@host.domain' or [ 'Personal Name <user1@host.domain>', 'user2@host.domain' ]

Webhook Action

edit

A watch action that connects to a web server and listens on a specific port. The webhook action supports both HTTP and HTTPS connections. See Table 12, “Webhook Action Attributes” for the supported attributes.

Configuring Webhook Actions

edit

You configure webhook actions in a watch’s actions array. Action-specific attributes are specified using the webhook keyword.

The following snippet shows a simple webhook action definition:

"actions" : {
  "my_webhook" : { 
    "transform" : { ... }, 
    "throttle_period" : "5m", 
    "webhook" : {
      "method" : "POST", 
      "host" : "mylisteningserver", 
      "port" : 9200, 
      "path": ":/{{ctx.watch_id}", 
      "body" : "{{ctx.watch_id}}:{{ctx.payload.hits.total}}" 
    }
  }
}

The id of the action

An optional transform to transform the payload before executing the webhook action

An optional throttle period for the action (5 minutes in this example)

The HTTP method to use when connecting to the host

The host to connect to

The port to connect to

The path (URI) to use in the HTTP request

The body to send with the request

You can use basic authentication when sending a request to a secured webservice. For example:

"actions" : {
  "my_webhook" : {
    "webhook" : {
      "auth" : {
        "basic" : {
          "username" : "<username>", 
          "password" : "<password>" 
        }
      }
      "method" : "POST",
      "host" : "mylisteningserver",
      "port" : 9200,
      "path": ":/{{ctx.watch_id}",
      "body" : "{{ctx.watch_id}}:{{ctx.payload.hits.total}}"
    }
  }
}

The username

The corresponding password

By default, both the username and the password are stored in the .watches index in plain text. When Shield is installed, Watcher can be configured to encrypt the password before storing it.

You can also use PKI-based authentication when submitting requests to a cluster secured with Shield. When you use PKI-based authentication instead of HTTP basic auth, you don’t need to store any authentication information in the watch itself. To use PKI-based authentication, you configure the SSL keystore path and password for Watcher in elasticsearch.yml. If the watcher.http.ssl.keystore.path and watcher.http.ssl.keystore password are not set, the Watcher HTTP client falls back to the Shield settings, shield.ssl.keystore.path and shield.ssl.keystore.password.

Query Parameters

edit

You can specify query parameters to send with the request with the params field. This field simply holds an object where the keys serve as the parameter names and the values serve as the parameter values:

"actions" : {
  "my_webhook" : {
    "webhook" : {
      "method" : "POST",
      "host" : "mylisteningserver",
      "port" : 9200,
      "path": ":/alert",
      "params" : {
        "watch_id" : "{{ctx.watch_id}}" 
      }
    }
  }
}

The parameter values can contain templated strings.

Custom Request Headers

edit

You can specify request headers to send with the request with the headers field. This field simply holds an object where the keys serve as the header names and the values serve as the header values:

"actions" : {
  "my_webhook" : {
    "webhook" : {
      "method" : "POST",
      "host" : "mylisteningserver",
      "port" : 9200,
      "path": ":/alert/{{ctx.watch_id}}",
      "headers" : {
        "Content-Type" : "application/yaml" 
      },
      "body" : "count: {{ctx.payload.hits.total}}"
    }
  }
}

The header values can contain templated strings.

Table 12. Webhook Action Attributes

Name Required Default Description

scheme

no

http

The connection scheme. Valid values are: http or https.

host

yes

-

The host to connect to.

port

yes

-

The port the HTTP service is listening on.

path

no

-

The URL path. The path can be static text or include Mustache templates. URL query string parameters must be specified via the request.params attribute.

method

no

get

The HTTP method. Valid values are: head, get, post, put and delete.

headers

no

-

The HTTP request headers. The header values can be static text or include Mustache templates.

params

no

-

The URL query string parameters. The parameter values can be static text or include Mustache templates.

auth

no

-

Authentication related HTTP headers. Currently, only basic authentication is supported.

body

no

-

The HTTP request body. The body can be static text or include Mustache templates. When not specified, an empty body is sent.

proxy.host

no

-

The proxy host to use when connecting to the host.

proxy.port

no

-

The proxy port to use when connecting to the host.

connection_timeout

no

10s

The timeout for setting up the http connection. If the connection could not be set up within this time, the action will timeout and fail. It is also possible to configure the default connection timeout for all http connection timeouts.

read_timeout

no

10s

The timeout for reading data from http connection. If no response was received within this time, the action will timeout and fail. It is also possible to configure the default read timeout for all http connection timeouts.

Index Action

edit

A watch Action that enable you to index data in Elasticsearch. See Table 13, “Index Action Attributes” for the supported attributes.

Configuring Index Actions

edit

The following snippet shows a simple index action definition:

"actions" : {
  "index_payload" : { 
    "transform": { ... },
    "index" : {
      "index" : "my-index", 
      "doc_type" : "my-type" 
    }
  }
}

The id of the action

An optional transform to transform the payload and prepare the data that should be indexed

The elasticsearch index to store the data to

The document type to store the data as

Table 13. Index Action Attributes

Name Required Default Description

index

yes

-

The Elasticsearch index to index into. are supported

doc_type

yes

-

The type of the document the data will be indexed as.

execution_time_field

no

_timestamp

The field that will store/index the watch execution time. When not set or when set to _timestamp, the execution time will serve as the document’s timestamp.

timeout

no

60s

The timeout for waiting for the index api call to return. If no response is returned within this time, the index action times out and fails. This setting overrides the default internal index/bulk operations timeouts.

Multi-Document Support

edit

Like with all other actions, you can use a transform to replace the current execution context payload with another and by that change the document that will end up indexed.

The index action plays well with transforms with its support for the special _doc payload field.

When resolving the document to be indexed, the index action first looks up for a _doc field in the payload. When not found, the payload is indexed as a single document.

When a _doc field exists, if the field holds an object, it is extracted and indexed as a single document. If the field holds an array of objects, each object is treated as a document and the index aciton indexes all of them in a bulk.

Logging Action

edit

A watch action that simply logs text to the standard Elasticsearch logs. See Table 14, “Logging Action Attributes” for the supported attributes.

This action is primarily used during development and debugging.

Configuring Logging Actions

edit

You configure logging actions in a watch’s actions array. Action-specific attributes are specified using the logging keyword.

The following snippet shows a simple logging action definition:

"actions" : {
  "log" : { 
    "transform" : { ... }, 
    "logging" : {
      "text" : "executed at {{ctx.execution_time}}" 
    }
  }
}

The id of the action.

An optional transform to transform the payload before executing the logging action.

The text to be logged.

Table 14. Logging Action Attributes

Name Required Default Description

text

yes

-

The text that should be logged. Can be static text or include Mustache templates.

category

no

watcher.actions.logging

The category under which the text will be logged.

level

no

info

The logging level. Valid values are: error, warn, info, debug and trace.

HipChat Action

edit

A watch action that sends messages to HipChat rooms or users. To use the HipChat action, you need to configure at least one HipChat account in Watcher. For information about configuring accounts, see Configuring Watcher to Send Notifications to HipChat.

Configuring HipChat Actions

edit

You configure HipChat actions in a watch’s actions array. Action-specific attributes are specified using the hipchat keyword. You must specify the message attribute for all hipchat actions. If you omit the account attribute, the message is sent using the default HipChat account configured in elasticsearch.yml.

For example, the following action is configured to send messages using a HipChat account that uses the integration profile. Because this type of account can only send messages to a particular room, the only required attribute is the message itself.

"actions" : {
  "notify-hipchat" : {
    "transform" : { ... },
    "throttle_period" : "5m",
    "hipchat" : {
      "account" : "integration-account", 
      "message" : {
        "body" : "Encountered  {{ctx.payload.hits.total}} errors in the last 5 minutes (facepalm)", 
        "format" : "text",
        "color" : "red",
        "notify" : true
      }
    }
  }
}

The name of a HipChat account configured in elasticsearch.yml.

The message you want to send to HipChat.

To send messages with a HipChat account that uses the user profile, you need to specify what rooms and users you want to send the message to. For example, the following action is configured to send messages to the mission-control and devops rooms as well as the user website-admin@example.com. (To send to multiple users or rooms, specify an array of strings.)

"actions" : {
  "notify-hipchat" : {
    "transform" : { ... },
    "throttle_period" : "5m",
    "hipchat" : {
      "account" : "user-account",
      "message" : {
        "room" : [ "mission-control", "devops" ],
        "user" : "website-admin@example.com",
        "body" : "Encountered  {{ctx.payload.hits.total}} errors in the last 5 minutes (facepalm)",
        "format" : "text",
        "color" : "red",
        "notify" : true
      }
    }
  }
}

To send messages with a HipChat account that uses the v1 profile, you need to specify what room or rooms you want to send the message to. For example, the following action is configured to send messages to the server-status room. (To send to multiple rooms, specify an array of strings.)

"actions" : {
  "notify-hipchat" : {
    "transform" : { ... },
    "throttle_period" : "5m",
    "hipchat" : {
      "account" : "v1-account",
      "message" : {
        "from" : "Watcher",
        "room" : [ "server-status", "infra-team" ],
        "body" : "Encountered  {{ctx.payload.hits.total}} errors in the last 5 minutes (facepalm)",
        "format" : "text",
        "color" : "red",
        "notify" : true
      }
    }
  }
}

HipChat Action Attributes

edit

The following table lists the attributes you can set for hipchat actions.

Name

Required

Default

Description

account

no

Default account

The HipChat account to use to send the message.

`message.body `

yes

-

The message content. Can contain up to 1000 characters.

message.format

no

html

The format of the message: text or html.

message.color

no

yellow

The background color of the notification in the room: gray, green, purple, red, yellow.

message.notify

no

false

Indicates whether people in the room should be actively notified

message.from

no

the watch ID

The name that appears as the notification sender. Only valid for accounts that use the v1 profile.

message.room

no

-

The rooms that the notification should go to. Accepts a string value or an array of string values. Must be specified when using the v1 profile. At least one room or user must be specified when using the user profile. Not valid for the integration profile.

message.user

no

-

The users that the notification should go to. Accepts a string value or an array of string values. At least one room or user must be specified when using the user profile. Not valid for the integration or v1 profiles.

Slack Action

edit

A watch action that sends messages to a Slack team’s channels or users. To use the Slack action, you need to configure at least one Slack account in Watcher. For information about configuring accounts, see Configuring Watcher to Send Notifications to Slack.

Configuring Slack Actions

edit

You configure Slack actions in a watch’s actions array. Action-specific attributes are specified using the slack keyword.

The following snippet shows a simple slack action definition:

"actions" : {
  "notify-slack" : {
    "transform" : { ... },
    "throttle_period" : "5m",
    "slack" : {
      "message" : {
        "to" : [ "#admins", "@chief-admin" ], 
        "text" : "Encountered  {{ctx.payload.hits.total}} errors in the last 5 minutes (facepalm)" 
      }
    }
  }
}

The channels and users you want to send the message to.

The content of the message.

Using Attachments to Format Slack Messages

edit

In addition to sending simple text-based messages, you can use the Slack attachment mechanism to send formatted messages. Watcher leverages Slack attachments to enable you to dynamically populate templated messages from the watch payload.

The following snippet shows a standard message attachment.

"actions" : {
  "notify-slack" : {
    "throttle_period" : "5m",
    "slack" : {
      "account" : "team1",
      "message" : {
        "from" : "watcher",
        "to" : [ "#admins", "@chief-admin" ],
        "text" : "System X Monitoring",
        "attachments" : [
          {
            "title" : "Errors Found",
            "text" : "Encountered  {{ctx.payload.hits.total}} errors in the last 5 minutes (facepalm)",
            "color" : "danger"
          }
        ]
      }
    }
  }
}

To define an attachment template that is dynamically populated from the watch payload, you specify dynamic_attachments in the watch action. For example, a dynamic attachment could reference histogram buckets in the payload and build an attachment per bucket.

In the following example, the watch input executes a search with a date histogram aggregation and the Slack action:

  1. Transforms the payload to a list where each item in the list holds the month, the user count for that month, and the color that represents the sentiment associated with that count (good or danger).
  2. Defines an attachment template that references items in the list generated by the transform. 
"input" : {
  "search" : {
    "request" : {
      "body" : {
        "aggs" : {
          "users_per_month" : {
            "date_histogram" : {
              "field" : "@timestamp",
              "interval" : "1m"
            }
          }
        }
      }
    }
  }
},
...
"actions" : {
  "notify-slack" : {
    "throttle_period" : "5m",
    "transform" : {
      "script" : "return [ items : ctx.payload.hits.aggs.users_per_month.buckets.collect { [ month : it.key_as_string, count : it.doc_count, color : it.doc_count < 100 ? 'danger' : 'good' ] }]"
    },
    "slack" : {
      "account" : "team1",
      "message" : {
        "from" : "watcher",
        "to" : [ "#admins", "@chief-admin" ],
        "text" : "System X Monitoring",
        "dynamic_attachments" : {
          "list_path" : "ctx.payload.items" 
          "attachment_template" : {
            "title" : "{{month}}", 
            "text" : "Users Count: {{count}}",
            "color" : "{{color}}"
          }
        }
      }
    }
  }
}

The list generated by the action’s transform.

The parameter placeholders refer to attributes in each item of the list generated by the transform.

Slack Action Attributes

edit
Name Required Description

from

no

The sender name to display in the Slack message. Overrides the incoming webhook’s configured name.

to

yes

The channels and users you want to send the message to. Channel names must start with # and user names must start with @. Accepts a string value or an array of string values.

icon

no

The icon to display in the Slack messages. Overrides the incoming webhook’s configured icon. Accepts a public URL to an image.

text

yes

The message content.

attachments

no

Slack message attachments. Message attachments enable you to create more richly-formatted messages. Specified as as array as defined in the Slack attachments documentation.

dynamic_attachments

no

Slack message attachments that can be populated dynamically based on the current watch payload. For more information, see Using Attachments to Format Slack Messages.
« Condition Transform »