Anatomy of a Watch
Elasticsearch has many use cases and often those use cases involve knowing when an event that is streaming into the system meets certain conditions and having additional data from the event at our disposal. For these scenarios, we developed Watcher. This component is very powerful and flexible alerting tool, however, because it is so configurable sometimes reviewing the documentation can be daunting and seem very complex.
In this article, I’ll present the main concepts and describe the basics of Watcher to create alerts from a simple configuration for testing and a more real world implementation. By the end of these steps, you will not only have installed and configured Watcher with active alerts but you will understand the fundamentals necessary to tailor Watcher to your environment.
For this demonstration, the following software versions were used:
- Elasticsearch 2.4.0
- Watcher 2.4.0
- CentOS 6.7
- Kibana 4.6.1 with Sense*
*not required for these examples but convenient for submitting commands.
Installing the Watcher plugin
The installation instructions below assumes that your Elasticsearch nodes have access to the Internet. If they do not, you can manually download the .zip, tar.gz or .deb packages from the Elastic site.
*If you have not already done so, you’ll need to install the license plugin prior to installing and configuring watcher.
Navigate to your Elasticsearch installation directory and run the plugin tool
bin/plugin install license
*The license plugin comes with an auto-generated 30 day trial license for all Elastic plugins.
Install the Watcher plugin:
bin/plugin install watcher
Repeat this procedure on every node in the cluster.
For more information on installation options:
https://www.elastic.co/guide/en/watcher/current/installing-watcher.html
Configuring Watcher
For this article, We’ll be using a generic gmail account to send the alerts. Watcher is capable of sending the alerts to different targets such as Slack, HipChat, and PagerDuty. If you are interested in sending the alerts to different targets, you can find information in the Administering Watcher section of the documentation.
For more information see:
https://www.elastic.co/guide/en/watcher/current/administering-watcher.html
To use a gmail account as in the examples to follow, you must disable the two factor authentication to allow Watcher to send alerts from the account.
https://support.google.com/accounts/answer/6010255?hl=en
To configure the account for Watcher, add the following in elasticsearch.yml substituting your account credentials. The settings are indent sensitive, use spaces not tabs:
#------WATCHER CONFIG------
watcher.actions.email.service.account:
work:
profile: standard
email_defaults:
from: my.account@gmail.com
smtp:
auth: true
starttls.enable: true
host: smtp.gmail.com
port: 587
user: my.account@gmail.com
password: mypassword
If you plan to use scripting in the watch, you’ll also need to enable scripting by adding the following to the elasticsearch.yml file:
#------SCRIPTING CONFIG ----- script.engine.groovy.inline.elasticsearch-watcher_watch: on
You may already have enabled scripting for other uses with the following, if you have, then the above setting is not needed.
#------SCRIPTING CONFIG ----- script.inline: on script.indexed: on
Once you have installed the plugin on all nodes and have set the configurations above, you will need to restart each node for the settings to take effect.
What is a Watch?
A watch is basically an alert configuration. It is where we define the schedule, input, conditions, and actions that control what Watcher will do. The documentation examples may appear complex, however, they can be broken down into sections that ask just four basic questions from Watcher:
- When should I check?
- Where should I check?
- What should I check for?
- How should I tell you?
We’ll go through how to answer these questions for Watcher and wrap up the configuration at the end so we can submit it to Elasticsearch.
When should I check?
This configuration is answered by using the following syntax. In the example below, we want Watcher to check to every 5 minutes. This is done with the trigger parameter by specifying a schedule with an interval.
"trigger" : {
"schedule" : { "interval" : "5m" }
}
*There are several ways that you can specify the triggers such as by specifying certain times, days, etc. and also by specifying an interval as we have done in this example.
For more information on triggers:
https://www.elastic.co/guide/en/watcher/current/trigger.html
Where should I check?
This question is answered by using the input parameter. There are several types of inputs we can use but for our purposes, we will use two types, the simple input, which we will use for basic testing, and the search input to get data from an index in Elasticsearch. Since the inputs are loaded into the watch context, you can use the loaded information later as conditions, actions or transforms.
Example #1
In this example, we’re using the simple input which is just basically an inline document. We’ll use this to test our Watcher configuration and demonstrate how to pull a value, “red”, from our field, “my_field”, and inject it into the email alert in a later section.
"input" : {
"simple" : { "my_field" : "red" }
}
Example #2
Using this example, we’re going to use the search input type to specify an index, “my_monitored_index”, and a basic query that will retrieve all the documents that have a date stamp within the last 10 minutes in the “my_date” field. Here we use the request portion to specify what indices we want and the body portion is where we’ll define our query. The query section uses the same syntax as the _search body that a REST request would use. In this case, we are using a range query to get the data we want to look at.
"input" : {
"search" : {
"request" : {
"indices" : [ "my_monitored_index" ],
"body" : {
"query" : {
"range" : {
"my_date" : { "gte" : "now-10m" }
}
}
}
}
}
}
For more information on input types:
https://www.elastic.co/guide/en/watcher/current/input.html
What should I check for?
The answer for this is optional. You can actually just have Watcher send an alert without any conditions and it will default to the always type. For instance, if all you wanted was to have a count of documents in an index sent to you every day, etc. In our example watches, we are going to specify a condition in one and in the other we are going to omit this section for testing basic alerting.
To specify what to look for, we use the condition parameter and denote what type of condition type we want to use (always, never, script and compare). It can be very simple or we can create very complex conditions using a script type.
In this example, we are going to look for field, “my_field”, to have a value of “red” by using the compare type with the ctx.payload variable which we can then use to reference “my_field”.
"condition" : {
"compare" : {
"ctx.payload.hits.hits.0._source.my_field" : { "eq" : "red” }
}
}
For more information on condition types, variables and operators:
https://www.elastic.co/guide/en/watcher/current/condition.html
How should I tell you?
This is where you tell Watcher what it should do with the information it has found. Watcher can perform one or multiple actions and those actions can be injected with information about what it has found. In this example, we are going to use our previously configured gmail account in the elasticsearch.yml file to send an email with the value of “my_field” in the email body. This action is for our conditional example.
"actions" : {
"my_action" : {
"email" : {
"to" : "your_alert_mailbox@yourdomain.com",
"subject" : "Hello From Your Elasticsearch Cluster",
"body" : "This is the conditional watch. The value of my_field is: {{ ctx.payload.hits.hits.0._source.my_field }} "
}
}
}
Putting It All Together
Now that we have answered the different questions, we can now create a wrapper around the different parts and put them in Elasticsearch. The wrapper is just specifying the watcher endpoint and the name that we will give our watch. You can either use the Sense plugin in Kibana (which I use in the examples below) or you can use cURL through REST API.
The basic syntax is like this:
PUT _watcher/watch/my_watch_name
{
<TRIGGER>,
<INPUT>,
<CONDITION>,
<ACTIONS>
}
Example Watch #1 : Testing a basic alert and injecting a variable from a simple input type.
Using the our prior work, we just need to put it together into our wrapper template and index it into Elasticsearch. Since this watch doesn’t have any conditions, it will just fire and send an email every five minutes with the information.
PUT _watcher/watch/my_simple_watch
{
"trigger" : {
"schedule" : { "interval" : "5m" }
},
"input" : {
"simple" : { "my_field" : "red" }
},
"actions" : {
"my_action" : {
"email" : {
"to" : "my_alert_mailbox@mydomain.com",
"subject" : "Hello From Your Elasticsearch Cluster",
"body" : "This is the simple watch. The value of my_field is: {{ ctx.payload.my_field }} "
}
}
}
}
Example Watch #2 : Testing an alert with a condition defined with data from an index.
In this watch, we will include all of the sections and index a sample document to test the condition we set, mainly looking for documents with the value of “red” in “my_field”.
PUT _watcher/watch/my_conditional_watch
{
"trigger" : {
"schedule" : { "interval" : "5m" }
},
"input" : {
"search" : {
"request" : {
"search_type" : "query_and_fetch",
"indices" : [ "my_monitored_index" ],
"body" : {
"query" : {
"range" : {
"my_date" : { "gte" : "now-10m" }
}
}
}
}
}
},
"condition" : {
"compare" : {
"ctx.payload.hits.hits.0._source.my_field" : { "eq" : "red" }
}
},
"actions" : {
"my_action" : {
"email" : {
"to" : "your_alert_mailbox@yourdomain.com",
"subject" : "Hello From Your Elasticsearch Cluster",
"body" : "This is the conditional watch. The value of my_field is: {{ ctx.payload.hits.hits.0._source.my_field }} "
}
}
}
}
To test this watch, we’ll just need to index a simple document into Elasticsearch that has the value we’re looking for but first, let’s define the mapping for our index so that we can set the data type for the my_date field and allow us to do date math on it.
PUT /my_monitored_index
{
"mappings": {
"mytype": {
"properties": {
"my_field" : {
"type" : "string"
},
"my_date": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss"
}
}
}
}
}
Now we index a simple document. You will need to update the date below to match the time and date of your test in UTC. Try different tests such as changing the color value in my_field or setting the my_date field to a time outside the 10 minute window that we’ve configured in the watch.
PUT /my_monitored_index/mytype/1
{
"my_field" : "red",
"my_date" : "2016-09-14 23:00:00"
}
You should receive an email alert when the condition is met. In fact, you may already be tired of receiving these alerts and you want to get rid of them.
You can do this easily by using the API as in the example below:
DELETE _watcher/watch/my_simple_watch
Testing and Debugging Tools
You can force execution of a watch by using the _execute endpoint:
POST _watcher/watch/my_simple_watch/_execute
*The _execute endpoint also allows for different debugging parameters such as modifying actions, ignoring the condition, using alternative input, etc.
For information on testing and debugging options:
https://www.elastic.co/guide/en/watcher/current/api-rest.html#api-rest-execute-watch
Summary
Watcher is a powerful and highly configurable component of the Elastic Stack. These examples are meant to demystify the steps needed to configure a watch. From here, you can use different options to catch a variety of conditions from different sources and send them to multiple targets. For example, you can tell Watcher “Hey, I want you to check the log data coming in from my firewall every minute and see if within the last 5 minutes there were over 10 failed login attempts, if there was, send an email to our security group with the users list and notify the security chat room in Slack”
For detailed information about Watcher:
https://www.elastic.co/guide/en/watcher/current/introduction.html
For further information about the Watcher REST API:
https://www.elastic.co/guide/en/watcher/current/api-rest.html
Using the Watcher context:
https://www.elastic.co/guide/en/watcher/current/scripts-templates.html#watch-execution-context
Visualizing Watcher History in Kibana:
https://www.elastic.co/guide/en/watcher/current/watch-history.html#monitoring-watches