Watcher Beta Goes Public! You Know, For Alerting...
Today we’re delighted to release the first public beta of Watcher – our commercial alerting and notification product. In the same spirit of Shield, Watcher is a standalone plugin for Elasticsearch.
Elasticsearch enables you to extract invaluable knowledge and insights from changes in your data, in realtime. But with the fast moving markets we all operate in today, it is not enough. In order for you to stay ahead of the game, you also need to do something with that knowledge – take actions that will directly translate to tangible values. And act fast. Act at realtime.
This is where Watcher complements Elasticsearch. While with Elasticsearch alone you need to be “proactively proactive” when it comes to analyzing your data, Watcher enables you to be “reactively proactive.” Hmm… what?
Yes! With Elasticsearch alone you need to continually be there to run the analytics, interpret the results and act on them. With Watcher that is no longer the case. Watcher will monitor the data for you, act on the changes and/or notify you about important events – freeing you up to focus on other important tasks, while letting you stay ahead of the game.
Here are a few examples of tasks you can define in Watcher:
- Watch indexed log events: Every 4 minutes, search for any errors that occurred in the last 5 minutes. Send me an email if more than 3 errors are found.
- Watch metrics data: Every 4 minutes, retrieve the average CPU usage over the last 5 minutes. Notify me when it goes above 75%.
- Watch indexed Tweets: Every minute, search for any mentions of earthquakes in the last 5 minutes. Notify me if more than 5 mentions are found in the same location.
- Watch incoming helpdesk events: Every 4 minutes, search for any new severity 1 tickets that were opened in the last 5 minutes. Use our pager service to notify me if any such tickets are found.
Watcher was designed to serve all the scenarios above and many more. And we’re very eager to see all the different use cases you use it for.
Next, I’ll show how you’d go about using Watcher. But first, if you haven’t registered to the beta program, don’t miss out and register now. Once registered, you will receive the installation instructions by email, and as you will find out, installing Watcher is as easy as installing any other Elasticsearch plugin.
How do I work with Watcher?
Watcher adds a new set of APIs to Elasticsearch to manage Watches. It’s probably best to explain the concept of a Watch with an example. In this example, suppose you are indexing your web traffic logs in Elasticsearch. Let's first create a logs index and enable the _timestamp mappings, such that a timestamp will be associated with each log event that we’ll index.
PUT logs
{
"mappings" : {
"event" : {
"_timestamp" : { "enabled" : true }
}
}
}
Now that we have the logs index ready, let’s add our first Watch. The immediate thing that comes to mind in this setup is the ability to pick up on errors in your logs as close to realtime as possible. In other words, you would like to watch newly indexed logs and look for any log events with an error status.
The following snippet registers a new watch that searches for any errors in the log entries in the last 5 minutes:
PUT /_watcher/watch/error_status
{
"trigger" : {
"schedule" : { "interval" : "5s" }
},
"input" : {
"search" : {
"request" : {
"indices" : [ "logs" ],
"body" : {
"query" : {
"filtered" : {
"query" : { "match" : { "status" : "error" }},
"filter" : { "range" : { "_timestamp" : { "from" : "now-5m" }}}
}
}
}
}
}
}
}
If you look closer at the watch definition above, you’ll see that we defined this watch with a schedule such that it will be executed every 5 seconds (I chose 5 seconds here just for the sake of the example, a more realistic value would have been 5 or 10 minutes).
The input loads data for the watch (what we like to refer to as payload). It does that by executing a search request over the logs index, and looks for events in the last 5 minutes with an “error” status.
Once this watch is added to Watcher, the schedule will immediately kick in and start ticking. Every 5 seconds the watch will execute and will search for errors. Although it looks like nothing is happening, in the background Watcher records all these executions in Watch History indices. And just like any other indices in Elasticsearch, you can explore them using the normal search API. Executing the following search request will return the last 10 watch executions:
GET /.watch_history*/_search
{
"sort" : [
{ "execution_result.execution_time" : "desc" }
]
}
OK… Searching for errors and then searching the watch history is a good start, but that’s just the beginning. It’s very likely you want to do something with the search results - find out if any errors occurred, and if they did, do something about it. So let’s change our watch to do exactly that:
PUT /_watcher/watch/error_status
{
"trigger" : {
"schedule" : { "interval" : "10s" }
},
"input" : {
"search" : {
"request" : {
"indices" : [ "logs" ],
"body" : {
"query" : {
"filtered" : {
"query" : { "match" : { "status" : "error" }},
"filter" : { "range" : { "_timestamp" : { "from" : "now-5m" }}}
}
}
}
}
}
},
"condition" : {
"compare" : {
"ctx.payload.hits.total" : { "gt" : 0 }
}
},
"actions" : {
"log_event" : {
"logging" : {
"text" : "found {{ctx.payload.hits.total}} errors in the logs"
}
}
}
}
We added two new elements to our watch: a condition and actions. The condition is a check that is evaluated over the watch payload. In our case, the payload is the search results and the condition checks if any errors were found.
When the condition evaluates to true, the configured actions are executed. Here we defined a very simple logging action. This simple action logs text to the standard Elasticsearch logs (use the logging action at development time for testing and debugging).
You probably noticed that the logged text is actually a (Mustache) template, which allows us to include the number of hits in the log messages. As you learn more about Watcher you will see that many constructs in Watcher accept templates, making Watcher highly customizable.
Even now, after the watch was updated, the action is still not executing. The reason for this is simply because there are no log events with an error status. You can verify that the condition has not been met by simply searching the watch history for a watch record with a “met” condition that was executed in the last 5 seconds:
GET /.watch_history*/_search
{
"query" : {
"bool" : {
"must" : [
{ "match" : { "execution_result.condition.met" : true }},
{ "range" : { "execution_result.execution_time" : { "from" : "now-5s"}}}
]
}
}
}
Running the above will return no results. Now, let’s put a new event, this time with an error status:
POST /logs/event
{
"status" : "error"
}
Within 5 seconds, you should see the log message in your Elasticsearch logs. If you search the watch history again, the results of the recent watch executions will be returned:
GET /.watch_history*/_search
{
"query" : {
"bool" : {
"must" : [
{ "match" : { "execution_result.condition.met" : true }},
{ "range" : { "execution_result.execution_time" : { "from" : "now-10s"}}}
]
}
}
}
Congrats! You’ve successfully created your first Watch.
What’s in a Watch?
By now you probably have a good feeling what a Watch is. Each Watch has its own purpose and lifecycle, and its execution is defined by a well-defined set of constructs: Trigger, Input, Condition and Actions. Each of these constructs represents an abstraction with potentially many different implementations.
The goal with Watcher was to build a system that not only naturally integrates with Elasticsearch, but is also extensible and prepared to serve a vast variety of use cases – from the simplest to the most complex requirements one might have. For the first public release we carefully chose an initial set of constructs that we believe will provide immediate value to you. You can view these constructs as the most basic building blocks for your watches:
- Schedule Trigger triggers the watches based on a schedule. Can be as simple as a fixed interval or more complex schedules based on cron expressions.
- Inputs load the initial payload for the condition to evaluate:
- search - Searches Elasticsearch, leveraging the full Elasticsearch query DSL capabilities
- http - Loads the data from an HTTP endpoint (making all Elasticsearch APIs accessible)
- simple - Static data that can be defined in the watch definition itself
- Conditions evaluate the loaded payload and determine whether the configured actions should be executed:
- always/never - Simple conditions that don’t actually check anything but simply always evaluate to either true or false
- compare - Compares a value in the loaded payload to a given value (seen in the example above)
- script - Leverages Elasticsearch’s scripting capabilities for more advance predicate logic
- Actions execute once the condition is met:
- email - Sends an email notification using your preferred SMTP service
- webhook - Notifies a 3rd party using a webhook (simple HTTP requests)
- index - Indexes the loaded payload back to Elasticsearch
- logging - Logs a message in the Elasticsearch logs (mostly used for development as seen above)
Where to now?
As mentioned, if you haven’t done so yet, you can register here to get access to the beta program. We highly recommend you read the online documentation to learn more about Watcher and try it out. We’re very curious to see how you are going to use it and what are you going to use it for, and we would absolutely love to get your feedback. For this we have set up a dedicated Watcher category in our forums.
Happy watching!