WARNING: Version 5.1 of the Elastic Stack has passed its EOL date.
This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.
Watch Log Data for Errors
editWatch Log Data for Errors
editYou can easily configure a watch that periodically checks your log data for error conditions:
- Schedule the watch and define an input to search your log data for error events.
- Add a condition that checks to see if any errors were found.
- Take action if there are any errors.
Schedule the Watch and Add an Input
editA watch schedule controls how often a watch is triggered. The watch input gets the data that you want to evaluate.
To periodically search your log data and load the results into the watch, you
use an interval schedule and a search
input. For example, the following Watch searches the logs
index for errors
every 10 seconds:
PUT _xpack/watcher/watch/errors_in_logs { "trigger" : { "schedule" : { "interval" : "10s" } }, "input" : { "search" : { "request" : { "indices" : [ "logs" ], "body" : { "query" : { "match" : { "message": "error" } } } } } } }
Schedules are typically configured to run less frequently. This example sets the interval to 10 seconds so you can easily see the watches being triggered. Since this watch runs so frequently, don’t forget to delete the watch when you’re done experimenting. |
If you check the watch history you’ll see that the watch is being triggered every 10 seconds. However, the search isn’t returning any results so nothing is loaded into the watch payload.
For example, the following snippet gets the last ten watch executions (a.k.a watch records) from the watch history:
GET .watcher-history*/_search?pretty { "sort" : [ { "result.execution_time" : "desc" } ] }
Add a Condition
editA condition evaluates the data you’ve loaded into the watch and determines if any action is required. Since you’ve defined an input that loads log errors into the watch, you can define a condition that checks to see if any errors were found.
For example, you could add a condition that checks to see if the search input returned any hits.
PUT _xpack/watcher/watch/errors_in_logs { "trigger" : { "schedule" : { "interval" : "10s" }}, "input" : { "search" : { "request" : { "indices" : [ "logs" ], "body" : { "query" : { "match" : { "message": "error" } } } } } }, "condition" : { "compare" : { "ctx.payload.hits.total" : { "gt" : 0 }} } }
The compare condition lets you easily compare against values in the execution context without enabling dynamic scripting. |
The condition result is recorded as part of the watch_record
each time the watch
executes. Since there are currently no log events in the logs
index, the watch
condition will not be met. If you search the history for watch executions where
the condition was met during the last 5 seconds, there are no hits:
GET .watcher-history*/_search?pretty { "query" : { "bool" : { "must" : [ { "match" : { "result.condition.met" : true }}, { "range" : { "result.execution_time" : { "from" : "now-10s" }}} ] } } }
For the condition in the example to evaluate to true
, you need to add an event
to the logs
index that contains an error. For example, the following snippet
adds a 404 error to the logs
index:
POST logs/event { "timestamp" : "2015-05-17T18:12:07.613Z", "request" : "GET index.html", "status_code" : 404, "message" : "Error: File not found" }
Once you add this event, the next time the watch executes its condition will
evaluate to true
. You can verify this by searching the watch history:
XGET .watcher-history*/_search?pretty { "query" : { "bool" : { "must" : [ { "match" : { "result.condition.met" : true }}, { "range" : { "result.execution_time" : { "from" : "now-10s" }}} ] } } }
Take Action
editRecording watch_records
in the watch history is nice, but the real power of
Watcher is being able to do something when the watch condition is met. The
watch’s actions define what to do when the watch condition evaluates
to true
--you can send emails, call third-party webhooks, write documents to an
Elasticsearch or log messages to the standards Elasticsearch log files.
For example, you could add an action to write a message to the Elasticsearch log when an error is detected.
PUT _xpack/watcher/watch/log_error_watch { "trigger" : { "schedule" : { "interval" : "10s" }}, "input" : { "search" : { "request" : { "indices" : [ "logs" ], "body" : { "query" : { "match" : { "message": "error" } } } } } }, "condition" : { "compare" : { "ctx.payload.hits.total" : { "gt" : 0 }} }, "actions" : { "log_error" : { "logging" : { "text" : "Found {{ctx.payload.hits.total}} errors in the logs" } } } }
Delete the Watch
editSince the log_error_watch
is configured to run every 10 seconds, make sure you
delete it when you’re done experimenting. Otherwise, the noise from this sample
watch will make it hard to see what else is going on in your watch history and
log file.
To remove the watch, use the DELETE watch API:
DELETE _xpack/watcher/watch/log_error_watch