Watch Log Data for Errors

You can easily configure a watch that periodically checks your log data for error conditions:

A watch schedule controls how often a watch is triggered. The watch input gets the data that you want to evaluate.

To periodically search your log data and load the results into the watch, you use an interval schedule and a search input. For example, the following Watch searches the logs index for errors every 10 seconds:

curl -XPUT 'http://localhost:9200/_watcher/watch/log_error_watch' -d '{
  "trigger" : {
    "schedule" : { "interval" : "10s" } 
  },
  "input" : {
    "search" : {
      "request" : {
        "indices" : [ "logs" ],
        "body" : {
          "query" : {
            "match" : { "message": "error" }
          }
        }
      }
    }
  }
}'

If you check the watch history you’ll see that the watch is being triggered every 10 seconds. However, the search isn’t returning any results so nothing is loaded into the watch payload.

For example, the following snippet gets the last ten watch executions (a.k.a watch records) from the watch history:

curl -XGET 'http://localhost:9200/.watch_history*/_search?pretty' -d '{
  "sort" : [
    { "result.execution_time" : "desc" }
  ]
}'

A condition evaluates the data you’ve loaded into the watch and determines if any action is required. Since you’ve defined an input that loads log errors into the watch, you can define a condition that checks to see if any errors were found.

For example, you could add a condition that simply checks to see if the search input returned any hits.

curl -XPUT 'http://localhost:9200/_watcher/watch/log_error_watch' -d '{
  "trigger" : { "schedule" : { "interval" : "10s" } },
  "input" : {
    "search" : {
      "request" : {
        "indices" : [ "logs" ],
        "body" : {
          "query" : {
            "match" : { "message": "error" }
          }
        }
      }
    }
  },
  "condition" : {
    "compare" : { "ctx.payload.hits.total" : { "gt" : 0 }} 
  }
}'

The condition result is recorded as part of the watch_record each time the watch executes. Since there are currently no log events in the logs index, the watch condition will not be met. If you search the history for watch executions where the condition was met during the last 5 seconds, there are no hits:

curl -XGET 'http://localhost:9200/.watch_history*/_search?pretty' -d '{
  "query" : {
    "bool" : {
      "must" : [
        { "match" : { "result.condition.met" : true }},
        { "range" : { "result.execution_time" : { "from" : "now-10s"}}}
      ]
    }
  }
}'

For the condition in the example above to evaluate to true, you need to add an event to the logs index that contains an error.

For example, the following snippet adds a 404 error to the logs index:

curl -XPOST 'http://localhost:9200/logs/event' -d '{
    "timestamp" : "2015-05-17T18:12:07.613Z",
    "request" : "GET index.html",
    "status_code" : 404,
    "message" : "Error: File not found"
}'

Once you add this event, the next time the watch executes its condition will evaluate to true. You can verify this by searching the watch history:

curl -XGET 'http://localhost:9200/.watch_history*/_search?pretty' -d '{
  "query" : {
    "bool" : {
      "must" : [
        { "match" : { "result.condition.met" : true }},
        { "range" : { "result.execution_time" : { "from" : "now-10s"}}}
      ]
    }
  }
}'

Recording watch_records in the watch history is nice, but the real power of Watcher is being able to do something when the watch condition is met. The watch’s actions define what to do when the watch condition evaluates to true--you can send emails, call third-party webhooks, write documents to an Elasticsearch or log messages to the standards Elasticsearch log files.

For example, you could add an action to write a message to the Elasticsearch log when an error is detected.

curl -XPUT 'http://localhost:9200/_watcher/watch/log_error_watch' -d '{
  "trigger" : { "schedule" : { "interval" : "10s" } },
  "input" : {
    "search" : {
      "request" : {
        "indices" : [ "logs" ],
        "body" : {
          "query" : {
            "match" : { "message": "error" }
          }
        }
      }
    }
  },
  "condition" : {
    "compare" : { "ctx.payload.hits.total" : { "gt" : 0 }}
  },
  "actions" : {
    "log_error" : {
      "logging" : {
        "text" : "Found {{ctx.payload.hits.total}} errors in the logs"
      }
    }
  }
}'

Since the log_error_watch is configured to run every 10 seconds, make sure you delete it when you’re done experimenting. Otherwise, the noise from this sample watch will make it hard to see what else is going on in your watch history and log file.

To remove the watch, use the DELETE watch API:

curl -XDELETE 'http://localhost:9200/_watcher/watch/log_error_watch'