Monitoring Watch Execution

Whenever a watch is triggered, a watch_record document is created and added to the watch history index. A new history index is created daily with a name of the form .watch_history-YYYY.MM.dd. You can search the watch history like any other Elasticsearch index or use Kibana to monitor and visualize watch execution.

A watch record’s _source field contains all of the information about the watch execution:

watch_id: The name of the watch that was triggered.
trigger_event: How the watch was triggered (manual or schedule) and the watch’s scheduled time and actual trigger time.
input: The input type (http, search, or simple) and definition.
condition: The condition type (always, never, or script) and definition.
state: The state of the watch execution (execution_not_needed, executed, throttled).
result: The results of each phase of the watch execution. Shows the input payload, condition status, transform status (if defined), and actions status.

While you can perform read operations on the watch history and manage the daily indices as needed, you should never perform write operations on a watch history index. If you have Shield installed, we recommend only allowing users read access to the watch history index.

Monitoring Watches with Kibana

edit

You can use Kibana to monitor the watch history and create visualizations of the watches that have executed over time.

To monitor watches with Kibana:

Go to the Kibana Settings > Indices tab. For example, http://localhost:5601/#/settings/indices.
Enter .watch_history* in the Index name or pattern field.
Click in the Time field name field and select trigger_event.triggered_time.
Go to the Discover tab to see the most recently executed watches.

You can create visualizations and add them to a Kibana dashboard to track what watches are being triggered and identify trends.

For example you could create a dashboard to:

Track triggered watches over time, broken down by top watch.
Identify top senders, priorities, and keywords for email actions.
Identify top webhook targets and status codes.

watcher kibana dashboard

Searching the Watch History

edit

To get the watch history for a particular day, search that day’s watch history index:

GET .watch_history-2015.05.11/_search
{
  "query" : { "match_all" : {}}
}

To get all of the watch records that reference a particular watch, search the watch_id field:

GET .watch_history*/_search
{
  "query" : { "match" : { "watch_id": "rss_watch" }}
}

To get all of the watch records for watches that were throttled, search the state field.

GET .watch_history*/_search
{
  "query" : { "match" : { "state": "throttled" }}
}

To get a date histogram over all triggered watches within a particular time range.

GET .watch_history*/_search?search_type=count
{
  "query": {
    "filtered": {
      "query": {
        "match_all": {}
      },
      "filter": {
        "range": {
          "trigger_event.triggered_time": {
            "gte": 1430438400000,
            "lte": 1431820800000
          }
        }
      }
    }
  },
  "aggs": {
    "records_per_minute": {
      "date_histogram": {
        "field": "trigger_event.triggered_time",
        "interval": "1m",
        "min_doc_count": 0,
        "extended_bounds": {
          "min": 1430438400000,
          "max": 1431820800000
        }
      }
    }
  }
}

Managing Watch History Indexes

edit

You should establish a policy for how long you need to keep your watch history indexes. For example, you might simply delete the daily history indexes after 30 days. If you need to preserve the history but don’t need to maintain immediate access to it, you can close the index or take a snapshot and then delete it.

Elasticsearch Curator provides a convenient CLI for managing time-series indices.

You can also set up a watch to manage your watch history indexes. For example, the following watch that runs daily and uses a webhook action to delete history indexes older than seven days.

PUT _watcher/watch/manage_history
{
  "metadata": {
    "keep_history_days": 7
  },
  "trigger": {
    "schedule": { "daily": { "at" : "00:01" }}
  },
  "input": {
    "simple": {}
  },
  "condition": {
    "always": {}
  },
  "transform": {
    "script" : "return [ indexToDelete : '/.watch_history-' + ctx.execution_time.minusDays(ctx.metadata.keep_history_days + 1).toString('yyyy.MM.dd') ]"
  },
  "actions": {
    "delete_old_index": {
      "webhook": {
        "method": "DELETE",
        "host": "localhost",
        "port": 9200,
        "path": "{{ctx.payload.indexToDelete}}"
      }
    }
  }
}

« Getting Watcher Statistics Managing Watches »