From version 5.0 onward, Watcher is part of X-Pack. For more information, see Alerting on cluster and index events.

« Java API Troubleshooting »

› ›

REST API

REST API

PUT Watch API

The PUT watch API either registers a new watch in watcher or update an existing one. Once registered, a new document will be added to the .watches index, representing the watch, and the watch’s trigger will immediately be registered with the relevant trigger engine (typically the scheduler, for the schedule trigger).

Putting a watch must be done via this API only. Do not put a watch directly to the .watches index using Elasticsearch’s Index API. When integrating with Shield, a best practice is to make sure no write privileges are granted to anyone over the .watches API.

The following example adds a watch with the my-watch id that has the following qualities:

The watch schedule triggers every minute.
The watch search input finds any 404 HTTP responses that occurred in the past five minutes.
The watch condition checks the search results for 404s.
The watch action sends an email if there are any 404s.

PUT _watcher/watch/my-watch
{
  "trigger" : {
    "schedule" : { "cron" : "0 0/1 * * * ?" }
  },
  "input" : {
    "search" : {
      "request" : {
        "indices" : [
          "logstash*"
        ],
        "body" : {
          "query" : {
            "filtered": {
              "query": {
                "match": { "response": 404 }
              },
              "filter": {
                "range": {
                  "@timestamp" : {
                    "from": "{{ctx.trigger.scheduled_time}}||-5m",
                    "to": "{{ctx.trigger.triggered_time}}"
                 }
                }
              }
            }
          }
        }
      }
    }
  },
  "condition" : {
    "script" : "ctx.payload.hits.total > 1"
  },
  "actions" : {
    "email_admin" : {
        "email" : {
          "to" : "admin@domain.host.com",
          "subject" : "404 recently encountered"
        }
      }
    }
}'

A watch has the following fields:

Name	Description
`trigger`	The trigger that defines when the watch should run
`input`	The input that defines the input that loads the data for the watch
`condition`	The condition that defines if the actions should be run
`actions`	The list of actions that will be run if the condition matches
`meta`	Metadata json that will be copied into the history entries.
`throttle_period`	The minimum time between actions being run, the default for this is 5 seconds. This default can be changed in the config file with the setting `watcher.throttle.period.default_period`.

Timeouts

When updating a watch while it is executing, the put action will block and wait for the watch execution to finish. Depending on the nature of the watch, in some situations this can take a while. For this reason, the put watch action is associated with a timeout that is set to 10 seconds by default. You can control this timeout by passing in the master_timeout parameter.

The following snippet shows how to change the default timeout of the put action to 30 seconds:

PUT _watcher/watch/my-watch?master_timeout=30s

Get Watch API

This API retrieves a watch by its id.

The following example gets a watch with my-watch id:

GET _watcher/watch/my-watch

This is an example of the output:

{
  "found": true,
  "_id": "my_watch",
  "_status": { 
    "last_checked": "2015-05-26T18:21:08.630Z",
    "last_met_condition": "2015-05-26T18:21:08.630Z",
    "actions": {
      "email_admin": {
        "ack_status": {
          "timestamp": "2015-05-26T18:21:09.982Z",
          "state": "acked"
        },
        "last_execution": {
          "timestamp": "2015-05-26T18:21:04.106Z",
          "successful": true
        },
        "last_successful_execution": {
          "timestamp": "2015-05-26T18:21:04.106Z",
          "successful": true
        },
        "last_throttle": {
          "timestamp": "2015-05-26T18:21:08.630Z",
          "reason": "throttling interval is set to [5 seconds] but time elapsed since last execution is [4 seconds and 530 milliseconds]"
        }
      }
    }
  },
  "watch": {
    "input": {
      "search": {
        "request": {
          "search_type": "query_then_fetch",
          "indices": [
            "logstash*"
          ],
          "types": [],
          "body": {
            "query": {
              "filtered": {
                "filter": {
                  "range": {
                    "@timestamp": {
                      "from": "{{ctx.trigger.scheduled_time}}||-5m",
                      "to": "{{ctx.trigger.triggered_time}}"
                    }
                  }
                },
                "query": {
                  "match": {
                    "response": 404
                  }
                }
              }
            }
          }
        }
      }
    },
    "condition": {
      "script": {
        "type": "inline",
        "lang": "groovy",
        "params": {},
        "script": "ctx.payload.hits.total > 1"
      }
    },
    "trigger": {
      "schedule": {
        "cron": "0 0/1 * * * ?"
      }
    },
    "actions": {
      "email_admin": {
        "email": {
          "subject": {
            "type": "inline",
            "lang": "mustache",
            "params": {},
            "script": "404 recently encountered"
          },
          "attach_data": false,
          "to": [
            "someone@domain.host.com"
          ]
        }
      }
    }
  }
}

The retrieved watch will return with its current status

Delete Watch API

The DELETE watch API removes a specific watch (identified by its id) from watcher. Once removed, the document representing the watch in the .watches index will be gone and it will never be executed again.

Please note that deleting a watch does not delete any watch execution records related to this watch from the Watch History.

Deleting a watch must be done via this API only. Do not delete the watch directly from the .watches index using Elasticsearch’s DELETE Document API. When integrating with Shield, a best practice is to make sure no write privileges are granted to anyone over the .watches API.

The following example deletes a watch with the my-watch id:

DELETE _watcher/watch/my-watch

This is a sample output

{
   "found": true,
   "_id": "my_watch",
   "_version": 10
}

Timeouts

When deleting a watch while it is executing, the delete action will block and wait for the watch execution to finish. Depending on the nature of the watch, in some situations this can take a while. For this reason, the delete watch action is associated with a timeout that is set to 10 seconds by default. You can control this timeout by passing in the master_timeout parameter.

The following snippet shows how to change the default timeout of the delete action to 30 seconds:

DELETE _watcher/watch/my-watch?master_timeout=30s

Execute Watch API

The execute watch API forces the execution of a stored watch. It can be used to force execution of the watch outside of its triggering logic, or to test the watch for debugging purposes.

The following example executes the my-watch watch:

POST _watcher/watch/my-watch/_execute

For testing and debugging purposes, you also have fine-grained control on how the watch is executed—execute the watch without executing all of its actions or by simply simulating them. You can also force execution by ignoring the watch’s condition and control whether a watch record would be written to the watch history after execution.

This API supports the following fields:

Name	Required	Default	Description
trigger_data	no		This structure is parsed as the data of the trigger event that will be used during the watch execution
ignore_condition	no	false	When set to `true`, the watch execution uses the Always Condition.
alternative_input	no	null	When present, the watch uses this object as a payload instead of executing its own input.
action_modes	no	null	Determines how to handle the watch actions as part of the watch execution. See Action Execution Modes for more information.
record_execution	no	false	When set to `true`, the watch record representing the watch execution result is persisted to the `.watch_history` index for the current time. In addition, the status of the watch is updated, possbily throttling subsequent executions.
watch	no	null	When present, this watch is used instead of the one specified in the request. This watch is not persisted to the index and record_execution cannot be set.

The following example shows a comprehensive example of executing the my-watch watch:

POST _watcher/watch/my-watch/_execute
{
  "trigger_data" : { 
     "triggered_time" : "now",
     "scheduled_time" : "now"
  },
  "alternative_input" : { 
    "foo" : "bar"
  },
  "ignore_condition" : true, 
  "action_modes" : {
    "my-action" : "force_simulate" 
  },
  "record_execution" : true 
}

	The triggered and schedule times are provided.
	The input as defined by the watch is ignored and instead the provided input will be used as the execution payload.
	The condition as defined by the watch will be ignored and will be assumed to evaluate to `true`.
	Forces the simulation of `my-action`. Forcing the simulation means that throttling is ignored and the watch is simulated by Watcher instead of being executed normally.
	The execution of the watch will create a watch record in the watch history, and the throttling state of the watch will potentially be updated accordingly.

This is an example of the output:

{
   "_id": "my-watch_0-2015-06-02T23:17:55.124Z", 
   "watch_record": { 
      "watch_id": "my-watch",
      "trigger_event": {
         "type": "manual",
         "triggered_time": "2015-06-02T23:17:55.124Z",
         "manual": {
            "schedule": {
               "scheduled_time": "2015-06-02T23:17:55.124Z"
            }
         }
      },
      "state": "executed",
      "input": {
         "search": {
            "request": {
               "search_type": "query_then_fetch",
               "indices": [
                  "logstash*"
               ],
               "types": [],
               "body": {
                  "query": {
                     "filtered": {
                        "query": {
                           "match": {
                              "response": 404
                           }
                        },
                        "filter": {
                           "range": {
                              "@timestamp": {
                                 "from": "{{ctx.trigger.scheduled_time}}||-5m",
                                 "to": "{{ctx.trigger.triggered_time}}"
                              }
                           }
                        }
                     }
                  }
               }
            }
         }
      },
      "condition": {
         "script": "ctx.payload.hits.total > 1"
      },
      "result": { 
         "execution_time": "2015-06-02T23:17:55.124Z",
         "execution_duration": 12608,
         "input": {
            "type": "simple",
            "payload": {
               "foo": "bar"
            }
         },
         "condition": {
            "type": "always",
            "met": true
         },
         "actions": [
            {
               "id": "email_admin",
               "type" : "email"
               "status" : "success"
               "email": {
                  "account": "gmail",
                  "email": {
                     "id": "my-watch_0-2015-05-30T01:14:05.319Z",
                     "from": "watcher@example.com",
                     "sent_date": "2015-05-30T01:14:05.319Z",
                     "to": [
                        "admin@domain.host.com"
                     ],
                     "subject": "404 recently encountered"
                  }
               }
            }
         ]
      }
   }
}

	The id of the watch record as it would be stored in the `.watch_history` index.
	The watch record document as it would be stored in the `.watch_history` index.
	The watch execution results.

Action Execution Modes

Action modes define how actions will be handled during the watch execution. There are five possible modes an action can be associated with:

Name	Description
simulate	The action execution will be simulated. Each action type define its own simulation mode. For example, The email action will create the email that would have been sent but will not actually send it. In this mode, the action may be throttled if the current state of the watch indicates it should be.
force_simulate	Similar to the the `simulate` mode, except the action will not be throttled even if the current state of the watch indicates it should be.
execute	Executes the action as it would have been executed if the watch would have been triggered by its own trigger. The execution may be throttled if the current state of the watch indicates it should be.
force_execute	Similar to the `execute` mode, except the action ill not be throttled even if the current state of the watch indicates it should be.
skip	The action will be skipped and won’t be executed or simulated. Effectively forcing the action to be throttled.

You can set a different execution mode for every action by simply associating the mode name with the action id:

POST _watcher/watch/my-watch/_execute
{
  "action_modes" : {
    "action1" : "force_simulate",
    "action2" : "skip"
  }
}

You can also associate a single execution mode with all the watch’s actions using _all as the action id:

POST _watcher/watch/my-watch/_execute
{
  "action_modes" : {
    "_all" : "force_execute"
  }
}

Inline Watch Execution

You can use the Execute API to execute watches that are not yet registered in Watcher by specifying the watch definition inline. This serves as great tool for testing and debugging your watches prior to adding them to Watcher.

The following example demonstrates how you can test a watch defintion:

POST _watcher/watch/_execute
{
  "watch" : {
    "trigger" : { "schedule" : { "interval" : "10s" } },
    "input" : {
      "search" : {
        "request" : {
          "indices" : [ "logs" ],
          "body" : {
            "query" : {
              "match" : { "message": "error" }
            }
          }
        }
      }
    },
    "condition" : {
      "compare" : { "ctx.payload.hits.total" : { "gt" : 0 }}
    },
    "actions" : {
      "log_error" : {
        "logging" : {
          "text" : "Found {{ctx.payload.hits.total}} errors in the logs"
        }
      }
    }
  }
}

All other settings for this API still apply take effect when inlining a watch. In the following snippet, while the watch is defined with a compare condition, during execution this condition will be ignored:

POST _watcher/watch/_execute
{
  "ignore_condition" : true,
  "watch" : {
    "trigger" : { "schedule" : { "interval" : "10s" } },
    "input" : {
      "search" : {
        "request" : {
          "indices" : [ "logs" ],
          "body" : {
            "query" : {
              "match" : { "message": "error" }
            }
          }
        }
      }
    },
    "condition" : {
      "compare" : { "ctx.payload.hits.total" : { "gt" : 0 }}
    },
    "actions" : {
      "log_error" : {
        "logging" : {
          "text" : "Found {{ctx.payload.hits.total}} errors in the logs"
        }
      }
    }
  }
}

Ack Watch API

Acknowledging a watch enables you to manually throttle execution of the watch’s actions. An action’s acknowledgement state is stored in the _status.actions.<id>.ack.state structure.

The current status of a watch and the state of its actions is returned with the watch definition when you call the Get Watch API:

GET _watcher/watch/<watch_id>

The action state of a newly-created watch is awaits_successful_execution.

"_status": {
   ...
   "actions": {
    "action_id": {
     "ack": {
      "timestamp": "2015-05-26T18:04:27.723Z",
      "state": "awaits_successful_execution"
     },
     ...
    }
   }
}

When the watch runs and the condition matches, the value of the ack.state changes to ackable:

"_status": {
   ...
   "actions": {
    "action_id": {
     "ack": {
      "timestamp": "2015-05-26T18:19:08.758Z",
      "state": "ackable"
     },
     ...
    }
   }
}

Acknowledging the watch action (using the ACK API) sets the value of the ack.state to acked:

"_status": {
   ...
   "actions": {
    "action_id": {
     "ack": {
      "timestamp": "2015-05-26T18:21:09.982Z",
      "state": "acked"
     },
     ...
    }
   }
}

Acknowledging an action throttles further executions of that action until its ack.state is reset to awaits_successful_execution. This happens when the watch’s condition is checked and is not met (the condition evaluates to false).

The following snippet shows how to ack a watch action identified by its id. In this example, the watch id is my-watch and the id of the action being acknowledged is my-action:

PUT _watcher/watch/my-watch/my-action/_ack

As a response to this request, the full watch status is returned:

{
   "_status": {
      "last_checked": "2015-05-26T18:21:08.630Z",
      "last_met_condition": "2015-05-26T18:21:08.630Z",
      "actions": {
         "my-action": {
            "ack_status": {
               "timestamp": "2015-05-26T18:21:09.982Z",
               "state": "acked"
            },
            "last_execution": {
               "timestamp": "2015-05-26T18:21:04.106Z",
               "successful": true
            },
            "last_successful_execution": {
               "timestamp": "2015-05-26T18:21:04.106Z",
               "successful": true
            },
            "last_throttle": {
               "timestamp": "2015-05-26T18:21:08.630Z",
               "reason": "throttling interval is set to [5 seconds] but time elapsed since last execution is [4 seconds and 530 milliseconds]"
            }
         }
      }
   }
}

You can acknowledge multiple actions by assigning the actions parameter a comma-separated list of action ids:

PUT _watcher/watch/my-watch/action1,action2/_ack

To acknowledge all of a watch’s actions, simply omit the actions parameter:

PUT _watcher/watch/my-watch/_ack

Timeouts

If you acknowledge a watch while it is executing, the ack action blocks and waits for the watch execution to finish. For some watches, this can take a significant amount of time. By default, the ack watch action has a timeout of 10 seconds. You can change the timeout setting by specifying the master_timeout parameter.

The following snippet shows how to change the default timeout of the ack action to 30 seconds:

PUT _watcher/watch/my-watch/_ack?master_timeout=30s

Info API

The watcher info API gives basic version information on the watcher plugin that is installed.

The following example queries the info API.

GET _watcher

A successful call returns a JSON structure similar to the following example:

{
   "version": {
      "name" : "1.0.0-Beta2"
      "number": "1.0.0.Beta2", 
      "build_hash": "41f64213d2d370bf66f0e9b839a30a19", 
      "build_timestamp": "2015-04-07T13:34:42Z", 
      "build_snapshot": true 
   }
}

	The build number of the plugin
	The build hash of the plugin
	The time the plugin was built
	Whether or not this plugin was a development snapshot build

Stats API

The watcher stats API returns information on the aspects of watcher on your cluster.

The watcher stats API supports the following request options

Name	Required	Default	Description
metric	no	null	What metric should be returned.

The supported metric values:

Metric	Description
executing_watches	Include the current executing watches in the response.
queued_watches	Include the watches queued for execution in the response.
_all	Include all metrics in the response.

The watcher stats API always returns basic metrics regardless of the metric option.

The following example queries the stats API including the basic metrics:

GET _watcher/stats

A successful call returns a JSON structure similar to the following example:

{
   "watcher_state": "started",  
   "watch_count": 1, 
   "execution_thread_pool": {
      "size": 1000, 
      "max_size": 1 
   }
}

	The current state of watcher. May be either `started`, `starting` or `stopped`.
	The number of watches currently registered in watcher.
	The number of watches that were triggered and currently queued for execution.
	The largest size of the execution thread pool indicating the largest number of concurrent executing watches.

Current executing watches metric

The current executing watches metric gives insight into the watches that are currently being executed by Watcher. Per watch that is executing information is shared, like the watch_id, when execution started and at what phase the execution is.

To include this metric, the metric option should be set to executing_watches or _all.

The following example specifies the metric option as a query string argument and will include the basic metrics and metrics about the current watches being executed:

GET _watcher/stats?metric=executing_watches

The following example specifies the metric option as part of the url path:

GET _watcher/stats/current_watches

An example of a successful json response that captures a watch in execution:

{
   "watcher_state": "started",
   "watch_count": 2,
   "execution_thread_pool": {
      "queue_size": 1000,
      "max_size": 20
   },
   "current_watches": [ 
      {
         "watch_id": "slow_condition", 
         "watch_record_id": "slow_condition_3-2015-05-13T07:42:32.179Z", 
         "triggered_time": "2015-05-12T11:53:51.800Z", 
         "execution_time": "2015-05-13T07:42:32.179Z", 
         "execution_phase": "condition" 
      }
   ]
}

	A list of all the Watches that are currently being executed by Watcher. In case of an empty array no executing watches had been captured. The captured watches are sorted by execution time in descending order. So the longest running watch is always on top.
	The id of the watch being executed.
	The id of the watch record.
	The time the watch was triggered by the trigger engine.
	The time the watch was executed. This is just before the input is being executed.
	The current execution phase the watch is in. Can be `input`, `condition` or `action`.

Queued watches metric

When a watch triggers it is being prepared for execution and when there is capacity the watch get executed. If many watches trigger concurrently and there is no capacity to execute then watches are queued up. These watches are then queued for execution. The queued watches metric gives insight what watches are queued for execution.

To include this metric, the metric option should include queued_watches or _all.

The following example specifies the queued_watches metric option and will include the basic metrics and the watches queued for execution:

GET _watcher/stats/queued_watches

An example of a successful json response that captures a watch in execution:

{
   "watcher_state": "started",
   "watch_count": 10,
   "execution_thread_pool": {
      "queue_size": 1000,
      "max_size": 20
   },
   "queued_watches": [ 
         {
            "watch_id": "slow_condition4", 
            "watch_record_id": "slow_condition4_223-2015-05-21T11:59:59.811Z", 
            "triggered_time": "2015-05-21T11:59:59.811Z", 
            "execution_time": "2015-05-21T11:59:59.811Z" 
         },
      ...
   ]
}

	A list of all the Watches that are queued for execution. In case of an empty array no watches are queued for execution.
	The id of the watch queued for execution.
	The id of the watch record.
	The time the watch was triggered by the trigger engine.
	The time the watch was went into a queued state.

Stop API

The stop watcher API stops the watcher service if the service is running, as in the following example:

PUT _watcher/_stop

Watcher returns the following response if the request is successful:

{
   "acknowledged": true
}

Start API

The start watcher API starts the watcher service if the service is not already running, as in the following example:

PUT _watcher/_start

Watcher returns the following response if the request is successful:

{
   "acknowledged": true
}

Restart API

The restart watcher API stops, then starts the watcher service, as in the following example:

PUT _watcher/_restart

Watcher returns the following response if the request is successful:

{
   "acknowledged": true
}

« Java API Troubleshooting »