REST API
editREST API
editPUT Watch API
editThe PUT watch API either registers a new watch in watcher or update an existing one. Once registered, a new document
will be added to the .watches
index, representing the watch, and the watch’s trigger will immediately be registered
with the relevant trigger engine (typically the scheduler, for the schedule
trigger).
Putting a watch must be done via this API only. Do not put a watch directly to the .watches
index
using Elasticsearch’s Index API. When integrating with Shield, a best practice is to make sure
no write
privileges are granted to anyone over the .watches
API.
The following example adds a watch with the my-watch
id that has the following qualities:
- The watch schedule triggers every minute.
- The watch search input finds any 404 HTTP responses that occurred in the past five minutes.
- The watch condition checks the search results for 404s.
- The watch action sends an email if there are any 404s.
PUT _watcher/watch/my-watch { "trigger" : { "schedule" : { "cron" : "0 0/1 * * * ?" } }, "input" : { "search" : { "request" : { "indices" : [ "logstash*" ], "body" : { "query" : { "filtered": { "query": { "match": { "response": 404 } }, "filter": { "range": { "@timestamp" : { "from": "{{ctx.trigger.scheduled_time}}||-5m", "to": "{{ctx.trigger.triggered_time}}" } } } } } } } } }, "condition" : { "script" : "ctx.payload.hits.total > 1" }, "actions" : { "email_admin" : { "email" : { "to" : "admin@domain.host.com", "subject" : "404 recently encountered" } } } }'
A watch has the following fields:
Name | Description |
---|---|
|
The trigger that defines when the watch should run |
|
The input that defines the input that loads the data for the watch |
|
The condition that defines if the actions should be run |
|
The list of actions that will be run if the condition matches |
|
Metadata json that will be copied into the history entries. |
|
The minimum time between actions being run, the default for this is 5 seconds. This default can be changed in the config file with the setting |
Timeouts
editWhen updating a watch while it is executing, the put action will block and wait for the watch execution
to finish. Depending on the nature of the watch, in some situations this can take a while. For this reason,
the put watch action is associated with a timeout that is set to 10 seconds by default. You can control this
timeout by passing in the master_timeout
parameter.
The following snippet shows how to change the default timeout of the put action to 30 seconds:
PUT _watcher/watch/my-watch?master_timeout=30s
Controlling Default Active State
editWhen adding a watch you can also define its initial active state. You do that by
setting the active
parameter. The following command add a watch and sets it to be inactive by default:
PUT _watcher/watch/my-watch?active=false
If you omit the active
parameter, the watch is set to the active state by default.
Get Watch API
editThis API retrieves a watch by its id.
The following example gets a watch with my-watch
id:
GET _watcher/watch/my-watch
This is an example of the output:
{ "found": true, "_id": "my_watch", "_status": { "last_checked": "2015-05-26T18:21:08.630Z", "last_met_condition": "2015-05-26T18:21:08.630Z", "actions": { "email_admin": { "ack_status": { "timestamp": "2015-05-26T18:21:09.982Z", "state": "acked" }, "last_execution": { "timestamp": "2015-05-26T18:21:04.106Z", "successful": true }, "last_successful_execution": { "timestamp": "2015-05-26T18:21:04.106Z", "successful": true }, "last_throttle": { "timestamp": "2015-05-26T18:21:08.630Z", "reason": "throttling interval is set to [5 seconds] but time elapsed since last execution is [4 seconds and 530 milliseconds]" } } } }, "watch": { "input": { "search": { "request": { "search_type": "query_then_fetch", "indices": [ "logstash*" ], "types": [], "body": { "query": { "filtered": { "filter": { "range": { "@timestamp": { "from": "{{ctx.trigger.scheduled_time}}||-5m", "to": "{{ctx.trigger.triggered_time}}" } } }, "query": { "match": { "response": 404 } } } } } } } }, "condition": { "script": { "type": "inline", "lang": "groovy", "params": {}, "script": "ctx.payload.hits.total > 1" } }, "trigger": { "schedule": { "cron": "0 0/1 * * * ?" } }, "actions": { "email_admin": { "email": { "subject": { "type": "inline", "lang": "mustache", "params": {}, "script": "404 recently encountered" }, "to": [ "someone@domain.host.com" ] } } } } }
Delete Watch API
editThe DELETE watch API removes a specific watch (identified by its id
) from watcher. Once removed, the document
representing the watch in the .watches
index will be gone and it will never be executed again.
Please note that deleting a watch does not delete any watch execution records related to this watch from the Watch History.
Deleting a watch must be done via this API only. Do not delete the watch directly from the .watches
index
using Elasticsearch’s DELETE Document API. When integrating with Shield, a best practice is to make sure
no write
privileges are granted to anyone over the .watches
API.
The following example deletes a watch with the my-watch
id:
DELETE _watcher/watch/my-watch
This is a sample output
{ "found": true, "_id": "my_watch", "_version": 10 }
Timeouts
editWhen deleting a watch while it is executing, the delete action will block and wait for the watch execution
to finish. Depending on the nature of the watch, in some situations this can take a while. For this reason,
the delete watch action is associated with a timeout that is set to 10 seconds by default. You can control this
timeout by passing in the master_timeout
parameter.
The following snippet shows how to change the default timeout of the delete action to 30 seconds:
DELETE _watcher/watch/my-watch?master_timeout=30s
Execute Watch API
editThe execute watch API forces the execution of a stored watch. It can be used to force execution of the watch outside of its triggering logic, or to test the watch for debugging purposes.
The following example executes the my-watch
watch:
POST _watcher/watch/my-watch/_execute
For testing and debugging purposes, you also have fine-grained control on how the watch is executed—execute the watch without executing all of its actions or by simply simulating them. You can also force execution by ignoring the watch’s condition and control whether a watch record would be written to the watch history after execution.
This API supports the following fields:
Name | Required | Default | Description |
---|---|---|---|
trigger_data |
no |
This structure is parsed as the data of the trigger event that will be used during the watch execution |
|
ignore_condition |
no |
false |
When set to |
alternative_input |
no |
null |
When present, the watch uses this object as a payload instead of executing its own input. |
action_modes |
no |
null |
Determines how to handle the watch actions as part of the watch execution. See Action Execution Modes for more information. |
record_execution |
no |
false |
When set to |
watch |
no |
null |
When present, this watch is used instead of the one specified in the request. This watch is not persisted to the index and record_execution cannot be set. |
The following example shows a comprehensive example of executing the my-watch
watch:
POST _watcher/watch/my-watch/_execute { "trigger_data" : { "triggered_time" : "now", "scheduled_time" : "now" }, "alternative_input" : { "foo" : "bar" }, "ignore_condition" : true, "action_modes" : { "my-action" : "force_simulate" }, "record_execution" : true }
The triggered and schedule times are provided. |
|
The input as defined by the watch is ignored and instead the provided input will be used as the execution payload. |
|
The condition as defined by the watch will be ignored and will
be assumed to evaluate to |
|
Forces the simulation of |
|
The execution of the watch will create a watch record in the watch history, and the throttling state of the watch will potentially be updated accordingly. |
This is an example of the output:
{ "_id": "my-watch_0-2015-06-02T23:17:55.124Z", "watch_record": { "watch_id": "my-watch", "trigger_event": { "type": "manual", "triggered_time": "2015-06-02T23:17:55.124Z", "manual": { "schedule": { "scheduled_time": "2015-06-02T23:17:55.124Z" } } }, "state": "executed", "input": { "search": { "request": { "search_type": "query_then_fetch", "indices": [ "logstash*" ], "types": [], "body": { "query": { "filtered": { "query": { "match": { "response": 404 } }, "filter": { "range": { "@timestamp": { "from": "{{ctx.trigger.scheduled_time}}||-5m", "to": "{{ctx.trigger.triggered_time}}" } } } } } } } } }, "condition": { "script": "ctx.payload.hits.total > 1" }, "result": { "execution_time": "2015-06-02T23:17:55.124Z", "execution_duration": 12608, "input": { "type": "simple", "payload": { "foo": "bar" } }, "condition": { "type": "always", "met": true }, "actions": [ { "id": "email_admin", "type" : "email" "status" : "success" "email": { "account": "gmail", "email": { "id": "my-watch_0-2015-05-30T01:14:05.319Z", "from": "watcher@example.com", "sent_date": "2015-05-30T01:14:05.319Z", "to": [ "admin@domain.host.com" ], "subject": "404 recently encountered" } } } ] } } }
The id of the watch record as it would be stored in the |
|
The watch record document as it would be stored in the |
|
The watch execution results. |
Action Execution Modes
editAction modes define how actions will be handled during the watch execution. There are five possible modes an action can be associated with:
Name | Description |
---|---|
simulate |
The action execution will be simulated. Each action type define its own simulation mode. For example, The email action will create the email that would have been sent but will not actually send it. In this mode, the action may be throttled if the current state of the watch indicates it should be. |
force_simulate |
Similar to the the |
execute |
Executes the action as it would have been executed if the watch would have been triggered by its own trigger. The execution may be throttled if the current state of the watch indicates it should be. |
force_execute |
Similar to the |
skip |
The action will be skipped and won’t be executed or simulated. Effectively forcing the action to be throttled. |
You can set a different execution mode for every action by simply associating the mode name with the action id:
POST _watcher/watch/my-watch/_execute { "action_modes" : { "action1" : "force_simulate", "action2" : "skip" } }
You can also associate a single execution mode with all the watch’s actions using _all
as the action id:
POST _watcher/watch/my-watch/_execute { "action_modes" : { "_all" : "force_execute" } }
Inline Watch Execution
editYou can use the Execute API to execute watches that are not yet registered in Watcher by specifying the watch definition inline. This serves as great tool for testing and debugging your watches prior to adding them to Watcher.
The following example demonstrates how you can test a watch defintion:
POST _watcher/watch/_execute { "watch" : { "trigger" : { "schedule" : { "interval" : "10s" } }, "input" : { "search" : { "request" : { "indices" : [ "logs" ], "body" : { "query" : { "match" : { "message": "error" } } } } } }, "condition" : { "compare" : { "ctx.payload.hits.total" : { "gt" : 0 }} }, "actions" : { "log_error" : { "logging" : { "text" : "Found {{ctx.payload.hits.total}} errors in the logs" } } } } }
All other settings for this API still apply take effect when inlining a watch.
In the following snippet, while the watch is defined with a compare
condition,
during execution this condition will be ignored:
POST _watcher/watch/_execute { "ignore_condition" : true, "watch" : { "trigger" : { "schedule" : { "interval" : "10s" } }, "input" : { "search" : { "request" : { "indices" : [ "logs" ], "body" : { "query" : { "match" : { "message": "error" } } } } } }, "condition" : { "compare" : { "ctx.payload.hits.total" : { "gt" : 0 }} }, "actions" : { "log_error" : { "logging" : { "text" : "Found {{ctx.payload.hits.total}} errors in the logs" } } } } }
Ack Watch API
editAcknowledging a watch enables you to manually throttle
execution of the watch’s actions. An action’s acknowledgement state is stored in the
_status.actions.<id>.ack.state
structure.
The current status of a watch and the state of its actions is returned with the watch definition when you call the Get Watch API:
GET _watcher/watch/<watch_id>
The action state of a newly-created watch is awaits_successful_execution
.
"_status": { ... "actions": { "action_id": { "ack": { "timestamp": "2015-05-26T18:04:27.723Z", "state": "awaits_successful_execution" }, ... } } }
When the watch runs and the condition matches, the value of the ack.state
changes
to ackable
:
"_status": { ... "actions": { "action_id": { "ack": { "timestamp": "2015-05-26T18:19:08.758Z", "state": "ackable" }, ... } } }
Acknowledging the watch action (using the ACK API) sets the value of the ack.state
to acked
:
"_status": { ... "actions": { "action_id": { "ack": { "timestamp": "2015-05-26T18:21:09.982Z", "state": "acked" }, ... } } }
Acknowledging an action throttles further executions of that action until its
ack.state
is reset to awaits_successful_execution
. This happens when the watch’s condition
is checked and is not met (the condition evaluates to false
).
The following snippet shows how to ack a watch action identified by its id. In this example, the
watch id is my-watch
and the id of the action being acknowledged is my-action
:
PUT _watcher/watch/my-watch/my-action/_ack
As a response to this request, the full watch status is returned:
{ "_status": { "last_checked": "2015-05-26T18:21:08.630Z", "last_met_condition": "2015-05-26T18:21:08.630Z", "actions": { "my-action": { "ack_status": { "timestamp": "2015-05-26T18:21:09.982Z", "state": "acked" }, "last_execution": { "timestamp": "2015-05-26T18:21:04.106Z", "successful": true }, "last_successful_execution": { "timestamp": "2015-05-26T18:21:04.106Z", "successful": true }, "last_throttle": { "timestamp": "2015-05-26T18:21:08.630Z", "reason": "throttling interval is set to [5 seconds] but time elapsed since last execution is [4 seconds and 530 milliseconds]" } } } } }
You can acknowledge multiple actions by assigning the actions
parameter a
comma-separated list of action ids:
PUT _watcher/watch/my-watch/action1,action2/_ack
To acknowledge all of a watch’s actions, simply omit the actions
parameter:
PUT _watcher/watch/my-watch/_ack
Timeouts
editIf you acknowledge a watch while it is executing, the ack action blocks and waits for the watch
execution to finish. For some watches, this can take a significant amount of time. By default,
the ack watch action has a timeout of 10 seconds. You can change the timeout setting by
specifying the master_timeout
parameter.
The following snippet shows how to change the default timeout of the ack action to 30 seconds:
PUT _watcher/watch/my-watch/_ack?master_timeout=30s
Activate Watch API
editA watch can be either active or inactive. This API enables you to activate a currently inactive watch.
The status of an inactive watch is returned with the watch definition when you call the Get Watch API:
GET _watcher/watch/<watch_id>
"_status": { "state" : { "active" : false, "timestamp" : "2015-08-20T12:21:32.734Z" } "actions": { ... } } }
You can activate the watch by executing the following API call:
PUT _watcher/watch/<watch_id>/_activate
The new state of the watch is returned as part of its overall status.
"_status": { "state" : { "active" : true, "timestamp" : "2015-09-04T08:39:46.816Z" } "actions": { ... } } }
Deactivate Watch API
editA watch can be either active or inactive. This API enables you to deactivate a currently active watch.
The status of an active watch is returned with the watch definition when you call the Get Watch API:
GET _watcher/watch/<watch_id>
"_status": { "state" : { "active" : true, "timestamp" : "2015-08-20T12:21:32.734Z" } "actions": { ... } } }
You can deactivate the watch by executing the following API call:
PUT _watcher/watch/<watch_id>/_deactivate
The new state of the watch is returned as part of its overall status.
"_status": { "state" : { "active" : false, "timestamp" : "2015-09-04T08:39:46.816Z" } "actions": { ... } } }
Info API
editThe watcher info API gives basic version information on the watcher plugin that is installed.
The following example queries the info
API.
GET _watcher
A successful call returns a JSON structure similar to the following example:
Stats API
editThe watcher stats
API returns information on the aspects of watcher on your cluster.
The watcher stats
API supports the following request options
Name | Required | Default | Description |
---|---|---|---|
metric |
no |
null |
What metric should be returned. |
The supported metric values:
Metric | Description |
---|---|
executing_watches |
Include the current executing watches in the response. |
queued_watches |
Include the watches queued for execution in the response. |
_all |
Include all metrics in the response. |
The watcher stats
API always returns basic metrics regardless of the metric
option.
The following example queries the stats
API including the basic metrics:
GET _watcher/stats
A successful call returns a JSON structure similar to the following example:
{ "watcher_state": "started", "watch_count": 1, "execution_thread_pool": { "size": 1000, "max_size": 1 } }
The current state of watcher. May be either |
|
The number of watches currently registered in watcher. |
|
The number of watches that were triggered and currently queued for execution. |
|
The largest size of the execution thread pool indicating the largest number of concurrent executing watches. |
Current executing watches metric
editThe current executing watches metric gives insight into the watches that are currently being executed by Watcher.
Per watch that is executing information is shared, like the watch_id
, when execution started and at what phase the
execution is.
To include this metric, the metric
option should be set to executing_watches
or _all
.
The following example specifies the metric
option as a query string argument and will include the basic metrics and
metrics about the current watches being executed:
GET _watcher/stats?metric=executing_watches
The following example specifies the metric
option as part of the url path:
GET _watcher/stats/current_watches
An example of a successful json response that captures a watch in execution:
{ "watcher_state": "started", "watch_count": 2, "execution_thread_pool": { "queue_size": 1000, "max_size": 20 }, "current_watches": [ { "watch_id": "slow_condition", "watch_record_id": "slow_condition_3-2015-05-13T07:42:32.179Z", "triggered_time": "2015-05-12T11:53:51.800Z", "execution_time": "2015-05-13T07:42:32.179Z", "execution_phase": "condition" } ] }
A list of all the Watches that are currently being executed by Watcher. In case of an empty array no executing watches had been captured. The captured watches are sorted by execution time in descending order. So the longest running watch is always on top. |
|
The id of the watch being executed. |
|
The id of the watch record. |
|
The time the watch was triggered by the trigger engine. |
|
The time the watch was executed. This is just before the input is being executed. |
|
The current execution phase the watch is in. Can be |
Queued watches metric
editWhen a watch triggers it is being prepared for execution and when there is capacity the watch get executed. If many watches trigger concurrently and there is no capacity to execute then watches are queued up. These watches are then queued for execution. The queued watches metric gives insight what watches are queued for execution.
To include this metric, the metric
option should include queued_watches
or _all
.
The following example specifies the queued_watches
metric option and will include the basic metrics and
the watches queued for execution:
GET _watcher/stats/queued_watches
An example of a successful json response that captures a watch in execution:
{ "watcher_state": "started", "watch_count": 10, "execution_thread_pool": { "queue_size": 1000, "max_size": 20 }, "queued_watches": [ { "watch_id": "slow_condition4", "watch_record_id": "slow_condition4_223-2015-05-21T11:59:59.811Z", "triggered_time": "2015-05-21T11:59:59.811Z", "execution_time": "2015-05-21T11:59:59.811Z" }, ... ] }
Stop API
editThe stop
watcher API stops the watcher service if the service is running, as in the following example:
PUT _watcher/_stop
Watcher returns the following response if the request is successful:
{ "acknowledged": true }
Start API
editThe start
watcher API starts the watcher service if the service is not already running, as in the following example:
PUT _watcher/_start
Watcher returns the following response if the request is successful:
{ "acknowledged": true }
Restart API
editThe restart
watcher API stops, then starts the watcher service, as in the following example:
PUT _watcher/_restart
Watcher returns the following response if the request is successful:
{ "acknowledged": true }