Watching event data
editWatching event data
editIf you are indexing event data, such as log messages, network traffic, or a web feed, you can create a watch to email notifications when certain events occur. For example, if you index a feed of RSVPs for meetup events happening around the world, you can create a watch that alerts you to interesting events.
To index the meetup data, you can use Logstash to ingest live data from the Meetup.com streaming API, http://stream.meetup.com/2/rsvps
.
To ingest this data with Logstash:
- Download Logstash and unpack the archive file.
-
Create a Logstash configuration file that uses the Logstash standard input and the Logstash standard output and save it in
logstash-{version}
directory aslivestream.conf
: -
To start indexing the meetup data, pipe the RSVP stream into Logstash and specify your
livestream.conf
configuration file.curl http://stream.meetup.com/2/rsvps | bin/logstash -f livestream.conf
Now that you’re indexing the meetup RSVPs, you can set up a watch that lets you know about events you might be interested in. For example, let’s create a watch that runs every hour, looks for events that talk about about Open Source, and sends an email with information about the events.
To set up the watch:
-
Specify how often you want to run the watch by adding a schedule trigger to the watch:
{ "trigger": { "schedule": { "interval": "1h" } },
-
Load data into the watch payload by creating an input that searches the meetup data for events that have Open Source as a topic. You can use aggregations to group the data by city, consolidate references to the same events, and sort the events by date.
"input": { "search": { "request": { "indices": [ "logstash" ], "body": { "size": 0, "query": { "bool": { "filter": [ { "range": { "@timestamp": { "gte": "now-3h" } } }, { "match": { "group.group_topics.topic_name": "Open Source" } } ] } }, "aggs": { "group_by_city": { "terms": { "field": "group.group_city.keyword", "size": 5 }, "aggs": { "group_by_event": { "terms": { "field": "event.event_url.keyword", "size": 5 }, "aggs": { "get_latest": { "terms": { "field": "@timestamp", "size": 1, "order": { "_key": "desc" } }, "aggs": { "group_by_event_name": { "terms": { "field": "event.event_name.keyword" } } } } } } } } } } } } },
logstash
is the default index alias for the Logstash indices containing the meetup data. By default, the Logstash index lifecycle management (ILM) policy rolls this alias to a new index when the index size reaches 50GB or becomes 30 days old. For more information, see ILM defaults in Logstash.Find all of the RSVPs with
Open Source
as a topic.Group the RSVPs by city.
Consolidate multiple RSVPs for the same event.
Sort the events so the latest events are listed first.
Group the events by name.
-
To determine whether or not there are any Open Source events, add a compare condition that checks the watch payload to see if there were any search hits.
"compare" : { "ctx.payload.hits.total" : { "gt" : 0 }}
-
To send an email when Open Source events are found, add an email action:
"actions": { "email_me": { "throttle_period": "10m", "email": { "from": "<from:email address>", "to": "<to:email address>", "subject": "Open Source Events", "body": { "html": "Found events matching Open Source: <ul>{{#ctx.payload.aggregations.group_by_city.buckets}}< li>{{key}} ({{doc_count}})<ul>{{#group_by_event.buckets}} <li><a href=\"{{key}}\">{{get_latest.buckets.0.group_by_event_name.buckets.0.key}}</a> ({{doc_count}})</li>{{/group_by_event.buckets}}</ul></li> {{/ctx.payload.aggregations.group_by_city.buckets}}</ul>" } } } }
To enable Watcher to send emails, you must configure an email account in elasticsearch.yml
. For more information, see Configuring email accounts.
The complete watch looks like this:
PUT _watcher/watch/meetup { "trigger": { "schedule": { "interval": "1h" } }, "input": { "search": { "request": { "indices": [ "logstash" ], "body": { "size": 0, "query": { "bool": { "filter": [ { "range": { "@timestamp": { "gte": "now-3h" } } }, { "match": { "group.group_topics.topic_name": "Open Source" } } ] } }, "aggs": { "group_by_city": { "terms": { "field": "group.group_city.keyword", "size": 5 }, "aggs": { "group_by_event": { "terms": { "field": "event.event_url.keyword", "size": 5 }, "aggs": { "get_latest": { "terms": { "field": "@timestamp", "size": 1, "order": { "_key": "desc" } }, "aggs": { "group_by_event_name": { "terms": { "field": "event.event_name.keyword" } } } } } } } } } } } } }, "condition": { "compare": { "ctx.payload.hits.total": { "gt": 0 } } }, "actions": { "email_me": { "throttle_period": "10m", "email": { "from": "username@example.org", "to": "recipient@example.org", "subject": "Open Source events", "body": { "html": "Found events matching Open Source: <ul>{{#ctx.payload.aggregations.group_by_city.buckets}}<li>{{key}} ({{doc_count}})<ul>{{#group_by_event.buckets}}<li><a href=\"{{key}}\">{{get_latest.buckets.0.group_by_event_name.buckets.0.key}}</a> ({{doc_count}})</li>{{/group_by_event.buckets}}</ul></li>{{/ctx.payload.aggregations.group_by_city.buckets}}</ul>" } } } } }
The email body can include Mustache templates to reference data in the watch payload. By default,it will be sanitized to block dangerous content. |
|
Replace the |
|
Replace the |
Now that you’ve created your watch, you can use the
_execute
API to run it without waiting for the schedule to trigger execution:
POST _watcher/watch/meetup/_execute