Watching the status of an Elasticsearch cluster

You can easily configure a basic watch to monitor the health of your Elasticsearch cluster:

A watch schedule controls how often a watch is triggered. The watch input gets the data that you want to evaluate.

The simplest way to define a schedule is to specify an interval. For example, the following schedule runs every 10 seconds:

PUT _watcher/watch/cluster_health_watch
{
  "trigger" : {
    "schedule" : { "interval" : "10s" } 
  }
}

To get the status of your cluster, you can call the Elasticsearch cluster health API:

response = client.cluster.health(
  pretty: true
)
puts response

GET _cluster/health?pretty

To load the health status into your watch, you simply add an HTTP input that calls the cluster health API:

PUT _watcher/watch/cluster_health_watch
{
  "trigger" : {
    "schedule" : { "interval" : "10s" }
  },
  "input" : {
    "http" : {
      "request" : {
        "host" : "localhost",
        "port" : 9200,
        "path" : "/_cluster/health"
      }
    }
  }
}

If you’re using Security, then you’ll also need to supply some authentication credentials as part of the watch configuration:

PUT _watcher/watch/cluster_health_watch
{
  "trigger" : {
    "schedule" : { "interval" : "10s" }
  },
  "input" : {
    "http" : {
      "request" : {
        "host" : "localhost",
        "port" : 9200,
        "path" : "/_cluster/health",
        "auth": {
          "basic": {
            "username": "elastic",
            "password": "x-pack-test-password"
          }
        }
      }
    }
  }
}

It would be a good idea to create a user with the minimum privileges required for use with such a watch configuration.

Depending on how your cluster is configured, there may be additional settings required before the watch can access your cluster such as keystores, truststores, or certificates. For more information, see Watcher settings.

If you check the watch history, you’ll see that the cluster status is recorded as part of the watch_record each time the watch executes.

For example, the following request retrieves the last ten watch records from the watch history:

response = client.search(
  index: '.watcher-history*',
  body: {
    sort: [
      {
        "result.execution_time": 'desc'
      }
    ]
  }
)
puts response

GET .watcher-history*/_search
{
  "sort" : [
    { "result.execution_time" : "desc" }
  ]
}

A condition evaluates the data you’ve loaded into the watch and determines if any action is required. Since you’ve defined an input that loads the cluster status into the watch, you can define a condition that checks that status.

For example, you could add a condition to check to see if the status is RED.

PUT _watcher/watch/cluster_health_watch
{
  "trigger" : {
    "schedule" : { "interval" : "10s" } 
  },
  "input" : {
    "http" : {
      "request" : {
       "host" : "localhost",
       "port" : 9200,
       "path" : "/_cluster/health"
      }
    }
  },
  "condition" : {
    "compare" : {
      "ctx.payload.status" : { "eq" : "red" }
    }
  }
}

If you check the watch history, you’ll see that the condition result is recorded as part of the watch_record each time the watch executes.

To check to see if the condition was met, you can run the following query.

response = client.search(
  index: '.watcher-history*',
  pretty: true,
  body: {
    query: {
      match: {
        "result.condition.met": true
      }
    }
  }
)
puts response

GET .watcher-history*/_search?pretty
{
  "query" : {
    "match" : { "result.condition.met" : true }
  }
}

Recording watch_records in the watch history is nice, but the real power of Watcher is being able to do something in response to an alert. A watch’s actions define what to do when the watch condition is true—​you can send emails, call third-party webhooks, or write documents to an Elasticsearch index or log when the watch condition is met.

For example, you could add an action to index the cluster status information when the status is RED.

PUT _watcher/watch/cluster_health_watch
{
  "trigger" : {
    "schedule" : { "interval" : "10s" }
  },
  "input" : {
    "http" : {
      "request" : {
       "host" : "localhost",
       "port" : 9200,
       "path" : "/_cluster/health"
      }
    }
  },
  "condition" : {
    "compare" : {
      "ctx.payload.status" : { "eq" : "red" }
    }
  },
  "actions" : {
    "send_email" : {
      "email" : {
        "to" : "username@example.org",
        "subject" : "Cluster Status Warning",
        "body" : "Cluster status is RED"
      }
    }
  }
}

For Watcher to send email, you must configure an email account in your elasticsearch.yml configuration file and restart Elasticsearch. To add an email account, set the xpack.notification.email.account property.

For example, the following snippet configures a single Gmail account named work:

xpack.notification.email.account:
  work:
    profile: gmail
    email_defaults:
      from: <email> 
    smtp:
      auth: true
      starttls.enable: true
      host: smtp.gmail.com
      port: 587
      user: <username> 
      password: <password> 

You can check the watch history or the status_index to see that the action was performed.

response = client.search(
  index: '.watcher-history*',
  pretty: true,
  body: {
    query: {
      match: {
        "result.condition.met": true
      }
    }
  }
)
puts response

GET .watcher-history*/_search?pretty
{
  "query" : {
    "match" : { "result.condition.met" : true }
  }
}

Since the cluster_health_watch is configured to run every 10 seconds, make sure you delete it when you’re done experimenting. Otherwise, you’ll spam yourself indefinitely.

To remove the watch, use the delete watch API:

response = client.watcher.delete_watch(
  id: 'cluster_health_watch'
)
puts response

DELETE _watcher/watch/cluster_health_watch