Cluster Alerts for Elasticsearch Issues: Cluster Alerts in X-Pack Monitoring
Editor's Note (August 3, 2021): This post uses deprecated features. Please reference the map custom regions with reverse geocoding documentation for current instructions.
“Alert me when my Elasticsearch cluster state is red!"
You asked, we heard.
Some of our users have been creating Watches based on the data that X-Pack monitoring collects to get real-time alerts. We then thought, ‘Wouldn’t it be nice if we created some Watches for the common cluster problems and make everyone’s life easier?’
So we did. We worked hard to automatically surface potential issues within your Elastic Stack. We hope you like it!
For this first-class feature in X-Pack monitoring, we leveraged X-Pack alerting via Watcher to periodically query the monitoring data, identify issues and provide alerts for critical issues.
When you click on the Monitoring app, you will see any active Cluster Alerts as part of the overview of your Elastic Stack.
Under Top Cluster Alerts, we see that “Elasticsearch cluster status is yellow” with a link to allocate missing replica shards. Clicking the link takes you to the index listing page, which has information about indices that have unassigned shards. There needs to be more than one node in a cluster so that replica shards could be assigned. To resolve this issue, we need to add a second node to join this single-node cluster. Anything that makes the Elasticsearch health turn green will make this Cluster Alert go away - most of the time, that means adding another node to host the replica shards.
In the 5.4 release, we’re shipping with the following four Watches to get this rolling:
- Yellow or red Elasticsearch cluster state
- Mismatching versions of Elasticsearch nodes
- Mismatching versions of different Kibana instances
- Mismatching versions of Logstash nodes in your cluster
In future releases, we plan to provide alerts for X-Pack license expiration approaching, Elasticsearch shards approaching maximum size limits, CPU, memory, and disk utilization, and the holy grail: nodes joining and leaving the cluster. We know that one size does not fit all, we plan on working on customizable thresholds in the future.
We are also actively working to enable E-mail configuration from Kibana for the built-in Watches. In the future, we will let you set notifications via Slack, HipChat, PagerDuty, Jira and Webhook integrations.
When you're on vacation on a beautiful beach, you can fully enjoy your time and relax knowing that your Elastic Stack issues will be kept under close watch with Cluster Alerts in X-Pack monitoring.
To try out this new feature, get started today with the latest release with a trial license, where you can take a full advantage of all X-Pack features. If you have any questions or requests, please let us know via our X-Pack discuss forum.