Alerting and Actions

Alerting allows you to detect complex conditions within different Kibana apps and trigger actions when those conditions are met. Alerting is integrated with APM, Metrics, Security, Uptime, can be centrally managed from the Management UI, and provides a set of built-in actions and alerts (known as stack alerts) for you to use.

Alerts work by running checks on a schedule to detect conditions. When a condition is met, the alert tracks it as an alert instance and responds by triggering one or more actions. Actions typically involve interaction with Kibana services or third party integrations. Connectors allow actions to talk to these services and integrations. This section describes all of these elements and how they operate together.

An alert specifies a background task that runs on the Kibana server to check for specific conditions. It consists of three main parts:

For example, when monitoring a set of servers, an alert might check for average CPU usage > 0.9 on each server for the two minutes (condition), checked every minute (schedule), sending a warning email message via SMTP with subject CPU on {{server}} is high (action).

The following sections each part of the alert is described in more detail.

Under the hood, Kibana alerts detect conditions by running javascript function on the Kibana server, which gives it flexibility to support a wide range of detections, anything from the results of a simple Elasticsearch query to heavy computations involving data from multiple sources or external systems.

These detections are packaged and exposed as alert types. An alert type hides the underlying details of the detection, and exposes a set of parameters to control the details of the conditions to detect.

For example, an index threshold alert type lets you specify the index to query, an aggregation field, and a time window, but the details of the underlying Elasticsearch query are hidden.

See Alert types for the types of alerts provided by Kibana and how they express their conditions.

Alert schedules are defined as an interval between subsequent checks, and can range from a few seconds to months.

Actions are invocations of Kibana services or integrations with third-party systems, that run as background tasks on the Kibana server when alert conditions are met.

When defining actions in an alert, you specify:

The result is a template: all the parameters needed to invoke a service are supplied except for specific values that are only known at the time the alert condition is detected.

In the server monitoring example, the email action type is used, and server is mapped to the body of the email, using the template string CPU on {{server}} is high.

When the alert detects the condition, it creates an alert instance containing the details of the condition, renders the template with these details such as server name, and executes the action on the Kibana server by invoking the email action type.

See Action and connector types for details on the types of actions provided by Kibana.

When checking for a condition, an alert might identify multiple occurrences of the condition. Kibana tracks each of these alert instances separately and takes action per instance.

Using the server monitoring example, each server with average CPU > 0.9 is tracked as an alert instance. This means a separate email is sent for each server that exceeds the threshold.

Since actions are taken per instance, alerts can end up generating a large number of actions. Take the following example where an alert is monitoring three servers every minute for CPU usage > 0.9:

In the above example, three emails are sent for server X123 in the span of 3 minutes for the same condition. Often it’s desirable to suppress frequent re-notification. Operations like muting and re-notification throttling can be applied at the instance level. If we set the alert re-notify interval to 5 minutes, we reduce noise by only getting emails for new servers that exceed the threshold:

Actions often involve connecting with services inside Kibana or integrations with third-party systems. Rather than repeatedly entering connection information and credentials for each action, Kibana simplifies action setup using connectors.

Connectors provide a central place to store connection information for services and integrations. For example if four alerts send email notifications via the same SMTP service, they all reference the same SMTP connector. When the SMTP settings change they are updated once in the connector, instead of having to update four alerts.

An alert consists of conditions, actions, and a schedule. When conditions are met, alert instances are created that render actions and invoke them. To make action setup and update easier, actions refer to connectors that centralize the information used to connect with Kibana services and third-party integrations.

Kibana alerting and Elasticsearch alerting are both used to detect conditions and can trigger actions in response, but they are completely independent alerting systems.

This section will clarify some of the important differences in the function and intent of the two systems.

Functionally, Kibana alerting differs in that:

At a higher level, Kibana alerts allow rich integrations across use cases like APM, Metrics, Security, and Uptime. Pre-packaged alert types simplify setup, hide the details complex domain-specific detections, while providing a consistent interface across Kibana.

If you are using an on-premises Elastic Stack deployment:

If you are using an on-premises Elastic Stack deployment with security:

To access alerting in a space, a user must have access to one of the following features:

See feature privileges for more information on configuring roles that provide access to these features. Also note that a user will need read privileges for the Actions and Connectors feature to attach actions to an alert or to edit an alert that has an action attached to it.

Alerts and connectors are isolated to the Kibana space in which they were created. An alert or connector created in one space will not be visible in another.

Alerts, including all background detection and the actions they generate are authorized using an API key associated with the last user to edit the alert. Upon creating or modifying an alert, an API key is generated for that user, capturing a snapshot of their privileges at that moment in time. The API key is then used to run all background tasks associated with the alert including detection checks and executing actions.

For security reasons you may wish to limit the extent to which Kibana can connect to external services. Action settings allows you to disable certain Action and connector types and whitelist the hostnames that Kibana can connect with.