How to set up monitoring

Learn how to configure your deployments for observability, which includes metric and log collection, troubleshooting views, and cluster alerts to automate performance monitoring.

These steps are helpful to set yourself up for success by making monitoring readily available and to automate alerts for the future.

As you manage, monitor, and troubleshoot your deployment, make sure you have an understanding of the shared responsibilities between Elastic and yourself, so you know what you need to do to keep your deployments running smoothly.

You may also consider subscribing to incident notices reported on the Elasticsearch Add-On for Heroku status page.

After you have created a new deployment, you should enable shipping logs and metrics to a monitoring deployment:

Optionally, turn on audit logging to capture security-related events, such as authentication failures, refused connections, and data-access events through the proxy. To turn on audit logging, edit your deployment’s elasticsearch.yml file to add these lines:

xpack.security.audit.enabled: true
# xpack.security.audit.logfile.events.include: _all
# xpack.security.audit.logfile.events.emit_request_body: true

The last two lines are commented out for now but left there as placeholders to easily turn on in the future. These two settings generate large logs, but can be helpful to turn on temporarily when troubleshooting traffic request bodies.

From the monitoring deployment, you can now view your deployment’s health in Kibana using Stack Monitoring:

To learn more about what Elasticsearch monitoring metrics are available, take a look at the different tabs. For example:

Some performance metrics are also available directly in the Elasticsearch Add-On for Heroku console and don’t require looking at your monitoring deployment. If you’re ever in a rush to determine if there is a performance problem, you can get a quick overview by going to the Performance page from your deployment menu:

If you suspect a performance issue, you can use your monitoring deployment to investigate what is going in Kibana:

Navigate to the Discover or Stream pages to check if you’ve misconfigured your SAML authentication setup by filtering for WARN and ERROR level logs and viewing the specific message fields, for example:

You can also use this page to test how problematic proxy traffic requests show up in audit logs. To illustrate, create a spurious test request from the Elasticsearch API console:

You will get this request reported as a new log. Audit logs do not currently report the HTTP response status code, but they do report a correlating event.action column:

You should take advantage of the default Elastic Stack monitoring alerts that are available out-of-the-box. You don’t have to do anything other than enable shipping logs and metrics to have them made available to you (which you did earlier on).

On top of these default alerts that write to indices you can investigate, you might want to add some custom actions, such as a connector for Slack notifications. To set up these notifications, you first configure a Slack connector and then append it to the default alerts and actions. From Kibana:

Now, when your CPU usage alert goes off, you will also get a Slack notification to investigate if your cluster is experiencing a traffic blip or if you need to scale out. (You can automate the latter with deployment autoscaling).

As a managed service, Elastic Cloud is here to help you manage the maintenance and upkeep. As part of your responsibilities, you should monitor deployment health on an ongoing basis. There are two main activities to perform:

When issues come up that you need to troubleshoot, you’ll frequently start with the same queries to determine which rabbit hole to investigate further, such as _cluster/health to determine overall deployment health.

You can run this query and many others from the API consoles available via:

You can also learn more about the queries you should run for your deployment by reading our blog Managing and Troubleshooting Elasticsearch Memory.