Enable logging and monitoring

The deployment logging and monitoring feature lets you monitor your deployment in Kibana by shipping logs and metrics to a monitoring deployment. You can:

Monitoring consists of two components:

The steps in this section cover only the enablement of the monitoring and logging features in Elasticsearch Add-On for Heroku. For more information on how to use the monitoring features, refer to Monitor a cluster.

Some limitations apply when you use monitoring on Elasticsearch Add-On for Heroku. To learn more, check the monitoring restrictions and limitations.

For production use, you should send your deployment logs and metrics to a dedicated monitoring deployment. Monitoring indexes logs and metrics into Elasticsearch and these indexes consume storage, memory, and CPU cycles like any other index. By using a separate monitoring deployment, you avoid affecting your other production deployments and can view the logs and metrics even when a production deployment is unavailable.

How many monitoring deployments you use depends on your requirements:

Logs and metrics that get sent to a dedicated monitoring Elasticsearch deployment may not be cleaned up automatically and might require some additional steps to remove excess data periodically.

When you enable monitoring in Elasticsearch Add-On for Heroku, your monitoring indices are retained for a certain period by default. After the retention period has passed, the monitoring indices are deleted automatically. The retention period is configured in the .monitoring-8-ilm-policy index lifecycle policy. To view or edit the policy open Kibana Stack management > Data > Index Lifecycle Policies.

When you enable self-monitoring in Elasticsearch Add-On for Heroku, your monitoring indices are retained for a certain period by default. After the retention period has passed, the monitoring indices are deleted automatically. Monitoring data is retained for three days by default or as specified by the xpack.monitoring.history.duration user setting.

To retain monitoring indices as is without deleting them automatically, you must disable the cleaner service by adding a disabled local exporter in your cluster settings.

For example

PUT /_cluster/settings
{
    "persistent": {
        "xpack.monitoring.exporters.__no-default-local__": {
            "type": "local",
            "enabled": false
        }
    }
}

When monitoring for production use, where you configure your deployments to send monitoring data to a dedicated monitoring deployment for indexing, this retention period does not apply. Monitoring indices on a dedicated monitoring deployment are retained until you remove them. There are three options open to you:

An ILM policy is pre-configured to manage log retention. The policy can be adjusted according to your requirements.

When sending monitoring data to a deployment, you can configure Index Lifecycle Management (ILM) to manage retention of your monitoring and logging indices. When sending logs to a deployment, an ILM policy is pre-configured to manage log retention and the policy can be customized to your needs.

Elasticsearch Add-On for Heroku manages the installation and configuration of the monitoring agent for you. When you enable monitoring on a deployment, you are configuring where the monitoring agent for your current deployment should send its logs and metrics.

To enable monitoring on your deployment:

With monitoring enabled for your deployment, you can access the logs and stack monitoring through Kibana.

Alternatively, you can access logs and metrics directly on the Kibana Logs and Stack Monitoring pages in the target monitoring deployment. You can also create an elastic-cloud-logs-* data view (formerly index pattern) to view your deployment’s logs in the Kibana Discover tab. Several fields are available for you to view logs based on key details, such as the source deployment:

When shipping logs to a monitoring deployment there are more logging features available to you. These features include:

After you’ve enabled log delivery on your deployment, you can add the Elasticsearch user settings to enable these features.

After you’ve enabled log delivery on your deployment, you can add the Kibana user settings to enable this feature.

Enabling log collection also supports collecting and indexing the following types of logs from other components in your deployments:

Enterprise Search

App Search

APM

Fleet and Elastic Agent

The ˆ*ˆ indicates that we also index the archived files of each type of log.

Check the respective product documentation for more information about the logging capabilities of each product.

With logging and monitoring enabled for a deployment, metrics are collected for Elasticsearch, Kibana, Enterprise Search, and APM with Fleet Server.

Audit logs are useful for tracking security events on your Elasticsearch and/or Kibana clusters. To enable Elasticsearch audit logs on your deployment:

A plan change will run on your deployment. When it finishes, audit logs will be delivered to your monitoring deployment.