Elasticsearch output plugin

You can run Elasticsearch on your own hardware or use our hosted Elasticsearch Service that is available on AWS, GCP, and Azure. Try the Elasticsearch Service for free.

Example: Basic default configuration

output {
    elasticsearch {
        hosts => "hostname"
        data_stream => "true"
    }
}

This example shows the minimal settings for processing data streams. Events with data_stream.*` fields are routed to the appropriate data streams. If the fields are missing, routing defaults to logs-generic-logstash.

Example: Customize data stream name

output {
    elasticsearch {
        hosts => "hostname"
        data_stream => "true"
        data_stream_type => "metrics"
        data_stream_dataset => "foo"
        data_stream_namespace => "bar"
    }
}

The Elasticsearch action to perform. Valid actions are:

For more details on actions, check out the Elasticsearch bulk API documentation.

Authenticate using Elasticsearch API key. Note that this option also requires SSL/TLS, which can be enabled by supplying a cloud_id, a list of HTTPS hosts, or by setting ssl_enabled => true.

Format is id:api_key where id and api_key are as returned by the Elasticsearch Create API key API.

HTTP Path to perform the _bulk requests to * This default bulk path is the concatenation of the value of path parameter and /_bulk?filter_path=errors,items.*.error,items.*.status * The filter_path query parameter is appended to the bulk path to reduce the payload between logstash and elasticsearch. However, if a custom filter_path query parameter is included in the bulk_path setting, then that value will be used.

The SHA-256 fingerprint of an SSL Certificate Authority to trust, such as the autogenerated self-signed CA for an Elasticsearch cluster.

Cloud authentication string ("<username>:<password>" format) is an alternative for the user/password pair.

For more details, check out the Logstash-to-Cloud documentation.

Cloud ID, from the Elastic Cloud web console. If set hosts should not be used.

For more details, check out the Logstash-to-Cloud documentation.

The gzip compression level. Setting this value to 0 disables compression. The compression level must be in the range of 1 (best speed) to 9 (best compression).

Increasing the compression level will reduce the network usage but will increase the CPU usage.

Defines whether data will be indexed into an Elasticsearch data stream. The other data_stream_* settings will be used only if this setting is enabled.

Logstash handles the output as a data stream when the supplied configuration is compatible with data streams and this value is set to auto.

Automatically routes events by deriving the data stream name using specific event fields with the %{[data_stream][type]}-%{[data_stream][dataset]}-%{[data_stream][namespace]} format.

If enabled, the data_stream.* event fields will take precedence over the data_stream_type, data_stream_dataset, and data_stream_namespace settings, but will fall back to them if any of the fields are missing from the event.

The data stream dataset used to construct the data stream at index time.

The data stream namespace used to construct the data stream at index time.

Automatically adds and syncs the data_stream.* event fields if they are missing from the event. This ensures that fields match the name of the data stream that is receiving events.

The data stream type used to construct the data stream at index time. Currently, only logs, metrics, synthetics and traces are supported.

List single-action error codes from Elasticsearch’s Bulk API that are considered valid to move the events into the dead letter queue. This list is an addition to the ordinary error codes considered for this feature, 400 and 404. It’s considered a configuration error to re-use the same predefined codes for success, DLQ or conflict. The option accepts a list of natural numbers corresponding to HTTP errors codes.

If enabled, failed index name interpolation events go into dead letter queue.

Enable doc_as_upsert for update mode. Create a new document with source if document_id doesn’t exist in Elasticsearch.

The document ID for the index. Useful for overwriting existing entries in Elasticsearch with the same ID.

This sets the document type to write events to. Generally you should try to write only similar events to the same type. String expansion %{foo} works here. If you don’t set a value for this option:

Controls this plugin’s compatibility with the Elastic Common Schema (ECS), including the installation of ECS-compatible index templates. The value of this setting affects the default values of:

Pass a set of key value pairs as the headers sent in each request to an elasticsearch node. The headers will be used for any kind of request (_bulk request, template installation, health checks and sniffing). These custom headers will be overidden by settings like compression_level.

HTTP Path where a HEAD request is sent when a backend is marked down the request is sent in the background to see if it has come back again before it is once again eligible to service requests. If you have custom firewall rules you may need to change this

Sets the host(s) of the remote instance. If given an array it will load balance requests across the hosts specified in the hosts parameter. Remember the http protocol uses the http address (eg. 9200, not 9300).

Examples:

`"127.0.0.1"`
`["127.0.0.1:9200","127.0.0.2:9200"]`
`["http://127.0.0.1"]`
`["https://127.0.0.1:9200"]`
`["https://127.0.0.1:9200/mypath"]` (If using a proxy on a subpath)

Exclude dedicated master nodes from the hosts list to prevent Logstash from sending bulk requests to the master nodes. This parameter should reference only data or client nodes in Elasticsearch.

Any special characters present in the URLs here MUST be URL escaped! This means # should be put in as %23 for instance.

Setting true enables gzip compression level 1 on requests.

This setting allows you to reduce this plugin’s outbound network traffic by compressing each bulk request to Elasticsearch.

The default setting of auto will automatically enable Index Lifecycle Management, if the Elasticsearch cluster is running Elasticsearch version 7.0.0 or higher with the ILM feature enabled, and disable it otherwise.

Setting this flag to false will disable the Index Lifecycle Management feature, even if the Elasticsearch cluster supports ILM. Setting this flag to true will enable Index Lifecycle Management feature, if the Elasticsearch cluster supports it. This is required to enable Index Lifecycle Management on a version of Elasticsearch earlier than version 7.0.0.

Pattern used for generating indices managed by Index Lifecycle Management. The value specified in the pattern will be appended to the write alias, and incremented automatically when a new index is created by ILM.

Date Math can be used when specifying an ilm pattern, see Rollover API docs for details.

Modify this setting to use a custom Index Lifecycle Management policy, rather than the default. If this value is not set, the default policy will be automatically installed into Elasticsearch

The rollover alias is the alias where indices managed using Index Lifecycle Management will be written to.

The index to write events to. This can be dynamic using the %{foo} syntax. The default value will partition your indices by day so you can more easily delete old data or only search specific date ranges. Indexes may not contain uppercase characters. For weekly indexes ISO 8601 format is recommended, eg. logstash-%{+xxxx.ww}. Logstash uses Joda formats and the @timestamp field of each event is being used as source for the date.

From Logstash 1.3 onwards, a template is applied to Elasticsearch during Logstash’s startup if one with the name template_name does not already exist. By default, the contents of this template is the default template for logstash-%{+YYYY.MM.dd} which always matches indices based on the pattern logstash-*. Should you require support for other index names, or would like to change the mappings in the template in general, a custom template can be specified by setting template to the path of a template file.

Setting manage_template to false disables this feature. If you require more control over template creation, (e.g. creating indices dynamically based on field names) you should set manage_template to false and use the REST API to apply your templates manually.

Pass a set of key value pairs as the URL query string. This query string is added to every host listed in the hosts configuration. If the hosts list contains urls that already have query strings, the one specified here will be appended.

For child documents, ID of the associated parent. This can be dynamic using the %{foo} syntax.

Password to authenticate to a secure Elasticsearch cluster

HTTP Path at which the Elasticsearch server lives. Use this if you must run Elasticsearch behind a proxy that remaps the root path for the Elasticsearch HTTP API lives. Note that if you use paths as components of URLs in the hosts field you may not also set this field. That will raise an error at startup

Set which ingest pipeline you wish to execute for an event. You can also use event dependent configuration here like pipeline => "%{[@metadata][pipeline]}". The pipeline parameter won’t be set if the value resolves to empty string ("").

While the output tries to reuse connections efficiently we have a maximum. This sets the maximum number of open connections the output will create. Setting this too low may mean frequently closing / opening connections which is bad.

While the output tries to reuse connections efficiently we have a maximum per endpoint. This sets the maximum number of open connections per endpoint the output will create. Setting this too low may mean frequently closing / opening connections which is bad.

Set the address of a forward HTTP proxy. This setting accepts only URI arguments to prevent leaking credentials. An empty string is treated as if proxy was not set. This is useful when using environment variables e.g. proxy => '${LS_PROXY:}'.

How frequently, in seconds, to wait between resurrection attempts. Resurrection is the process by which backend endpoints marked down are checked to see if they have come back to life

Set initial interval in seconds between bulk retries. Doubled on each retry up to retry_max_interval

Set max interval in seconds between bulk retries.

The number of times Elasticsearch should internally retry an update/upserted document.

A routing override to be applied to all processed events. This can be dynamic using the %{foo} syntax.

Set script name for scripted update mode

Example:

    output {
      elasticsearch {
        script => "ctx._source.message = params.event.get('message')"
      }
    }

Set the language of the used script. When using indexed (stored) scripts on Elasticsearch 6.0 and higher, you must set this parameter to "" (empty string).

Define the type of script referenced by "script" variable inline : "script" contains inline script indexed : "script" contains the name of script directly indexed in elasticsearch file : "script" contains the name of script stored in elasticsearch’s config directory

Set variable name passed to script (scripted update)

if enabled, script is in charge of creating non-existent document (scripted update)

Defines the list of Elasticsearch errors that you don’t want to log. A useful example is when you want to skip all 409 errors which are document_already_exists_exception.

    output {
      elasticsearch {
        silence_errors_in_log => ["document_already_exists_exception"]
      }
    }

This setting asks Elasticsearch for the list of all cluster nodes and adds them to the hosts list. For Elasticsearch 5.x and 6.x any nodes with http.enabled (on by default) will be added to the hosts list, excluding master-only nodes.

How long to wait, in seconds, between sniffing attempts

HTTP Path to be used for the sniffing requests the default value is computed by concatenating the path value and "_nodes/http" if sniffing_path is set it will be used as an absolute path do not use full URL here, only paths, e.g. "/sniff/_nodes/http"

SSL certificate to use to authenticate the client. This certificate should be an OpenSSL-style X.509 certificate file.

The .cer or .pem files to validate the server’s certificate.

The list of cipher suites to use, listed by priorities. Supported cipher suites vary depending on the Java and protocol versions.

Enable SSL/TLS secured communication to Elasticsearch cluster. Leaving this unspecified will use whatever scheme is specified in the URLs listed in hosts or extracted from the cloud_id. If no explicit protocol is specified plain HTTP will be used.

OpenSSL-style RSA private key that corresponds to the ssl_certificate.

Set the keystore password

The keystore used to present a certificate to the server. It can be either .jks or .p12

The format of the keystore file. It must be either jks or pkcs12.

List of allowed SSL/TLS versions to use when establishing a connection to the Elasticsearch cluster.

For Java 8 'TLSv1.3' is supported only since 8u262 (AdoptOpenJDK), but requires that you set the LS_JAVA_OPTS="-Djdk.tls.client.protocols=TLSv1.3" system property in Logstash.

Set the truststore password

The truststore to validate the server’s certificate. It can be either .jks or .p12.

The format of the truststore file. It must be either jks or pkcs12.

Defines how to verify the certificates presented by another party in the TLS connection:

full validates that the server certificate has an issue date that’s within the not_before and not_after dates; chains to a trusted Certificate Authority (CA), and has a hostname or IP address that matches the names within the certificate.

none performs no certificate validation.

You can set the path to your own template here, if you so desire. If not set, the included template will be used.

The default setting of auto will use index template API to create index template, if the Elasticsearch cluster is running Elasticsearch version 8.0.0 or higher, and use legacy template API otherwise.

Setting this flag to legacy will use legacy template API to create index template. Setting this flag to composable will use index template API to create index template.

This configuration option defines how the template is named inside Elasticsearch. Note that if you have used the template management features and subsequently change this, you will need to prune the old template manually, e.g.

curl -XDELETE <http://localhost:9200/_template/OldTemplateName?pretty>

where OldTemplateName is whatever the former setting was.

The template_overwrite option will always overwrite the indicated template in Elasticsearch with either the one indicated by template or the included one. This option is set to false by default. If you always want to stay up to date with the template provided by Logstash, this option could be very useful to you. Likewise, if you have your own template file managed by puppet, for example, and you wanted to be able to update it regularly, this option could help there as well.

Please note that if you are using your own customized version of the Logstash template (logstash), setting this to true will make Logstash to overwrite the "logstash" template (i.e. removing all customized settings)

Set the timeout, in seconds, for network operations and requests sent Elasticsearch. If a timeout occurs, the request will be retried.

Set upsert content for update mode. Create a new document with this parameter as json string if document_id doesn’t exists

Username to authenticate to a secure Elasticsearch cluster

How long to wait before checking for a stale connection to determine if a keepalive request is needed. Consider setting this value lower than the default, possibly to 0, if you get connection errors regularly.

This client is based on Apache Commons. Here’s how the Apache Commons documentation describes this option: "Defines period of inactivity in milliseconds after which persistent connections must be re-validated prior to being leased to the consumer. Non-positive value passed to this method disables connection validation. This check helps detect connections that have become stale (half-closed) while kept inactive in the pool."

The version to use for indexing. Use sprintf syntax like %{my_version} to use a field value here. See the versioning support blog for more information.

The version_type to use for indexing. See the versioning support blog and Version types in the Elasticsearch documentation.

The .cer or .pem file to validate the server’s certificate.

The keystore used to present a certificate to the server. It can be either .jks or .p12

Set the keystore password

Enable SSL/TLS secured communication to Elasticsearch cluster. Leaving this unspecified will use whatever scheme is specified in the URLs listed in hosts or extracted from the cloud_id. If no explicit protocol is specified plain HTTP will be used.

Option to validate the server’s certificate. Disabling this severely compromises security. For more information on disabling certificate verification please read https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf

The truststore to validate the server’s certificate. It can be either .jks or .p12. Use either :truststore or :cacert.

Set the truststore password

Disable or enable metric logging for this specific plugin instance. By default we record all the metrics we can, but you can disable metrics collection for a specific plugin.

Add a unique ID to the plugin configuration. If no ID is specified, Logstash will generate one. It is strongly recommended to set this ID in your configuration. This is particularly useful when you have two or more plugins of the same type. For example, if you have 2 elasticsearch outputs. Adding a named ID in this case will help in monitoring Logstash when using the monitoring APIs.

output {
  elasticsearch {
    id => "my_plugin_id"
  }
}