elasticsearch_java

The Elasticsearch action to perform. Valid actions are: index, delete.

Use of this setting REQUIRES you also configure the document_id setting because delete actions all require a document id.

What does each action do?

For more details on actions, check out the Elasticsearch bulk API documentation

The .cer or .pem file to validate the server’s certificate

The name of your cluster if you set it on the Elasticsearch side. Useful for discovery when using node or transport protocols. By default, it looks for a cluster named elasticsearch. Equivalent to the Elasticsearch option cluster.name

The codec used for output data. Output codecs are a convenient method for encoding your data before it leaves the output, without needing a separate filter in your Logstash pipeline.

Enable doc_as_upsert for update mode create a new document with source if document_id doesn’t exists

The document ID for the index. Useful for overwriting existing entries in Elasticsearch with the same ID.

The document type to write events to. Generally you should try to write only similar events to the same type. String expansion %{foo} works here. Unless you set document_type, the event type will be used if it exists otherwise the document type will be assigned the value of logs

This plugin uses the bulk index api for improved indexing performance. To make efficient bulk api calls, we will buffer a certain number of events before flushing that out to Elasticsearch. This setting controls how many events will be buffered before sending a batch of events.

For the node protocol, if you do not specify host, it will attempt to use multicast discovery to connect to Elasticsearch. If multicast is disabled in Elasticsearch, you must include the hostname or IP address of the host(s) to use for Elasticsearch unicast discovery. Remember the node protocol uses the transport address (eg. 9300, not 9200). "127.0.0.1" ["127.0.0.1:9300","127.0.0.2:9300"] When setting hosts for node protocol, it is important to confirm that at least one non-client node is listed in the host list. Also keep in mind that the host parameter when used with the node protocol is for discovery purposes only (not for load balancing). When multiple hosts are specified, it will contact the first host to see if it can use it to discover the cluster. If not, then it will contact the second host in the list and so forth. With the node protocol, Logstash will join the Elasticsearch cluster as a node client (which has a copy of the cluster state) and this node client is the one that will automatically handle the load balancing of requests across data nodes in the cluster. If you are looking for a high availability setup, our recommendation is to use the transport protocol (below), set up multiple client nodes and list the client nodes in the host parameter.

For the transport protocol, it will load balance requests across the hosts specified in the host parameter. Remember the transport protocol uses the transport address (eg. 9300, not 9200). "127.0.0.1" ["127.0.0.1:9300","127.0.0.2:9300"] There is also a sniffing option (see below) that can be used with the transport protocol to instruct it to use the host to sniff for "alive" nodes in the cluster and automatically use it as the hosts list (but will skip the dedicated master nodes). If you do not use the sniffing option, it is important to exclude dedicated master nodes from the host list to prevent Logstash from sending bulk requests to the master nodes. So this parameter should only reference either data or client nodes.

For the http protocol, it will load balance requests across the hosts specified in the host parameter. Remember the http protocol uses the http address (eg. 9200, not 9300). "127.0.0.1" ["127.0.0.1:9200","127.0.0.2:9200"] It is important to exclude dedicated master nodes from the host list to prevent LS from sending bulk requests to the master nodes. So this parameter should only reference either data or client nodes.

The amount of time since last flush before a flush is forced.

This setting helps ensure slow event rates don’t get stuck in Logstash. For example, if your flush_size is 100, and you have received 10 events, and it has been more than idle_flush_time seconds since the last flush, Logstash will flush those 10 events automatically.

This helps keep both fast and slow log streams moving along in near-real-time.

The index to write events to. This can be dynamic using the %{foo} syntax. The default value will partition your indices by day so you can more easily delete old data or only search specific date ranges. Indexes may not contain uppercase characters. For weekly indexes ISO 8601 format is recommended, eg. logstash-%{+xxxx.ww}

The index type to write events to. Generally you should try to write only similar events to the same type. String expansion %{foo} works here.

Deprecated in favor of document_type field.

The keystore used to present a certificate to the server It can be either .jks or .p12

Set the truststore password

Starting in Logstash 1.3 (unless you set option manage_template to false) a default mapping template for Elasticsearch will be applied, if you do not already have one set to match the index pattern defined (default of logstash-%{+YYYY.MM.dd}), minus any variables. For example, in this case the template will be applied to all indices starting with logstash-*

If you have dynamic templating (e.g. creating indices based on field names) then you should set manage_template to false and use the REST API to upload your templates manually.

This setting no longer does anything. It exists to keep config validation from failing. It will be removed in future versions.

Set max retry for each event

The name/address of the host to bind to for Elasticsearch clustering. Equivalent to the Elasticsearch option network.host option. This MUST be set for either protocol to work (node or transport)! The internal Elasticsearch node will bind to this ip. This ip MUST be reachable by all nodes in the Elasticsearch cluster

The node name Elasticsearch will use when joining a cluster.

By default, this is generated internally by the ES client.

The port for Elasticsearch transport to use.

If you do not set this, the following defaults are used: * protocol => transport - port 9300-9305 * protocol => node - port 9300-9305

Choose the protocol used to talk to Elasticsearch.

The node protocol (default) will connect to the cluster as a normal Elasticsearch node (but will not store data). If you use the node protocol, you must permit bidirectional communication on the port 9300 (or whichever port you have configured).

If you do not specify the host parameter, it will use multicast for Elasticsearch discovery. While this may work in a test/dev environment where multicast is enabled in Elasticsearch, we strongly recommend using unicast in Elasticsearch. To connect to an Elasticsearch cluster with multicast disabled, you must include the host parameter (see relevant section above).

The transport protocol will connect to the host you specify and will not show up as a node in the Elasticsearch cluster. This is useful in situations where you cannot permit connections outbound from the Elasticsearch cluster to this Logstash server.

All protocols will use bulk requests when talking to Elasticsearch.

Set max interval between bulk retries

Set retry policy for events that failed to send

A routing override to be applied to all processed events. This can be dynamic using the %{foo} syntax.

Enable cluster sniffing (transport only). Asks host for the list of all cluster nodes and adds them to the hosts list Equivalent to the Elasticsearch option client.transport.sniff

Validate the server’s certificate Disabling this severely compromises security For more information read https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf

You can set the path to your own template here, if you so desire. If not set, the included template will be used.

This configuration option defines how the template is named inside Elasticsearch. Note that if you have used the template management features and subsequently change this, you will need to prune the old template manually, e.g.

curl -XDELETE <http://localhost:9200/_template/OldTemplateName?pretty>

where OldTemplateName is whatever the former setting was.

Overwrite the current template with whatever is configured in the template and template_name directives.

This sets the local port to bind to. Equivalent to the Elasticsrearch option transport.tcp.port

The JKS truststore to validate the server’s certificate Use either :truststore or :cacert

Set the truststore password

Set upsert content for update mode create a new document with this parameter as json string if document_id doesn’t exists

The number of workers to use for this output. Note that this setting may not be useful for all outputs.