elasticsearch
editelasticsearch
editThis plugin is the recommended method of storing logs in Elasticsearch. If you plan on using the Kibana web interface, you’ll want to use this output.
This output only speaks the HTTP protocol. HTTP is the preferred protocol for interacting with Elasticsearch as of Logstash 2.0. We strongly encourage the use of HTTP over the node protocol for a number of reasons. HTTP is only marginally slower, yet far easier to administer and work with. When using the HTTP protocol one may upgrade Elasticsearch versions without having to upgrade Logstash in lock-step. For those still wishing to use the node or transport protocols please see the elasticsearch_java output plugin.
You can learn more about Elasticsearch at https://www.elastic.co/products/elasticsearch
Retry Policy
editThe retry policy has changed significantly in the 2.2.0 release. This plugin uses the Elasticsearch bulk API to optimize its imports into Elasticsearch. These requests may experience either partial or total failures.
The following errors are retried infinitely:
- Network errors (inability to connect)
- 429 (Too many requests) and
- 503 (Service unavailable) errors
409 exceptions are no longer retried. Please set a higher retry_on_conflict
value if you experience 409 exceptions.
It is more performant for Elasticsearch to retry these exceptions than this plugin.
DNS Caching
editThis plugin uses the JVM to lookup DNS entries and is subject to the value of networkaddress.cache.ttl, a global setting for the JVM.
As an example, to set your DNS TTL to 1 second you would set
the LS_JAVA_OPTS
environment variable to -Dnetwordaddress.cache.ttl=1
.
Keep in mind that a connection with keepalive enabled will not reevaluate its DNS value while the keepalive is in effect.
Synopsis
editThis plugin supports the following configuration options:
Required configuration options:
elasticsearch { }
Available configuration options:
Setting | Input type | Required | Default value |
---|---|---|---|
string, one of |
No |
|
|
a valid filesystem path |
No |
||
No |
|
||
No |
|
||
No |
|||
No |
|||
No |
|
||
No |
|
||
No |
|
||
No |
|
||
a valid filesystem path |
No |
||
No |
|||
No |
|
||
No |
|
||
No |
|
||
No |
|||
No |
|
||
<<,>> |
No |
||
No |
|
||
No |
|||
No |
|
||
No |
|
||
string, one of |
No |
|
|
No |
|
||
No |
|
||
No |
|
||
No |
|
||
No |
|||
No |
|
||
a valid filesystem path |
No |
||
No |
|
||
No |
|
||
No |
|||
a valid filesystem path |
No |
||
No |
|||
No |
|
||
No |
|||
No |
|
Details
edit
action
edit-
Value can be any of:
index
,delete
,create
,update
-
Default value is
"index"
The Elasticsearch action to perform. Valid actions are:
- index: indexes a document (an event from Logstash).
- delete: deletes a document by id (An id is required for this action)
- create: indexes a document, fails if a document by that id already exists in the index.
-
update: updates a document by id. Update has a special case where you can upsert — update a
document if not already present. See the
upsert
option -
A sprintf style string to change the action based on the content of the event. The value
%{[foo]}
would use the foo field for the action
For more details on actions, check out the Elasticsearch bulk API documentation
cacert
edit- Value type is path
- There is no default value for this setting.
The .cer or .pem file to validate the server’s certificate
codec
edit- Value type is codec
-
Default value is
"plain"
The codec used for output data. Output codecs are a convenient method for encoding your data before it leaves the output, without needing a separate filter in your Logstash pipeline.
doc_as_upsert
edit- Value type is boolean
-
Default value is
false
Enable doc_as_upsert
for update mode.
Create a new document with source if document_id
doesn’t exist in Elasticsearch
document_id
edit- Value type is string
- There is no default value for this setting.
The document ID for the index. Useful for overwriting existing entries in Elasticsearch with the same ID.
document_type
edit- Value type is string
- There is no default value for this setting.
The document type to write events to. Generally you should try to write only
similar events to the same type. String expansion %{foo}
works here.
Unless you set document_type, the event type will be used if it exists
otherwise the document type will be assigned the value of logs
flush_size
edit- Value type is number
-
Default value is
500
This plugin uses the bulk index API for improved indexing performance. In Logstashes >= 2.2 this setting defines the maximum sized bulk request Logstash will make You you may want to increase this to be in line with your pipeline’s batch size. If you specify a number larger than the batch size of your pipeline it will have no effect, save for the case where a filter increases the size of an inflight batch by outputting events.
In Logstashes ⇐ 2.1 this plugin uses its own internal buffer of events.
This config option sets that size. In these older logstashes this size may
have a significant impact on heap usage, whereas in 2.2+ it will never increase it.
To make efficient bulk API calls, we will buffer a certain number of
events before flushing that out to Elasticsearch. This setting
controls how many events will be buffered before sending a batch
of events. Increasing the flush_size
has an effect on Logstash’s heap size.
Remember to also increase the heap size using LS_HEAP_SIZE
if you are sending big documents
or have increased the flush_size
to a higher value.
hosts
edit- Value type is array
-
Default value is
["127.0.0.1"]
Sets the host(s) of the remote instance. If given an array it will load balance requests across the hosts specified in the hosts
parameter.
Remember the http
protocol uses the http address (eg. 9200, not 9300).
"127.0.0.1"
["127.0.0.1:9200","127.0.0.2:9200"]
["https://127.0.0.1:9200"]
["https://127.0.0.1:9200/mypath"]
(If using a proxy on a subpath)
It is important to exclude dedicated master nodes from the hosts
list
to prevent LS from sending bulk requests to the master nodes. So this parameter should only reference either data or client nodes in Elasticsearch.
idle_flush_time
edit- Value type is number
-
Default value is
1
The amount of time since last flush before a flush is forced.
This setting helps ensure slow event rates don’t get stuck in Logstash.
For example, if your flush_size
is 100, and you have received 10 events,
and it has been more than idle_flush_time
seconds since the last flush,
Logstash will flush those 10 events automatically.
This helps keep both fast and slow log streams moving along in near-real-time.
index
edit- Value type is string
-
Default value is
"logstash-%{+YYYY.MM.dd}"
The index to write events to. This can be dynamic using the %{foo}
syntax.
The default value will partition your indices by day so you can more easily
delete old data or only search specific date ranges.
Indexes may not contain uppercase characters.
For weekly indexes ISO 8601 format is recommended, eg. logstash-%{+xxxx.ww}
keystore
edit- Value type is path
- There is no default value for this setting.
The keystore used to present a certificate to the server. It can be either .jks or .p12
keystore_password
edit- Value type is password
- There is no default value for this setting.
Set the truststore password
manage_template
edit- Value type is boolean
-
Default value is
true
Starting in Logstash 1.3 (unless you set option manage_template
to false)
a default mapping template for Elasticsearch will be applied, if you do not
already have one set to match the index pattern defined (default of
logstash-%{+YYYY.MM.dd}
), minus any variables. For example, in this case
the template will be applied to all indices starting with logstash-*
If you have dynamic templating (e.g. creating indices based on field names)
then you should set manage_template
to false and use the REST API to upload
your templates manually.
max_retries
(DEPRECATED)
edit- Value type is number
-
Default value is
3
DEPRECATED This setting no longer does anything. It will be marked obsolete in a future version.
parent
edit- Value type is string
-
Default value is
nil
For child documents, ID of the associated parent. This can be dynamic using the %{foo}
syntax.
password
edit- Value type is password
- There is no default value for this setting.
Password to authenticate to a secure Elasticsearch cluster
path
edit- Value type is string
-
Default value is
"/"
HTTP Path at which the Elasticsearch server lives. Use this if you must run Elasticsearch behind a proxy that remaps the root path for the Elasticsearch HTTP API lives.
proxy
edit<li> Value type is <<string,string>> * There is no default value for this setting.
Set the address of a forward HTTP proxy.
Can be either a string, such as http://localhost:123
or a hash in the form
of {host: 'proxy.org' port: 80 scheme: 'http'}
.
Note, this is NOT a SOCKS proxy, but a plain HTTP proxy
retry_max_interval
edit- Value type is number
-
Default value is
2
Set max interval between bulk retries.
retry_max_items
(DEPRECATED)
edit- DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
- Value type is number
-
Default value is
500
routing
edit- Value type is string
- There is no default value for this setting.
A routing override to be applied to all processed events.
This can be dynamic using the %{foo}
syntax.
script_type
edit-
Value can be any of:
inline
,indexed
,file
-
Default value is
["inline"]
Define the type of script referenced by "script" variable inline : "script" contains inline script indexed : "script" contains the name of script directly indexed in elasticsearch file : "script" contains the name of script stored in elasticseach’s config directory
script_var_name
edit- Value type is string
-
Default value is
"event"
Set variable name passed to script (scripted update)
scripted_upsert
edit- Value type is boolean
-
Default value is
false
if enabled, script is in charge of creating non-existent document (scripted update)
sniffing
edit- Value type is boolean
-
Default value is
false
This setting asks Elasticsearch for the list of all cluster nodes and adds them to the hosts list.
Note: This will return ALL nodes with HTTP enabled (including master nodes!). If you use
this with master nodes, you probably want to disable HTTP on them by setting
http.enabled
to false in their elasticsearch.yml. You can either use the sniffing
option or
manually enter multiple Elasticsearch hosts using the hosts
parameter.
sniffing_delay
edit- Value type is number
-
Default value is
5
How long to wait, in seconds, between sniffing attempts
ssl
edit- Value type is boolean
- There is no default value for this setting.
Enable SSL/TLS secured communication to Elasticsearch cluster. Leaving this unspecified will use whatever scheme is specified in the URLs listed in hosts. If no explicit protocol is specified plain HTTP will be used. If SSL is explicitly disabled here the plugin will refuse to start if an HTTPS URL is given in hosts
ssl_certificate_verification
edit- Value type is boolean
-
Default value is
true
Option to validate the server’s certificate. Disabling this severely compromises security. For more information on disabling certificate verification please read https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf
template
edit- Value type is path
- There is no default value for this setting.
You can set the path to your own template here, if you so desire. If not set, the included template will be used.
template_name
edit- Value type is string
-
Default value is
"logstash"
This configuration option defines how the template is named inside Elasticsearch. Note that if you have used the template management features and subsequently change this, you will need to prune the old template manually, e.g.
curl -XDELETE <http://localhost:9200/_template/OldTemplateName?pretty>
where OldTemplateName
is whatever the former setting was.
template_overwrite
edit- Value type is boolean
-
Default value is
false
The template_overwrite option will always overwrite the indicated template in Elasticsearch with either the one indicated by template or the included one. This option is set to false by default. If you always want to stay up to date with the template provided by Logstash, this option could be very useful to you. Likewise, if you have your own template file managed by puppet, for example, and you wanted to be able to update it regularly, this option could help there as well.
Please note that if you are using your own customized version of the Logstash template (logstash), setting this to true will make Logstash to overwrite the "logstash" template (i.e. removing all customized settings)
timeout
edit- Value type is number
- There is no default value for this setting.
Set the timeout for network operations and requests sent Elasticsearch. If a timeout occurs, the request will be retried.
truststore
edit- Value type is path
- There is no default value for this setting.
The JKS truststore to validate the server’s certificate.
Use either :truststore
or :cacert
truststore_password
edit- Value type is password
- There is no default value for this setting.
Set the truststore password
upsert
edit- Value type is string
-
Default value is
""
Set upsert content for update mode.
Create a new document with this parameter as json string if document_id
doesn’t exists