WARNING: Version 1.2 of Filebeat has passed its EOL date.
This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.
Elasticsearch Output
editElasticsearch Output
editWhen you specify Elasticsearch for the output, the Beat sends the transactions directly to Elasticsearch by using the Elasticsearch HTTP API.
Example configuration:
output: elasticsearch: # The Elasticsearch cluster hosts: ["http://localhost:9200"] # Comment this option if you don't want to store the topology in # Elasticsearch. The default is false. # This option makes sense only for Packetbeat # save_topology: false # Optional index name. The default is "filebeat" and generates # [filebeat-]YYYY.MM.DD keys. index: "filebeat" # tls configuration. By default is off. tls: # List of root certificates for HTTPS server verifications certificate_authorities: ["/etc/pki/root/ca.pem"] # Certificate for TLS client authentication certificate: "/etc/pki/client/cert.pem" # Client Certificate Key certificatekey: "/etc/pki/client/cert.key"
To enable SSL, just add https
to all URLs defined under hosts.
output: elasticsearch: # The Elasticsearch cluster hosts: ["https://localhost:9200"] # Comment this option if you don't want to store the topology in # Elasticsearch. The default is false. # This option makes sense only for Packetbeat # save_topology: false # HTTP basic auth username: "admin" password: "s3cr3t"
If the Elasticsearch nodes are defined by IP:PORT
, then add protocol: https
to the yaml file.
output: elasticsearch: # The Elasticsearch cluster hosts: ["localhost"] # Optional http or https. Default is http protocol: "https" # Comment this option if you don't want to store the topology in # Elasticsearch. The default is false. # This option makes sense only for Packetbeat # save_topology: false # HTTP basic auth username: "admin" password: "s3cr3t"
Elasticsearch Output Options
editYou can specify the following options in the elasticsearch
section:
hosts
editThe list of Elasticsearch nodes to connect to. The events are distributed to
these nodes in round robin order. If one node becomes unreachable, the event is
automatically sent to another node. Each Elasticsearch node can be defined as a URL
or IP:PORT
.
For example: http://192.15.3.2
, https://es.found.io:9230
or 192.24.3.2:9300
.
If no port is specified, 9200
is used.
output: elasticsearch: # The Elasticsearch cluster hosts: ["10.45.3.2:9220", "10.45.3.1:9230"] # Optional http or https. Default is http protocol: https # HTTP Path at which each Elasticsearch server lives path: /elasticsearch
In the previous example, the Elasticsearch nodes are available at https://10.45.3.2:9220/elasticsearch
and
https://10.45.3.1:9230/elasticsearch
.
worker
editThe number of workers per configured host publishing events to Elasticsearch. This is best used with load balancing mode enabled. Example: If you have 2 hosts and 3 workers, in total 6 workers are started (3 for each host).
host (DEPRECATED)
editThe host of the Elasticsearch server. This option is deprecated because it is replaced by hosts.
port (DEPRECATED)
editThe port of the Elasticsearch server. This option is deprecated because it is replaced by hosts.
username
editThe basic authentication username for connecting to Elasticsearch.
password
editThe basic authentication password for connecting to Elasticsearch.
protocol
editThe name of the protocol Elasticsearch is reachable on. The options are:
http
or https
. The default is http
. However, if you specify a URL for
hosts, the value of protocol
is overridden by whatever scheme you
specify in the URL.
path
editAn HTTP path prefix that is prepended to the HTTP API calls. This is useful for the cases where Elasticsearch listens behind an HTTP reverse proxy that exports the API under a custom prefix.
proxy_url
editThe URL of the proxy to use when connecting to the Elasticsearch servers. The value may be either a complete URL or a "host[:port]", in which case the "http" scheme is assumed. If a value is not specified through the configuration file then proxy environment variables are used. See the golang documentation for more information about the environment variables.
index
editThe index root name to write events to. The default is the Beat name. For example "filebeat" generates "[filebeat-]YYYY.MM.DD" indexes (for example, "filebeat-2015.04.26").
template
editThe index template to use for setting mappings in Elasticsearch. By default, template loading is disabled. If you leave this option disabled, you must load the template by running the load script that is provided with Filebeat.
If you enable the template
config option, you can adjust the following settings to load your
own template or overwrite an existing one:
-
name
-
The name of the template. The default is
filebeat
. -
path
- The path to the template file.
-
overwrite
- A boolean that specifies whether to overwrite the existing template. The default is false.
For example:
output: elasticsearch: hosts: ["localhost:9200"] name: "filebeat" path: "filebeat.template.json" overwrite: false
max_retries
editThe number of times to retry publishing an event after a publishing failure.
After the specified number of retries, the events are typically dropped.
Some Beats, such as Filebeat, ignore the max_retries
setting and retry until all
events are published.
Set max_retries
to a value less than 0 to retry until all events are published.
The default is 3.
bulk_max_size
editThe maximum number of events to bulk in a single Elasticsearch bulk API index request. The default is 50.
If the Beat sends single events, the events are collected into batches. If the Beat publishes
a large batch of events (larger than the value specified by bulk_max_size
), the batch is
split.
Specifying a larger batch size can improve performance by lowering the overhead of sending events. However big batch sizes can also increase processing times, which might result in API errors, killed connections, timed-out publishing requests, and, ultimately, lower throughput.
Setting bulk_max_size
to values less than or equal to 0 disables buffering in libbeat. When buffering is disabled,
Beats that publish single events (such as Packetbeat and Topbeat) send each event directly to
Elasticsearch. Beats that publish data in batches (such as Filebeat) send events in batches based on the
spooler size.
timeout
editThe http request timeout in seconds for the Elasticsearch request. The default is 90.
flush_interval
editThe number of seconds to wait for new events between two bulk API index requests.
If bulk_max_size
is reached before this interval expires, additional bulk index
requests are made.
save_topology
editA Boolean that specifies whether the topology is kept in Elasticsearch. The default is false.
This option is relevant for Packetbeat only.
topology_expire
editThe time to live in seconds for the topology information that is stored in Elasticsearch. The default is 15 seconds.
tls
editConfiguration options for TLS parameters like the certificate authority to use
for HTTPS-based connections. If the tls
section is missing, the host CAs are used for HTTPS connections to
Elasticsearch.
See TLS Options for more information.