IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« Elasticsearch output settings Kafka output settings »

› › ›

Logstash output settings

edit

Logstash output settings

edit

Specify these settings to send data over a secure connection to Logstash. You must also configure a Logstash pipeline that reads encrypted data from Elastic Agents and sends the data to Elasticsearch. Follow the in-product steps to configure the Logstash pipeline.

In the Fleet Output settings, make sure that the Logstash output type is selected.

Before using the Logstash output, you need to make sure that for any integrations that have been added to your Elastic Agent policy, the integration assets have been installed on the destination cluster. Refer to Install and uninstall Elastic Agent integration assets for the steps to add integration assets.

To learn how to generate certificates, refer to Configure SSL/TLS for the Logstash output.

To receive the events in Logstash, you also need to create a Logstash configuration pipeline. The Logstash configuration pipeline listens for incoming Elastic Agent connections, processes received events, and then sends the events to Elasticsearch.

The following example configures a Logstash pipeline that listens on port 5044 for incoming Elastic Agent connections and routes received events to Elasticsearch.

The Logstash pipeline definition below is an example. Please refer to the Additional Logstash configuration required steps when creating the Logstash output in the Fleet outputs page.

input {
  elastic_agent {
    port => 5044
    enrich => none # don't modify the events' schema at all
    ssl => true
    ssl_certificate_authorities => ["<ca_path>"]
    ssl_certificate => "<server_cert_path>"
    ssl_key => "<server_cert_key_in_pkcs8>"
    ssl_verify_mode => "force_peer"
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9200"] 
    # cloud_id => "..."
    data_stream => "true"
    api_key => "<api_key>" 
    data_stream => true
    ssl => true
    # cacert => "<elasticsearch_ca_path>"
  }
}

	The Elasticsearch server and the port (`9200`) where Elasticsearch is running.
	The API Key obtained from the Logstash output creation steps in Fleet.

Logstash hosts	The addresses your Elastic Agents will use to connect to Logstash. Use the format `host:port`. Click add row to specify additional Logstash addresses. Examples: `192.0.2.0:5044` `mylogstashhost:5044` Refer to the Fleet Server documentation for default ports and other configuration details.
Server SSL certificate authorities	The CA certificate to use to connect to Logstash. This is the CA used to generate the certificate and key for Logstash. Copy and paste in the full contents for the CA certificate. This setting is optional.
Client SSL certificate	The certificate generated for the client. Copy and paste in the full contents of the certificate. This is the certificate that all the agents will use to connect to Logstash. In cases where each client has a unique certificate, the local path to that certificate can be placed here. The agents will pick the certificate in that location when establishing a connection to Logstash.
Client SSL certificate key	The private key generated for the client. This must be in PKCS 8 key. Copy and paste in the full contents of the certificate key. This is the certificate key that all the agents will use to connect to Logstash. In cases where each client has a unique certificate key, the local path to that certificate key can be placed here. The agents will pick the certificate key in that location when establishing a connection to Logstash. To prevent unauthorized access the certificate key is stored as a secret value. While secret storage is recommended, you can choose to override this setting and store the key as plain text in the agent policy definition. Secret storage requires Fleet Server version 8.12 or higher. Note that this setting can also be stored as a secret value or as plain text for preconfigured outputs. See Preconfiguration settings in the Kibana Guide to learn more.
Proxy	Select a proxy URL for Elastic Agent to connect to Logstash. To learn about proxy configuration, refer to Using a proxy server with Elastic Agent and Fleet.
Advanced YAML configuration	YAML settings that will be added to the Logstash output section of each policy that uses this output. Make sure you specify valid YAML. The UI does not currently provide validation. See Advanced YAML configuration for descriptions of the available settings.
Make this output the default for agent integrations	When this setting is on, Elastic Agents use this output to send data if no other output is set in the agent policy. Output to Logstash is not supported for agent integrations in a policy used by Fleet Server or APM.
Make this output the default for agent monitoring	When this setting is on, Elastic Agents use this output to send agent monitoring data if no other output is set in the agent policy. Output to Logstash is not supported for agent monitoring in a policy used by Fleet Server or APM.

Advanced YAML configuration

edit

Setting	Description
`backoff.init`	(string) The number of seconds to wait before trying to reconnect to Logstash after a network error. After waiting `backoff.init` seconds, Elastic Agent tries to reconnect. If the attempt fails, the backoff timer is increased exponentially up to `backoff.max`. After a successful connection, the backoff timer is reset. Default: `1s`
`backoff.max`	(string) The maximum number of seconds to wait before attempting to connect to Elasticsearch after a network error. Default: `60s`
`bulk_max_size`	(int) The maximum number of events to bulk in a single Logstash request. Events can be collected into batches. Elastic Agent will split batches larger than `bulk_max_size` into multiple batches. Specifying a larger batch size can improve performance by lowering the overhead of sending events. However big batch sizes can also increase processing times, which might result in API errors, killed connections, timed-out publishing requests, and, ultimately, lower throughput. Set this value to `0` to turn off the splitting of batches. When splitting is turned off, the queue determines the number of events to be contained in a batch. Default: `2048`
`compression_level`	(int) The gzip compression level. Set this value to `0` to disable compression. The compression level must be in the range of `1` (best speed) to `9` (best compression). Increasing the compression level reduces network usage but increases CPU usage.
`escape_html`	(boolean) Configures escaping of HTML in strings. Set to `true` to enable escaping. Default: `false`
`index`	(string) The index root name to write events to.
`loadbalance`	If `true` and multiple Logstash hosts are configured, the output plugin load balances published events onto all Logstash hosts. If `false`, the output plugin sends all events to one host (determined at random) and switches to another host if the selected one becomes unresponsive. With `loadbalance` enabled: Elastic Agent reads batches of events and sends each batch to one Logstash worker dynamically, based on a work-queue shared between the outputs. If a connection drops, Elastic Agent takes the disconnected Logstash worker out of its pool. Elastic Agent tries to reconnect. If it succeeds, it re-adds the Logstash worker to the pool. If one of the Logstash nodes is slow but "healthy", it sends a keep-alive signal until the full batch of data is processed. This prevents Elastic Agent from sending further data until it receives an acknowledgement signal back from Logstash. Elastic Agent keeps all events in memory until after that acknowledgement occurs. Without `loadbalance` enabled: Elastic Agent picks a random Logstash host and sends batches of events to it. Due to the random algorithm, the load on the Logstash nodes should be roughly equal. In case of any errors, Elastic Agent picks another Logstash node, also at random. If a connection to a host fails, the host is retried only if there are errors on the new connection. Default: `false` Example: outputs: default: type: logstash hosts: ["localhost:5044", "localhost:5045"] loadbalance: true
`max_retries`	(int) The number of times to retry publishing an event after a publishing failure. After the specified number of retries, the events are typically dropped. Set `max_retries` to a value less than 0 to retry until all events are published. Default: `3`
`pipelining`	(int) The number of batches to send asynchronously to Logstash while waiting for an ACK from Logstash. The output becomes blocking after the specified number of batches are written. Specify `0` to turn off pipelining. Default: `2`
`proxy_use_` `local_resolver`	(boolean) Determines whether Logstash hostnames are resolved locally when using a proxy. If `false` and a proxy is used, name resolution occurs on the proxy server. Default: `false`
`queue.mem.events`	The number of events the queue can store. This value should be evenly divisible by the smaller of `queue.mem.flush.min_events` or `bulk_max_size` to avoid sending partial batches to the output. Default: `3200 events`
`queue.mem.flush.min_events`	`flush.min_events` is a legacy parameter, and new configurations should prefer to control batch size with `bulk_max_size`. As of 8.13, there is never a performance advantage to limiting batch size with `flush.min_events` instead of `bulk_max_size` Default: `1600 events`
`queue.mem.flush.timeout`	(int) The maximum wait time for `queue.mem.flush.min_events` to be fulfilled. If set to 0s, events are available to the output immediately. Default: `10s`
`slow_start`	(boolean) If `true`, only a subset of events in a batch of events is transferred per transaction. The number of events to be sent increases up to `bulk_max_size` if no error is encountered. On error, the number of events per transaction is reduced again. Default: `false`
`timeout`	(string) The number of seconds to wait for responses from the Logstash server before timing out. Default: `30s`
`ttl`	(string) Time to live for a connection to Logstash after which the connection will be reestablished. This setting is useful when Logstash hosts represent load balancers. Because connections to Logstash hosts are sticky, operating behind load balancers can lead to uneven load distribution across instances. Specify a TTL on the connection to achieve equal connection distribution across instances. Default: `0` (turns off the feature) The `ttl` option is not yet supported on an asynchronous Logstash client (one with the `pipelining` option set).
`worker`	(int) The number of workers per configured host publishing events. Example: If you have two hosts and three workers, in total six workers are started (three for each host). Default: `1`

« Elasticsearch output settings Kafka output settings »