Logstash output

edit

The Logstash output is currently only supported for Elastic Agents in standalone mode. Fleet-managed agents are not supported.

The Logstash output uses an internal protocol to send events directly to Logstash over TCP. Logstash provides additional parsing, transformation, and routing of data collected by Elastic Agent.

Compatibility: This output works with all compatible versions of Logstash. Refer to the Elastic Support Matrix.

This example configures a Logstash output called default in the elastic-agent.yml file:

outputs:
  default:
    type: logstash
    hosts: ["127.0.0.1:5044"] 

The Logstash server and the port (5044) where Logstash is configured to listen for incoming Elastic Agent connections.

To receive the events in Logstash, you also need to create a Logstash configuration pipeline. The Logstash configuration pipeline listens for incoming Elastic Agent connections, processes received events, and then sends the events to Elasticsearch.

The following example configures a Logstash pipeline that listens on port 5044 for incoming Elastic Agent connections and routes received events to Elasticsearch:

input {
  elastic_agent {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"] 
    data_stream => "true"
  }
}

The Elasticsearch server and the port (9200) where Elasticsearch is running.

For more information about configuring Logstash, refer to Configuring Logstash and Elastic Agent input plugin.

Logstash output configuration settings

edit

The logstash output supports the following settings, grouped by category. Many of these settings have sensible defaults that allow you to run Elastic Agent with minimal configuration.

Commonly used settings

edit
Setting Description

enabled

(boolean) Enables or disables the output. If set to false, the output is disabled.

escape_html

(boolean) Configures escaping of HTML in strings. Set to true to enable escaping.

Default: false

hosts

(list) The list of known Logstash servers to connect to. If load balancing is disabled, but multiple hosts are configured, one host is selected randomly (there is no precedence). If one host becomes unreachable, another one is selected randomly.

All entries in this list can contain a port number. If no port is specified, 5044 is used.

proxy_url

(string) The URL of the SOCKS5 proxy to use when connecting to the Logstash servers. The value must be a URL with a scheme of socks5://. The protocol used to communicate to Logstash is not based on HTTP, so you cannot use a web proxy.

If the SOCKS5 proxy server requires client authentication, embed a username and password in the URL as shown in the example.

When using a proxy, hostnames are resolved on the proxy server instead of on the client. To change this behavior, set proxy_use_local_resolver.

outputs:
  default:
    type: logstash
    hosts: ["remote-host:5044"]
    proxy_url: socks5://user:password@socks5-proxy:2233

proxy_use_ local_resolver

(boolean) Determines whether Logstash hostnames are resolved locally when using a proxy. If false and a proxy is used, name resolution occurs on the proxy server.

Default: false

Authentication settings

edit

Settings for authenticating with Logstash.

To use SSL, you must also configure the Elastic Agent input plugin for Logstash to use SSL/TLS.

There are a number of SSL configuration settings available depending on whether you are configuring the client, server, or both. See the following tables for available settings:

For more information about using certificates, refer to Encrypt traffic in a self-managed cluster.

Table 4. Common configuration options

Setting Description

ssl.ca_sha256

(string) This configures a certificate pin that you can use to ensure that a specific certificate is part of the verified chain.

The pin is a base64 encoded string of the SHA-256 of the certificate.

This check is not a replacement for the normal SSL validation, but it adds additional validation. If this setting is used with verification_mode set to none, the check will always fail because it will not receive any verified chains.

ssl.cipher_suites

(list) The list of cipher suites to use. The first entry has the highest priority. If this option is omitted, the Go crypto library’s default suites are used (recommended). Note that TLS 1.3 cipher suites are not individually configurable in Go, so they are not included in this list.

The following cipher suites are available:

  • ECDHE-ECDSA-AES-128-CBC-SHA
  • ECDHE-ECDSA-AES-128-CBC-SHA256: TLS 1.2 only. Disabled by default.
  • ECDHE-ECDSA-AES-128-GCM-SHA256: TLS 1.2 only.
  • ECDHE-ECDSA-AES-256-CBC-SHA
  • ECDHE-ECDSA-AES-256-GCM-SHA384: TLS 1.2 only.
  • ECDHE-ECDSA-CHACHA20-POLY1305: TLS 1.2 only.
  • ECDHE-ECDSA-RC4-128-SHA: Disabled by default. RC4 not recommended.
  • ECDHE-RSA-3DES-CBC3-SHA
  • ECDHE-RSA-AES-128-CBC-SHA
  • ECDHE-RSA-AES-128-CBC-SHA256: TLS 1.2 only. Disabled by default.
  • ECDHE-RSA-AES-128-GCM-SHA256: TLS 1.2 only.
  • ECDHE-RSA-AES-256-CBC-SHA
  • ECDHE-RSA-AES-256-GCM-SHA384: TLS 1.2 only.
  • ECDHE-RSA-CHACHA20-POLY1205: TLS 1.2 only.
  • ECDHE-RSA-RC4-128-SHA: Disabled by default. RC4 not recommended.
  • RSA-3DES-CBC3-SHA
  • RSA-AES-128-CBC-SHA
  • RSA-AES-128-CBC-SHA256: TLS 1.2 only. Disabled by default.
  • RSA-AES-128-GCM-SHA256: TLS 1.2 only.
  • RSA-AES-256-CBC-SHA
  • RSA-AES-256-GCM-SHA384: TLS 1.2 only.
  • RSA-RC4-128-SHA: Disabled by default. RC4 not recommended.

Here is a list of acronyms used in defining the cipher suites:

  • 3DES: Cipher suites using triple DES
  • AES-128/256: Cipher suites using AES with 128/256-bit keys.
  • CBC: Cipher using Cipher Block Chaining as block cipher mode.
  • ECDHE: Cipher suites using Elliptic Curve Diffie-Hellman (DH) ephemeral key exchange.
  • ECDSA: Cipher suites using Elliptic Curve Digital Signature Algorithm for authentication.
  • GCM: Galois/Counter mode is used for symmetric key cryptography.
  • RC4: Cipher suites using RC4.
  • RSA: Cipher suites using RSA.
  • SHA, SHA256, SHA384: Cipher suites using SHA-1, SHA-256 or SHA-384.

ssl.curve_types

(list) The list of curve types for ECDHE (Elliptic Curve Diffie-Hellman ephemeral key exchange).

The following elliptic curve types are available:

  • P-256
  • P-384
  • P-521
  • X25519

ssl.enabled

(boolean) Enables or disables the SSL configuration.

Default: true

SSL settings are disabled if either enabled is set to false or the ssl section is missing.

ssl.supported_protocols

(list) List of allowed SSL/TLS versions. If the SSL/TLS server supports none of the specified versions, the connection will be dropped during or after the handshake. The list of allowed protocol versions include: SSLv3, TLSv1 for TLS version 1.0, TLSv1.0, TLSv1.1, TLSv1.2, and TLSv1.3.

Default: [TLSv1.1, TLSv1.2, TLSv1.3]

Table 5. Client configuration options

Setting Description

ssl.certificate

(string) The path to the certificate for SSL client authentication. This setting is only required if client_authentication is specified. If certificate is not specified, client authentication is not available, and the connection might fail if the server requests client authentication. If the SSL server does not require client authentication, the certificate will be loaded, but not requested or used by the server.

Example:

ssl.certificate: "/path/to/cert.pem"

When this setting is configured, the ssl.key setting is also required.

Specify a path, or embed a certificate directly in the YAML configuration:

ssl.certificate: |
    -----BEGIN CERTIFICATE-----
    CERTIFICATE CONTENT APPEARS HERE
    -----END CERTIFICATE-----

ssl.certificate _authorities

(list) The list of root certificates for verifications (required). If certificate_authorities is empty or not set, the system keystore is used. If certificate_authorities is self-signed, the host system needs to trust that CA cert as well.

Example:

ssl.certificate_authorities: ["/path/to/root/ca.pem"]

Specify a list of files that Elastic Agent will read, or embed a certificate directly in the YAML configuration:

ssl.certificate_authorities:
  - |
    -----BEGIN CERTIFICATE-----
    CERTIFICATE CONTENT APPEARS HERE
    -----END CERTIFICATE-----

ssl.key

(string) The client certificate key used for client authentication. Only required if client_authentication is configured.

Example:

ssl.key: "/path/to/cert.key"

Specify a path, or embed the private key directly in the YAML configuration:

ssl.key: |
    -----BEGIN PRIVATE KEY-----
    KEY CONTENT APPEARS HERE
    -----END PRIVATE KEY-----

ssl.key_passphrase

(string) The passphrase used to decrypt an encrypted key stored in the configured key file.

ssl.verification _mode

(string) Controls the verification of server certificates. Valid values are:

full
Verifies that the provided certificate is signed by a trusted authority (CA) and also verifies that the server’s hostname (or IP address) matches the names identified within the certificate.
strict
Verifies that the provided certificate is signed by a trusted authority (CA) and also verifies that the server’s hostname (or IP address) matches the names identified within the certificate. If the Subject Alternative Name is empty, it returns an error.
certificate
Verifies that the provided certificate is signed by a trusted authority (CA), but does not perform any hostname verification.
none
Performs no verification of the server’s certificate. This mode disables many of the security benefits of SSL/TLS and should only be used after cautious consideration. It is primarily intended as a temporary diagnostic mechanism when attempting to resolve TLS errors; its use in production environments is strongly discouraged.

Default: full

Table 6. Server configuration options

Setting Description

ssl.certificate

(string) The path to the certificate for SSL server authentication. If the certificate is not specified, startup will fail.

Example:

ssl.certificate: "/path/to/server/cert.pem"

When this setting is configured, the key setting is also required.

Specify a path, or embed a certificate directly in the YAML configuration:

ssl.certificate: |
    -----BEGIN CERTIFICATE-----
    CERTIFICATE CONTENT APPEARS HERE
    -----END CERTIFICATE-----

ssl.certificate _authorities

(list) The list of root certificates for client verifications is only required if client_authentication is configured. If certificate_authorities is empty or not set, and client_authentication is configured, the system keystore is used. If certificate_authorities is self-signed, the host system needs to trust that CA cert too.

Example:

ssl.certificate_authorities: ["/path/to/root/ca.pem"]

Specify a list of files that Elastic Agent will read, or embed a certificate directly in the YAML configuration:

ssl.certificate_authorities:
  - |
    -----BEGIN CERTIFICATE-----
    CERTIFICATE CONTENT APPEARS HERE
    -----END CERTIFICATE-----

ssl.client_ authentication

(string) Configures client authentication. The valid options are:

none
Disables client authentication.
optional
When a client certificate is supplied, the server will verify it.
required
Requires clients to provide a valid certificate.

Default: required (if certificate_authorities is set); otherwise, none

ssl.key

(string) The server certificate key used for authentication (required).

Example:

ssl.key: "/path/to/server/cert.key"

Specify a path, or embed the private key directly in the YAML configuration:

ssl.key: |
    -----BEGIN PRIVATE KEY-----
    KEY CONTENT APPEARS HERE
    -----END PRIVATE KEY-----

ssl.key_passphrase

(string) The passphrase used to decrypt an encrypted key stored in the configured key file.

ssl.renegotiation

(string) Configures the type of TLS renegotiation to support. The valid options are:

never
Disables renegotiation.
once
Allows a remote server to request renegotiation once per connection.
freely
Allows a remote server to request renegotiation repeatedly.

Default: never

ssl.verification _mode

(string) Controls the verification of client certificates. Valid values are:

full
Verifies that the provided certificate is signed by a trusted authority (CA) and also verifies that the server’s hostname (or IP address) matches the names identified within the certificate.
strict
Verifies that the provided certificate is signed by a trusted authority (CA) and also verifies that the server’s hostname (or IP address) matches the names identified within the certificate. If the Subject Alternative Name is empty, it returns an error.
certificate
Verifies that the provided certificate is signed by a trusted authority (CA), but does not perform any hostname verification.
none
Performs no verification of the server’s certificate. This mode disables many of the security benefits of SSL/TLS and should only be used after cautious consideration. It is primarily intended as a temporary diagnostic mechanism when attempting to resolve TLS errors; its use in production environments is strongly discouraged.

Default: full

Performance tuning settings

edit

Settings that may affect performance.

Setting Description

backoff.init

(string) The number of seconds to wait before trying to reconnect to Logstash after a network error. After waiting backoff.init seconds, Elastic Agent tries to reconnect. If the attempt fails, the backoff timer is increased exponentially up to backoff.max. After a successful connection, the backoff timer is reset.

Default: 1s

backoff.max

(string) The maximum number of seconds to wait before attempting to connect to Elasticsearch after a network error.

Default: 60s

bulk_max_size

(int) The maximum number of events to bulk in a single Logstash request.

Events can be collected into batches. Elastic Agent will split batches larger than bulk_max_size into multiple batches.

Specifying a larger batch size can improve performance by lowering the overhead of sending events. However big batch sizes can also increase processing times, which might result in API errors, killed connections, timed-out publishing requests, and, ultimately, lower throughput.

Set this value to 0 to turn off the splitting of batches. When splitting is turned off, the queue determines the number of events to be contained in a batch.

Default: 2048

compression_level

(int) The gzip compression level. Set this value to 0 to disable compression. The compression level must be in the range of 1 (best speed) to 9 (best compression).

Increasing the compression level reduces network usage but increases CPU usage.

Default: 3

loadbalance

If true and multiple Logstash hosts are configured, the output plugin load balances published events onto all Logstash hosts. If false, the output plugin sends all events to one host (determined at random) and switches to another host if the selected one becomes unresponsive.

Default: false

Example:

outputs:
  default:
    type: logstash
    hosts: ["localhost:5044", "localhost:5045"]
    loadbalance: true

max_retries

(int) The number of times to retry publishing an event after a publishing failure. After the specified number of retries, the events are typically dropped.

Set max_retries to a value less than 0 to retry until all events are published.

Default: 3

pipelining

(int) The number of batches to send asynchronously to Logstash while waiting for an ACK from Logstash. The output becomes blocking after the specified number of batches are written. Specify 0 to turn off pipelining.

Default: 2

slow_start

(boolean) If true, only a subset of events in a batch of events is transferred per transaction. The number of events to be sent increases up to bulk_max_size if no error is encountered. On error, the number of events per transaction is reduced again.

Default: false

timeout

(string) The number of seconds to wait for responses from the Logstash server before timing out.

Default: 30s

ttl

(string) Time to live for a connection to Logstash after which the connection will be reestablished. This setting is useful when Logstash hosts represent load balancers. Because connections to Logstash hosts are sticky, operating behind load balancers can lead to uneven load distribution across instances. Specify a TTL on the connection to achieve equal connection distribution across instances.

Default: 0 (turns off the feature)

The ttl option is not yet supported on an async Logstash client (one with the pipelining option set).

worker

(int) The number of workers per configured host publishing events to {output-type}. This is best used with load balancing mode enabled. Example: If you have two hosts and three workers, in total six workers are started (three for each host).

Default: 1