IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Fleet settings Logstash output settings »
Elastic Docs ›Fleet and Elastic Agent Guide [8.13] ›Centrally manage Elastic Agents in Fleet ›Fleet settings

Elasticsearch output settings

edit

Elasticsearch output settings

edit

Specify these settings to send data over a secure connection to Elasticsearch. In the Fleet Output settings, make sure that Elasticsearch output type is selected.

Hosts

The Elasticsearch URLs where Elastic Agents will send data. By default, Elasticsearch is exposed on the following ports:

9200
Default Elasticsearch port for self-managed clusters
443
Default Elasticsearch port for Elastic Cloud

Examples:

  • https://192.0.2.0:9200
  • https://1d7a52f5eb344de18ea04411fe09e564.fleet.eu-west-1.aws.qa.cld.elstc.co:443
  • https://[2001:db8::1]:9200

Refer to the Fleet Server documentation for default ports and other configuration details.

Elasticsearch CA trusted fingerprint

HEX encoded SHA-256 of a CA certificate. If this certificate is present in the chain during the handshake, it will be added to the certificate_authorities list and the handshake will continue normally. To learn more about trusted fingerprints, refer to the Elasticsearch security documentation.

Proxy

Select a proxy URL for Elastic Agent to connect to Elasticsearch. To learn about proxy configuration, refer to Using a proxy server with Elastic Agent and Fleet.

Advanced YAML configuration

YAML settings that will be added to the Elasticsearch output section of each policy that uses this output. Make sure you specify valid YAML. The UI does not currently provide validation.

See Advanced YAML configuration for descriptions of the available settings.

Make this output the default for agent integrations

When this setting is on, Elastic Agents use this output to send data if no other output is set in the agent policy.

Make this output the default for agent monitoring

When this setting is on, Elastic Agents use this output to send agent monitoring data if no other output is set in the agent policy.

Performance tuning

Choose one of the menu options to tune your Elastic Agent performance when sending data to an Elasticsearch output. You can optimize for throughput, scale, latency, or you can choose a balanced (the default) set of performance specifications. Refer to Performance tuning settings for details about the setting values and their potential impact on performance.

You can also use the Advanced YAML configuration field to set custom values. Note that if you adjust any of the performance settings described in the following Advanced YAML configuration section, the Performance tuning option automatically changes to Custom and cannot be changed.

Performance tuning preset values take precedence over any settings that may be defined separately. If you want to change any setting, you need to use the Custom Performance tuning option and specify the settings in the Advanced YAML configuration field.

For example, if you would like to use the balanced preset values except that you prefer a higher compression level, you can do so as follows:

  1. In Fleet, open the Settings tab.
  2. In the Outputs section, select Add output to create a new output, or select the edit icon to edit an existing output.
  3. In the Add new output or the Edit output flyout, set Performance tuning to Custom.

  4. Refer to the list of performance tuning preset values, and add the settings you prefer into the Advanced YAML configuration field. For the balanced presets, the yaml configuration would be as shown:

    bulk_max_size: 1600
worker: 1
queue.mem.events: 3200
queue.mem.flush.min_events: 1600
queue.mem.flush.timeout: 10s
compression_level: 1
connection_idle_timeout: 3s
  5. Adjust any settings as preferred. For example, you can update the compression_level setting to 4.

When you create an Elastic Agent policy using this output, the output will use the balanced preset options except with the higher compression level, as specified.

Advanced YAML configuration

edit
Setting Description

backoff.init

(string) The number of seconds to wait before trying to reconnect to Elasticsearch after a network error. After waiting backoff.init seconds, Elastic Agent tries to reconnect. If the attempt fails, the backoff timer is increased exponentially up to backoff.max. After a successful connection, the backoff timer is reset.

Default: 1s

backoff.max

(string) The maximum number of seconds to wait before attempting to connect to Elasticsearch after a network error.

Default: 60s

bulk_max_size

(int) The maximum number of events to bulk in a single Elasticsearch bulk API index request.

Events can be collected into batches. Elastic Agent will split batches larger than bulk_max_size into multiple batches.

Specifying a larger batch size can improve performance by lowering the overhead of sending events. However big batch sizes can also increase processing times, which might result in API errors, killed connections, timed-out publishing requests, and, ultimately, lower throughput.

Setting bulk_max_size to values less than or equal to 0 turns off the splitting of batches. When splitting is disabled, the queue decides on the number of events to be contained in a batch.

Default: 1600

compression_level

(int) The gzip compression level. Set this value to 0 to disable compression. The compression level must be in the range of 1 (best speed) to 9 (best compression).

Increasing the compression level reduces network usage but increases CPU usage.

max_retries

(int) The number of times to retry publishing an event after a publishing failure. After the specified number of retries, the events are typically dropped.

Set max_retries to a value less than 0 to retry until all events are published.

Default: 3

queue.mem.events

The number of events the queue can store. This value should be evenly divisible by the smaller of queue.mem.flush.min_events or bulk_max_size to avoid sending partial batches to the output.

Default: 3200 events

queue.mem.flush.min_events

flush.min_events is a legacy parameter, and new configurations should prefer to control batch size with bulk_max_size. As of 8.13, there is never a performance advantage to limiting batch size with flush.min_events instead of bulk_max_size

Default: 1600 events

queue.mem.flush.timeout

(int) The maximum wait time for queue.mem.flush.min_events to be fulfilled. If set to 0s, events are available to the output immediately.

Default: 10s

timeout

(string) The HTTP request timeout in seconds for the Elasticsearch request.

Default: 90s

worker

(int) The number of workers per configured host publishing events. Example: If you have two hosts and three workers, in total six workers are started (three for each host).

Default: 1

Performance tuning settings

edit

Table 1. Performance tuning preset values

Configuration Balanced Optimized for Throughput Optimized for Scale Optimized for Latency

bulk_max_size

1600

1600

1600

50

worker

1

4

1

1

queue.mem.events

3200

12800

3200

4100

queue.mem.flush.min_events

1600

1600

1600

2050

queue.mem.flush.timeout

10

5

20

1

compression_level

1

1

1

1

connection_idle_timeout

3

15

1

60

For descriptions of each setting, refer to Advanced YAML configuration. For the queue.mem.events, queue.mem.flush.min_events and queue.mem.flush.timeout settings, refer to the internal queue configuration settings in the Filebeat documentation.

Balanced represents the new default setting (out of the box behaviour). Relative to Balanced, Optimized for throughput setting will improve EPS by 4 times, Optimized for Scale will perform on par and Optimized for Latency will show a 20% degredation in EPS (Events Per Second). These relative performance numbers were calculated from a performance testbed which operates in a controlled setting ingesting a large log file.

As mentioned, the custom preset allows you to input your own set of parameters for a finer tuning of performance. The following table is a summary of a few data points and how the resulting EPS compares to the Balanced setting mentioned above.

These presets apply only to agents on version 8.12.0 or later.

Table 2. Performance tuning: EPS data

worker bulk_max_size queue.mem_events queue.mem.flush.min_events queue.mem.flush.timeout connection_idle_timeout Relative EPS

1

1600

3200

1600

5

15

1x

1

2048

4096

2048

5

15

1x

1

4096

8192

4096

5

15

1x

2

1600

6400

1600

5

15

2x

2

2048

8192

2048

5

15

2x

2

4096

16384

4096

5

15

2x

4

1600

12800

1600

5

15

3.6x

4

2048

16384

2048

5

15

3.6x

4

4096

32768

4096

5

15

3.6x

8

1600

25600

1600

5

15

5.3x

8

2048

32768

2048

5

15

5.1x

8

4096

65536

4096

5

15

5.2x

16

1600

51200

1600

5

15

5.3x

16

2048

65536

2048

5

15

5.2x

16

4096

131072

4096

5

15

5.3x
« Fleet settings Logstash output settings »