WARNING: Version 5.4 of Filebeat has passed its EOL date.

This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.

« Specifying Multiple Prospectors YAML Tips and Gotchas »

› ›

Load Balancing

edit

Load Balancing

edit

Filebeat provides configuration options that you can use to fine tune load balancing when sending events to multiple hosts.

To enable load balancing, you specify loadbalance: true when you configure the output. For example:

output.logstash:
  hosts: ["localhost:5044", "localhost:5045"]
  loadbalance: true

The loadbalance option is available for Redis, Logstash, and Elasticsearch outputs. The Kafka output handles load balancing internally.

Filebeat can send events in a few different modes:

Send events to one host after another:

By default, when you configure Filebeat to send events to multiple hosts (loadbalance: true), Filebeat will send the events to one host after another. This mode requires the least memory and CPU usage. This is not true load balancing in the sense that Filebeat doesn’t account for the load being processed by each host.

The load balancer also supports multiple workers per host. The default is worker: 1. If you increase the number of workers, additional network connections will be used. The total number of workers participating in load balancing is number of hosts * workers.

Example:
```
filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/*.log
output.logstash:
  hosts: ["localhost:5044", "localhost:5045"]
  loadbalance: true
  worker: 2
```
In this example, there are 4 workers participating in load balancing.
Send events to N hosts in lock-step:

You can configure Filebeat to send events to N hosts in lock-step by setting spool_size = N * bulk_max_size. In lock-step mode, the batch collected by the spooler is split up into smaller batches of size bulk_max_size. These smaller batches are load balanced between available connections. Filebeat waits for all sub-batches to be published before it retrieves another batch from the spooler.

This mode requires more memory and CPU usage than the previous mode.

Example:
```
filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/*.log
filebeat.spool_size: 4096
output.logstash:
  hosts: ["localhost:5044", "localhost:5045"]
  loadbalance: true
  bulk_max_size: 2048
```

« Specifying Multiple Prospectors YAML Tips and Gotchas »