Configure the internal queue
editConfigure the internal queue
editMetricbeat uses an internal queue to store events before publishing them. The queue is responsible for buffering and combining events into batches that can be consumed by the outputs. The outputs will use bulk operations to send a batch of events in one transaction.
You can configure the type and behavior of the internal queue by setting
options in the queue
section of the metricbeat.yml
config file. Only one
queue type can be configured.
This sample configuration sets the memory queue to buffer up to 4096 events:
queue.mem: events: 4096
Configure the memory queue
editThe memory queue keeps all events in memory.
The memory queue waits for the output to acknowledge or drop events. If the queue is full, no new events can be inserted into the memory queue. Only after the signal from the output will the queue free up space for more events to be accepted.
The memory queue is controlled by the parameters flush.min_events
and flush.timeout
. If
flush.timeout
is 0s
or flush.min_events
is 0
or 1
then events can be sent by the output as
soon as they are available. If the output supports a bulk_max_size
parameter it controls the
maximum batch size that can be sent.
If flush.min_events
is greater than 1
and flush.timeout
is greater than 0s
, events will only
be sent to the output when the queue contains at least flush.min_events
events or the
flush.timeout
period has expired. In this mode the maximum size batch that that can be sent by the
output is flush.min_events
. If the output supports a bulk_max_size
parameter, values of
bulk_max_size
greater than flush.min_events
have no effect. The value of flush.min_events
should be evenly divisible by bulk_max_size
to avoid sending partial batches to the output.
This sample configuration forwards events to the output if 512 events are available or the oldest available event has been waiting for 5s in the queue:
queue.mem: events: 4096 flush.min_events: 512 flush.timeout: 5s
Configuration options
editYou can specify the following options in the queue.mem
section of the metricbeat.yml
config file:
events
editNumber of events the queue can store. This value should be evenly divisible by flush.min_events
to
avoid sending partial batches to the output.
The default value is 4096 events.
flush.min_events
editMinimum number of events required for publishing. If this value is set to 0 or 1, events are
available to the output immediately. If this value is greater than 1 the output must wait for the
queue to accumulate this minimum number of events or for flush.timeout
to expire before
publishing. When greater than 1
this value also defines the maximum possible batch that can be
sent by the output.
The default value is 2048.
flush.timeout
editMaximum wait time for flush.min_events
to be fulfilled. If set to 0s, events are available to the
output immediately.
The default value is 1s.
Configure the disk queue
editThe disk queue stores pending events on the disk rather than main memory. This allows Beats to queue a larger number of events than is possible with the memory queue, and to save events when a Beat or device is restarted. This increased reliability comes with a performance tradeoff, as every incoming event must be written and read from the device’s disk. However, for setups where the disk is not the main bottleneck, the disk queue gives a simple and relatively low-overhead way to add a layer of robustness to incoming event data.
To enable the disk queue with default settings, specify a maximum size:
queue.disk: max_size: 10GB
The queue will use up to the specified maximum size on disk. It will only use as much space as required. For example, if the queue is only storing 1GB of events, then it will only occupy 1GB on disk no matter how high the maximum is. Queue data is deleted from disk after it has been successfully sent to the output.
Configuration options
editYou can specify the following options in the queue.disk
section of the
metricbeat.yml
config file:
path
editThe path to the directory where the disk queue should store its data files. The directory is created on startup if it doesn’t exist.
The default value is "${path.data}/diskqueue"
.
max_size
(required)
editThe maximum size the queue should use on disk. Events that exceed this maximum will either pause their input or be discarded, depending on the input’s configuration.
A value of 0
means that no maximum size is enforced, and the queue can
grow up to the amount of free space on the disk. This value should be used
with caution, as completely filling a system’s main disk can make it
inoperable. It is best to use this setting only with a dedicated data or
backup partition that will not interfere with Metricbeat or the rest
of the host system.
The default value is 10GB
.
segment_size
editData added to the queue is stored in segment files. Each segment contains some number of events waiting to be sent to the outputs, and is deleted when all its events are sent. By default, segment size is limited to 1/10 of the maximum queue size. Using a smaller size means that the queue will use more data files, but they will be deleted more quickly after use. Using a larger size means some data will take longer to delete, but the queue will use fewer auxiliary files. It is usually fine to leave this value unchanged.
The default value is max_size / 10
.
read_ahead
editThe number of events that should be read from disk into memory while waiting for an output to request them. If you find outputs are slowing down because they can’t read as many events at a time, adjusting this setting upward may help, at the cost of higher memory usage.
The default value is 512
.
write_ahead
editThe number of events the queue should accept and store in memory while waiting for them to be written to disk. If you find the queue’s memory use is too high because events are waiting too long to be written to disk, adjusting this setting downward may help, at the cost of reduced event throughput. On the other hand, if inputs are waiting or discarding events because they are being produced faster than the disk can handle, adjusting this setting upward may help, at the cost of higher memory usage.
The default value is 2048
.
retry_interval
editSome disk errors may block operation of the queue, for example a permission
error writing to the data directory, or a disk full error while writing an
event. In this case, the queue reports the error and retries after pausing
for the time specified in retry_interval
.
The default value is 1s
(one second).
max_retry_interval
editWhen there are multiple consecutive errors writing to the disk, the queue
increases the retry interval by factors of 2 up to a maximum of
max_retry_interval
. Increase this value if you are concerned about logging
too many errors or overloading the host system if the target disk becomes
unavailable for an extended time.
The default value is 30s
(thirty seconds).