Processing Details
editProcessing Details
editUnderstanding how Logstash works and how components interrelate can help you make better decisions when you are setting up or adjusting your Logstash environment. This section is designed to elevate concepts to assist with that level of knowledge.
This is a new section. We’re still working on it.
Event ordering
editBy design and by default, Logstash does not guarantee event order. Reordering can occur in two places:
- Events within a batch can be reordered during filter processing.
- In-flight batches can be reordered when one or more batches are processed faster than others.
When maintaining event order is important, use a single worker and set pipeline.ordered ⇒ true. This approach ensures that batches are computed one-after-the-other, and that events maintain their order within the batch.
pipeline.ordered setting
editThe pipeline.ordered
setting in logstash.yml
gives you more control over event ordering for single worker pipelines.
auto
automatically enables ordering if the pipeline.workers
setting is also
set to 1
. true
will enforce ordering on the pipeline and prevent logstash
from starting if there are multiple workers. false
will disable the processing
required to preserve order. Ordering will not be guaranteed, but you save the
processing cost required to preserve order.
Java pipeline initialization time
editThe Java pipeline initialization time appears in the startup logs at INFO level. Initialization time is the time it takes to compile the pipeline config and instantiate the compiled execution for all workers.
Reserved fields in Logstash events
editSome fields in Logstash events are reserved, or are required to adhere to a certain shape. Using these fields can cause runtime exceptions when the event API or plugins encounter incompatible values.
A key/value map. Ruby-based Plugin API: value is an org.jruby.RubyHash. Java-based Plugin API: value is an org.logstash.ConvertedMap. In serialized form (such as JSON): a key/value map where the keys must be strings and the values are not constrained to a particular type. |
|
|
An object holding representation of a specific moment in time. Ruby-based Plugin API: value is an org.jruby.RubyTime. Java-based Plugin API: value is a java.time.Instant. In serialized form (such as JSON) or when setting with Event#set: an ISO8601-compliant String value is acceptable. |
|
A string, holding an integer value. |
|
An array of distinct strings |