Configure the Kafka output

The enabled config is a boolean setting to enable or disable the output. If set to false, the output is disabled.

The default value is true.

The list of Kafka broker addresses from where to fetch the cluster metadata. The cluster metadata contain the actual Kafka brokers events are published to.

Kafka protocol version that Packetbeat will request when connecting. Defaults to 1.0.0.

Valid values are all kafka releases in between 0.8.2.0 and 2.6.0.

The protocol version controls the Kafka client features available to Packetbeat; it does not prevent Packetbeat from connecting to Kafka versions newer than the protocol version.

See Compatibility for information on supported versions.

The username for connecting to Kafka. If username is configured, the password must be configured as well.

The password for connecting to Kafka.

The SASL mechanism to use when connecting to Kafka. It can be one of:

If sasl.mechanism is not set, PLAIN is used if username and password are provided. Otherwise, SASL authentication is disabled.

To use GSSAPI mechanism to authenticate with Kerberos, you must leave this field empty, and use the kerberos options.

The Kafka topic used for produced events.

You can set the topic dynamically by using a format string to access any event field. For example, this configuration uses a custom field, fields.log_topic, to set the topic for each event:

topic: '%{[fields.log_topic]}'

See the topics setting for other ways to set the topic dynamically.

An array of topic selector rules. Each rule specifies the topic to use for events that match the rule. During publishing, Packetbeat sets the topic for each event based on the first matching rule in the array. Rules can contain conditionals, format string-based fields, and name mappings. If the topics setting is missing or no rule matches, the topic field is used.

Rule settings:

The following example sets the topic based on whether the message field contains the specified string:

output.kafka:
  hosts: ["localhost:9092"]
  topic: "logs-%{[agent.version]}"
  topics:
    - topic: "critical-%{[agent.version]}"
      when.contains:
        message: "CRITICAL"
    - topic: "error-%{[agent.version]}"
      when.contains:
        message: "ERR"

This configuration results in topics named critical-9.0.0-beta1, error-9.0.0-beta1, and logs-9.0.0-beta1.

Optional formatted string specifying the Kafka event key. If configured, the event key can be extracted from the event using a format string.

See the Kafka documentation for the implications of a particular choice of key; by default, the key is chosen by the Kafka cluster.

Kafka output broker event partitioning strategy. Must be one of random, round_robin, or hash. By default the hash partitioner is used.

random.group_events: Sets the number of events to be published to the same partition, before the partitioner selects a new partition by random. The default value is 1 meaning after each event a new partition is picked randomly.

round_robin.group_events: Sets the number of events to be published to the same partition, before the partitioner selects the next partition. The default value is 1 meaning after each event the next partition will be selected.

hash.hash: List of fields used to compute the partitioning hash value from. If no field is configured, the events key value will be used.

hash.random: Randomly distribute events if no hash or key value can be computed.

All partitioners will try to publish events to all partitions by default. If a partition’s leader becomes unreachable for the beat, the output might block. All partitioners support setting reachable_only to overwrite this behavior. If reachable_only is set to true, events will be published to available partitions only.

A header is a key-value pair, and multiple headers can be included with the same key. Only string values are supported. These headers will be included in each produced Kafka message.

output.kafka:
  hosts: ["localhost:9092"]
  topic: "logs-%{[agent.version]}"
  headers:
    - key: "some-key"
      value: "some value"
    - key: "another-key"
      value: "another value"

The configurable ClientID used for logging, debugging, and auditing purposes. The default is "beats".

Output codec configuration. If the codec section is missing, events will be json encoded.

See Change the output codec for more information.

Kafka metadata update settings. The metadata do contain information about brokers, topics, partition, and active leaders to use for publishing.

The number of times to retry publishing an event after a publishing failure. After the specified number of retries, the events are typically dropped.

Set max_retries to a value less than 0 to retry until all events are published.

The default is 3.

The number of seconds to wait before trying to republish to Kafka after a network error. After waiting backoff.init seconds, Packetbeat tries to republish. If the attempt fails, the backoff timer is increased exponentially up to backoff.max. After a successful publish, the backoff timer is reset. The default is 1s.

The maximum number of seconds to wait before attempting to republish to Kafka after a network error. The default is 60s.

The maximum number of events to bulk in a single Kafka request. The default is 2048.

Duration to wait before sending bulk Kafka request. 0 is no delay. The default is 0.

The number of seconds to wait for responses from the Kafka brokers before timing out. The default is 30 (seconds).

The maximum duration a broker will wait for number of required ACKs. The default is 10s.

Per Kafka broker number of messages buffered in output pipeline. The default is 256.

The keep-alive period for an active network connection. If 0s, keep-alives are disabled. The default is 0 seconds.

Sets the output compression codec. Must be one of none, snappy, lz4, gzip and zstd. The default is gzip.

Sets the compression level used by gzip. Setting this value to 0 disables compression. The compression level must be in the range of 1 (best speed) to 9 (best compression).

Increasing the compression level will reduce the network usage but will increase the cpu usage.

The default value is 4.

The maximum permitted size of JSON-encoded messages. Bigger messages will be dropped. The default value is 1000000 (bytes). This value should be equal to or less than the broker’s message.max.bytes.

The ACK reliability level required from broker. 0=no response, 1=wait for local commit, -1=wait for all replicas to commit. The default is 1.

Note: If set to 0, no ACKs are returned by Kafka. Messages might be lost silently on error.

Configuration options for SSL parameters like the root CA for Kafka connections. The Kafka host keystore should be created with the -keyalg RSA argument to ensure it uses a cipher supported by Filebeat’s Kafka library. See SSL for more information.

Configuration options for Kerberos authentication.

See Kerberos for more information.