Azure eventhub input
editAzure eventhub input
editThis functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.
Users can make use of the azure-eventhub input in order to read messages from an azure eventhub.
The azure-eventhub input implementation is based on the the event processor host (EPH is intended to be run across multiple processes and machines while load balancing message consumers more on this here https://github.com/Azure/azure-event-hubs-go#event-processor-host, https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-event-processor-host).
State such as leases on partitions and checkpoints in the event stream are shared between receivers using an Azure Storage container. For this reason, as a prerequisite to using this input, users will have to create or use an existing storage account.
Example configuration:
filebeat.inputs: - type: azure-eventhub eventhub: "insights-operational-logs" consumer_group: "test" connection_string: "Endpoint=sb://....." storage_account: "azureeph" storage_account_key: "....." storage_account_container: ""
Configuration options
editThe azure-eventhub input supports the following configuration:
eventhub
editThe name of the eventhub users would like to read from.
consumer_group
editOptional, we recommend using a dedicated consumer group for the azure input. Reusing consumer groups among non-related consumers can cause unexpected behavior and possibly lost events.
connection_string
editThe connection string required to communicate with Event Hubs, steps here https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string.
A Blob Storage account is required in order to store/retrieve/update the offset or state of the eventhub messages. This means that after stopping filebeat it can start back up at the spot that it stopped processing messages.
storage_account
editThe name of the storage account.
storage_account_key
editThe storage account key, this key will be used to authorize access to data in your storage account.
storage_account_container
editOptional, the name of the storage account container you would like to store the offset information in.