Azure Module

The instructions below assume that you have Logstash, Elasticsearch, and Kibana running locally. You can also run Logstash, Elasticsearch, and Kibana on separate hosts.

The Elastic Stack version 6.4 (or later) is required for this module.

The Azure module uses the azure_event_hubs input plugin to consume logs and metrics from your Azure environment. It is installed by default with Logstash 6.4 (or later). Basic understanding of the plugin and options is helpful when you set up the Azure module. See the azure_event_hubs input plugin documentation for more information.

Elastic products are available to download and easy to install.

Azure Monitor should be configured to stream logs to one or more Event Hubs. Logstash will need to access these Event Hubs instances to consume your Azure logs and metrics. See Microsoft Azure resources at the end of this topic for links to Microsoft Azure documentation.

The configuration in the logstash.yml file is shared between Event Hubs. Basic configuration is recommended for most use cases

modules:
  - name: azure
    var.elasticsearch.hosts: localhost:9200
    var.kibana.host: localhost:5601
    var.input.azure_event_hubs.consumer_group: "logstash" 
    var.input.azure_event_hubs.storage_connection: "DefaultEndpointsProtocol=https;AccountName=instance1..." 
    var.input.azure_event_hubs.threads: 9 
    var.input.azure_event_hubs.event_hub_connections:
      - "Endpoint=sb://...EntityPath=insights-operational-logs" 
      - "Endpoint=sb://...EntityPath=insights-metrics-pt1m" 
      - "Endpoint=sb://...EntityPath=insights-logs-blocks"
      - "Endpoint=sb://...EntityPath=insights-logs-databasewaitstatistics"
      - "Endpoint=sb://...EntityPath=insights-logs-errors"
      - "Endpoint=sb://...EntityPath=insights-logs-querystoreruntimestatistics"
      - "Endpoint=sb://...EntityPath=insights-logs-querystorewaitstatistics"
      - "Endpoint=sb://...EntityPath=insights-logs-timeouts"

The basic configuration requires the var.input.azure_event_hubs. prefix before a configuration option. Notice the notation for the threads option.

Advanced configuration in the logstash.yml file supports Event Hub specific options. Advanced configuration is available for more granular tuning of threading and Blob Storage usage across multiple Event Hubs. Advanced configuration is not necessary or recommended for most use cases. Use it only if it is required for your deployment scenario.

You must define the header array with name in the first position. You can define other options in any order. The per Event Hub configuration takes precedence. Any values not defined per Event Hub use the global config value.

In this example threads, consumer_group, and storage_connection will be applied to each of the configured Event Hubs. Note that decorate_events is defined in both the global and per Event Hub configuration. The per Event Hub configuration takes precedence, and the global configuration is effectively ignored when the per Event Hub setting is present.

modules:
  - name: azure
    var.elasticsearch.hosts: localhost:9200
    var.kibana.host: localhost:5601
    var.input.azure_event_hubs.decorate_events: true 
    var.input.azure_event_hubs.threads: 9 
    var.input.azure_event_hubs.consumer_group: "logstash"
    var.input.azure_event_hubs.storage_connection: "DefaultEndpointsProtocol=https;AccountName=instance1..."
    var.input.azure_event_hubs.event_hubs:
      - ["name",                                    "initial_position",  "storage_container",  "decorate_events",  "event_hub_connection"]                                   
      - ["insights-operational-logs",                 "TAIL",              "activity-logs1",    "true",             "Endpoint=sb://...EntityPath=insights-operational-logs"]
      - ["insights-operational-logs",                 "TAIL",              "activity_logs2",  "true",             "Endpoint=sb://...EntityPath=insights-operational-logs"]   
      - ["insights-metrics-pt1m",                     "TAIL",              "dbmetrics",         "true",             "Endpoint=sb://...EntityPath=insights-metrics-pt1m"]
      - ["insights-logs-blocks",                      "TAIL",              "dbblocks",          "true",             "Endpoint=sb://...EntityPath=insights-logs-blocks"]
      - ["insights-logs-databasewaitstatistics",      "TAIL",              "dbwaitstats",       "false",            "Endpoint=sb://...EntityPath=insights-logs-databasewaitstatistics"]
      - ["insights-logs-errors",                      "HEAD",              "dberrors",          "true",             "Endpoint=sb://...EntityPath=insights-logs-errors"
      - ["insights-logs-querystoreruntimestatistics", "TAIL",              "dbstoreruntime",    "true",             "Endpoint=sb://...EntityPath=insights-logs-querystoreruntimestatistics"]
      - ["insights-logs-querystorewaitstatistics",    "TAIL",              "dbstorewaitstats",  "true",             "Endpoint=sb://...EntityPath=insights-logs-querystorewaitstatistics"]
      - ["insights-logs-timeouts",                    "TAIL",              "dbtimeouts",        "true",             "Endpoint=sb://...EntityPath=insights-logs-timeouts"]

The advanced configuration doesn’t require a prefix before a per Event Hub configuration option. Notice the notation for the initial_position option.

An Azure Blob Storage account is an essential part of Azure-to-Logstash configuration. It is required for users who want to scale out multiple Logstash instances to consume from Event Hubs.

A Blob Storage account is a central location that enables multiple instances of Logstash to work together to process events. It records the offset (location) of processed events. On restart, Logstash resumes processing exactly where it left off.

Configuration notes:

Sample Blob Storage connection string:

DefaultEndpointsProtocol=https;AccountName=logstash;AccountKey=ETOPnkd/hDAWidkEpPZDiXffQPku/SZdXhPSLnfqdRTalssdEuPkZwIcouzXjCLb/xPZjzhmHfwRCGo0SBSw==;EndpointSuffix=core.windows.net

Find the connection string to Blob Storage here: Azure Portal-> Blob Storage account -> Access keys.

Here are some guidelines to help you achieve a successful deployment, and avoid data conflicts that can cause lost events.

Run this command from the Logstash directory:

bin/logstash --setup

The --modules azure option starts a Logstash pipeline for ingestion from Azure Event Hubs. The --setup option creates an azure-* index pattern in Elasticsearch and imports Kibana dashboards and visualizations.

Run this command from the Logstash directory:

bin/logstash

Defines the per Event Hubs configuration for the advanced configuration.

The advanced configuration uses event_hub_connection instead of event_hub_connections. The event_hub_connection option is defined per Event Hub.

List of connection strings that identifies the Event Hubs to be read. Connection strings include the EntityPath for the Event Hub.

Interval in seconds to write checkpoints during batch processing. Checkpoints tell Logstash where to resume processing after a restart. Checkpoints are automatically written at the end of each batch, regardless of this setting.

Writing checkpoints too frequently can slow down processing unnecessarily.

Consumer group used to read the Event Hub(s). Create a consumer group specifically for Logstash. Then ensure that all instances of Logstash use that consumer group so that they can work together properly.

Adds metadata about the Event Hub, including Event Hub name, consumer_group, processor_host, partition, offset, sequence, timestamp, and event_size.

When first reading from an Event Hub, start from this position:

If storage_connection is set, the initial_position value is used only the first time Logstash reads from the Event Hub.

Number of seconds to look back to find the initial position for pre-existing events. This option is used only if initial_position is set to look_back. If storage_connection is set, this configuration applies only the first time Logstash reads from the Event Hub.

Maximum number of events retrieved and processed together. A checkpoint is created after each batch. Increasing this value may help with performance, but requires more memory.

Connection string for blob account storage. Blob account storage persists the offsets between restarts, and ensures that multiple instances of Logstash process different partitions. When this value is set, restarts resume where processing left off. When this value is not set, the initial_position value is used on every restart.

We strongly recommend that you define this value for production environments.

Name of the storage container used to persist offsets and allow multiple instances of Logstash to work together.

To avoid overwriting offsets, you can use different storage containers. This is particularly important if you are monitoring two Event Hubs with the same name. You can use the advanced configuration model to configure different storage containers.

Total number of threads used to process events. The value you set here applies to all Event Hubs. Even with advanced configuration, this value is a global setting, and can’t be set per event hub.

The number of threads should be the number of Event Hubs plus one or more. See Best practices for more information.

Common options

The following configuration options are supported by all modules:

Many of the logs contain a properties top level field. This is often where the most interesting data lives. There is not a fixed schema between log types for properties fields coming from different sources.

For example, one log may have properties.type where one log sets this a String type and another sets this an Integer type. To avoid mapping errors, the original properties field is moved to <azure.group>_<azure_category>_properties.<original_key>. For example properties.type may end up as sql_diagnostics_Errors_properties.type or activity_log_Security_properties.type depending on the group/category where the event originated.