Azure Logs Integration (v2 preview)

The Azure Logs integration (v2 preview) collects logs from selected Azure services, such as Microsoft Entra ID (Sign-in, Audit, Identity Protection, and Provisioning logs), Azure Spring Apps, Azure Firewall, Microsoft Graph Activity, and several others.

You can then visualize that data in Kibana, create alerts if something goes wrong, and reference data when troubleshooting an issue.

For example, to detect possible brute force sign-in attacks, you can install the Azure Logs integration to send Azure sign-in logs to Elastic. Then, by setting up a new rule in the Elastic Observability Logs app, you can be alerted when the number of failed sign-in attempts exceeds a certain threshold.

You may also want to plan your Azure capacity better. Send Azure Activity logs to Elastic to track and visualize when your virtual machines fail to start because they exceed the quota limit.

The Azure Logs integration (v2 preview) introduces a new architecture that allows you to forward logs from multiple Azure services to the same event hub.

                                                             ┌─────────────────┐
                                                             │  activity logs  │
                                                          ┌─▶│ <<data stream>> │
                                                          │  └─────────────────┘
                                                          │
┌───────────────┐   ┌─────────────┐  ┌─────────────────┐  │  ┌─────────────────┐
│     logs      │   │   Elastic   │  │ events (router) │  │  │  firewall logs  │
│ <<event hub>> │──▶│    Agent    │─▶│ <<data stream>> │──┼─▶│ <<data stream>> │
└───────────────┘   └─────────────┘  └─────────────────┘  │  └─────────────────┘
                                                          │
                                                          │  ┌─────────────────┐
                                                          │  │   signin logs   │
                                                          └─▶│ <<data stream>> │
                                                             └─────────────────┘

The integration will automatically detect the log category and forward the logs to the appropriate data stream. When the integration v2 preview cannot find a matching data stream for a log category, it forwards the logs to the platform logs data stream.

Under the hood, the v2 preview uses only one azure-eventhub input per event hub. The v2 preview avoids contention and inefficiencies from using multiple inputs with the same event hub, which is typical of the v1 architecture. With the v2 preview, you can still assign the agent policy to multiple Elastic Agents to scale out the logs processing.

The Azure Logs integration (v2 preview) collects logs.

Logs help you keep a record of events that happen on your Azure account. Log data streams collected by the Azure Logs integration include Activity, Platform, Microsoft Entra ID (Sign-in, Audit, Identity Protection, Provisioning), Microsoft Graph Activity, and Spring Apps logs.

You need Elasticsearch to store and search for your data and Kibana to visualize and manage it. You can use our recommended hosted Elasticsearch Service on Elastic Cloud or self-manage the Elastic Stack on your hardware.

Before using the Azure integration, you will need:

Azure diagnostic settings allow you to export metrics and logs from a source service (or resource) to one destination for analysis and long-term storage.

   ┌────────────────────┐      ┌──────────────┐     ┌─────────────────┐
   │ Microsoft Entra ID │      │  Diagnostic  │     │    Event Hub    │
   │    <<source>>      │─────▶│   settings   │────▶│ <<destination>> │
   └────────────────────┘      └──────────────┘     └─────────────────┘

Examples of source services:

The diagnostic settings support several destination types. The Elastic Agent requires diagnostic settings configured with an event hub as the destination.

Azure Event Hubs is a data streaming platform and event ingestion service that can receive and temporarily store millions of events.

Elastic Agent with the Azure Logs integration will consume logs published in the Event Hubs service.

  ┌────────────────┐      ┌────────────┐
  │     adlogs     │      │  Elastic   │
  │ <<Event Hub>>  │─────▶│   Agent    │
  └────────────────┘      └────────────┘

To learn more about Event Hubs, refer to Features and terminology in Azure Event Hubs.

The Storage account is a versatile Azure service that allows you to store data in various storage types, including blobs, file shares, queues, tables, and disks.

The Azure Logs integration requires a Storage Account container to work. The integration uses the Storage Account container for checkpointing; it stores data about the Consumer Group (state, position, or offset) and shares it among the Elastic Agents. Sharing such information allows multiple Elastic Agents assigned to the same agent policy to work together, enabling horizontal scaling of the logs processing when required.

  ┌────────────────┐                     ┌────────────┐
  │     adlogs     │        logs         │  Elastic   │
  │ <<Event Hub>>  │────────────────────▶│   Agent    │
  └────────────────┘                     └────────────┘
                                                │
                       consumer group info      │
  ┌────────────────┐   (state, position, or     │
  │   azurelogs    │         offset)            │
  │ <<container>>  │◀───────────────────────────┘
  └────────────────┘

The Elastic Agent automatically creates one container for each enabled integration. In the container, the Agent will create one blob for each existing partition on the event hub.

For example, if you enable one integration to fetch data from an event hub with four partitions, the Agent will create the following:

The information stored in the blobs is small (usually < 300 bytes per blob) and accessed relatively frequently. Elastic recommends using the Hot storage tier.

You need to keep the Storage Account container as long as you need to run the integration with the Elastic Agent. If you delete a storage account container, the Elastic Agent will stop working and create a new one the next time it starts. By deleting a storage account container, the Elastic Agent will lose track of the last message processed and start processing messages from the beginning of the event hub retention period.

With the Azure Logs integration (v2 preview), you can forward logs from multiple Azure services to the same event hub. The integration will automatically detect the log category and forward the logs to the appropriate data stream.

                                                             ┌─────────────────┐
                                                             │  activity logs  │
                                                          ┌─▶│ <<data stream>> │
                                                          │  └─────────────────┘
                                                          │
┌───────────────┐   ┌─────────────┐  ┌─────────────────┐  │  ┌─────────────────┐
│     logs      │   │   Elastic   │  │ events (router) │  │  │  firewall logs  │
│ <<event hub>> │──▶│    Agent    │─▶│ <<data stream>> │──┼─▶│ <<data stream>> │
└───────────────┘   └─────────────┘  └─────────────────┘  │  └─────────────────┘
                                                          │
                                                          │  ┌─────────────────┐
                                                          │  │   signin logs   │
                                                          └─▶│ <<data stream>> │
                                                             └─────────────────┘

Before adding the integration, you must complete the following tasks.

The event hub receives the logs exported from the Azure service and makes them available for the Elastic Agent to read.

Here’s a high-level overview of the required steps:

For a detailed step-by-step guide, check the quickstart Create an event hub using Azure portal.

Take note of the event hub Name, which you will use later when specifying an eventhub in the integration settings.

In the integration settings, you should use the event hub name (not the Event Hubs namespace name) as the value for the event hub option.

If you are new to Event Hubs, think of the Event Hubs namespace as the cluster and the event hub as the topic. You will typically have one cluster and multiple topics.

If you are familiar with Kafka, here’s a conceptual mapping between the two:

The number of partitions is essential to balance the event hub cost and performance.

Here are a few examples with one or multiple agents, with recommendations on picking the correct number of partitions for your use case.

With a single Agent deployment, increasing the number of partitions on the event hub is the primary driver in scale-up performances. The Agent creates one worker for each partition.

┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐    ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐

│                         │    │                         │

│   ┌─────────────────┐   │    │   ┌─────────────────┐   │
    │   partition 0   │◀───────────│    consumer     │
│   └─────────────────┘   │    │   └─────────────────┘   │
    ┌─────────────────┐            ┌─────────────────┐
│   │   partition 1   │◀──┼────┼───│    consumer     │   │
    └─────────────────┘            └─────────────────┘
│   ┌─────────────────┐   │    │   ┌─────────────────┐   │
    │   partition 2   │◀────────── │    consumer     │
│   └─────────────────┘   │    │   └─────────────────┘   │
    ┌─────────────────┐            ┌─────────────────┐
│   │   partition 3   │◀──┼────┼───│    consumer     │   │
    └─────────────────┘            └─────────────────┘
│                         │    │                         │

│                         │    │                         │

└ Event Hub ─ ─ ─ ─ ─ ─ ─ ┘    └ Agent ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘

With more than one Agent, setting the number of partitions is crucial. The agents share the existing partitions to scale out performance and improve availability.

The number of partitions must be at least the number of agents.

┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐    ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐

│                         │    │   ┌─────────────────┐   │
                            ┌──────│    consumer     │
│   ┌─────────────────┐   │ │  │   └─────────────────┘   │
    │   partition 0   │◀────┘      ┌─────────────────┐
│   └─────────────────┘   │ ┌──┼───│    consumer     │   │
    ┌─────────────────┐     │      └─────────────────┘
│   │   partition 1   │◀──┼─┘  │                         │
    └─────────────────┘         ─Agent─ ─ ─ ─ ─ ─ ─ ─ ─ ─
│   ┌─────────────────┐   │    ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐
    │   partition 2   │◀────┐
│   └─────────────────┘   │ │  │  ┌─────────────────┐    │
    ┌─────────────────┐     └─────│    consumer     │
│   │   partition 3   │◀──┼─┐  │  └─────────────────┘    │
    └─────────────────┘     │     ┌─────────────────┐
│                         │ └──┼──│    consumer     │    │
                                  └─────────────────┘
│                         │    │                         │

└ Event Hub ─ ─ ─ ─ ─ ─ ─ ┘    └ Agent ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘

Create an event hub with at least two partitions. Two partitions allow low-volume deployment to support high availability with two agents. Consider creating four partitions or more to handle medium-volume deployments with availability.

To learn more about event hub partitions, read an in-depth guide from Microsoft at Quickstart: Create an event hub using Azure portal.

To learn more about event hub partition from the performance perspective, check the scalability-focused document at Event Hubs scalability.

With the Azure Logs integration (v2 preview), Elastic strongly recommends creating one event hub and using it for all Azure services.

For example, if you plan to collect Microsoft Entra ID and Azure Firewall logs, create one event hub and use it for both services.

Here’s a high-level diagram of the solution:

┌────────────────┐   ┌───────────────┐
│  MS Entra ID   │   │  Diagnostic   │
│  <<service>>   │──▶│   Settings    │─┐
└────────────────┘   └───────────────┘ │
                                       │  ┌───────────────┐   ┌─────────────┐
                                       │  │     logs      │   │   Elastic   │
                                       ├─▶│ <<event hub>> │──▶│    Agent    │
                                       │  └───────────────┘   └─────────────┘
┌────────────────┐   ┌───────────────┐ │
│ Azure Firewall │   │  Diagnostic   │ │
│  <<service>>   │──▶│   Settings    │─┘
└────────────────┘   └───────────────┘

The Azure Logs integration (v2 preview) will automatically detect the log category and forward the logs to the appropriate data stream.

Like all other event hub clients, Elastic Agent needs a consumer group name to access the event hub.

A Consumer Group is an entire event hub’s view (state, position, or offset). Consumer groups enable multiple agents to have a separate view of the event stream and to read the logs independently at their own pace and with their offsets.

Consumer groups allow multiple Elastic Agents assigned to the same agent policy to work together; this enables horizontal scaling of the logs processing when required.

In most cases, you can use the default consumer group named $Default. If $Default is already used by other applications, you can create a consumer group dedicated to the Azure Logs integration.

The Elastic Agent requires a connection string to access the event hub and fetch the exported logs. The connection string contains details about the event hub used and the credentials required to access it.

To get the connection string for your Event Hubs namespace:

Create a new Shared Access Policy (SAS):

When the SAS Policy is ready, select it to display the information panel.

Take note of the Connection string–primary key, which you will use later when specifying a connection_string in the integration settings.

The diagnostic settings export the logs from Azure services to a destination and in order to use Azure Logs integration, it must be an event hub.

To create a diagnostic settings to export logs:

In the diagnostic settings page you must select the source log categories you want to export and then select their destination.

Each Azure service exports a well-defined list of log categories. Check the individual integration doc to learn which log categories the integration supports.

Select the subscription and the Event Hubs namespace you previously created. Select the event hub dedicated to this integration.

  ┌───────────────┐   ┌──────────────┐   ┌───────────────┐      ┌───────────┐
  │  MS Entra ID  │   │  Diagnostic  │   │     adlogs    │      │  Elastic  │
  │  <<service>>  ├──▶│   Settings   │──▶│ <<Event Hub>> │─────▶│   Agent   │
  └───────────────┘   └──────────────┘   └───────────────┘      └───────────┘

The Elastic Agent stores the event hub checkpoint information in a storage account container. Storing checkpoint information in a container allows agents to share message processing and resume from the last processed message after a restart.

To create the storage account:

This is the final diagram of the setup for collecting Activity logs from the Azure Monitor service.

 ┌───────────────┐   ┌──────────────┐   ┌────────────────┐         ┌───────────┐
 │  MS Entra ID  │   │  Diagnostic  │   │     adlogs     │  logs   │  Elastic  │
 │  <<service>>  ├──▶│   Settings   │──▶│ <<Event Hub>>  │────────▶│   Agent   │
 └───────────────┘   └──────────────┘   └────────────────┘         └───────────┘
                                                                          │
                     ┌──────────────┐          consumer group info        │
                     │  azurelogs   │          (state, position, or       │
                     │<<container>> │◀───────────────offset)──────────────┘
                     └──────────────┘

The Elastic Agent can use a single Storage Account to store the checkpoint information for multiple integrations.

CRITICAL: make sure to use a different storage_account_container for each integration. The Elastic Agent uses the integration name and the event hub name to uniquely identify the container that holds the blobs with the checkpoint information.

┌─────────────────────────────────┐      ┌──────────────────────────────────────────┐
│                                 │      │                                          │
│    ┌─────────────────────┐      │      │  ┌────────────────────────────────────┐  │
│    │   azure-eventhub    │      │      │  │  filebeat-activitylogs-eventhub-1  │  │
│    │      <<input>>      │──────┼──────┼─▶│           <<container>>            │  │
│    └─────────────────────┘      │      │  └────────────────────────────────────┘  │
│    ┌─────────────────────┐      │      │  ┌────────────────────────────────────┐  │
│    │   azure-eventhub    │      │      │  │   filebeat-signinlogs-eventhub-2   │  │
│    │      <<input>>      │──────┼──────┼─▶│           <<container>>            │  │
│    └─────────────────────┘      │      │  └────────────────────────────────────┘  │
│    ┌─────────────────────┐      │      │  ┌────────────────────────────────────┐  │
│    │   azure-eventhub    │      │      │  │   filebeat-auditlogs-eventhub-3    │  │
│    │      <<input>>      │──────┼──────┼─▶│           <<container>>            │  │
│    └─────────────────────┘      │      │  └────────────────────────────────────┘  │
│                                 │      │                                          │
└─Elastic Agent───────────────────┘      └─Storage Account──────────────────────────┘

When you run the Elastic Agent behind a firewall, you must allow traffic on ports 5671 and 5672 for the event hub and port 443 for the Storage Account container to ensure proper communication with the necessary components.

┌────────────────────────────────┐  ┌───────────────────┐  ┌───────────────────┐
│                                │  │                   │  │                   │
│ ┌────────────┐   ┌───────────┐ │  │  ┌──────────────┐ │  │ ┌───────────────┐ │
│ │ diagnostic │   │ event hub │ │  │  │azure-eventhub│ │  │ │ activity logs │ │
│ │  setting   │──▶│           │◀┼AMQP─│  <<input>>   │─┼──┼▶│<<data stream>>│ │
│ └────────────┘   └───────────┘ │  │  └──────────────┘ │  │ └───────────────┘ │
│                                │  │          │        │  │                   │
│                                │  │          │        │  │                   │
│                                │  │          │        │  │                   │
│         ┌─────────────┬─────HTTPS─┼──────────┘        │  │                   │
│ ┌───────┼─────────────┼──────┐ │  │                   │  │                   │
│ │       │             │      │ │  │                   │  │                   │
│ │       ▼             ▼      │ │  └─Agent─────────────┘  └─Elastic Cloud─────┘
│ │ ┌──────────┐  ┌──────────┐ │ │
│ │ │    0     │  │    1     │ │ │
│ │ │ <<blob>> │  │ <<blob>> │ │ │
│ │ └──────────┘  └──────────┘ │ │
│ │                            │ │
│ │                            │ │
│ └─Storage Account Container──┘ │
│                                │
│                                │
└─Azure──────────────────────────┘

Port 5671 and 5672 are commonly used for secure communication with the event hub. These ports are used to receive events. The Elastic Agent can establish a secure connection with the event hub by allowing traffic on these ports.

For more information, check the following documents:

The Elastic Agent uses port 443 for secure communication with the Storage Account container. By allowing traffic on port 443, the Elastic Agent can securely access and interact with the Storage Account container, essential for storing and retrieving checkpoint data for each event hub partition.

Optionally, you can restrict the traffic to the following domain names:

*.servicebus.windows.net
*.blob.core.windows.net
*.cloudapp.net

Use the following settings to configure the Azure Logs integration when you add it to Fleet.

eventhub : string A fully managed, real-time data ingestion service. Elastic recommends using only letters, numbers, and the hyphen (-) character for event hub names to maximize compatibility. You can use existing event hubs having underscores (_) in the event hub name; in this case, the integration will replace underscores with hyphens (-) when it uses the event hub name to create dependent Azure resources behind the scenes (e.g., the storage account container to store event hub consumer offsets). Elastic also recommends using a separate event hub for each log type as the field mappings of each log type differ. Default value insights-operational-logs.

consumer_group : string Enable the publish/subscribe mechanism of Event Hubs with consumer groups. A consumer group is a view (state, position, or offset) of an entire event hub. Consumer groups enable multiple consuming applications to each have a separate view of the event stream, and to read the stream independently at their own pace and with their own offsets. Default value: $Default

connection_string : string

The connection string is required to communicate with Event Hubs. See Get an Event Hubs connection string for more information.

A Blob Storage account is required to store/retrieve/update the offset or state of the event hub messages. This allows the integration to start back up when it stopped processing messages.

storage_account : string The name of the storage account that the state/offsets will be stored and updated.

storage_account_key : string The storage account key. Used to authorize access to data in your storage account.

storage_account_container : string The storage account container where the integration stores the checkpoint data for the consumer group. It is an advanced option to use with extreme care. You MUST use a dedicated storage account container for each Azure log type (activity, sign-in, audit logs, and others). DO NOT REUSE the same container name for more than one Azure log type. See Container Names for details on naming rules from Microsoft. The integration generates a default container name if not specified.

resource_manager_endpoint : string Optional. By default, the integration uses the Azure public environment. To override this and use a different Azure environment, users can provide a specific resource manager endpoint

Examples:

This setting can also be used to define your own endpoints, like for hybrid cloud models.

Azure services have been observed occasionally sending malformed JSON documents. These logs can disrupt the expected JSON formatting and lead to parsing issues during processing.

To address this issue, the advanced settings section of each data stream offers two sanitization options:

Malformed logs can be identified by:

Known data streams that might produce malformed logs:

Visit the page for each individual Azure Logs integration to see details about exported fields and sample events.