New Beats modules for metrics from Azure Monitor, Azure activity logs, and AD activity reports

We are happy to announce that Elastic Stack 7.5 brings support for the Azure Monitor service in Metricbeat and Azure activity logs and AD (Active Directory) activity reports in Filebeat.

Introducing the Azure module in Metricbeat

The Azure Monitor service collects fundamental metrics and logs from a variety of sources in Azure, such as applications, subscriptions, tenants, and more. Using the new Azure Metricbeat module, users can retrieve all the metrics Azure Monitor exposes and forward them to any of the outputs that Metricbeat supports.

The Azure module includes three metricsets:

• The monitor metricset, which allows users to retrieve all types of metrics from specified resources. Using filters like metric namespace, name, aggregations and dimensions, users  can limit the metrics to just that they are interested in.
• The compute_vm and compute_vm_scaleset metricsets allow users to collect all available metrics from virtual machines and virtual machine scale sets. The metric sampling duration set in the metricsets is 5 minutes so we recommend users to select a collecting interval (period) of 300s or multiples of 300s.

Configuring the Metricbeat Azure module

The following command enables the Azure module configuration:

metricbeat modules enable azure

The module uses the Azure Monitor REST API to retrieve metrics, therefore, all requests must be authenticated with Azure Active Directory (Azure AD). 

The required credentials for authorization are:

• client_id represents the unique identifier for the application (also known as Application Id)
• client_secret represents the client/application secret/key
• subscription_id represents the unique identifier for the azure subscription 
• tenant_id represents the unique identifier of the Azure Active Directory instance

One approach to retrieve these values is to create an Azure AD service principal. For a more detailed walk-through on generating an Azure service principal users can use the Azure PowerShell tool or they can create one via the Azure portal.

We recommend using a dedicated Azure service principal and making sure that the role assigned to the application contains reading permissions to the monitor service and the resources . 

Example of a complete Azure module configuration (watch out for word wrap):

- module: azure 
  metricsets: 
   - monitor 
  enabled: true 
  period: 300s 
  client_id: 'd3d1ab91-2707-4058-8274-05cd96339728' 
  client_secret: 'password' 
  tenant_id: 'd3d1ab91-2707-4058-8274-05cd96339728' 
  subscription_id: 'd3d1ab91-2707-4058-8274-05cd96339728' 
  refresh_list_interval: 600s 
  resources: 
  - resource_id: "subscriptions/d3d1ab91-2707-4058-8274-05cd96339728/resourceGroups/obs-infrastructure/providers/Microsoft.ApiManagement/service/apimanagement" 
    metrics: 
    - name: "Requests" 
      namespace: "Microsoft.ApiManagement/service" 
      aggregations: ["Maximum"] 
      timegrain: "PT1M" 
      dimensions: 
      - name: "Hostname" 
        value: "apimanagement.azure-api.net" 
    - name: ["Capacity", "Requests"] 
      namespace: "Microsoft.ApiManagement/service" 
      aggregations: ["Average"] 
      dimensions: 
      - name: "Location" 
        value: "West Europe" 
  - resource_group: "obs-infrastructure" 
    resource_type: "Microsoft.Compute/virtualMachines" 
    metrics: 
    - name: "*" 
      namespace: "Microsoft.Compute/virtualMachines" 
  - resource_query: "resourceType eq 'Microsoft.DocumentDb/databaseAccounts' and name eq 'dbaccount'" 
    metrics: 
    - name: ["DataUsage", "DocumentCount", "DocumentQuota"] 
      namespace: "Microsoft.DocumentDb/databaseAccounts" 
      aggregations: ["Average", "Total"]

We provide several options to select the monitored resources so users can choose between the following options:

•  resource_id - the fully qualified ID of the resource which includes the resource name and resource type 

Since users can have a greater number of resources they would like to gather metrics on, in order to reduce the verbosity, they can also go for the following options:

• resource_group and resource_type - where metrics from all the resources of the selected resource type that are contained by the entered resource group will be returned.
• resource_query - users can enter a specific query to filter resources on (as defined by the $filter in the Azure REST API resources documentation ).

We also offer options to filter the metrics to be collected:

• name - users can list specific metric names or enter "*" instead which will consider all metrics inside the namespace provided
• namespace - a list of the supported metrics and namespaces, for  specific guest os metrics or custom metrics will be categorized in additional namespaces
• aggregations - Azure Monitor stores all metrics at one-minute granularity intervals, but, to limit the number of raw values users will have to emit and pay for in Azure Monitor, the values are locally pre-aggregated. Each metric has a list of supported aggregations and one primary (most relevant) aggregation. Users can list the aggregations they would like to collect values of or ignore this option entirely which will return the primary aggregation value for the metric.
• dimensions - they are a list of name value pairs that help describe additional characteristics about the metrics. This configuration setting is optional as there are metrics that do not contain any dimensions.
• timegrain - represents the metric sampling duration, metrics can have several timegrain values assigned, if not value is entered a default one is provided.

Azure resource details will be collected at each fetch call (period interval), this means a number of Azure REST calls will be executed each time. If the Azure users are adding/removing resources that match the configuration options while metricbeat is running, these resources will be reflected in Metricbeat as well. If users would like to reduce the number of API calls due performance and rate/ cost reasons they can use the refresh_list_interval configuration option and set an interval the list of resources is being updated.

For the compute_vm and compute_vm_scaleset metricsets, metrics will be collected from all the virtual machines and virtual machine scalesets in the entire subscription if no resource options are configured.

- module: azure 
  metricsets: 
  - compute_vm 
  enabled: true 
  period: 300s 
  client_id: '${AZURE_CLIENT_ID:""}' 
  client_secret: '${AZURE_CLIENT_SECRET:""}' 
  tenant_id: '${AZURE_TENANT_ID:""}' 
  subscription_id: '${AZURE_SUBSCRIPTION_ID:""}' 
  refresh_list_interval: 600s

- module: azure
  metricsets:
  - compute_vm_scaleset
  enabled: true
  period: 300s
  client_id: '${AZURE_CLIENT_ID:""}'
  client_secret: '${AZURE_CLIENT_SECRET:""}'
  tenant_id: '${AZURE_TENANT_ID:""}'
  subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'
  refresh_list_interval: 600s

To filter by resource group, users can add the following configuration:

resources: 
  - resource_group: "test_resource_group"

Dashboards and visualizations

These two metricsets come with dashboards and visualizations that help users view the most relevant metric information. Some dashboards expose extended metrics  that users will need to explicitly enable in Azure.

To enable memory, ASP.NET, or SQLServer extended metrics, users can follow the documented steps for virtual machines and the documentation for virtual machine scalesets.

The [Metricbeat Azure] Compute VMs overview dashboard offers a visual representation of the CPU percentage, incoming and outgoing traffic, average of bytes read/written from/to disk and the disk read/write IOPS. Additional memory metrics are displayed for the Windows vm (if the guest os metrics are collected). 

The [Metricbeat Azure] VM Guest metrics overview dashboard provides visual information on the ASP.NET applications running, SQL Server connections, memory usage and read/write stats.

The [Metricbeat Azure] VM Scale Sets overview dashboard offers a visual representation of the CPU percentage, incoming and outgoing traffic, average of bytes and operations read/written from/to disk.  Additional memory metrics are displayed for the Windows VM (if the guest os metrics are collected).

Introducing the Azure module in Filebeat

With the Filebeat Azure module, users can retrieve Azure activity logs and AD activity reports.

A prerequisite for this functionality is that these logs should be exported to event hubs with an  enabled kafka option as the module will collect the logs from these event hubs.

The activity logs provide insight into subscription-level events that have occurred in Azure. This includes a range of data, from Azure Resource Manager operational data to updates on Service Health events.

To export the activity logs to an event hub in the Azure Portal users can follow these steps.

The Azure module fileset that collects these logs is the activitylogs fileset.

The AD activity reports include the sign-in logs which provide information about the usage of managed applications and user sign-in activities and the audit logs which provide traceability through logs for all changes done by various features within Azure AD.

To export the Azure AD reports to an event hub in the Azure Portal users can follow these steps. 

The Azure module filesets that collect these two types of logs are named signinlogs and auditlogs, respectively.

Configuring the Azure module in Filebeat

The following command enables the Azure module configuration :

filebeat modules enable azure

Example configuration:

- module: azure 
  activitylogs: 
   enabled: true 
   var: 
     namespace: "eventhubs.servicebus.windows.net:9093" 
     eventhub: ["insights-operational-logs"] 
     consumer_group: "$Default" 
     connection_string: "" 
  auditlogs: 
   enabled: true 
   var: 
     namespace: "eventhubs.servicebus.windows.net:9093" 
     eventhub: ["insights-logs-auditlogs"] 
     consumer_group: "$Default" 
     connection_string: "" 
  signinlogs: 
   enabled: true 
   var: 
     namespace: "eventhubs.servicebus.windows.net:9093" 
     eventhub: ["insights-logs-signinlogs"] 
     consumer_group: "$Default" 
     connection_string: ""

• namespace contains the Azure event hub namespace FQDN followed by the kafka protocol
• eventhub contains the event hub name the logs are exported to
• consumer_group contains the consumer group name, we advise having a dedicated consumer group for the filebeat module.
• connection_string contains the connection string required to communicate with Event Hubs, users can find it inside the SAS policy area. 

Dashboards and visualizations

The Azure module comes with several visualizations and dashboards where users can view resource activity and monitor signin and audit events.

The [Filebeat Azure] Cloud Overview offers a visual representation of the total number of users/resources/resource groups/subscriptions mentioned in the logs. The dashboards comes with several overviews of the activity, sign in alert and service health levels.

The [Filebeat Azure] Alerts Overview provides an outline of the activity alerts count and overview and lists the most affected resources.

The [Filebeat Azure] User Activity provides an outline of the user list and activity and top resources and resource types affected. 

What’s next

With the monitor metricset users can already have access to all metrics exposed by the Azure Monitor. 

To further simplify access to this data we will continue adding more service specific integrations with Azure and enriching these modules with new metricsets and filesets that will monitor the most relevant Azure services. We encourage you to test these new modules out, either in your own cluster or in a free trial of Elasticsearch Service.

We are very interested in your feedback so let us know what you think about this first step at supporting Azure with Beats in the Beats forum.