Monitor Microsoft Azure with Beats

Log in to the Azure portal.

Select a region and then click Review + create.

After your deployment is complete, click Go to resource. Here you can see and configure your deployment details. To access the cluster, click Kibana.

To see if there is any available data, click Observability. There should be no data yet, but next, you will ingest logs.

On to the Elasticsearch resource page in Azure, click Ingest logs and metrics from Azure Services.

Check both checkboxes and click Save.

Go to your Elastic resource and click Virtual machines.

Select the VMs that you want to collect logs and metrics from, click Install Extension, and then click OK.

Wait until it is installed and sending data (if the list does not update, click Refresh ). To see the logs from the VM, open Logs Explorer (find Logs Explorer in the global search field).

To view VM metrics, go to Infrastructure inventory and then select a VM. (To open Infrastructure inventory, find Infrastructure in the main menu or use the global search field.)

Go to the Azure Management Portal. Search and click on Azure Active Directory.

Click on App registrations in the navigation pane of the selected Active Directory and then click on New registration.

Type the name of your application (this tutorial uses monitor-azure) and click on Register (leave all the other options with the default value).

Copy the Application (client) ID, and save it for future reference. This id is required to configure Metricbeat to connect to your Azure account.

Click on Certificates & secrets. Then, click on New client secret to create a new security key.

Type a key description and select a key duration in the expire list. Click on Add to create a client secret. The next page will display the key value under the Value field. Copy the secret and save it (along with your Client ID) for future reference.

On Azure Portal, search and click on Subscriptions.

In the Select field, type the description name of the configured service principal (monitor-azure).

On the Elasticsearch resource page, click Manage changes in Elastic Cloud.

Copy the Cloud ID and keep it safe. You will use it later.

Click Security and then click Reset password. Confirm, and copy the password. Keep it safe as you will use it later.

Use the Metricbeat keystore to store secure settings. Store the Cloud ID in the keystore.

./metricbeat keystore create
echo -n "<Your Deployment Cloud ID>" | ./metricbeat keystore add CLOUD_ID --stdin

To store metrics in Elasticsearch with minimal permissions, create an API key to send data from Metricbeat to Elasticsearch Service. Log into Kibana (you can do so from the Cloud Console without typing in any permissions) and find Dev Tools in the global search field. From the Console, send the following request:

POST /_security/api_key
{
  "name": "metricbeat-monitor",
  "role_descriptors": {
    "metricbeat_writer": {
      "cluster": ["monitor", "read_ilm"],
      "index": [
        {
          "names": ["metricbeat-*"],
          "privileges": ["view_index_metadata", "create_doc"]
        }
      ]
    }
  }
}

The response contains an api_key and an id field, which can be stored in the Metricbeat keystore in the following format: id:api_key.

echo -n "IhrJJHMB4JmIUAPLuM35:1GbfxhkMT8COBB4JWY3pvQ" | ./metricbeat keystore add ES_API_KEY --stdin

To see if both settings have been stored, run the following command:

./metricbeat keystore list

To configure Metricbeat to output to Elasticsearch Service, edit the metricbeat.yml configuration file. Add the following lines to the end of the file.

cloud.id: ${CLOUD_ID}
output.elasticsearch:
  api_key: ${ES_API_KEY}

Finally, test if the configuration is working. If it is not working, verify if you used the right credentials and add them again.

./metricbeat test output

The azure module configuration needs three ids and one secret. Use the commands below to store each one of them in the keystore.

echo -n "<client_id>" | ./metricbeat keystore add AZURE_CLIENT_ID --stdin
echo -n "<client_secret>" | ./metricbeat keystore add AZURE_CLIENT_SECRET --stdin
echo -n "<tenant_id>" | ./metricbeat keystore add AZURE_TENANT_ID --stdin
echo -n "<subscription_id>" | ./metricbeat keystore add AZURE_SUBSCRIPTION_ID --stdin

Enable the Azure module.

./metricbeat modules enable azure

Edit the modules.d/azure.yml file to collect billing metrics.

- module: azure
  metricsets:
  - billing  
  enabled: true
  period: 24h  
  client_id: '${AZURE_CLIENT_ID:""}'
  client_secret: '${AZURE_CLIENT_SECRET:""}'
  tenant_id: '${AZURE_TENANT_ID:""}'
  subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'
  refresh_list_interval: 600s

To check if Metricbeat can collect data, test the input by running the following command:

./metricbeat test modules azure

Metricbeat will print billing metrics to the terminal, if the setup is correct.

When the input and output are ready, start Metricbeat to collect the data.

./metricbeat -e

Finally, log into Kibana and open the [Metricbeat Azure] Billing overview dashboard. Keep in mind it collects data every 24 hours.