Monitor Google Cloud Platform

Use the Metricbeat keystore to store secure settings. Store the Cloud ID in the keystore.

./metricbeat keystore create
echo -n "<Your Deployment Cloud ID>" | ./metricbeat keystore add CLOUD_ID --stdin

To store metrics in Elasticsearch with minimal permissions, create an API key to send data from Metricbeat to Elasticsearch Service. Log into Kibana (you can do so from the Cloud Console without typing in any permissions) and select Management → Dev Tools. Send the following request:

POST /_security/api_key
{
  "name": "metricbeat-monitor",
  "role_descriptors": {
    "metricbeat_writer": {
      "cluster": ["monitor", "read_ilm"],
      "index": [
        {
          "names": ["metricbeat-*"],
          "privileges": ["view_index_metadata", "create_doc"]
        }
      ]
    }
  }
}

The response contains an api_key and an id field, which can be stored in the Metricbeat keystore in the following format: id:api_key.

echo -n "IhrJJHMB4JmIUAPLuM35:1GbfxhkMT8COBB4JWY3pvQ" | ./metricbeat keystore add ES_API_KEY --stdin

To see if both settings have been stored, run the following command:

./metricbeat keystore list

To configure Metricbeat to output to Elasticsearch Service, edit the metricbeat.yml configuration file. Add the following lines to the end of the file.

cloud.id: ${CLOUD_ID}
output.elasticsearch:
  api_key: ${ES_API_KEY}

Finally, test if the configuration is working. If it is not working, verify if you used the right credentials and add them again.

./metricbeat test output

Enable the GCP module.

./metricbeat modules enable gcp

Edit the modules.d/gcp.yml file to configure which metrics to collect.

- module: gcp
  metricsets:
    - compute 
  zone: "" 
  project_id: "your-project-id" 
  period: 1m 
  credentials_file_path: "/home/ubuntu/credentials.json" 

To check if Metricbeat can collect data, test the input by running the following command:

./metricbeat test modules gcp

Metricbeat will print GCP metrics to the terminal, if the setup is correct.

When the input and output are ready, start Metricbeat to collect the data.

./metricbeat -e

Finally, log into Kibana and open the [Metricbeat GCP] Compute Overview dashboard.

Use the Filebeat keystore to store secure settings. Store the Cloud ID in the keystore.

./filebeat keystore create
echo -n "<Your Deployment Cloud ID>" | ./filebeat keystore add CLOUD_ID --stdin

To store logs in Elasticsearch with minimal permissions, create an API key to send data from Filebeat to Elasticsearch Service. Log into Kibana (you can do so from the Cloud Console without typing in any permissions) and select Management → Dev Tools. Send the following request:

POST /_security/api_key
{
  "name": "filebeat-monitor-gcp",
  "role_descriptors": {
    "filebeat_writer": {
      "cluster": [
        "monitor",
        "read_ilm",
        "cluster:admin/ingest/pipeline/get", 
        "cluster:admin/ingest/pipeline/put" 
      ],
      "index": [
        {
          "names": ["filebeat-*"],
          "privileges": ["view_index_metadata", "create_doc"]
        }
      ]
    }
  }
}

The response contains an api_key and an id field, which can be stored in the Filebeat keystore in the following format: id:api_key.

echo -n "IhrJJHMB4JmIUAPLuM35:1GbfxhkMT8COBB4JWY3pvQ" | ./filebeat keystore add ES_API_KEY --stdin

To see if both settings have been stored, run the following command:

./filebeat keystore list

To configure Filebeat to output to Elasticsearch Service, edit the filebeat.yml configuration file. Add the following lines to the end of the file.

cloud.id: ${CLOUD_ID}
output.elasticsearch:
  api_key: ${ES_API_KEY}

Finally, test if the configuration is working. If it is not working, verify that you used the right credentials and, if necessary, add them again.

./filebeat test output

Go to the Logs Router page to configure GCP to export logs to a Pub/Sub topic. Use the search bar to find the page:

To set up the logs routing sink, click Create sink. Set sink name as monitor-gcp-audit-sink. Select the Cloud Pub/Sub topic as the sink service and Create new Cloud Pub/Sub topic named monitor-gcp-audit:

Finally, under Choose logs to include in sink, add logName:"cloudaudit.googleapis.com" (it includes all audit logs). Click create sink. It will look something like the following:

Now go to the Pub/Sub page to add a subscription to the topic you just created. Use the search bar to find the page:

To add a subscription to the monitor-gcp-audit topic click Create subscription:

Set monitor-gcp-audit-sub as the Subscription ID and leave the Delivery type as pull:

Finally, scroll down and click Create.

Now that GCP is configured to export audit logs, enable Filebeat Google Cloud module.

./filebeat modules enable gcp

Edit the modules.d/gcp.yml file with the following configurations.

- module: gcp
  vpcflow:
    enabled: false 
  firewall:
    enabled: false 
  audit:
    enabled: true 
    var.project_id: "elastic-education" 
    var.topic: "monitor-gcp-audit" 
    var.subscription_name: "monitor-gcp-audit-sub" 
    var.credentials_file: "/home/ubuntu/credentials.json" 

Start Filebeat to collect the logs.

./filebeat -e

Finally, log into Kibana and open the [Filebeat GCP] Audit dashboard.