Google Cloud Platform carbon metricset

edit

This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.

The carbon metricset is designed to collect Carbon Footprint data from GCP BigQuery monthly cost detail table. BigQuery is a fully-managed, serverless data warehouse.

Cloud Carbon export to BigQuery enables you to export detailed Google Cloud carbon footprint data (such as carbon produced by tier and service) automatically throughout the month to a BigQuery dataset that you specify. Then you can access your Cloud Carbon data from BigQuery for detailed analysis using Metricbeat. Please see export carbon footprint data to BigQuery for more details on how to export carbon footprint data.

Metricset-specific configuration notes

edit
  • dataset_id: (Required) Dataset ID that points to the top-level container which contains the actual carbon footprint tables.
  • table_pattern: (Optional) The name of the table where carbon footprint data is stored. Default to carbon_footprint.

Configuration example

edit
- module: gcp
  metricsets:
    - carbon
  period: 24h
  project_id: "your project id"
  credentials_file_path: "your JSON credentials file path"
  dataset_id: "dataset id"
  table_name: "table name"

Fields

For a description of each field in the metricset, see the exported fields section.

Here is an example document generated by this metricset:

{
  "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud.account.id": "01475F-5B1080-1137E7",
    "cloud.project.id": "elastic-bi",
    "cloud.project.name": "elastic-containerlib-prod",
    "cloud.provider": "gcp",
    "event": {
      "dataset": "gcp.carbon",
      "duration": 115000,
      "module": "gcp"
    },
    "gcp": {
      "carbon": {
        "project_id": "containerlib-prod-12763",
        "project_name": "elastic-containerlib-prod",
        "service_id": "24E6-581D-38E5",
        "service_description": "BigQuery",
        "footprint.scope1":          4.044,
        "footprint.scope2.location": 1.797,
        "footprint.scope2.market":   null,
        "footprint.scope3":          2.337,
        "footprint.offsets":         null
      }
    },
    "metricset": {
      "name": "carbon",
      "period": 10000
    },
    "service": {
      "type": "gcp"
    }
}