Google Cloud Platform dataproc metricset
editGoogle Cloud Platform dataproc metricset
editThis functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.
Dataproc metricset fetches metrics from Dataproc in Google Cloud Platform.
The dataproc metricset contains all metrics exported from the GCP Dataproc Monitoring API. The field names have been left untouched for people already familiar with them.
You can specify a single region to fetch metrics like us-central1. Be aware that GCP Storage does not use zones so us-central1-a will return nothing. If no region is specified, it will return metrics from all buckets.
Metrics
editHere is a list of metrics collected by dataproc metricset:
-
dataproc.batch.spark.executors: Indicates the number of Batch Spark executors. -
dataproc.cluster.hdfs.datanodes: Indicates the number of HDFS DataNodes that are running inside a cluster. -
dataproc.cluster.hdfs.storage_capacity: Indicates capacity of HDFS system running on cluster in GB. -
dataproc.cluster.hdfs.storage_utilization: The percentage of HDFS storage currently used. -
dataproc.cluster.hdfs.unhealthy_blocks: Indicates the number of unhealthy blocks inside the cluster. -
dataproc.cluster.job.completion_time: The time jobs took to complete from the time the user submits a job to the time Dataproc reports it is completed. -
dataproc.cluster.job.duration: The time jobs have spent in a given state. -
dataproc.cluster.job.failed_count: Indicates the number of jobs that have failed on a cluster. -
dataproc.cluster.job.running_count: Indicates the number of jobs that are running on a cluster. -
dataproc.cluster.job.submitted_count: Indicates the number of jobs that have been submitted to a cluster. -
dataproc.cluster.operation.completion_time: The time operations took to complete from the time the user submits a operation to the time Dataproc reports it is completed. -
dataproc.cluster.operation.duration: The time operations have spent in a given state. -
dataproc.cluster.operation.failed_count: Indicates the number of operations that have failed on a cluster. -
dataproc.cluster.operation.running_count: Indicates the number of operations that are running on a cluster. -
dataproc.cluster.operation.submitted_count: Indicates the number of operations that have been submitted to a cluster. -
dataproc.cluster.yarn.allocated_memory_percentage: The percentage of YARN memory is allocated. -
dataproc.cluster.yarn.apps: Indicates the number of active YARN applications. -
dataproc.cluster.yarn.containers: Indicates the number of YARN containers. -
dataproc.cluster.yarn.memory_size: Indicates the YARN memory size in GB. Sampled every 60 seconds. -
dataproc.cluster.yarn.nodemanagers: Indicates the number of YARN NodeManagers running inside cluster. -
dataproc.cluster.yarn.pending_memory_size: The current memory request, in GB, that is pending to be fulfilled by the scheduler. -
dataproc.cluster.yarn.virtual_cores: Indicates the number of virtual cores in YARN.
For a description of each field in the metricset, see the exported fields section.
Here is an example document generated by this metricset:
{
"@timestamp": "2016-05-23T08:05:34.853Z",
"cloud": {
"account": {
"id": "elastic-apm"
},
"provider": "gcp"
},
"event": {
"dataset": "gcp.dataproc",
"duration": 115000,
"module": "gcp"
},
"gcp": {
"labels": {
"metrics": {
"storage_class": "MULTI_REGIONAL"
},
"resource": {
"bucket_name": "artifacts.elastic-apm.appspot.com",
"location": "us"
}
},
"dataproc": {
"cluster": {
"hdfs": {
"datanodes": {
"value": 15
}
}
}
}
},
"metricset": {
"name": "dataproc",
"period": 10000
},
"service": {
"type": "gcp"
}
}