Amazon EMR

Collect logs and metrics for Amazon EMR service with Elastic Agent


Version	2.30.1 (View all)
Compatible Kibana version(s)	8.15.2 or higher
Supported Serverless project types What's this?	Security Observability
Subscription level What's this?	Basic

The Amazon EMR integration allows you to monitor Amazon EMR — a fully managed big data processing and analytics service.

Use the Amazon EMR integration to collect metrics related to your EMR instances. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference the metrics when troubleshooting an issue.

For example, you could use this data to track Amazon EMR cluster progress and cluster storage. Then you can alert when utilization for an instance crosses a predefined threshold.

IMPORTANT: Extra AWS charges on API requests will be generated by this integration. Check API Requests for more details.

Data streams

The Amazon EMR integration collects two types of data: metrics and logs.

Metrics give you insight into the state of Amazon EMR. The metrics collected by the Amazon EMR integration include cluster progress, cluster state, cluster or node storage, and more. See more details in the Metrics reference

Logs help you keep a record of events happening in Amazon EMR. Logs collected by the Amazon EMR integration include the cluster status, node status details and more.

Requirements

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.

Before using any AWS integration you will need:

AWS Credentials to connect with your AWS account.
AWS Permissions to make sure the user you're using to connect has permission to share the relevant data.

For more details about these requirements, see the AWS integration documentation.

Setup

Use this integration if you only need to collect data from the Amazon EMR service.

If you want to collect data from two or more AWS services, consider using the AWS integration. When you configure the AWS integration, you can collect data from as many AWS services as you'd like.

For step-by-step instructions on how to set up an integration, see the Getting started guide.

Metrics reference

An example event for emr looks as following:

{
    "@timestamp": "2022-07-26T21:43:00.000Z",
    "agent": {
        "ephemeral_id": "cdaaaabb-be7e-432f-816b-bda019fd7c15",
        "id": "2d4b09d0-cdb6-445e-ac3f-6415f87b9864",
        "name": "docker-fleet-agent",
        "type": "metricbeat",
        "version": "8.3.2"
    },
    "aws": {
        "cloudwatch": {
            "namespace": "AWS/ElasticMapReduce"
        },
        "dimensions": {
            "JobFlowId": "j-3LRBO17JBA7H9"
        },
        "elasticmapreduce": {
            "metrics": {
                "IsIdle": {
                    "avg": 1
                }
            }
        }
    },
    "cloud": {
        "account": {
            "id": "428152502467",
            "name": "elastic-beats"
        },
        "provider": "aws",
        "region": "eu-central-1"
    },
    "data_stream": {
        "dataset": "aws.emr_metrics",
        "namespace": "default",
        "type": "metrics"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "2d4b09d0-cdb6-445e-ac3f-6415f87b9864",
        "snapshot": false,
        "version": "8.3.2"
    },
    "event": {
        "agent_id_status": "verified",
        "dataset": "aws.emr_metrics",
        "duration": 11576777300,
        "ingested": "2022-07-26T21:47:48Z",
        "module": "aws"
    },
    "metricset": {
        "name": "cloudwatch",
        "period": 300000
    },
    "service": {
        "type": "aws"
    }
}

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields

Field	Description	Type	Unit	Metric Type
@timestamp	Event timestamp.	date
agent.id	Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.	keyword
aws.cloudwatch.namespace	The namespace specified when query cloudwatch api.	keyword
aws.dimensions.JobFlowId	Filters metrics by cluster ID.	keyword
aws.elasticmapreduce.metrics.AppsCompleted.sum	The number of applications submitted to YARN that have completed.	long		gauge
aws.elasticmapreduce.metrics.AppsFailed.sum	The number of applications submitted to YARN that have failed to complete.	long		gauge
aws.elasticmapreduce.metrics.AppsKilled.sum	The number of applications submitted to YARN that have been killed.	long		gauge
aws.elasticmapreduce.metrics.AppsPending.sum	The number of applications submitted to YARN that are in a pending state.	long		gauge
aws.elasticmapreduce.metrics.AppsRunning.sum	The number of applications submitted to YARN that are running.	long		gauge
aws.elasticmapreduce.metrics.AppsSubmitted.sum	The number of applications submitted to YARN.	long		gauge
aws.elasticmapreduce.metrics.AutoTerminationIsClusterIdle.avg	Indicates whether the cluster is in use.	long	percent	gauge
aws.elasticmapreduce.metrics.CapacityRemainingGB.sum	The amount of remaining HDFS disk capacity.	long	byte	gauge
aws.elasticmapreduce.metrics.ContainerAllocated.sum	The number of resource containers allocated by the ResourceManager.	long		gauge
aws.elasticmapreduce.metrics.ContainerPending.sum	The number of containers in the queue that have not yet been allocated.	long		gauge
aws.elasticmapreduce.metrics.ContainerPendingRatio.avg	The ratio of pending containers to containers allocated	long	percent	gauge
aws.elasticmapreduce.metrics.ContainerReserved.sum	The number of containers reserved.	long		gauge
aws.elasticmapreduce.metrics.CoreNodesPending.sum	The number of core nodes waiting to be assigned.	long		gauge
aws.elasticmapreduce.metrics.CoreNodesRequested.max	The target number of CORE nodes in a cluster as determined by managed scaling.	long		gauge
aws.elasticmapreduce.metrics.CoreNodesRunning.avg	The current number of CORE nodes running in a cluster.	long		gauge
aws.elasticmapreduce.metrics.CoreUnitsRequested.max	The target number of CORE units in a cluster as determined by managed scaling.	long		gauge
aws.elasticmapreduce.metrics.CoreUnitsRunning.avg	The current number of CORE units running in a cluster.	long		gauge
aws.elasticmapreduce.metrics.CoreVCPURequested.max	The target number of CORE vCPUs in a cluster as determined by managed scaling.	long		gauge
aws.elasticmapreduce.metrics.CoreVCPURunning.avg	The current number of CORE vCPUs running in a cluster.	long		gauge
aws.elasticmapreduce.metrics.CorruptBlocks.max	The number of blocks that HDFS reports as corrupted.	long		gauge
aws.elasticmapreduce.metrics.DfsPendingReplicationBlocks.sum	The status of block replication - blocks being replicated, age of replication requests, and unsuccessful replication requests.	long		gauge
aws.elasticmapreduce.metrics.HDFSBytesRead.sum	The number of bytes read from HDFS.	long	byte	gauge
aws.elasticmapreduce.metrics.HDFSBytesWritten.sum	The number of bytes written to HDFS.	long	byte	gauge
aws.elasticmapreduce.metrics.HDFSUtilization.avg	The percentage of HDFS storage currently used.	double	percent	gauge
aws.elasticmapreduce.metrics.IsIdle.avg	Indicates that a cluster is no longer performing work, but is still alive and accruing charges.	long	percent	gauge
aws.elasticmapreduce.metrics.LiveDataNodes.avg	The percentage of data nodes that are receiving work from Hadoop.	double	percent	gauge
aws.elasticmapreduce.metrics.MRActiveNodes.sum	The number of nodes presently running MapReduce tasks or jobs.	long		gauge
aws.elasticmapreduce.metrics.MRDecommissionedNodes.sum	The number of nodes allocated to MapReduce applications that have been marked in a DECOMMISSIONED state.	long		gauge
aws.elasticmapreduce.metrics.MRLostNodes.sum	The number of nodes allocated to MapReduce that have been marked in a LOST state.	long		gauge
aws.elasticmapreduce.metrics.MRRebootedNodes.sum	The number of nodes available to MapReduce that have been rebooted and marked in a REBOOTED state.	long		gauge
aws.elasticmapreduce.metrics.MRTotalNodes.sum	The number of nodes presently available to MapReduce jobs.	long		gauge
aws.elasticmapreduce.metrics.MRUnhealthyNodes.sum	The number of nodes available to MapReduce jobs marked in an UNHEALTHY state.	long		gauge
aws.elasticmapreduce.metrics.MemoryAllocatedMB.sum	The amount of memory allocated to the cluster.	long	byte	gauge
aws.elasticmapreduce.metrics.MemoryAvailableMB.sum	The amount of memory available to be allocated.	long	byte	gauge
aws.elasticmapreduce.metrics.MemoryReservedMB.sum	The amount of memory reserved.	long	byte	gauge
aws.elasticmapreduce.metrics.MemoryTotalMB.sum	The total amount of memory in the cluster.	long	byte	gauge
aws.elasticmapreduce.metrics.MissingBlocks.max	The number of blocks in which HDFS has no replicas.	long		gauge
aws.elasticmapreduce.metrics.MultiMasterInstanceGroupNodesRequested.sum	The number of requested master nodes.	long		gauge
aws.elasticmapreduce.metrics.MultiMasterInstanceGroupNodesRunning.sum	The number of running master nodes.	long		gauge
aws.elasticmapreduce.metrics.MultiMasterInstanceGroupNodesRunningPercentage.avg	The percentage of master nodes that are running over the requested master node instance count.	double	percent	gauge
aws.elasticmapreduce.metrics.PendingDeletionBlocks.sum	The number of blocks marked for deletion.	long		gauge
aws.elasticmapreduce.metrics.S3BytesRead.sum	The number of bytes read from Amazon S3.	long	byte	gauge
aws.elasticmapreduce.metrics.S3BytesWritten.sum	The number of bytes written to Amazon S3.	long	byte	gauge
aws.elasticmapreduce.metrics.TaskNodesRequested.max	The target number of TASK nodes in a cluster as determined by managed scaling.	long		gauge
aws.elasticmapreduce.metrics.TaskNodesRunning.avg	The current number of TASK nodes running in a cluster.	long		gauge
aws.elasticmapreduce.metrics.TaskUnitsRequested.max	The target number of TASK units in a cluster as determined by managed scaling.	long		gauge
aws.elasticmapreduce.metrics.TaskUnitsRunning.avg	The current number of TASK units running in a cluster.	long		gauge
aws.elasticmapreduce.metrics.TaskVCPURequested.max	The target number of TASK vCPUs in a cluster as determined by managed scaling.	long		gauge
aws.elasticmapreduce.metrics.TaskVCPURunning.avg	The current number of TASK vCPUs running in a cluster.	long		gauge
aws.elasticmapreduce.metrics.TotalLoad.sum	The total number of concurrent data transfers.	long		gauge
aws.elasticmapreduce.metrics.TotalNodesRequested.max	The target total number of nodes in a cluster as determined by managed scaling.	long		gauge
aws.elasticmapreduce.metrics.TotalNodesRunning.avg	The current total number of nodes available in a running cluster.	long		gauge
aws.elasticmapreduce.metrics.TotalNotebookKernels.sum	The total number of running and idle notebook kernels on the cluster.	long		gauge
aws.elasticmapreduce.metrics.TotalUnitsRequested.max	The target total number of units in a cluster as determined by managed scaling.	long		gauge
aws.elasticmapreduce.metrics.TotalUnitsRunning.avg	The current total number of units available in a running cluster.	long		gauge
aws.elasticmapreduce.metrics.TotalVCPURequested.max	The target total number of vCPUs in a cluster as determined by managed scaling.	long		gauge
aws.elasticmapreduce.metrics.TotalVCPURunning.avg	The current total number of vCPUs available in a running cluster.	long		gauge
aws.elasticmapreduce.metrics.UnderReplicatedBlocks.sum	The number of blocks that need to be replicated one or more times.	long		gauge
aws.elasticmapreduce.metrics.YARNMemoryAvailablePercentage.avg	The percentage of remaining memory available to YARN	double	percent	gauge
aws.metrics_names_fingerprint	Autogenerated ID representing the fingerprint of the list of metrics names. Applicable only for Amazon Data Firehose integration.	keyword
aws.tags	Tag key value pairs from aws resources.	flattened
cloud.account.id	The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.	keyword
cloud.image.id	Image ID for the cloud instance.	keyword
cloud.region	Region in which this host, resource, or service is located.	keyword
data_stream.dataset	Data stream dataset.	constant_keyword
data_stream.namespace	Data stream namespace.	constant_keyword
data_stream.type	Data stream type.	constant_keyword
event.module	Event module	constant_keyword
host.containerized	If the host is a container.	boolean
host.os.build	OS build information.	keyword
host.os.codename	OS codename, if any.	keyword

Logs reference