Amazon EC2

Collect logs and metrics for Amazon Elastic Compute Cloud service with Elastic Agent


Version	2.30.1 (View all)
Compatible Kibana version(s)	8.15.2 or higher
Supported Serverless project types What's this?	Security Observability
Subscription level What's this?	Basic

The Amazon EC2 integration allows you to monitor Amazon Elastic Compute Cloud (Amazon EC2)—a cloud compute platform.

Use the Amazon EC2 integration to collect logs and metrics related to your EC2 instances. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference the logs and metrics when troubleshooting an issue.

For example, you could use this data to track Amazon EC2 CPU utilization. Then you can alert when utilization for an instance crosses a predefined threshold.

IMPORTANT: Extra AWS charges on API requests will be generated by this integration. Check API Requests for more details.

Data streams

The Amazon EC2 integration collects two types of data: logs and metrics.

Logs help you keep a record of events happening in Amazon EC2. Logs collected by the Amazon EC2 integration include the region in which an instance is running, the operating system architecture, container information, and more. See more details in the Logs reference.

Metrics give you insight into the state of your Amazon EC2 instances. Metrics collected by the Amazon EC2 integration include the Amazon EC2 instance ID, the number of earned CPU credits that an instance has accrued since it was launched or started, and more. See more details in the Metrics reference.

Requirements

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.

Before using any AWS integration you will need:

AWS Credentials to connect with your AWS account.
AWS Permissions to make sure the user you're using to connect has permission to share the relevant data.

For more details about these requirements, please take a look at the AWS integration documentation.

Setup

Use this integration if you only need to collect data from the Amazon EC2 service.

If you want to collect data from two or more AWS services, consider using the AWS integration. When you configure the AWS integration, you can collect data from as many AWS services as you'd like.

For step-by-step instructions on how to set up an integration, see the Getting started guide.

Advanced options

CloudWatch

The CloudWatch logs input has several advanced options to fit specific use cases.

Latency

AWS CloudWatch Logs sometimes takes extra time to make the latest logs available to clients like the Agent.

The CloudWatch integration offers the latency setting to address this scenario. Latency translates the query's time range to consider the CloudWatch Logs latency. For example, a 5m latency means the integration will query CloudWatch for logs available 5 minutes ago.

Number of workers

If you are collecting log events from multiple log groups using log_group_name_prefix, you should review the value of the number_of_workers.

The number_of_workers setting defines the number of workers assigned to reading from log groups. Each log group matching the log_group_name_prefix requires a worker to keep log ingestion as close to real-time as possible. For example, if log_group_name_prefix matches five log groups, then number_of_workers should be set to 5. The default value is 1.

Logs reference

The ec2 data stream supports both EC2 logs stored in AWS CloudWatch and EC2 logs stored in Amazon S3. For logs stored in S3, you must export logs from log groups to an Amazon S3 bucket which has SQS notification setup already.

With this data stream, EC2 logs will be parsed into fields like ip_address and process.name. For logs from other services, please use the AWS CloudWatch integration.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields

Field	Description	Type
@timestamp	Event timestamp.	date
aws.ec2.ip_address	The internet address of the requester.	keyword
aws.s3.bucket.arn	The AWS S3 bucket ARN.	keyword
aws.s3.bucket.name	The AWS S3 bucket name.	keyword
aws.s3.object.key	The AWS S3 Object key.	keyword
cloud.image.id	Image ID for the cloud instance.	keyword
data_stream.dataset	Data stream dataset.	constant_keyword
data_stream.namespace	Data stream namespace.	constant_keyword
data_stream.type	Data stream type.	constant_keyword
event.module	Event module	constant_keyword
host.containerized	If the host is a container.	boolean
host.os.build	OS build information.	keyword
host.os.codename	OS codename, if any.	keyword
input.type	Input type	keyword
log.offset	Log offset	long

An example event for ec2 looks as following:

{
    "@timestamp": "2020-02-20T07:01:01.000Z",
    "agent": {
        "ephemeral_id": "d86fb0e5-bdd6-4b91-8a3e-3535b80a4118",
        "id": "acba78ef-1401-4689-977c-d8c2e5d6a8fa",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.10.1"
    },
    "aws": {
        "ec2": {
            "ip_address": "ip-172-31-81-156"
        },
        "s3": {
            "bucket": {
                "arn": "arn:aws:s3:::elastic-package-aws-bucket-36908",
                "name": "elastic-package-aws-bucket-36908"
            },
            "object": {
                "key": "ec2.log"
            }
        }
    },
    "cloud": {
        "provider": "",
        "region": "us-east-1"
    },
    "data_stream": {
        "dataset": "aws.ec2_logs",
        "namespace": "ep",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "acba78ef-1401-4689-977c-d8c2e5d6a8fa",
        "snapshot": false,
        "version": "8.10.1"
    },
    "event": {
        "agent_id_status": "verified",
        "dataset": "aws.ec2_logs",
        "ingested": "2023-11-03T15:00:09Z",
        "original": "2020-02-20T07:01:01.000Z Feb 20 07:01:01 ip-172-31-81-156 systemd: Stopping User Slice of root."
    },
    "input": {
        "type": "aws-s3"
    },
    "log": {
        "file": {
            "path": "https://elastic-package-aws-bucket-36908.s3.us-east-1.amazonaws.com/ec2.log"
        },
        "offset": 0
    },
    "message": "Stopping User Slice of root.",
    "process": {
        "name": "systemd"
    },
    "tags": [
        "preserve_original_event",
        "forwarded",
        "aws-ec2-logs"
    ]
}

Metrics reference