aws module
editaws module
editThis module periodically fetches monitoring metrics from AWS Cloudwatch using GetMetricData API for AWS services. Note: extra AWS charges on GetMetricData API requests will be generated by this module.
The default metricset is ec2
.
Module-specific configuration notes
editThis module environment variable AWS_ACCESS_KEY_ID
, AWS_SECRET_ACCESS_KEY
, AWS_SESSION_TOKEN
and AWS_REGION
references in the config file to set values that need to be configurable during deployment.
There are two different kinds of AWS credentials can be used here: access keys
and temporary security credentials
.
AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
are the two parts of access keys
. They are long-term credentials for
an IAM user or the AWS account root user. Please see
AWS Access Keys
and Secret Access Keys for more details. temporary security credentials
has a limited lifetime and consists of an access key ID,
a secret access key, and a security token which typically returned from GetSessionToken
. MFA-enabled IAM users would
need to submit an MFA code while calling GetSessionToken
. aws_default_region
is to set the region for SDK to use. Please
see Temporary Security Credentials for more details.
sts get-session-token
AWS CLI can be used to generate temporary credentials. For example. with MFA-enabled:
aws> sts get-session-token --serial-number arn:aws:iam::1234:mfa/your-email@example.com --token-code 456789 --duration-seconds 129600
Because temporary security credentials are short term, after they expire, the user needs to generate new ones and modify the aws.yml config file with the new credentials. Unless live reloading feature is enabled for Metricbeat, the user needs to manually restart Metricbeat after updating the config file in order to continue collecting Cloudwatch metrics. This will cause data loss if the config file is not updated with new credentials before the old ones expire. For Metricbeat, we recommend users to use access keys in config file to enable aws module making AWS api calls without have to generate new temporary credentials and update the config frequently.
IAM policy is an entity that defines permissions to an object within your AWS environment. Specific permissions needs to be added into the IAM user’s policy to authorize Metricbeat to collect AWS monitoring metrics. Please see documentation under each metricset for required permissions.
By default, Amazon EC2 sends metric data to CloudWatch every 5 minutes. With this basic monitoring, period
in aws module
configuration should be larger or equal than 300s
. If period
is set to be less than 300s
, the same cloudwatch metrics
will be collected more than once which will cause extra fees without getting more granular metrics. For example, in US East (N. Virginia)
region, it costs
$0.01/1000 metrics requested using GetMetricData. Please see AWS Cloudwatch Pricing
for more details. To avoid unnecessary charges, period
is preferred to be set to 300s
or multiples of 300s
, such as
600s
and 900s
.
For more granular monitoring data you can enable detailed monitoring on the instance to get metrics every 1 minute. Please see
Enabling Detailed Monitoring for instructions
on how to enable detailed monitoring. With detailed monitoring enabled, period
in aws module configuration can be any number
larger than 60s
. Since AWS sends metric data to CloudWatch in 1-minute periods, setting metricbeat module period
less
than 60s
will cause extra API requests which means extra charges on AWS. To avoid unnecessary charges, period
is
preferred to be set to 60s
or multiples of 60s
, such as 120s
and 180s
.
Here is an example of aws metricbeat module configuration:
metricbeat.modules: - module: aws period: 300s metricsets: ["ec2"] access_key_id: '${AWS_ACCESS_KEY_ID}' secret_access_key: '${AWS_SECRET_ACCESS_KEY}' session_token: '${AWS_SESSION_TOKEN}' default_region: '${AWS_REGION:us-west-1}'
Metricsets
editThe following Metricsets are already included:
ec2
editBy default, Amazon EC2 sends metric data to CloudWatch every 5 minutes. With this basic monitoring, period
in aws module
configuration should be larger or equal than 300s
. If period
is set to be less than 300s
, the same cloudwatch metrics
will be collected more than once which will cause extra fees without getting more granular metrics. For example, in US East (N. Virginia)
region, it costs
$0.01/1000 metrics requested using GetMetricData. Please see AWS Cloudwatch Pricing
for more details. To avoid unnecessary charges, period
is preferred to be set to 300s
or multiples of 300s
, such as
600s
and 900s
. For more granular monitoring data you can enable detailed monitoring on the instance to get metrics every 1 minute. Please see
Enabling Detailed Monitoring for instructions
on how to enable detailed monitoring. With detailed monitoring enabled, period
in aws module configuration can be any number
larger than 60s
. Since AWS sends metric data to CloudWatch in 1-minute periods, setting metricbeat module period
less
than 60s
will cause extra API requests which means extra charges on AWS. To avoid unnecessary charges, period
is
preferred to be set to 60s
or multiples of 60s
, such as 120s
and 180s
.
The AWS module comes with a predefined dashboard. For example:
Example configuration
editThe aws module supports the standard configuration options that are described in Specify which modules to run. Here is an example configuration:
metricbeat.modules: - module: aws period: 300s metricsets: - "ec2" access_key_id: '${AWS_ACCESS_KEY_ID:""}' secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}' session_token: '${AWS_SESSION_TOKEN:""}' default_region: '${AWS_REGION:us-west-1}'
Metricsets
editThe following metricsets are available: