Custom AWS Logs

Collect raw logs from AWS S3 or CloudWatch with Elastic Agent.

What is an Elastic integration?
Version
1.4.1 (View all)
Compatible Kibana version(s)
8.12.0 or higher
Supported Serverless project types

What's this?
Security
Observability
Subscription level
What's this?
Basic
Level of support
What's this?
Elastic

The custom AWS input integration offers users two ways to collect logs from AWS: from an S3 bucket (with or without SQS notification) and from CloudWatch. Custom ingest pipelines may be added by adding the name to the pipeline configuration option, creating custom ingest pipelines can be done either through the API or the Ingest Node Pipeline UI.

Collecting logs from S3 bucket

When collecting logs from S3 bucket is enabled, users can retrieve logs from S3 objects that are pointed to by S3 notification events read from an SQS queue or directly polling list of S3 objects in an S3 bucket.

The use of SQS notification is preferred: polling list of S3 objects is expensive in terms of performance and costs and should be preferably used only when no SQS notification can be attached to the S3 buckets. This input integration also supports S3 notification from SNS to SQS.

SQS notification method is enabled setting queue_url configuration value. S3 bucket list polling method is enabled setting bucket_arn configuration value and number_of_workers value. Both queue_url and bucket_arn cannot be set at the same time and at least one of the two value must be set.

Collecting logs from CloudWatch

When collecting logs from CloudWatch is enabled, users can retrieve logs from all log streams in a specific log group. filterLogEvents AWS API is used to list log events from the specified log group. Amazon CloudWatch Logs can be used to store log files from Amazon Elastic Compute Cloud(EC2), AWS CloudTrail, Route53, and other sources.

Changelog

VersionDetailsKibana version(s)

1.4.1

Bug fix View pull request
Remove ingest pipeline input option.

8.12.0 or higher

1.4.0

Enhancement View pull request
Update file_selectors field to be able to receive multiline configuration

8.12.0 or higher

1.3.1

Bug fix View pull request
Update max_number_of_messages parameter description

8.12.0 or higher

1.3.0

Enhancement View pull request
Improve clarity and validity of system testing.

8.12.0 or higher

1.2.0

Enhancement View pull request
Add a custom settings variable to allow passing additional settings to the input.

8.12.0 or higher

1.1.0

Enhancement View pull request
Update the package format_version to 3.0.2 and enable 'secret' for the sensitive fields

8.12.0 or higher

1.0.0

Enhancement View pull request
Release package as GA.

8.0.0 or higher

0.6.0

Enhancement View pull request
Update the package format_version to 3.0.0.

—

0.5.1

Enhancement View pull request
Remove duplicated number_of_workers settings

—

0.5.0

Enhancement View pull request
Add permissions to reroute events to logs-- for generic datastream

—

0.4.0

Enhancement View pull request
Add multiline support for using s3 input

—

0.3.3

Enhancement View pull request
Added categories and/or subcategories.

—

0.3.2

Enhancement View pull request
Add required field number of workers to support non aws buckets, and add default value.

—

0.3.1

Bug fix View pull request
Add latency config parameter for aws-cloudwatch input

—

0.3.0

Enhancement View pull request
Expose Default Region setting to UI

—

0.2.5

Bug fix View pull request
Set default endpoint to empty string

—

0.2.4

Bug fix View pull request
Fix proxy URL documentation rendering.

—

0.2.3

Bug fix View pull request
Fix misspelling of Log Stream Prefix variable in manifest for aws-cloudwatch input

—

0.2.2

Bug fix View pull request
update readme file

—

0.2.1

Bug fix View pull request
Add kibana version constraint

—

0.2.0

Enhancement View pull request
Move s3 input and cloudwatch input into the same generic data stream

—

0.1.0

Enhancement View pull request
initial release

—

On this page