New Elastic and Amazon S3 Storage Lens Integration: Simplify management, control costs, and reduce risk

As data and applications grow at an exponential rate — they produce logs, metrics and traces at a rapid pace — making storage and management even more complex. The new Elastic and Amazon Simple Storage Service (Amazon S3) Storage Lens integration provides customers with a complete organization-wide view of Amazon S3 usage and activity metrics alongside their other AWS and on-premises data sets.

SREs and DevOps engineers can now use the powerful search and analytics capabilities of the Elastic search platform — including dashboarding capabilities, built-in machine learning analysis, and threshold-based alerts — to identify trends and correlate their Amazon S3 Storage Lens data with other performance indicators across their environment. These insights can also be leveraged to optimize storage costs by identifying and migrating infrequently accessed objects to less expensive storage.

Let’s deep dive further into Elastic’s integration with S3 Storage Lens and how to easily ship S3 Storage Lens metrics to Elastic Cloud.

Get crystal clear insight into storage usage

Amazon S3 delivers cost-efficient and flexible storage; however, optimizing the utilization and cost-efficiency of ever-growing storage buckets across a large organization can present challenges. With Elastic, DevOps and infrastructure teams can view, aggregate, and analyze S3 Storage Lens metrics alongside security events, application metrics, and system logs to make informed decisions about storage optimization.

Elastic Observability delivers unified visibility across the entire cloud and on-premises ecosystem so users can monitor and react to events happening anywhere in their environment. Elastic brings logs, metrics, and traces together, at scale in a single stack with machine learning to help detect performance issues and surface root causes — before end users are affected.

• View metrics from S3 Storage Lens alongside other AWS data sets — like Amazon Elastic Compute Cloud (Amazon EC2) and Amazon S3 Glacier — as well as other cloud and on-premises data sets.
• Use time series modeling to detect anomalies in current data and forecast trends based on historical data.
• Get notified if Amazon S3 object storage metrics goes above or below a certain threshold within a set time interval using threshold alerts to determine storage over- and under-utilization, from the organizational-level down to the Amazon S3 account or bucket.
• Keep costs in check and under budget by identifying archive-ready objects that can be migrated to other lower-cost storage like Amazon S3 Glacier or Amazon S3 Glacier Deep Archive.
• Optimize performance and availability with extensive capabilities across infrastructure monitoring, APM, and end user experience monitoring, including end-to-end distributed tracing, error tracking, service dependency mapping, uptime, and API health.

Users can install the Elastic and S3 Storage Lens integration straight from the Kibana web UI, which contains prebuilt alerts and interactive dashboards that help organizations quickly get the most value out of their data. Elastic offers different analysis options, from customizable prebuilt dashboards and fully customizable dashboards to use case-specific UIs that are tailored for practitioner workflows.

Detect, investigate, and respond to evolving threats

Since Elastic offers a common schema and single repository, the same observability data from S3 Storage Lens and other data sets can also be used for extended detection and response (XDR) to drive mean time to detection towards zero. Elastic Security brings together SIEM and endpoint security, allowing organizations to ingest and retain large volumes of data from diverse sources, store and search data for longer, and augment threat hunting with detections and machine learning. Eliminate data silos, reduce alert fatigue, and ready the organization to stop threats quickly, and at cloud scale.

Store data, logs, and metrics cost-effectively

Most organizations retain historical data to better understand patterns and improve their standard of operations, but often are challenged to reduce storage costs while maintaining data accessibility. Good news: Elastic enables organizations to retain large amounts of historical data in low-cost object storage like Amazon S3 — while keeping it fully active and searchable. This allows organizations to safely keep their AWS and on-premises logs, events, and metrics for usage, analysis, and compliance purposes. Keep data — at any granularity, for any length of time — and then scale as the data grows. Data management and tiering is automated through index lifecycle management and autoscaling capabilities, based on the organization’s data performance, resilience, and retention requirements.

Getting started with Elastic and Amazon S3 Storage Lens

This reference architecture diagram is a relatively standard depiction of shipping multiple S3 buckets metrics from Cloudwatch metrics to Elastic Cloud.

Implementation Steps:

Go to Management->Integrations in Kibana UI and search for S3 Storage Lens. Click the AWS S3 Storage Lens integration to see more details, click Add AWS S3 Storage Lens and provide the appropriate configuration, and then click Save integration.

AWS S3 Storage Lens metrics are published into CloudWatch by AWS once every 24 hours. Elastic agents will collect data every day, by default, or less frequently based on agent configuration.

Explore collected metrics for various custom use cases using Kibana Lens.

Wrapping up

Ready to reduce operational complexity, and analyze Amazon S3 object storage data with other logs, metrics, and traces together from a single pane of glass? This streamlined integration with AWS is just one example of how Elastic is constantly iterating to provide customers with a frictionless experience, enabling them to run where and how they want.

For more information on other AWS integrations, visit the Elastic integrations library. Or, start your own 7-day free trial by signing up via AWS Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on AWS around the world. Your AWS Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with AWS.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.