Up to 70% metrics storage savings with TSDS enabled integrations in Elastic Observability

The latest versions of Elastic Observability’s most popular observability integrations now use the storage cost-efficient time series index mode for metrics by default. Kubernetes, Nginx, System, AWS, Azure, RabbitMQ, Redis, and more popular Elastic Observability integrations are time series data stream (TSDS) enabled integrations.

Elastic^® released the time series data stream functionality to GA in 8.7. Elastic time series data stream (TSDS) stores metrics in indices optimized for a time series database (TSDB), which is optimized to store time series metrics. Elastic’s also got a significant optimization in 8.7 with additional features, reducing storage costs for metrics in the region of 70%. Read how TSDS works and how we use it in Elastic Observability, as well as how you can use it for your own metrics.

The three features for TSDB that were made GA in 8.7 are:

• Downsampling reduces the footprint of time series by storing them at lower granularity, giving you greater control in deciding how much historical metric data can be stored within your budget. And since downsampling “just works” in Kibana^® with 8.7, it also speeds up visualizations using Lens, Timelion, or TSVB on large data sets.
• Time series data streams (TSDS) leverage specific organization of time series data (e.g., by timestamp and dimension fields) to sort and store metric data more efficiently — 30% less disk space in our benchmarks!
• Synthetic source reduces storage space by not saving the original document data in _source, and instead reconstructing it from doc_values when needed. We observed space saving of 40%, depending on index configuration.

By leveraging Elasticsearch's Time Series Data Stream (TSDS) for our integrations, Elastic Observability helps realize storage savings for your key metrics data being ingested.

At the time of the release of 8.9, upwards to 40% of the Elastic integrations have a version released to use time series index mode out of the box. These include but are not limited to: Kubernetes, Nginx, System, AWS, Kinesis, Lambda, and most of the integrations that collect large numbers of metrics.

When using the TSDS enabled version for an integration collecting metrics, the benefits that you can realize are the following:

• Up to 70% less disk space: With TSDS seamlessly enabled in the integration for your platforms, you can enjoy up to a significant 70% reduction in disk storage for your metrics data. This reduction translates to cost savings and increased scalability for your infrastructure. 
• Streamlined data management: TSDS simplifies the storage and retrieval of your time stamped metrics data, making it effortless to organize and analyze your valuable insights. 
• Out-of-the-box functionality: Thanks to the native integration of TSDS, you can leverage its powerful features without any user input or additional configuration. The integration seamlessly optimizes storage efficiency for your metrics data, allowing you to focus on deriving meaningful insights.

This process of enabling integrations to use time series index mode will continue, and these will be released outside of the Elastic release cycle, essentially updated when ready. This will apply to Elastic Cloud, as any on-prem or self-managed cloud deployments will have to wait for the next release to upgrade.

When you add a document to a TSDS, Elasticsearch adds the document to the appropriate backing index based on its @timestamp value. As a result, a TSDS can add documents to any TSDS backing index that can receive writes. This applies even if the index isn’t the most recent backing index.

Applications are scaling and data is growing at an exponential rate, along with the cost to store it. Most organizations need to make a difficult decision of what data to keep or delete to stay under budget. With Elastic’s optimizations, your usage of cloud storage (such as S3) will be lowered and reduce the need to move data to “cold” storage. 

This will naturally allow you to scale metrics storage for more longer-term metrics data, which will help analyze patterns (long-term analysis), reduce MTTx, and improve your application’s overall performance. Read more details on how TSDS works. 

All you have to do is upgrade the integration version (with the upgrade integration policies selected) to the time series enabled version. This will unlock the time series indexing mode going forward!

The latest version of the following integrations is now TSDB enabled:

• ActiveMQ
• Apache
• AWS
  • Billing, Dynamod, EBS, ECS, ELB, Firewall, Kinesis, Lambda, Nat Gateway, RDS, Redshift, S3, SNS, SQS, Transit Gateway, Usage, VPN, S3 Storage Lens
• Couchdb
• Docker
• Elasticsearch
• IBM MQ
• IIS
• Influxdb
• Kafka
• Kubernetes
• Microsoft SQL server
• Nginx
• Oracle
• RabbitMQ
• Redis
• System
• Vsphere

Get started today and experience the benefits of storing your metrics in time series mode in the most popular and powerful store for logs!

For more insights, including a benchmark comparison for storing metrics using time series versus standard data stream, visit this article.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.