Serve more with Serverless

As we think about the next decade, we recognize the need for a simpler user experience that still delivers lightning-fast performance. We know many Elastic users want full control to deploy and scale, but others want more simplicity. SOC analysts want to protect their organizations, not scale shards for better threat detection. Developers want to build search applications, not tune infrastructure for faster querying. SREs want to ensure reliability online, not set configurations to help minimize downtime. We may love managing clusters, but you don't have to! Elastic’s serverless architecture removes the operational responsibility, so you can say goodbye to managing clusters, configuring shards, scaling, and setting up ILM. Just bring your data and queries, and the platform handles all the scaling and management. 

Tired of hearing that you can't have faster scalability with longer data retention periods and still balance costs and reduce complexity? Well, now you can. For many workloads, both scale and speed matter — whether it's investigating threats like SolarWinds, which have a long dwell time, identifying the root cause of an outage across hundreds of services, or using vector search to power generative AI workloads with retrieval augmented generation. 

That’s why our serverless architecture is based on a redesigned and reimagined Elasticsearch, which entirely decouples compute from storage and relies on object storage. Cloud object stores offer cost-effective scalability but introduce latency, requiring new techniques for speed. Thankfully, our years-long experience in optimizing Elasticsearch and Lucene index data structures for efficient caching, combined with enhanced query-time parallelization, overcomes this latency challenge. This means you can enjoy both speed and scale with built-in controls to easily balance speed and cost.

Elastic’s new serverless architecture marks a significant re-engineering of Elasticsearch. It was built to take advantage of the latest cloud-native services and deliver optimized product experiences with hassle-free administration. It provides the storage capacity of a data lake but with the fast search performance synonymous with Elasticsearch, and operational simplicity with hands-off cluster management and scale. The architecture is built on four key principles:

• Decoupled compute and storage
• Separate search and indexing tiers
• Inexpensive object storage as the system of record
• Low latency querying

Rather than relying on primary and replica instances to manage multiple workloads, Elastic’s serverless architecture supports distinct indexing and search tiers. This separation means workloads can be scaled independently and hardware can be selected and optimized for each use case. 

Additionally, this approach effectively addresses the persistent issue of search and indexing workloads interfering with each other. This makes it easier to optimize both performance and spend for any search use case or workload. This property is important to high-volume logging and security users who want to prevent heavy searches from disrupting indexing operations, and search users who want to use heavy index-time features for better relevance and search performance without affecting their search performance.

The serverless architecture relies on inexpensive object storage for greater scale while reducing storage costs. By taking advantage of object storage for persistence, Elasticsearch no longer needs to replicate indexing operations to one or more replicas for durability, thereby reducing indexing cost and data duplication. Instead, segments are persisted and replicated through object storage. This creates efficiencies for various requirements. For example, it cuts down storage expenses for the indexing tier by minimizing the data stored on local disks. The serverless architecture indexes directly to the object store, so only a fraction is retained as local data. For scenarios involving append-only operations, only specific metadata needs to be preserved for indexing, resulting in a substantial reduction in the local storage needed for indexing purposes.

Object stores can support massive amounts of data but aren't known for speed or low latency. So how does Elastic use object storage and maintain great query performance? Well, we’ve introduced several new capabilities to deliver fast performance. Segment level query parallelization reduces latency when retrieving data from the object store. This enables more requests to be quickly pushed to object stores like S3 when data is not in the local cache. Caching has also been made "smarter" through reusability and leveraging the optimal Lucene index formats for each type of data. These are just some of the novel capabilities that result in significant performance improvements in both the object store and caching layers.

We’re also taking this opportunity to build custom products for the serverless architecture for search, observability, and security. The purpose is to optimize for the unique needs of each workflow with a streamlined user experience. This includes faster and continuous onboarding, tighter integration of capabilities, and optimizing custom interfaces for the work of each use case. Notable highlights of each product include:

• Search: The serverless search experience focuses on ensuring developers can create exceptional search experiences out of the box, quickly and easily. APIs are front and center, combined with easy ways to ingest and bring data into Elasticsearch. These pipelines have been simplified to allow transformation and other tasks to be completed quickly. New language clients like Java, .NET, Python, and others have been created to reduce the initial learning curve and the steps required to complete tasks, along with in-line documentation — collectively creating a streamlined developer experience that helps developers get value more quickly.