ElasticGPT: Empowering our workforce with generative AI

Elastic’s IT team plays a foundational role in accelerating technology initiatives. And over the past 18 months, the focus has been geared toward enabling generative AI technology across the organization to improve employee efficiency. This includes several homegrown generative AI tools and capabilities as well as new and emerging innovations added across our enterprise offering. 

Throughout this journey, multiple business functions came together to identify the opportunity to use generative AI for employee experience, but several questions arose as we evolved in this space. These included:

• How can Elasticsearch combine with generative AI to make it easier for employees to find information across our enterprise data sources and systems?

• How do we bridge private company information with the power of large language models (LLMs) to generate relevant results while maintaining security and confidentiality?  

• Can we build a scalable solution that serves as a platform for multiple use cases while offering us the flexibility to use multiple LLMs? And how can we promote domain-specific innovation by providing this as a managed service across our organization?

• How can we enable capabilities to address time-consuming and redundant requests by introducing self-service workflows that deliver capacity gains across teams? 

• How can we avoid technical debt by optimizing spend, managing risk, and mitigating tool sprawl? 

Our core objective was simple: Build an internal, private, and secure generative AI tool using the Elastic Search AI Platform that can benefit all Elasticians for information retrieval and knowledge discovery. 

Considering we are Elastic, we knew how Elasticsearch’s product capabilities would complement the AI capabilities we wanted to implement. That said, the immediate challenge was honing in on the most impactful use cases, defining our data strategy, and remaining laser-focused on our core vision.

With our goal to build a solution that would benefit all Elastic employees, we knew we needed to start by examining what data sources would make it into our minimum viable product (MVP). We narrowed it down to two criteria: 1) the data sources should have detailed information that would benefit all employees, and 2) the data needs to be up to date so that relevant, real-time information is retrieved in the outputs. We landed on two data sources for our MVP:

1. Confluence data: Our internal Confluence site, Elastic Wiki, is a comprehensive internal resource for all things related to Elastic, including detailed information on our products, teams, technology, processes, policies, and company culture. 

2. ServiceNow data: We use ServiceNow knowledge articles to help address questions across various topics like policies, usage instructions, troubleshooting tips, and requesting support from teams like IT and HR.

Identifying the two data sources was simple, but governance was not as easy. Why?  Like most organizations with petabytes of data, we had lots of information in various locations, and we weren’t confident about data accuracy. This multidimensional problem was critical to solve to make sure that the quality of response was not negatively impacted by “dirty” or “noisy” data.

To tackle our data challenges, we developed a step-by-step framework: 

• Step 1: Capture and organize data. This entailed taking inventory of our data sources, organizing our information, and defining how and where this data would be used.  

• Step 2: Ensure data accuracy and relevancy. We had to define what information was reliable, so we built data criteria that classified information as “stale” (e.g., information that has not been updated for a defined period, artifacts without owners) and put forward a plan to archive this stale information or update it. This was critical to making sure that our RAG-based approach and architecture would be successful. RAG can manage conflicting information by actively retrieving the most up-to-date and relevant information across data sources to answer a query. Focusing on our data quality allowed us to ensure that answers to queries were reliable and trustworthy. 
• Step 3: Set ourselves up for the future. While we planned to start with two data sources, we knew this would grow over time. Getting our data governance framework in place early would help us grow faster as we scale.

ElasticGPT — our generative AI employee assistant built on a RAG framework — is designed to help Elasticians retrieve relevant information from natural language queries and offers efficient ways to summarize information for day-to-day tasks. With the two main proprietary data sources and company-wide information from our internal tools — Confluence and ServiceNow — users can easily find answers instead of searching across multiple sources for information.  

In the past, someone like John would spend their time (and often others’) trying to find information quickly.

ElasticGPT is powered by multiple products and capabilities built on the Elastic Search AI Platform. This includes Elasticsearch, a vector database, semantic search, enterprise connectors, Elastic Observability, application performance monitoring (APM), and Kibana. Each of these capabilities plays a special role in developing and monitoring ElasticGPT and allows us to use a RAG technique to build a reliable and accurate solution that securely uses confidential and proprietary data. These capabilities on the Elastic Search AI Platform are now known as the Elastic AI Ecosystem. 

Each of our Search AI Platform’s capabilities on our enterprise offering plays a crucial role in building out this solution:

1. Elastic Cloud: Our IT organization uses Google Cloud Platform (GCP) and Microsoft Azure Services for existing infrastructure. We used Elastic Cloud on GCP to build ElasticGPT. 

2. Enterprise connectors: We used our managed connectors to ingest our data sources (Confluence and BigQuery for ServiceNow) into Elastic. 

3. Elasticsearch: Using Elasticsearch as a vector database, we could easily break down data via a “chunking” process to help generative AI handle large volumes of data and deliver effective responses. With semantic search and vector search, we could efficiently retrieve the most relevant answers for conversation context. And with Elasticsearch’s storage capabilities, we could store all conversations and associated metadata like time stamps and user feedback in real time. 

4. Elastic Observability: Using our Elastic Observability stack, we implemented APM to track the performance and health of ElasticGPT. This includes capturing response times, error rates, and resource utilization to help us identify and resolve bottlenecks impacting the user experience.

5. Information security: To keep our data secure and compliant with our security policies, we implemented robust security measures, including an SSO authentication for all Elasticians. 

6. Kibana: With access to APM’s real user monitoring (RUM) data, we can collect metrics — such as user engagement, total conversations, failures, reported chats, model usage, and other key details — to track usage, performance, and other KPIs for ElasticGPT. 

When building ElasticGPT, we used our Microsoft Azure OpenAI subscription to integrate LLMs, such as GPT-4o and GPT-4o-mini, into our solution. Elasticsearch retrieves the query context, which is then passed to these LLMs, enabling them to generate highly relevant and readable text answers. Elasticsearch also made it easy to store additional context with each interaction, including the specific LLM model used, conversation threads, source references, and user feedback. 

By deploying on Kubernetes, we were able to ensure scalability and reliability by taking advantage of automated scaling based on demand, zero-downtime deployments, and comprehensive monitoring through Elastic Observability.

In this first phase of ElasticGPT, Elasticians globally are using the generative AI experience to find relevant information. Since the first 90 days of launch, we successfully answered nearly 10K queries with ElasticGPT and saw a 99% satisfaction rate based on chat feedback. The business impact of ElasticGPT is often dependent on the particular case. However, across the organization, Elasticians are empowered with this self-service experience to boost efficiency. Not only does this help field redundant questions or requests across the organization, but it can also lead to reduced support tickets for shared service teams, such as HR and IT. 

There are many use cases where employees use this tool as part of their daily workflows to summarize information, analyze data sets, generate drafts, and simply spark their creativity. Here are some specific use cases we’re seeing from early usage across the organization:

1. Product enablement: New hires use ElasticGPT to learn about our products and feature capabilities while existing employees catch up with Elastic’s innovation velocity, especially new releases and launches.

As we evaluate what’s next on our generative AI roadmap, specifically for ElasticGPT, we are thinking about it in the following dimensions:

• Expanding our knowledge base: We’re looking to add incremental data sources to our knowledge base to provide access to a wider range of information. 

• Scaling with a managed API service: Building on our approach of a “central landing point,” we want to make it easier for our business groups to get started with generative AI. We’re developing a managed API service that teams can use to experiment in this space. 

• Building function-specific experiences: We plan to add specialized internal models to support function-specific use cases like in finance and legal with different models using our Elastic inference APIs. This allows access to fine-tuned models for specific functions and out-of-the-box options, ensuring flexibility and precision for diverse business needs. 

• Incorporating automation: We plan to see how we can incorporate agentic workflow automation for manual and routine tasks with predefined goals. 

Thinking about building a generative AI tool for your workplace? A successful release requires planning and preparation. Before you begin your build, your organization should define a set of guidelines and frameworks around how and where it's safe to use generative AI tools. These guidelines outline important principles like ensuring the protection of sensitive data as well as considering aspects like the copyright of the created material. 

Generative AI tools can be powerful assets, but they also bring a new set of risks that should be considered. Establishing your guidelines, data strategy, and frameworks upfront will help you navigate discussions around what you will be doing as well as what you will not be doing. In other words, you’ll be able to define what capabilities and data sources will be in the scope of your build(s) and how those outputs should be used. 

Getting started with Elastic’s generative AI capabilities is easy using our AI Playground. You can unlock powerful generative AI capabilities using LLMs with a free 14-day trial. Test out ingesting your data, build a proof of concept, and kick the tires on Elastic's machine learning and RAG capabilities. Deploy any data in any cloud — or multiple clouds — in real time, at scale.

Dive deeper into how to operationalize generative AI at your organization or get started in the AI Playground.