How we built Automatic Import, Attack Discovery, and Elastic AI Assistant using LangChain

Elastic Security is delivering innovative AI features for security teams, accelerating migration from legacy SIEM to AI-driven security analytics. We’ve released three capabilities — Automatic Import, Attack Discovery, and Elastic AI Assistant — that apply generative AI to expedite labor-intensive SecOps tasks, and we’re just getting started.

Leveraging the Elastic Search AI Platform and generative AI, Elastic partners with LangChain, the de facto generative AI orchestration library, to build and deliver these features. This technical blog shares the engineering underpinnings of this work.

ES|QL, our new piped query language, added a whole new set of threat hunting and detection capabilities for Elastic Security users. Without AI, adopting it required learning the details of the query syntax and its functions. We wanted to simplify this process by enabling Elastic AI Assistant to generate ES|QL queries from natural language questions.

Facilitating ES|QL generation was one of the first core use cases for the integration of LangChain. Elastic AI Assistant generates ES|QL leveraging retrieval augmented generation (RAG) to provide rich context to the chosen LLM, enabling generation of a query based on the user’s input. LangGraph, a controllable agent orchestration framework, powers the end-to-end generation workflow.

Applying a modified version of the native Elasticsearch LangChain vector store component as part of the ES|QL generation graph allowed the team to leverage the Elastic Search AI Platform to retrieve the vectorized content necessary to formulate the query.

Below is a screenshot of LangSmith visualizing a trace of the resulting ES|QL generation LangGraph.

Automatic Import leverages LangGraph to generate the resulting integration package. With the combination of LangGraph and LLMs, users are now able to simply and quickly build stateful workflows. Below is a visual representation of the graph that powers Automatic Import.

Elastic Security users have the freedom to integrate the solution’s generative AI features with their LLM of choice. With Elastic’s open inference API and LangChain’s extensive chat model ecosystem, the Elastic Security team is quickly expanding customer LLM options.

Elastic Security is at the forefront of propelling security operations workflows with generative AI, leveraging the Elastic Search AI Platform and our partnership with LangChain. In parallel, Elastic Observability harnesses the Elastic Search AI Platform to deliver comprehensive LangChain tracing, along with end-to-end application tracing, logging, and metrics analysis. Discover how you can leverage Elastic Observability with OpenTelemetry to trace your LangChain applications effectively.

Interested in the impact? Read EMA’s views on AI-driven security analytics from Elastic Security.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use. 

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.