Open Security impact: Elastic AI Assistant

11-hand.jpg

Over the past couple of years, we’ve been discussing the benefits of an open and transparent approach to security — the idea that providing public access to details of our detection and prevention capabilities, code, documentation, and more, will bolster the security capabilities we’re able to provide to our customers. In this blog, we’ll be looking at some of the most recent ways our Open Security initiative has impacted our Elastic Security community, specifically with regard to the ever present topic of generative AI and large language models (LLM), such as those provided by OpenAI and Microsoft.

Elastic AI Assistant

Users of Endgame might be familiar with Artemis, the AI assisted chatbot. Artemis provides Endgame users with an interactive chat experience that enables security analysts of all experience levels to easily perform guided investigations on endpoint data.

The Elastic AI Assistant, available with the 8.8.1 release of the Elastic Stack, builds on the capabilities of Artemis by integrating the power of LLMs directly into the analyst workflow. This arms every analyst with tools to be more efficient at triage, analysis, and remediation in any environment. 

How does this relate to open security? What really sets us apart from the competition is the public availability of all things Elastic Security and how that knowledge is utilized by existing and new LLMs to make the Elastic AI Assistant an effective tool to modernize your security operations.

Data, detections, and AI — Oh my!

Let's start with data

Elastic Security is built on a foundation of the Elastic Common Schema (ECS), an open source schema that defines a common set of fields and data types for logs and metrics in Elasticsearch®. In short, source ip is always source.ip, process name is always process.name, etc.

In April 2023, Elastic contributed ECS to Open Telemetry and committed to joint development of a common schema. By contributing ECS to OpenTelemetry, we are working to create a mature common schema for metrics, logs, traces, and security events. Together with OTel, we will continue to develop and support that common schema going forward.

Contributing to ECS is open to everyone, and it is actually how I got started with Elastic five years ago. That open ethos encourages users, customers, and developers — all with varied domain expertise and requirements — to contribute their experience to the areas they are most passionate about.

Since ECS is completely open, LLMs trained on publicly available data tend to be highly knowledgeable about how Elastic Security stores and references data. For instance, prompting the Elastic AI Assistant for help with a query to analyze nginx network traffic quickly provides a novice user with the exact syntax required, as well as a full explanation of the ECS fields used in the query.

Elastic AI Assistant
Elastic AI Assistant writes and explains a query for Nginx traffic

Looks like we've got another mystery on our hands

Analyzing the never ending flood of suspicious events can be taxing to the most experienced analysts, and new hires often lack the security experience and institutional knowledge of business environments necessary to quickly triage and respond to threats. Elastic AI Assistant helps new and experienced analysts triage events faster, using information found in our public detection repository, context from alert rules, risk ratings, and MITRE ATT&CK® Tactic and Technique information. Summarizations can even include investigation suggestions based on the specific context of the alert. 

In the screenshot below, Elastic AI Assistant has summarized the “Suspicious DGA DNS Request” alert and provided initial investigation suggestions to analyze the potential implications of this alert.

Elastic AI Assistant summarizes an Elastic Security ML
Elastic AI Assistant summarizes an Elastic Security machine learning based alert identifying potential DGA traffic in DNS logs

Does it slice and dice, too?

Well, no, but it can help: 

  • Write detection rules 
  • Simplify SIEM migrations with fast and accurate rule conversions from other query languages 
  • Provide workflow suggestions for things like custom dashboards or ingest pipelines
  • Provide suggestions for which Agents to use for ingesting specific sources 

Customizable quick prompts enable our users to save and reuse the prompts that provide the most effective responses. In the example below, the Elastic AI Assistant has built an Event Query Language (EQL) rule from a general use case of detecting “data exfiltration attempts on linux systems.”

Elastic AI Assistant writes an EQL correlation rule
Elastic AI Assistant writes an EQL correlation rule for “data exfiltration on linux systems”

The power of generative AI and Open Security

Half the battle with generative AI is making sure it’s been trained on the right data. Our commitment to Open Security has made it as easy as possible for our clients to leverage the power of generative AI with the Elastic AI Assistant in their day-to-day operations, boosting the capabilities of novice and experienced users alike. 

Open Security means more to us than just a public GitHub repo. The combination of a common taxonomy to normalize events and alerts, public availability of all things Elastic Security, and the power of generative AI makes new levels of optimization and efficiency possible. Whether its accelerating architecture and migration to shorten ROI time frames, arming every analyst with the context to accelerate triage and reduce MTTR, or enabling novice and expert users alike to simplify daily operations; Open Security is a key component for modernizing your security operations with Elastic.

Get started today

Elastic AI Assistant is available now to all users. For more information on how to integrate it with your model of choice and begin harnessing the power of generative AI, read our documentation.

If you’d like to give it a spin for yourself, head to cloud.elastic.co and sign up for a free 14 day trial.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use. 

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.