Elastic is accelerating the adoption of AI-driven log analytics by automating the ingestion of custom logs, which is increasingly important as the deployment of GenAI-based applications grows. These custom data sources must be ingested, parsed, and indexed effortlessly, enabling broader visibility and more straightforward root cause analysis (RCA) without requiring effort from Site Reliability Engineers (SREs). Achieving visibility across an enterprise IT environment is inherently challenging for SREs due to constant growth and change, such as new applications, added systems, and infrastructure migrations to the cloud. Until now, the onboarding of custom data has been costly and complex for SREs. With automatic import, SREs can concentrate on deploying, optimizing, and improving applications.
Automatic Import uses generative AI to automate the development of custom data integrations, reducing the time required from several days to less than 10 minutes and significantly lowering the learning curve for onboarding data. Powered by the Elastic Search AI Platform, it provides model-agnostic access to leverage large language models (LLMs) and grounds answers in proprietary data through retrieval augmented generation (RAG). This capability is further enhanced by Elastic's expertise in enabling observability teams to utilize any type of data and the flexibility of its Search AI Lake. Arriving at a crucial time when organizations face an explosion of applications and telemetry data, such as logs, Automatic Import streamlines the initial stages of data migration by simplifying data collection and normalization. It also addresses the challenges of building custom connectors, which can otherwise delay deployments, issue analysis, and impact customer experiences.
Enhancing AI Powered Observability with Automatic Import
Automatic Import builds on Elastic Observability’s AI-driven log analytics innovations—such as anomaly detection, log rate and pattern analysis, and Elastic AI Assistant, and further automates and simplifies SRE’s workflows. Automatic Import applies generative AI to automate the creation of custom data integrations, allowing SREs to focus on logs and other telemetry data. While Elastic provides over 400+ prebuilt data integrations, automatic import allows SREs to extend integrations to fit their workflows and expand visibility into production environments.
In conjunction with automatic import, Elastic is introducing Elastic Express Migration, a commercial incentive program designed to overcome migration inertia from existing deployments and contracts, providing a faster adoption path for new customers.
Automatic Import leverages Elastic Common Schema (ECS) with public LLMs to process and analyze data in ECS format which is also part of OpenTelemetry. Once the data is in, SRE’s can leverage Elastic’s RAG-based AI Assistant to solve root cause analysis (RCA) challenges in dynamic, complex environments.
Configuring and using Automatic Import
Automatic Import is available to everyone with an Enterprise license. Here is how it works:
-
The user configures connectivity to an LLM and uploads sample data
-
Automatic Import then extrapolates what to expect from the data source. These log samples are paired with LLM prompts that have been honed by Elastic engineers to reliably produce conformant Elasticsearch ingest pipelines.
-
Automatic Import then iteratively builds, tests, and tweaks a custom ingest pipeline until it meets Elastic integration requirements.
Automatic Import powered by the Elastic Search AI Platform
Within minutes, a validated custom integration is created that accurately maps raw data into ECS and custom fields, populates contextual information (such as
Automatic Import currently supports Anthropic models via Elastic’s connector for Amazon Bedrock, and additional LLMs will be introduced soon. It supports JSON and NDJSON-based log formats currently.
Automatic Import workflow
SREs are constantly having to manage new tools and components that developers add into applications. Neo4j, is a database that doesn’t have an integration in Elastic. The following steps walk you through how to create an integration for Neo4j with automatic import:
- Start by navigating to Integrations->Create new integration.
- Provide a name and description for the new data source.
- Next, fill in other details and provide some sample data, anonymized as you see fit.
- Click “Analyze logs” to submit integration details, sample logs, and expert-written instructions from Elastic to the specified LLM, which builds the integration package using generative AI. Automatic Import then fine-tunes the integration in an automated feedback loop until it is validated to meet Elastic requirements.
- Review what automatic Import presents as recommended mappings to ECS fields and custom fields. You can easily adjust these settings if necessary.
- After finalizing the integration, add it to Elastic Agent or view it in Kibana. It is now available alongside your other integrations and follows the same workflows as prebuilt integrations.
- Upon deployment, you can begin analyzing newly ingested data immediately. Start by looking at the new Logs Explorer in Elastic Observability
Accelerate log-analytics with automatic import
Automatic Import lowers the time required to build and test custom data integrations from days to minutes, accelerating the switch to AI-driven log analytics. Elastic Observability pairs the unique power of Automatic Import with Elastic’s deep library of prebuilt data integrations, enabling wider visibility and fast data onboarding, along with AI-based features, such as the Elastic AI Assistant to accelerate RCA and reduce operational overhead.
Interested in our Express Migration program to level up to Elastic? Contact Elastic to learn more.
The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.
In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.
Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.