Skip to main content
Login
Start free trialContact Sales

Elastic’s new Custom Threat Intelligence integration

By

Chema Martinez

145070_-_Blog_header_image_Switching_from_the_Java_High_Level_(1).jpg

In modern cybersecurity, organizations must be prepared to handle a wide variety of threat intelligence data to stay ahead of emerging threats. To support this need, Elastic introduces the Custom Threat Intelligence integration, specifically designed for ingesting indicators of compromise (IoCs) in the STIX 2.1 format. By converting STIX data into the Elastic Common Schema (ECS), this integration makes it simple to bring threat data from diverse sources directly into Elastic for unified analysis and detection workflows.

This integration gives users the flexibility to gather threat intelligence from a range of sources, such as STIX-compliant APIs, TAXII 2.1 servers, and even log files in air-gapped environments.

Key features of the Custom Threat Intelligence integration

The Custom Threat Intelligence integration is built for ingesting and converting STIX data into ECS, allowing users to merge threat intelligence from various external sources into Elastic effortlessly. Below are the integration’s core features:

  • Conversion of STIX indicators to ECS: The integration is designed to ingest STIX 2.1 format data, converting each indicator into ECS-compatible fields. A default pipeline handles common STIX data mappings automatically, but users can extend functionality by adding custom pipelines for unique STIX indicators or formats as needed.

  • CEL input for API communication: The integration uses the Common Expression Language (CEL) input to enable flexible communication with STIX APIs. Through CEL programs, the integration can connect to STIX-compliant APIs and retrieve relevant threat intelligence data. This approach allows for highly customizable data ingestion, handling HTTP requests, responses, and data processing in a structured way.

  • Built-in TAXII 2.1 client for simplified data retrieval: For organizations relying on TAXII servers, the integration includes a built-in CEL program that acts as a TAXII 2.1 client. This client facilitates connections to TAXII 2.x servers, automatically retrieving STIX data with minimal configuration — just the TAXII server URL and any required authentication credentials.

  • Log file support for air-gapped environments: Recognizing the needs for air-gapped environments, the integration also supports reading STIX-based threat intelligence from log files. This enables organizations in isolated environments to incorporate threat intelligence without requiring direct internet access.

  • Built-in dashboard for visualizing indicators: The integration comes with a built-in dashboard to visualize ingested threat intelligence indicators, making it easy for users to view and interpret the threat data collected. This dashboard provides an organized view of indicators, helping analysts quickly assess threat intelligence patterns and enhance awareness.

Custom Threat Intelligence dashboard: IoCs overview
Custom Threat Intelligence dashboard: IoCs overview

Use case: Understanding a CEL program for custom API ingestion

For STIX APIs that do not follow a specific protocol like TAXII, CEL programs allow you to customize the retrieval and processing of threat data. Using CEL, you can define HTTP headers, query parameters, pagination, and payload processing to meet the requirements of the API — ensuring accurate and complete ingestion of threat intelligence.

To show how CEL programs work within the Custom Threat Intelligence integration, let’s break down a sample CEL program. This example shows how a CEL program can be used to retrieve data from a custom STIX API using the GET method and incorporates flexible authentication options based on the presence of an API key or Basic Authentication credentials.

state.with(
  request(
    "GET",
    (has(state.initial_interval) && state.initial_interval != "") ?
      (
        state.url.trim_right("/") + "?" + {
          "start": [(now() - duration(state.initial_interval)).format(time_layout.RFC3339)],
        }.format_query()
      )
    :
      state.url
  ).with(
    {
      "Header": {
        "Authorization": (has(state.api_key) && state.api_key != "") ?
          ["Bearer " + string(state.api_key)]
        : (state.?username.orValue("") != "" && state.?password.orValue("") != "") ?
          ["Basic " + (state.username + ":" + state.password).base64()]
        :
          []
      }
    }
  ).do_request().as(resp, (resp.StatusCode == 200) ?
    bytes(resp.Body).decode_json().as(body,
      {
        "events": body.objects.map(e,
          {
            "message": e.encode_json(),
          }
        ),
        "url": state.url,
        "api_key": state.?api_key.orValue(""),
        "username": state.?username.orValue(""),
        "password": state.?password.orValue(""),
      }
    )
  :
    {
      "events": {
        "error": {
          "code": string(resp.StatusCode),
          "id": string(resp.Status),
          "message": "GET:" +
          (
            (size(resp.Body) != 0) ?
              string(resp.Body)
            :
              string(resp.Status) + " (" + string(resp.StatusCode) + ")"
          ),
        },
      }
    }
  )
)

Breaking down the CEL program

  1. Request construction: The CEL program constructs an HTTP GET request. If the initial_interval is set, the request URL appends a query parameter specifying the start time — which is calculated from the current time minus the initial interval. This allows the integration to fetch only recent threat data.

  2. Header setup: The program dynamically builds the Authorization header to support different types of authentication:

    1. If an api_key is provided, it creates a Bearer token using Authorization: Bearer <api_key>.

    2. If the API requires Basic Authentication instead, it checks for username and password fields, encoding them in base64 as required by the Basic <encoded_credentials> format.

    3. If neither an API key nor username and password are available, the Authorization header is left empty.

  3. Data processing on successful response: If the response status is 200 (OK), the program decodes the JSON body of the response and maps each STIX object as an event in Elastic.

  4. Error handling: If the response status is not 200, the program generates an error event that includes the status code, response status, and any response body content. This error handling provides visibility into any issues with data retrieval.

CEL initial state

Each CEL program requires an initial state definition — which includes key parameters, such as the API URL, authentication tokens, and retrieval intervals. Following the example above, the CEL state for this case could look like this.

url: "https://stix-server.example.com"
username: "<USERNAME>"
password: "<PASSWORD>"
initial_interval: "120h"

When adding a CEL program to the Custom Threat Intelligence integration, ensure the Enable TAXII 2.1 toggle is disabled, and paste the CEL program and its initial state into the designated blocks in the integration’s configuration.

CEL configuration for the integration
CEL configuration for the integration

This example provides a generic template for using CEL programs to retrieve threat intelligence from STIX APIs. However, users will likely need to adapt the program based on the specific requirements of their STIX server — paying particular attention to query parameters, required headers, and the structure of the response body.

Bring threat data into your security infrastructure easily

Elastic’s Custom Threat Intelligence integration makes it easy to bring threat data into your security infrastructure — giving you more insights to catch and respond to threats faster. With support for STIX 2.1 indicators, connections to TAXII servers, and flexible CEL configurations, this integration is built to fit your specific needs — turning threat intelligence indicators into ECS format so that it works seamlessly across the Elastic Search AI Platform.

We’re excited for you to try out these features, explore new use cases, and share your feedback to help us continue evolving threat intelligence capabilities within Elastic.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Sign up for Elastic Cloud free trial

Spin up a fully loaded deployment on the cloud provider you choose. As the company behind Elasticsearch, we bring our features and support to your Elastic clusters in the cloud.

Start free trial