The Custom Threat Intelligence package is an integration designed to ingest threat intelligence IOCs in the STIX 2.1 format and convert them into the Elastic Common Schema (ECS) for seamless ingestion into Elasticsearch. It has been delivered to ingest threat intelligence data for those APIs that do not currently have an existing integration.

The integration comes with a default pipeline that automatically maps standard STIX 2.1 data into ECS fields. However, it also offers the flexibility to handle custom STIX data by allowing users to add custom pipelines accordingly.

Key features

Supported data sources

RESTful API:

• Connects to public or private RESTful APIs that provide threat intelligence in STIX 2.1 format.
• Supports standard HTTP methods for data retrieval, including GET and POST.

TAXII 2.1 Protocol:

• Acts as a TAXII client to connect to TAXII 2.x servers, enabling the collection of threat intelligence feeds in STIX format.
• It only supports collection models for TAXII data retrieval.

Log files:

• Ingests threat intelligence indicators provided in logfile, for air-gapped environments.

STIX 2.1 Compliance

Indicators are categorized based on their type and stored for further processing. Currently, the supported IOCs types are:

• Autonomous System
• Domain Name
• Email
• File
• IPv4
• IPv6
• URL
• Windows Registry
• x509 Certificates

The default pipeline is able to ingest other types of indicators, although they are not 100% mapped into ECS.

Configuration guidelines

Due to the lack of standards from some TI providers, it is possible that some extra configuration is required for certain use cases.

When connecting to STIX APIs, by default, the integration provides a native way of acting as a TAXII client. Therefore, for collecting data from TAXII 2.x servers no extra configuration is needed apart from providing the server URL, and authentication credentials when needed.

However, for APIs that don't follow a specific communication protocol. The correct ingestion of STIX data would require:

• Add a CEL program where API specifications are met. Pay special attention to HTTP headers, query parameters, pagination, and the processing of the payload.
• Add a initial state to be provided to the program. Generally, it would include the API URL, authentication parameters and intervals. More information can be found in the documentation.

By default the integration only supports STIX 2.1 indicators. This means that to process IOCs in other formats, the Restrict STIX 2.1 format option must be disabled, and a custom pipeline added to map the indicators correctly.

Expiration of Indicators of Compromise (IOCs)

The Custom Threat Intelligence integration supports IOC expiration. The ingested IOCs expire after certain duration. Based on the STIX 2.1 reference, the following options are available to determine the expiration of indicators:

• The valid_until field that indicates the time at which this Indicator should no longer be considered a valid indicator
• The revoked field that means that the indicator is no longer considered valid by the object creator.
• When missing valid_until and revoked, the indicator expires according to the default expiration set by IOC Expiration Duration configuration parameter. For more details, see Handling Orphaned IOCs.

The field stix.ioc_expiration_reason indicates which among the 3 methods stated above is the reason for indicator expiration.

An Elastic Transform is created to faciliate only active IOCs be available to the end users. This transform creates destination indices named logs-ti_custom_latest.dest_indicator-* which only contains active and unexpired IOCs. The latest destination index also has an alias named logs-ti_custom_latest.indicator. When querying for active indicators or setting up indicator match rules, only use the latest destination indices or the alias to avoid false positives from expired IOCs.

Handling orphaned IOCs

Some IOCs may never expire and will continue to stay in the latest destination indices logs-ti_custom_latest.dest_indicator-*. To avoid any false positives from such orphaned IOCs, users are allowed to configure IOC Expiration Duration parameter while setting up the integration. This parameter deletes any indicator ingested into destination indices logs-ti_custom_latest.dest_indicator-* after this specified duration is reached, defaults to 90d from source's @timestamp field. Note that IOC Expiration Duration parameter only exists to add a fail-safe default expiration in case IOCs never expire.

ILM Policy

To facilitate IOC expiration, source datastream-backed indices .ds-logs-ti_custom.indicator-* are allowed to contain duplicates from each polling interval. ILM policy is added to these source indices so it doesn't lead to unbounded growth. This means data in these source indices will be deleted after 5 days from ingested date.

Logs reference

indicator

The indicator dataset stores STIX 2.1 indicators processed into ECS.

Example

An example event for indicator looks as following:

{
    "@timestamp": "2015-05-15T09:12:16.432Z",
    "agent": {
        "ephemeral_id": "3179505d-300c-450f-9a13-6c48e97a6453",
        "id": "f2593b61-002f-42d8-b72b-c4181b590c92",
        "name": "elastic-agent-77457",
        "type": "filebeat",
        "version": "8.14.0"
    },
    "data_stream": {
        "dataset": "ti_custom.indicator",
        "namespace": "36027",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "f2593b61-002f-42d8-b72b-c4181b590c92",
        "snapshot": false,
        "version": "8.14.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "threat"
        ],
        "dataset": "ti_custom.indicator",
        "ingested": "2024-10-02T10:51:17Z",
        "kind": "enrichment",
        "original": "{\"type\":\"indicator\",\"spec_version\":\"2.1\",\"pattern_type\":\"stix\",\"id\":\"indicator--745e1537-b4f3-49da-9f64-df6b1b5df190\",\"created\":\"2015-05-15T09:12:16.432Z\",\"modified\":\"2015-05-15T09:12:16.432Z\",\"name\":\"Appendix E MD5 hash '002325a0a67fded0381b5648d7fe9b8e'\",\"description\":\"Test description.\",\"pattern\":\"[file:hashes.md5 = '002325a0a67fded0381b5648d7fe9b8e']\",\"indicator_types\":[\"malicious-activity\"],\"valid_from\":\"2015-05-15T09:12:16.432678Z\"}",
        "type": [
            "indicator"
        ]
    },
    "input": {
        "type": "filestream"
    },
    "log": {
        "file": {
            "device_id": "36",
            "inode": "125",
            "path": "/tmp/service_logs/stix-indicators-ndjson.log"
        },
        "offset": 3409
    },
    "related": {
        "hash": [
            "002325a0a67fded0381b5648d7fe9b8e"
        ]
    },
    "stix": {
        "created": "2015-05-15T09:12:16.432Z",
        "id": "indicator--745e1537-b4f3-49da-9f64-df6b1b5df190",
        "indicator_types": [
            "malicious-activity"
        ],
        "ioc_expiration_date": "2015-05-20T09:12:16.432Z",
        "ioc_expiration_duration": "5d",
        "ioc_expiration_reason": "Expiration set by Elastic from the integration's parameter `IOC Expiration Duration`",
        "modified": "2015-05-15T09:12:16.432Z",
        "pattern": "[file:hashes.md5 = '002325a0a67fded0381b5648d7fe9b8e']",
        "pattern_type": "stix",
        "spec_version": "2.1",
        "type": "indicator",
        "valid_from": "2015-05-15T09:12:16.432678Z"
    },
    "tags": [
        "preserve_original_event",
        "forwarded",
        "ti_custom-indicator"
    ],
    "threat": {
        "feed": {
            "name": "STIX Provider",
            "reference": "https://stix-example.com"
        },
        "indicator": {
            "description": "Test description.",
            "file": {
                "hash": {
                    "md5": [
                        "002325a0a67fded0381b5648d7fe9b8e"
                    ]
                }
            },
            "first_seen": "2015-05-15T09:12:16.432Z",
            "last_seen": "2015-05-15T09:12:16.432Z",
            "modified_at": "2015-05-15T09:12:16.432Z",
            "name": "Appendix E MD5 hash '002325a0a67fded0381b5648d7fe9b8e'",
            "type": "file"
        }
    }
}

Exported fields

Changelog