Recorded Future Integration

edit

Recorded Future Integration

edit

Version

1.28.0 (View all)

Compatible Kibana version(s)

8.13.0 or higher

Supported Serverless project types
What’s this?

Security
Observability

Subscription level
What’s this?

Basic

Level of support
What’s this?

Elastic

The Recorded Future integration fetches risklists from the Recorded Future API. It supports domain, hash, ip and url entities.

In order to use it you need to define the entity and list to fetch. Check with Recorded Future for the available lists for each entity. To fetch indicators from multiple entities, it’s necessary to define one integration for each.

Alternatively, it’s also possible to use the integration to fetch custom Fusion files by supplying the URL to the CSV file as the Custom URL configuration option.

Expiration of Indicators of Compromise (IOCs)
edit

The ingested IOCs expire after certain duration. An Elastic Transform is created to faciliate only active IOCs be available to the end users. This transform creates a destination index named logs-ti_recordedfuture_latest.threat-1 which only contains active and unexpired IOCs. The destination index also has an alias logs-ti_recordedfuture_latest.threat. When setting up indicator match rules, use this latest destination index to avoid false positives from expired IOCs. Please read ILM Policy below which is added to avoid unbounded growth on source .ds-logs-ti_recordedfuture.threat-* indices.

ILM Policy
edit

To facilitate IOC expiration, source datastream-backed indices .ds-logs-ti_recordedfuture.threat-* are allowed to contain duplicates from each polling interval. ILM policy is added to these source indices so it doesn’t lead to unbounded growth. This means data in these source indices will be deleted after 5 days from ingested date.

For large risklist downloads, adjust the timeout setting so that the Agent has enough time to download and process the risklist.

Example

An example event for threat looks as following:

{
    "@timestamp": "2024-08-02T06:24:04.201Z",
    "agent": {
        "ephemeral_id": "25d7a936-2b7c-4476-9181-82d1296ce9df",
        "id": "8299ae35-ee0e-4107-9acb-1b6acfdda1fb",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.13.0"
    },
    "data_stream": {
        "dataset": "ti_recordedfuture.threat",
        "namespace": "67234",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "8299ae35-ee0e-4107-9acb-1b6acfdda1fb",
        "snapshot": false,
        "version": "8.13.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "threat"
        ],
        "created": "2024-08-02T06:24:04.201Z",
        "dataset": "ti_recordedfuture.threat",
        "ingested": "2024-08-02T06:24:16Z",
        "kind": "enrichment",
        "original": "{\"EvidenceDetails\":\"{\\\"EvidenceDetails\\\": [{\\\"Name\\\": \\\"suspectedCncDnsName\\\", \\\"EvidenceString\\\": \\\"1 sighting on 1 source: ThreatFox Infrastructure Analysis. ThreatFox identified ubykou33.top as possible TA0011 (Command and Control) for CryptBot on December 26, 2023. Most recent link (Dec 26, 2023): https://threatfox.abuse.ch/ioc/1223634\\\", \\\"CriticalityLabel\\\": \\\"Unusual\\\", \\\"MitigationString\\\": \\\"\\\", \\\"Rule\\\": \\\"Historical Suspected C\\\\\u0026C DNS Name\\\", \\\"SourcesCount\\\": 1.0, \\\"Sources\\\": [\\\"source:sIoEOQ\\\"], \\\"Timestamp\\\": \\\"2023-12-26T17:06:29.000Z\\\", \\\"SightingsCount\\\": 1.0, \\\"Criticality\\\": 1.0}, {\\\"Name\\\": \\\"malwareSiteDetected\\\", \\\"EvidenceString\\\": \\\"2 sightings on 2 sources: External Sensor Data Analysis, Bitdefender. ubykou33.top is observed to be a malware site domain that navigates to malicious content including executables, drive-by infection sites, malicious scripts, viruses, trojans, or code.\\\", \\\"CriticalityLabel\\\": \\\"Unusual\\\", \\\"MitigationString\\\": \\\"\\\", \\\"Rule\\\": \\\"Historically Detected Malware Operation\\\", \\\"SourcesCount\\\": 2.0, \\\"Sources\\\": [\\\"source:kBB1fk\\\", \\\"source:d3Awkm\\\"], \\\"Timestamp\\\": \\\"2024-01-26T00:00:00.000Z\\\", \\\"SightingsCount\\\": 2.0, \\\"Criticality\\\": 1.0}, {\\\"Name\\\": \\\"malwareSiteSuspected\\\", \\\"EvidenceString\\\": \\\"1 sighting on 1 source: Bitdefender. Detected malicious behavior from an endpoint agent via global telemetry. Last observed on Jan 26, 2024.\\\", \\\"CriticalityLabel\\\": \\\"Unusual\\\", \\\"MitigationString\\\": \\\"\\\", \\\"Rule\\\": \\\"Historically Suspected Malware Operation\\\", \\\"SourcesCount\\\": 1.0, \\\"Sources\\\": [\\\"source:d3Awkm\\\"], \\\"Timestamp\\\": \\\"2024-01-26T00:00:00.000Z\\\", \\\"SightingsCount\\\": 1.0, \\\"Criticality\\\": 1.0}, {\\\"Name\\\": \\\"recentMalwareSiteDetected\\\", \\\"EvidenceString\\\": \\\"1 sighting on 1 source: External Sensor Data Analysis. ubykou33.top is observed to be a malware site domain that navigates to malicious content including executables, drive-by infection sites, malicious scripts, viruses, trojans, or code.\\\", \\\"CriticalityLabel\\\": \\\"Malicious\\\", \\\"MitigationString\\\": \\\"\\\", \\\"Rule\\\": \\\"Recently Detected Malware Operation\\\", \\\"SourcesCount\\\": 1.0, \\\"Sources\\\": [\\\"source:kBB1fk\\\"], \\\"Timestamp\\\": \\\"2024-05-08T23:11:43.601Z\\\", \\\"SightingsCount\\\": 1.0, \\\"Criticality\\\": 3.0}]}\",\"Name\":\"ubykou33.top\",\"Risk\":\"67\",\"RiskString\":\"4/52\"}",
        "risk_score": 67,
        "type": [
            "indicator"
        ]
    },
    "input": {
        "type": "httpjson"
    },
    "recordedfuture": {
        "evidence_details": [
            {
                "criticality": 1,
                "criticality_label": "Unusual",
                "evidence_string": "1 sighting on 1 source: ThreatFox Infrastructure Analysis. ThreatFox identified ubykou33.top as possible TA0011 (Command and Control) for CryptBot on December 26, 2023. Most recent link (Dec 26, 2023): https://threatfox.abuse.ch/ioc/1223634",
                "mitigation_string": "",
                "name": "suspectedCncDnsName",
                "rule": "Historical Suspected C&C DNS Name",
                "sightings_count": 1,
                "sources": [
                    "source:sIoEOQ"
                ],
                "sources_count": 1,
                "timestamp": "2023-12-26T17:06:29.000Z"
            },
            {
                "criticality": 1,
                "criticality_label": "Unusual",
                "evidence_string": "2 sightings on 2 sources: External Sensor Data Analysis, Bitdefender. ubykou33.top is observed to be a malware site domain that navigates to malicious content including executables, drive-by infection sites, malicious scripts, viruses, trojans, or code.",
                "mitigation_string": "",
                "name": "malwareSiteDetected",
                "rule": "Historically Detected Malware Operation",
                "sightings_count": 2,
                "sources": [
                    "source:kBB1fk",
                    "source:d3Awkm"
                ],
                "sources_count": 2,
                "timestamp": "2024-01-26T00:00:00.000Z"
            },
            {
                "criticality": 1,
                "criticality_label": "Unusual",
                "evidence_string": "1 sighting on 1 source: Bitdefender. Detected malicious behavior from an endpoint agent via global telemetry. Last observed on Jan 26, 2024.",
                "mitigation_string": "",
                "name": "malwareSiteSuspected",
                "rule": "Historically Suspected Malware Operation",
                "sightings_count": 1,
                "sources": [
                    "source:d3Awkm"
                ],
                "sources_count": 1,
                "timestamp": "2024-01-26T00:00:00.000Z"
            },
            {
                "criticality": 3,
                "criticality_label": "Malicious",
                "evidence_string": "1 sighting on 1 source: External Sensor Data Analysis. ubykou33.top is observed to be a malware site domain that navigates to malicious content including executables, drive-by infection sites, malicious scripts, viruses, trojans, or code.",
                "mitigation_string": "",
                "name": "recentMalwareSiteDetected",
                "rule": "Recently Detected Malware Operation",
                "sightings_count": 1,
                "sources": [
                    "source:kBB1fk"
                ],
                "sources_count": 1,
                "timestamp": "2024-05-08T23:11:43.601Z"
            }
        ],
        "list": "test",
        "name": "ubykou33.top",
        "risk_string": "4/52"
    },
    "tags": [
        "preserve_original_event",
        "forwarded",
        "recordedfuture"
    ],
    "threat": {
        "feed": {
            "name": "Recorded Future"
        },
        "indicator": {
            "provider": [
                "ThreatFox Infrastructure Analysis",
                "External Sensor Data Analysis",
                "Bitdefender"
            ],
            "scanner_stats": 5,
            "sightings": 5,
            "type": "domain-name",
            "url": {
                "domain": "ubykou33.top"
            }
        }
    }
}
Exported fields
Field Description Type

@timestamp

Event timestamp.

date

cloud.image.id

Image ID for the cloud instance.

keyword

data_stream.dataset

Data stream dataset name.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

input.type

Type of Filebeat input.

keyword

labels.is_ioc_transform_source

Indicates whether an IOC is in the raw source data stream, or the in latest destination index.

constant_keyword

log.flags

Flags for the log file.

keyword

log.offset

Offset of the entry in the log file.

long

recordedfuture.evidence_details.criticality

double

recordedfuture.evidence_details.criticality_label

keyword

recordedfuture.evidence_details.evidence_string

keyword

recordedfuture.evidence_details.mitigation_string

keyword

recordedfuture.evidence_details.name

keyword

recordedfuture.evidence_details.rule

keyword

recordedfuture.evidence_details.sightings_count

integer

recordedfuture.evidence_details.sources

keyword

recordedfuture.evidence_details.sources_count

integer

recordedfuture.evidence_details.timestamp

date

recordedfuture.list

User-configured risklist.

keyword

recordedfuture.name

Indicator value.

keyword

recordedfuture.risk_string

Details of risk rules observed.

keyword

threat.feed.name

Display friendly feed name

constant_keyword

threat.indicator.first_seen

The date and time when intelligence source first reported sighting this indicator.

date

threat.indicator.last_seen

The date and time when intelligence source last reported sighting this indicator.

date

threat.indicator.modified_at

The date and time when intelligence source last modified information for this indicator.

date

Changelog

edit
Changelog
Version Details Kibana version(s)

1.28.0

Enhancement (View pull request)
Do not remove event.original in main ingest pipeline.

8.13.0 or higher

1.27.0

Enhancement (View pull request)
Add "preserve_original_event" tag to documents with event.kind set to "pipeline_error".

8.13.0 or higher

1.26.3

Bug fix (View pull request)
Fix labels.is_ioc_transform_source values

8.13.0 or higher

1.26.2

Bug fix (View pull request)
Add missing fields in transform

8.13.0 or higher

1.26.1

Bug fix (View pull request)
Fix ECS date mapping on threat fields.

8.13.0 or higher

1.26.0

Enhancement (View pull request)
Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.

8.13.0 or higher

1.25.1

Bug fix (View pull request)
Adjust field mappings for transform destination index.

8.12.0 or higher

1.25.0

Enhancement (View pull request)
Decode Evidence_Details field. This is a breaking change since the mapping is changed.

8.12.0 or higher

1.24.0

Enhancement (View pull request)
Add destination index alias and fix docs.

8.12.0 or higher

1.23.0

Enhancement (View pull request)
Add dashboards and list field

8.12.0 or higher

1.22.0

Enhancement (View pull request)
Set sensitive values as secret.

8.12.0 or higher

1.21.0

Enhancement (View pull request)
Make threat.indicator.url.full available for rule detections.

8.8.0 or higher

1.20.2

Enhancement (View pull request)
Changed owners

8.8.0 or higher

1.20.1

Bug fix (View pull request)
Fix exclude_files pattern.

8.8.0 or higher

1.20.0

Enhancement (View pull request)
Limit request tracer log count to five.

8.8.0 or higher

1.19.0

Enhancement (View pull request)
ECS version updated to 8.11.0.

8.8.0 or higher

1.18.1

Bug fix (View pull request)
Fix the parse of providers information.

8.8.0 or higher

1.18.0

Enhancement (View pull request)
Improve event.original check to avoid errors if set.

8.8.0 or higher

1.17.0

Enhancement (View pull request)
ECS version updated to 8.10.0.

8.8.0 or higher

1.16.0

Enhancement (View pull request)
Add DLM policy. Add owner.type to package manifest. Update format_version to 3.0.0

Enhancement (View pull request)
Add tags.yml file so that integration’s dashboards and saved searches are tagged with "Security Solution" and displayed in the Security Solution UI.

8.8.0 or higher

1.15.1

Bug fix (View pull request)
Replace dotted YAML keys.

1.15.0

Enhancement (View pull request)
Update package-spec to 2.10.0.

8.8.0 or higher

1.14.0

Enhancement (View pull request)
Update package to ECS 8.9.0.

8.8.0 or higher

1.13.0

Enhancement (View pull request)
Document duration units.

8.8.0 or higher

1.12.0

Enhancement (View pull request)
Ensure event.kind is correctly set for pipeline errors.

8.8.0 or higher

1.11.0

Enhancement (View pull request)
Update package to ECS 8.8.0.

8.8.0 or higher

1.10.0

Enhancement (View pull request)
Add IOC field to transform source to easily filter destination indices with unexpired indicators

8.8.0 or higher

1.9.0

Enhancement (View pull request)
Support for IoC Expiration

8.8.0 or higher

1.8.0

Enhancement (View pull request)
Add a new flag to enable request tracing

8.7.1 or higher

1.7.0

Enhancement (View pull request)
Scrape provider details from evidence field.

8.0.0 or higher

1.6.0

Enhancement (View pull request)
Update package to ECS 8.7.0.

8.0.0 or higher

1.5.0

Enhancement (View pull request)
Update package to ECS 8.6.0.

8.0.0 or higher

1.4.1

Bug fix (View pull request)
Use ECS definition for threat.indicator.geo.location.

8.0.0 or higher

1.4.0

Enhancement (View pull request)
Update package to ECS 8.5.0.

1.3.0

Enhancement (View pull request)
Update package to ECS 8.4.0

8.0.0 or higher

1.2.1

Bug fix (View pull request)
Expose request timeout setting and increase it to 5m.

Bug fix (View pull request)
Do not fail on invalid URLs.

8.0.0 or higher

1.2.0

Enhancement (View pull request)
Update categories to include threat_intel.

8.0.0 or higher

1.1.0

Enhancement (View pull request)
Update package to ECS 8.3.0.

8.0.0 or higher

1.0.1

Enhancement (View pull request)
update readme added link to recorded future API documentation

8.0.0 or higher

1.0.0

Enhancement (View pull request)
Make GA

8.0.0 or higher

0.1.3

Enhancement (View pull request)
Update package descriptions

0.1.2

Enhancement (View pull request)
Add field mapping for event.created

0.1.1

Enhancement (View pull request)
Add documentation for multi-fields

0.1.0

Enhancement (View pull request)
Initial release