Collective Intelligence Framework v3

Ingest threat indicators from a Collective Intelligence Framework v3 instance with Elastic Agent.

What is an Elastic integration?
Version
1.14.4 (View all)
Compatible Kibana version(s)
8.13.0 or higher
Supported Serverless project types

What's this?
Security
Observability
Subscription level
What's this?
Basic
Level of support
What's this?
Community

This integration connects with the REST API from the running CIFv3 instance to retrieve indicators.

Expiration of Indicators of Compromise (IOCs)

Indicators are expired after a certain duration. An Elastic Transform is created for a source index to allow only active indicators to be available to the end users. The transform creates a destination index named logs-ti_cif3_latest.dest_feed* which only contains active and unexpired indicators. Destination indices are aliased to logs-ti_cif3_latest.feed. The indicator match rules and dashboards are updated to show only active indicators.

Indicator TypeIndicator Expiration Duration
ipv4-addr
45d
ipv6-addr
45d
domain-name
90d
url
365d
file
365d
All Other Types
Derived from IOC Expiration Duration setting

ILM Policy

To facilitate IOC expiration, source datastream-backed indices .ds-logs-ti_cif3.feed-* are allowed to contain duplicates. ILM policy logs-ti_cif3.feed-default_policy is added to these source indices so it doesn't lead to unbounded growth. This means data in these source indices will be deleted after 5 days from ingested date.

Data Streams

Feed

The CIFv3 integration collects threat indicators based on user-defined configuration including a polling interval, how far back in time it should look, and other filters like indicator type and tags.

CIFv3 confidence field values (0..10) are converted to ECS confidence (None, Low, Medium, High) in the following way:

CIFv3 ConfidenceECS Conversion
Beyond Range
None
0 - <3
Low
3 - <7
Medium
7 - 10
High

Exported fields

FieldDescriptionType
@timestamp
Event timestamp.
date
cif3.application
The application used by the indicator, such as telnet or ssh.
keyword
cif3.asn
AS Number of IP.
integer
cif3.asn_desc
AS Number org name.
keyword
cif3.cc
Country code of GeoIP.
keyword
cif3.city
GeoIP city information.
keyword
cif3.confidence
The confidence on a scale of 0-10 that the tags appropriately contextualize the indicator.
float
cif3.count
The number of times the same indicator has been reported with the same metadata by the same provider.
integer
cif3.deleted_at
The indicator expiration timestamp.
date
cif3.description
A description of the indicator.
keyword
cif3.expiration_duration
The configured expiration duration.
keyword
cif3.indicator
The value of the indicator, for example if the type is fqdn, this would be the value.
keyword
cif3.indicator_iprange
IPv4 or IPv6 IP Range.
ip_range
cif3.indicator_ipv4
IPv4 address.
ip
cif3.indicator_ipv4_mask
subnet mask of IPv4 CIDR.
integer
cif3.indicator_ipv6
singleton IPv6 address.
keyword
cif3.indicator_ipv6_mask
subnet mask of IPv6 CIDR.
integer
cif3.indicator_ssdeep_chunk
SSDEEP hash chunk.
text
cif3.indicator_ssdeep_chunksize
SSDEEP hash chunk size.
integer
cif3.indicator_ssdeep_double_chunk
SSDEEP hash double chunk.
text
cif3.itype
The indicator type, can for example be "ipv4, fqdn, email, url, sha256".
keyword
cif3.latitude
Latitude of GeoIP.
keyword
cif3.location
Lat/Long of GeoIP.
geo_point
cif3.longitude
Longitude of GeoIP.
keyword
cif3.portlist
The port or range of ports used by the indicator.
text
cif3.protocol
The protocol used by the indicator.
text
cif3.provider
The source of the indicator information.
keyword
cif3.rdata
Extra text or descriptive content related to the indicator such as OS, reverse lookup, etc.
keyword
cif3.reference
A reference URL with further info related to the indicator.
keyword
cif3.region
GeoIP region information.
keyword
cif3.tags
Comma-separated list of words describing the indicator such as "malware,exploit".
keyword
cif3.timezone
Timezone of GeoIP.
text
cif3.uuid
The ID of the indicator.
keyword
data_stream.dataset
Data stream dataset.
constant_keyword
data_stream.namespace
Data stream namespace.
constant_keyword
data_stream.type
Data stream type.
constant_keyword
event.dataset
Event dataset
constant_keyword
event.module
Name of the module this data is coming from.
constant_keyword
input.type
Type of Filebeat input.
keyword
labels.is_ioc_transform_source
Indicates whether an IOC is in the raw source data stream, or the in latest destination index.
constant_keyword
log.flags
Flags for the log file.
keyword
log.offset
Offset of the entry in the log file.
long
threat.feed.name
Display friendly feed name
constant_keyword
threat.indicator.first_seen
The date and time when intelligence source first reported sighting this indicator.
date
threat.indicator.last_seen
The date and time when intelligence source last reported sighting this indicator.
date
threat.indicator.modified_at
The date and time when intelligence source last modified information for this indicator.
date
threat.indicator.tls.client.ja3
An md5 hash that identifies clients based on their TLS handshake.
keyword

An example event for feed looks as following:

{
    "@timestamp": "2024-08-01T08:05:14.040Z",
    "agent": {
        "ephemeral_id": "b351d699-2fd0-49f7-99e1-a7a471a29a62",
        "id": "36b03887-7783-4bc4-b8c5-6f8997e4cd1a",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.13.0"
    },
    "cif3": {
        "deleted_at": "2022-09-03T20:25:53.000Z",
        "expiration_duration": "45d",
        "indicator": "20.206.75.106",
        "itype": "ipv4",
        "portlist": "443",
        "uuid": "ac240898-1443-4d7e-a98a-1daed220c162"
    },
    "data_stream": {
        "dataset": "ti_cif3.feed",
        "namespace": "26457",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "36b03887-7783-4bc4-b8c5-6f8997e4cd1a",
        "snapshot": false,
        "version": "8.13.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "threat"
        ],
        "created": "2024-08-01T08:05:14.040Z",
        "dataset": "ti_cif3.feed",
        "ingested": "2024-08-01T08:05:26Z",
        "kind": "enrichment",
        "original": "{\"application\":\"https\",\"asn\":8075,\"asn_desc\":\"microsoft-corp-msn-as-block\",\"cc\":\"br\",\"city\":\"campinas\",\"confidence\":10,\"count\":1,\"firsttime\":\"2022-07-20T20:25:53.000000Z\",\"group\":[\"everyone\"],\"indicator\":\"20.206.75.106\",\"indicator_ipv4\":\"20.206.75.106\",\"itype\":\"ipv4\",\"lasttime\":\"2022-07-20T20:25:53.000000Z\",\"latitude\":-22.9035,\"location\":[-47.0565,-22.9035],\"longitude\":-47.0565,\"portlist\":\"443\",\"protocol\":\"tcp\",\"provider\":\"sslbl.abuse.ch\",\"reference\":\"https://sslbl.abuse.ch/blacklist/sslipblacklist.csv\",\"region\":\"sao paulo\",\"reporttime\":\"2022-07-21T20:33:26.585967Z\",\"tags\":[\"botnet\"],\"timezone\":\"america/sao_paulo\",\"tlp\":\"white\",\"uuid\":\"ac240898-1443-4d7e-a98a-1daed220c162\"}",
        "type": [
            "indicator"
        ]
    },
    "input": {
        "type": "httpjson"
    },
    "network": {
        "protocol": "https",
        "transport": "tcp"
    },
    "related": {
        "ip": [
            "20.206.75.106"
        ]
    },
    "tags": [
        "preserve_original_event",
        "forwarded",
        "cif3-indicator",
        "botnet"
    ],
    "threat": {
        "indicator": {
            "as": {
                "number": 8075,
                "organization": {
                    "name": "microsoft-corp-msn-as-block"
                }
            },
            "confidence": "High",
            "first_seen": "2022-07-20T20:25:53.000Z",
            "geo": {
                "country_iso_code": "br",
                "location": {
                    "lat": -22.9035,
                    "lon": -47.0565
                },
                "region_name": "sao paulo",
                "timezone": "america/sao_paulo"
            },
            "ip": "20.206.75.106",
            "last_seen": "2022-07-20T20:25:53.000Z",
            "marking": {
                "tlp": "WHITE"
            },
            "modified_at": "2022-07-21T20:33:26.585967Z",
            "name": "20.206.75.106",
            "provider": "sslbl.abuse.ch",
            "reference": "https://sslbl.abuse.ch/blacklist/sslipblacklist.csv",
            "sightings": 1,
            "type": "ipv4-addr"
        }
    }
}

Changelog

VersionDetailsKibana version(s)

1.14.4

Bug fix View pull request
Use triple-brace Mustache templating when referencing variables in ingest pipelines.

8.13.0 or higher

1.14.3

Bug fix View pull request
Fix labels.is_ioc_transform_source values

8.13.0 or higher

1.14.2

Bug fix View pull request
Add missing fields in transform

8.13.0 or higher

1.14.1

Bug fix View pull request
Fix ECS date mapping on threat fields.

8.13.0 or higher

1.14.0

Enhancement View pull request
Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.

8.13.0 or higher

1.13.1

Bug fix View pull request
Adjust field mappings for transform destination index.

8.12.0 or higher

1.13.0

Enhancement View pull request
Improve handling of empty responses.

8.12.0 or higher

1.12.0

Enhancement View pull request
Support for IOC expiration

8.12.0 or higher

1.11.0

Enhancement View pull request
Set sensitive values as secret.

8.12.0 or higher

1.10.2

Bug fix View pull request
Clean up null handling

8.7.1 or higher

1.10.1

Enhancement View pull request
Changed owners

8.7.1 or higher

1.10.0

Enhancement View pull request
Limit request tracer log count to five.

8.7.1 or higher

1.9.0

Enhancement View pull request
ECS version updated to 8.11.0.

8.7.1 or higher

1.8.0

Enhancement View pull request
Set 'community' owner type.

8.7.1 or higher

1.7.0

Enhancement View pull request
ECS version updated to 8.10.0.

8.7.1 or higher

1.6.0

Enhancement View pull request
The format_version in the package manifest changed from 2.11.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added 'owner.type: elastic' to package manifest.

8.7.1 or higher

1.5.0

Enhancement View pull request
Add tags.yml file so that integration's dashboards and saved searches are tagged with "Security Solution" and displayed in the Security Solution UI.

8.7.1 or higher

1.4.0

Enhancement View pull request
Update package-spec to 2.9.0.

8.7.1 or higher

1.3.0

Enhancement View pull request
Update package to ECS 8.9.0.

8.7.1 or higher

1.2.0

Enhancement View pull request
Document duration units.

8.7.1 or higher

1.1.0

Enhancement View pull request
Document valid duration units.

8.7.1 or higher

1.0.0

Enhancement View pull request
Release Collective Intelligence Framework as GA.

8.7.1 or higher

0.8.0

Enhancement View pull request
Ensure event.kind is correctly set for pipeline errors.

—

0.7.0

Enhancement View pull request
Update package to ECS 8.8.0.

—

0.6.0

Enhancement View pull request
Add a new flag to enable request tracing

—

0.5.0

Enhancement View pull request
Update package to ECS 8.7.0.

—

0.4.1

Enhancement View pull request
Honor preserve_original_event setting.

—

0.4.0

Enhancement View pull request
Update package to ECS 8.6.0.

—

0.3.1

Bug fix View pull request
Use ECS definition for threat.indicator.geo.location.

—

0.3.0

Enhancement View pull request
Update package to ECS 8.5.0.

—

0.2.2

Bug fix View pull request
Remove duplicate field.

—

0.2.1

Enhancement View pull request
Fix documentation build error

—

0.2.0

Enhancement View pull request
Labelling with Threat Intelligence category

—

0.1.0

Enhancement View pull request
Initial draft of the package

—

On this page