Collective Intelligence Framework v3 Integration

Version	1.16.0 (View all)
Compatible Kibana version(s)	8.13.0 or higher
Supported Serverless project types What’s this?	Security Observability
Subscription level What’s this?	Basic
Level of support What’s this?	Community

This integration connects with the REST API from the running CIFv3 instance to retrieve indicators.

Expiration of Indicators of Compromise (IOCs)

edit

Indicators are expired after a certain duration. An Elastic Transform is created for a source index to allow only active indicators to be available to the end users. The transform creates a destination index named logs-ti_cif3_latest.dest_feed* which only contains active and unexpired indicators. Destination indices are aliased to logs-ti_cif3_latest.feed. The indicator match rules and dashboards are updated to show only active indicators.

Indicator Type	Indicator Expiration Duration
`ipv4-addr`	`45d`
`ipv6-addr`	`45d`
`domain-name`	`90d`
`url`	`365d`
`file`	`365d`
All Other Types	Derived from `IOC Expiration Duration` setting

ILM Policy

edit

To facilitate IOC expiration, source datastream-backed indices .ds-logs-ti_cif3.feed-* are allowed to contain duplicates. ILM policy logs-ti_cif3.feed-default_policy is added to these source indices so it doesn’t lead to unbounded growth. This means data in these source indices will be deleted after 5 days from ingested date.

Data Streams

edit

Feed

edit

The CIFv3 integration collects threat indicators based on user-defined configuration including a polling interval, how far back in time it should look, and other filters like indicator type and tags.

CIFv3 confidence field values (0..10) are converted to ECS confidence (None, Low, Medium, High) in the following way:

CIFv3 Confidence	ECS Conversion
Beyond Range	None
0 - <3	Low
3 - <7	Medium
7 - 10	High

Exported fields

Field	Description	Type
@timestamp	Event timestamp.	date
cif3.application	The application used by the indicator, such as telnet or ssh.	keyword
cif3.asn	AS Number of IP.	integer
cif3.asn_desc	AS Number org name.	keyword
cif3.cc	Country code of GeoIP.	keyword
cif3.city	GeoIP city information.	keyword
cif3.confidence	The confidence on a scale of 0-10 that the tags appropriately contextualize the indicator.	float
cif3.count	The number of times the same indicator has been reported with the same metadata by the same provider.	integer
cif3.deleted_at	The indicator expiration timestamp.	date
cif3.description	A description of the indicator.	keyword
cif3.expiration_duration	The configured expiration duration.	keyword
cif3.indicator	The value of the indicator, for example if the type is fqdn, this would be the value.	keyword
cif3.indicator_iprange	IPv4 or IPv6 IP Range.	ip_range
cif3.indicator_ipv4	IPv4 address.	ip
cif3.indicator_ipv4_mask	subnet mask of IPv4 CIDR.	integer
cif3.indicator_ipv6	singleton IPv6 address.	keyword
cif3.indicator_ipv6_mask	subnet mask of IPv6 CIDR.	integer
cif3.indicator_ssdeep_chunk	SSDEEP hash chunk.	text
cif3.indicator_ssdeep_chunksize	SSDEEP hash chunk size.	integer
cif3.indicator_ssdeep_double_chunk	SSDEEP hash double chunk.	text
cif3.itype	The indicator type, can for example be "ipv4, fqdn, email, url, sha256".	keyword
cif3.latitude	Latitude of GeoIP.	keyword
cif3.location	Lat/Long of GeoIP.	geo_point
cif3.longitude	Longitude of GeoIP.	keyword
cif3.portlist	The port or range of ports used by the indicator.	text
cif3.protocol	The protocol used by the indicator.	text
cif3.provider	The source of the indicator information.	keyword
cif3.rdata	Extra text or descriptive content related to the indicator such as OS, reverse lookup, etc.	keyword
cif3.reference	A reference URL with further info related to the indicator.	keyword
cif3.region	GeoIP region information.	keyword
cif3.tags	Comma-separated list of words describing the indicator such as "malware,exploit".	keyword
cif3.timezone	Timezone of GeoIP.	text
cif3.uuid	The ID of the indicator.	keyword
data_stream.dataset	Data stream dataset.	constant_keyword
data_stream.namespace	Data stream namespace.	constant_keyword
data_stream.type	Data stream type.	constant_keyword
event.dataset	Event dataset	constant_keyword
event.module	Name of the module this data is coming from.	constant_keyword
input.type	Type of Filebeat input.	keyword
labels.is_ioc_transform_source	Indicates whether an IOC is in the raw source data stream, or the in latest destination index.	constant_keyword
log.flags	Flags for the log file.	keyword
log.offset	Offset of the entry in the log file.	long
threat.feed.name	Display friendly feed name	constant_keyword
threat.indicator.first_seen	The date and time when intelligence source first reported sighting this indicator.	date
threat.indicator.last_seen	The date and time when intelligence source last reported sighting this indicator.	date
threat.indicator.modified_at	The date and time when intelligence source last modified information for this indicator.	date
threat.indicator.tls.client.ja3	An md5 hash that identifies clients based on their TLS handshake.	keyword

Example

An example event for feed looks as following:

{
    "@timestamp": "2024-08-01T08:05:14.040Z",
    "agent": {
        "ephemeral_id": "b351d699-2fd0-49f7-99e1-a7a471a29a62",
        "id": "36b03887-7783-4bc4-b8c5-6f8997e4cd1a",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.13.0"
    },
    "cif3": {
        "deleted_at": "2022-09-03T20:25:53.000Z",
        "expiration_duration": "45d",
        "indicator": "20.206.75.106",
        "itype": "ipv4",
        "portlist": "443",
        "uuid": "ac240898-1443-4d7e-a98a-1daed220c162"
    },
    "data_stream": {
        "dataset": "ti_cif3.feed",
        "namespace": "26457",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "36b03887-7783-4bc4-b8c5-6f8997e4cd1a",
        "snapshot": false,
        "version": "8.13.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "threat"
        ],
        "created": "2024-08-01T08:05:14.040Z",
        "dataset": "ti_cif3.feed",
        "ingested": "2024-08-01T08:05:26Z",
        "kind": "enrichment",
        "original": "{\"application\":\"https\",\"asn\":8075,\"asn_desc\":\"microsoft-corp-msn-as-block\",\"cc\":\"br\",\"city\":\"campinas\",\"confidence\":10,\"count\":1,\"firsttime\":\"2022-07-20T20:25:53.000000Z\",\"group\":[\"everyone\"],\"indicator\":\"20.206.75.106\",\"indicator_ipv4\":\"20.206.75.106\",\"itype\":\"ipv4\",\"lasttime\":\"2022-07-20T20:25:53.000000Z\",\"latitude\":-22.9035,\"location\":[-47.0565,-22.9035],\"longitude\":-47.0565,\"portlist\":\"443\",\"protocol\":\"tcp\",\"provider\":\"sslbl.abuse.ch\",\"reference\":\"https://sslbl.abuse.ch/blacklist/sslipblacklist.csv\",\"region\":\"sao paulo\",\"reporttime\":\"2022-07-21T20:33:26.585967Z\",\"tags\":[\"botnet\"],\"timezone\":\"america/sao_paulo\",\"tlp\":\"white\",\"uuid\":\"ac240898-1443-4d7e-a98a-1daed220c162\"}",
        "type": [
            "indicator"
        ]
    },
    "input": {
        "type": "httpjson"
    },
    "network": {
        "protocol": "https",
        "transport": "tcp"
    },
    "related": {
        "ip": [
            "20.206.75.106"
        ]
    },
    "tags": [
        "preserve_original_event",
        "forwarded",
        "cif3-indicator",
        "botnet"
    ],
    "threat": {
        "indicator": {
            "as": {
                "number": 8075,
                "organization": {
                    "name": "microsoft-corp-msn-as-block"
                }
            },
            "confidence": "High",
            "first_seen": "2022-07-20T20:25:53.000Z",
            "geo": {
                "country_iso_code": "br",
                "location": {
                    "lat": -22.9035,
                    "lon": -47.0565
                },
                "region_name": "sao paulo",
                "timezone": "america/sao_paulo"
            },
            "ip": "20.206.75.106",
            "last_seen": "2022-07-20T20:25:53.000Z",
            "marking": {
                "tlp": "WHITE"
            },
            "modified_at": "2022-07-21T20:33:26.585967Z",
            "name": "20.206.75.106",
            "provider": "sslbl.abuse.ch",
            "reference": "https://sslbl.abuse.ch/blacklist/sslipblacklist.csv",
            "sightings": 1,
            "type": "ipv4-addr"
        }
    }
}

Changelog

edit

Changelog

Version	Details	Kibana version(s)
1.16.0	Enhancement (View pull request) Do not remove `event.original` in main ingest pipeline.	8.13.0 or higher
1.15.0	Enhancement (View pull request) Add "preserve_original_event" tag to documents with `event.kind` set to "pipeline_error".	8.13.0 or higher
1.14.4	Bug fix (View pull request) Use triple-brace Mustache templating when referencing variables in ingest pipelines.	8.13.0 or higher
1.14.3	Bug fix (View pull request) Fix labels.is_ioc_transform_source values	8.13.0 or higher
1.14.2	Bug fix (View pull request) Add missing fields in transform	8.13.0 or higher
1.14.1	Bug fix (View pull request) Fix ECS date mapping on threat fields.	8.13.0 or higher
1.14.0	Enhancement (View pull request) Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.	8.13.0 or higher
1.13.1	Bug fix (View pull request) Adjust field mappings for transform destination index.	8.12.0 or higher
1.13.0	Enhancement (View pull request) Improve handling of empty responses.	8.12.0 or higher
1.12.0	Enhancement (View pull request) Support for IOC expiration	8.12.0 or higher
1.11.0	Enhancement (View pull request) Set sensitive values as secret.	8.12.0 or higher
1.10.2	Bug fix (View pull request) Clean up null handling	8.7.1 or higher
1.10.1	Enhancement (View pull request) Changed owners	8.7.1 or higher
1.10.0	Enhancement (View pull request) Limit request tracer log count to five.	8.7.1 or higher
1.9.0	Enhancement (View pull request) ECS version updated to 8.11.0.	8.7.1 or higher
1.8.0	Enhancement (View pull request) Set community owner type.	8.7.1 or higher
1.7.0	Enhancement (View pull request) ECS version updated to 8.10.0.	8.7.1 or higher
1.6.0	Enhancement (View pull request) The format_version in the package manifest changed from 2.11.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added owner.type: elastic to package manifest.	8.7.1 or higher
1.5.0	Enhancement (View pull request) Add tags.yml file so that integration’s dashboards and saved searches are tagged with "Security Solution" and displayed in the Security Solution UI.	8.7.1 or higher
1.4.0	Enhancement (View pull request) Update package-spec to 2.9.0.	8.7.1 or higher
1.3.0	Enhancement (View pull request) Update package to ECS 8.9.0.	8.7.1 or higher
1.2.0	Enhancement (View pull request) Document duration units.	8.7.1 or higher
1.1.0	Enhancement (View pull request) Document valid duration units.	8.7.1 or higher
1.0.0	Enhancement (View pull request) Release Collective Intelligence Framework as GA.	8.7.1 or higher
0.8.0	Enhancement (View pull request) Ensure event.kind is correctly set for pipeline errors.	—
0.7.0	Enhancement (View pull request) Update package to ECS 8.8.0.	—
0.6.0	Enhancement (View pull request) Add a new flag to enable request tracing	—
0.5.0	Enhancement (View pull request) Update package to ECS 8.7.0.	—
0.4.1	Enhancement (View pull request) Honor `preserve_original_event` setting.	—
0.4.0	Enhancement (View pull request) Update package to ECS 8.6.0.	—
0.3.1	Bug fix (View pull request) Use ECS definition for `threat.indicator.geo.location`.	—
0.3.0	Enhancement (View pull request) Update package to ECS 8.5.0.	—
0.2.2	Bug fix (View pull request) Remove duplicate field.	—
0.2.1	Enhancement (View pull request) Fix documentation build error	—
0.2.0	Enhancement (View pull request) Labelling with Threat Intelligence category	—
0.1.0	Enhancement (View pull request) Initial draft of the package	—

« Anomali Integration Custom Threat Intelligence integration »