Anomali Integration

Version	1.25.0 (View all)
Compatible Kibana version(s)	8.13.0 or higher
Supported Serverless project types What’s this?	Security Observability
Subscription level What’s this?	Basic
Level of support What’s this?	Elastic

The Anomali integration can fetch indicators from Anomali ThreatStream, a commercial Threat Intelligence service.

It has the following data streams:

intelligence Indicators retrieved from the Anomali ThreatStream API’s intelligence endpoint.
threatstream Indicators received from the Anomali ThreatStream Elastic Extension, which is additional software. This is deprecated.

Logs

edit

Expiration of Indicators of Compromise (IOCs)

edit

An Elastic Transform is created to provide a view of active indicators for end users. The transform creates destination indices that are accessible via the alias of the form logs-ti_anomali_latest.<datastreamname>. When querying for active indicators or setting up indicator match rules, use the alias to avoid false positives from expired indicators. The dashboards show only the latest indicators.

Handling Orphaned IOCs

edit

Indicator data from Anomali can contain information about deletion or expiry times. However, some Anomali IOCs may never expire and will continue to stay in the latest destination index. To avoid any false positives from such orphaned IOCs, users are allowed to configure an "IOC Expiration Duration" or "IOC Duration Before Deletion" parameter while setting up a policy. The value set there will limit the time that indicators are retained before deletion, but indicators may be removed earlier based on information from Anomali.

Destination index versioning and deleting older versions

edit

The destination indices created by the transform are versioned with an integer suffix such as -1, -2, for example, logs-ti_anomali_latest.intelligence-1.

Due to schema changes in the destination index, its version number may be incremented.

When this happens, the transform does not have the functionality to auto-delete the old index, so users must delete this old index manually. This is to ensure that duplicates are not present when using wildcard queries such as logs-ti_anomali_latest.intelligence-*. To delete an old index, follow the steps below (either for intelligence as below, or for the older threatstream equivalents):

After upgrading the integration to the latest version, check the current transform’s destination index version by navigating to: Stack Management -> Transforms -> logs-ti_anomali.latest_intelligence-default -> Details. Check the destination_index value.
Run GET _cat/indices?v and check if any older versions exist. Such as logs-ti_anomali_latest.intelligence-1
Run DELETE logs-ti_anomali_latest.intelligence-<OLDVERSION> to delete the old index.

ILM Policies

edit

To prevent unbounded growth of the source data streams logs-ti_opencti.<datastreamname>-*, index lifecycle management (ILM) policies will deletes records 5 days after ingestion.

Anomali ThreatStream API

edit

The Anomali ThreatStream API’s intelligence endpoint is the preferred source of indicators. This data will be be accessible using the alias logs-ti_anomali_latest.intelligence.

Example

An example event for intelligence looks as following:

{
    "@timestamp": "2024-10-02T16:04:31.789615115Z",
    "agent": {
        "ephemeral_id": "bfe2b7b4-003a-49a1-b51b-e41ece98943b",
        "id": "72cb3ab8-2baa-4ae5-9a62-ee752a56df42",
        "name": "elastic-agent-85520",
        "type": "filebeat",
        "version": "8.14.3"
    },
    "anomali": {
        "threatstream": {
            "can_add_public_tags": true,
            "confidence": 60,
            "deletion_scheduled_at": "2024-10-09T16:04:31.789615115Z",
            "expiration_ts": "9999-12-31T00:00:00.000Z",
            "feed_id": 0,
            "id": "232020126",
            "is_anonymous": false,
            "is_editable": false,
            "is_public": true,
            "itype": "apt_domain",
            "meta": {
                "severity": "very-high"
            },
            "owner_organization_id": 67,
            "retina_confidence": -1,
            "source_reported_confidence": 60,
            "status": "active",
            "threat_type": "apt",
            "type": "domain",
            "update_id": 100000001,
            "uuid": "0921be47-9cc2-4265-b896-c62a7cb91042",
            "value": "gen1xyz.com"
        }
    },
    "data_stream": {
        "dataset": "ti_anomali.intelligence",
        "namespace": "49937",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "72cb3ab8-2baa-4ae5-9a62-ee752a56df42",
        "snapshot": false,
        "version": "8.14.3"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "threat"
        ],
        "created": "2021-04-06T09:56:22.915Z",
        "dataset": "ti_anomali.intelligence",
        "ingested": "2024-10-02T16:04:31Z",
        "kind": "enrichment",
        "original": "{\"asn\":\"\",\"can_add_public_tags\":true,\"confidence\":60,\"created_by\":null,\"created_ts\":\"2021-04-06T09:56:22.915Z\",\"description\":null,\"expiration_ts\":\"9999-12-31T00:00:00.000Z\",\"feed_id\":0,\"id\":232020126,\"is_anonymous\":false,\"is_editable\":false,\"is_public\":true,\"itype\":\"apt_domain\",\"locations\":[],\"meta\":{\"detail2\":\"imported by user 136\",\"severity\":\"very-high\"},\"modified_ts\":\"2021-04-06T09:56:22.915Z\",\"org\":\"\",\"owner_organization_id\":67,\"rdns\":null,\"resource_uri\":\"/api/v2/intelligence/232020126/\",\"retina_confidence\":-1,\"sort\":[455403032],\"source\":\"Analyst\",\"source_locations\":[],\"source_reported_confidence\":60,\"status\":\"active\",\"subtype\":null,\"tags\":null,\"target_industry\":[],\"threat_type\":\"apt\",\"threatscore\":54,\"tlp\":null,\"trusted_circle_ids\":null,\"type\":\"domain\",\"update_id\":100000001,\"uuid\":\"0921be47-9cc2-4265-b896-c62a7cb91042\",\"value\":\"gen1xyz.com\",\"workgroups\":[]}",
        "severity": 9,
        "type": [
            "indicator"
        ]
    },
    "input": {
        "type": "cel"
    },
    "tags": [
        "preserve_original_event",
        "forwarded",
        "anomali-intelligence"
    ],
    "threat": {
        "indicator": {
            "confidence": "Medium",
            "marking": {
                "tlp": "WHITE"
            },
            "modified_at": "2021-04-06T09:56:22.915Z",
            "provider": "Analyst",
            "type": "domain-name",
            "url": {
                "domain": "gen1xyz.com"
            }
        }
    }
}

Exported fields

Field	Description	Type
@timestamp	Event timestamp.	date
anomali.threatstream.can_add_public_tags	Indicates whether a user can add public tags to a Threat Model entity.	boolean
anomali.threatstream.confidence	Level of certainty that an observable is of the reported indicator type. Confidence scores range from 0-100, in increasing order of confidence, and is assigned by ThreatStream based on several factors.	long
anomali.threatstream.deletion_scheduled_at	At this time the IOC will be deleted by the transform.	date
anomali.threatstream.expiration_ts	Time stamp of when intelligence will expire on ThreatStream, in UTC time. Note: expiration_ts can only be specified in an advanced search query.	date
anomali.threatstream.feed_id	Numeric ID of the threat feed that generated the indicator. feed_id = 0 for user-created indicators.	long
anomali.threatstream.id	Unique ID for the indicator. This identifier is assigned to the indicator when it is first created on ThreatStream. Unlike update_id, this identifier never changes as long as the indicator is available on ThreatStream.	keyword
anomali.threatstream.import_session_id	ID of import session in which the indicator was imported. import_session_id=0 if the indicator came in through a threat feed.	long
anomali.threatstream.is_anonymous	Whether the organization and user information is anonymized when the observable is accessed by users outside of the owner organization.	boolean
anomali.threatstream.is_editable	Indicates whether the imported entity can be updated by an intelligence source. This attribute is reserved for intelligence source providers and can be ignored.	boolean
anomali.threatstream.is_public	Visibility of the indicator—public or private. 0/False—if the indicator is private or belongs to a Trusted Circle 1/True—if the indicator is public Default: 0/False	boolean
anomali.threatstream.itype	Indicator type.	keyword
anomali.threatstream.meta.maltype	Tag that specifies the malware associated with an indicator.	keyword
anomali.threatstream.meta.registrant.address	Indicator domain WHOIS registrant address.	keyword
anomali.threatstream.meta.registrant.email	Indicator domain WHOIS registrant email.	keyword
anomali.threatstream.meta.registrant.name	Indicator domain WHOIS registrant name.	keyword
anomali.threatstream.meta.registrant.org	Indicator domain WHOIS registrant org.	keyword
anomali.threatstream.meta.registrant.phone	Indicator domain WHOIS registrant phone.	keyword
anomali.threatstream.meta.registration_created	Registration created.	date
anomali.threatstream.meta.registration_updated	Registration updated.	date
anomali.threatstream.meta.severity	Severity assigned to the indicator through machine-learning algorithms ThreatStream deploys. Possible values: low, medium, high, very-high	keyword
anomali.threatstream.owner_organization_id	ID of the (ThreatStream) organization that brought in the indicator, either through a threat feed or through the import process.	long
anomali.threatstream.rdns	Domain name (obtained through reverse domain name lookup) associated with the IP address that is associated with the indicator.	keyword
anomali.threatstream.retina_confidence	Confidence score assigned to the observable by Anomali machine learning algorithms.	long
anomali.threatstream.source_created	Time stamp of when the entity was created by its original source.	date
anomali.threatstream.source_modified	Time stamp of when the entity was last updated by its original source.	date
anomali.threatstream.source_reported_confidence	A risk score from 0 to 100, provided by the source of the indicator.	long
anomali.threatstream.status	Status assigned to the indicator. For example, active, inactive, falsepos.	keyword
anomali.threatstream.threat_type	Summarized threat type of the indicator. For example, malware, compromised, apt, c2, and so on.	keyword
anomali.threatstream.threatscore	Deprecated.	keyword
anomali.threatstream.trusted_circle_ids	IDs of the trusted circles with which the indicator is shared.	long
anomali.threatstream.type	Type of indicator—domain, email, ip, md5, string, url.	keyword
anomali.threatstream.update_id	An incrementing numeric identifier associated with each update to intelligence on ThreatStream.	long
anomali.threatstream.uuid	UUID (universally unique identifier) assigned to the observable for STIX compliance.	keyword
anomali.threatstream.value	Value of the observable. For example, 192.168.0.10 or http://www.google.com.	keyword
cloud.image.id	Image ID for the cloud instance.	keyword
data_stream.dataset	Data stream dataset name.	constant_keyword
data_stream.namespace	Data stream namespace.	constant_keyword
data_stream.type	Data stream type.	constant_keyword
event.dataset	Event dataset	constant_keyword
event.module	Event module	constant_keyword
host.containerized	If the host is a container.	boolean
host.os.build	OS build information.	keyword
host.os.codename	OS codename, if any.	keyword
input.type	Type of Filebeat input.	keyword
labels.is_ioc_transform_source	Indicates whether an IOC is in the raw source data stream, or the in latest destination index.	constant_keyword
threat.feed.dashboard_id	Dashboard ID used for Kibana CTI UI	constant_keyword
threat.feed.name	Display friendly feed name	constant_keyword

Anomali ThreatStream via the Elastic Extension

edit

This source of indicators is deprecated. New users should instead use the API source above. This source requires additional software, the Elastic Extension, to connect Anomali ThreatStream to this integration. It’s available on the ThreatStream download page.

Please refer to the documentation included with the extension for a detailed explanation on how to configure Anomali ThreatStream to send indicators to this integration.

Indicators ingested in this way will become accessible using the alias logs-ti_anomali_latest.threatstream.

Example

An example event for threatstream looks as following:

{
    "@timestamp": "2020-10-08T12:22:11.000Z",
    "agent": {
        "ephemeral_id": "2f4f6445-5077-4a66-8582-2c74e071b6dd",
        "id": "36b03887-7783-4bc4-b8c5-6f8997e4cd1a",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.13.0"
    },
    "anomali": {
        "threatstream": {
            "added_at": "2020-10-08T12:22:11.000Z",
            "classification": "public",
            "confidence": 20,
            "deleted_at": "2020-10-13T12:22:11.000Z",
            "detail2": "imported by user 184",
            "id": "3135167627",
            "import_session_id": "1400",
            "itype": "mal_domain",
            "resource_uri": "/api/v1/intelligence/P46279656657/",
            "severity": "high",
            "source_feed_id": "3143",
            "state": "active",
            "trusted_circle_ids": [
                "122"
            ],
            "update_id": "3786618776",
            "value_type": "domain"
        }
    },
    "data_stream": {
        "dataset": "ti_anomali.threatstream",
        "namespace": "44735",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "36b03887-7783-4bc4-b8c5-6f8997e4cd1a",
        "snapshot": false,
        "version": "8.13.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "threat"
        ],
        "dataset": "ti_anomali.threatstream",
        "ingested": "2024-08-01T07:49:22Z",
        "kind": "enrichment",
        "original": "{\"added_at\":\"2020-10-08T12:22:11\",\"classification\":\"public\",\"confidence\":20,\"country\":\"FR\",\"date_first\":\"2020-10-08T12:21:50\",\"date_last\":\"2020-10-08T12:24:42\",\"detail2\":\"imported by user 184\",\"domain\":\"d4xgfj.example.net\",\"id\":3135167627,\"import_session_id\":1400,\"itype\":\"mal_domain\",\"lat\":-49.1,\"lon\":94.4,\"org\":\"OVH Hosting\",\"resource_uri\":\"/api/v1/intelligence/P46279656657/\",\"severity\":\"high\",\"source\":\"Default Organization\",\"source_feed_id\":3143,\"srcip\":\"89.160.20.156\",\"state\":\"active\",\"trusted_circle_ids\":\"122\",\"update_id\":3786618776,\"value_type\":\"domain\"}",
        "severity": 7,
        "type": [
            "indicator"
        ]
    },
    "input": {
        "type": "http_endpoint"
    },
    "tags": [
        "preserve_original_event",
        "forwarded",
        "anomali-threatstream"
    ],
    "threat": {
        "indicator": {
            "as": {
                "organization": {
                    "name": "OVH Hosting"
                }
            },
            "confidence": "Low",
            "first_seen": "2020-10-08T12:21:50.000Z",
            "geo": {
                "country_iso_code": "FR",
                "location": {
                    "lat": -49.1,
                    "lon": 94.4
                }
            },
            "ip": "89.160.20.156",
            "last_seen": "2020-10-08T12:24:42.000Z",
            "marking": {
                "tlp": [
                    "WHITE"
                ]
            },
            "provider": "Default Organization",
            "type": "domain-name",
            "url": {
                "domain": "d4xgfj.example.net"
            }
        }
    }
}

Exported fields

Field	Description	Type
@timestamp	Event timestamp.	date
anomali.threatstream.added_at	Date when IOC was added.	date
anomali.threatstream.classification	Indicates whether an indicator is private or from a public feed and available publicly. Possible values: private, public.	keyword
anomali.threatstream.confidence	The measure of the accuracy (from 0 to 100) assigned by ThreatStream’s predictive analytics technology to indicators.	short
anomali.threatstream.deleted_at	Date when IOC was deleted/expired.	date
anomali.threatstream.detail2	Detail text for indicator.	text
anomali.threatstream.id	The ID of the indicator.	keyword
anomali.threatstream.import_session_id	ID of the import session that created the indicator on ThreatStream.	keyword
anomali.threatstream.itype	Indicator type. Possible values: "apt_domain", "apt_email", "apt_ip", "apt_url", "bot_ip", "c2_domain", "c2_ip", "c2_url", "i2p_ip", "mal_domain", "mal_email", "mal_ip", "mal_md5", "mal_url", "parked_ip", "phish_email", "phish_ip", "phish_url", "scan_ip", "spam_domain", "ssh_ip", "suspicious_domain", "tor_ip" and "torrent_tracker_url".	keyword
anomali.threatstream.maltype	Information regarding a malware family, a CVE ID, or another attack or threat, associated with the indicator.	wildcard
anomali.threatstream.md5	Hash for the indicator.	keyword
anomali.threatstream.resource_uri	Relative URI for the indicator details.	keyword
anomali.threatstream.severity	Criticality associated with the threat feed that supplied the indicator. Possible values: low, medium, high, very-high.	keyword
anomali.threatstream.source	Source for the indicator.	keyword
anomali.threatstream.source_feed_id	ID for the integrator source.	keyword
anomali.threatstream.state	State for this indicator.	keyword
anomali.threatstream.trusted_circle_ids	ID of the trusted circle that imported the indicator.	keyword
anomali.threatstream.update_id	Update ID.	keyword
anomali.threatstream.url	URL for the indicator.	keyword
anomali.threatstream.value_type	Data type of the indicator. Possible values: ip, domain, url, email, md5.	keyword
cloud.image.id	Image ID for the cloud instance.	keyword
data_stream.dataset	Data stream dataset name.	constant_keyword
data_stream.namespace	Data stream namespace.	constant_keyword
data_stream.type	Data stream type.	constant_keyword
event.dataset	Event dataset	constant_keyword
event.module	Event module	constant_keyword
host.containerized	If the host is a container.	boolean
host.os.build	OS build information.	keyword
host.os.codename	OS codename, if any.	keyword
input.type	Type of Filebeat input.	keyword
labels.is_ioc_transform_source	Indicates whether an IOC is in the raw source data stream, or the in latest destination index.	constant_keyword
log.flags	Flags for the log file.	keyword
log.offset	Offset of the entry in the log file.	long
threat.feed.dashboard_id	Dashboard ID used for Kibana CTI UI	constant_keyword
threat.feed.name	Display friendly feed name	constant_keyword
threat.indicator.first_seen	The date and time when intelligence source first reported sighting this indicator.	date
threat.indicator.last_seen	The date and time when intelligence source last reported sighting this indicator.	date
threat.indicator.modified_at	The date and time when intelligence source last modified information for this indicator.	date

Changelog

edit

Changelog

Version	Details	Kibana version(s)
1.25.0	Enhancement (View pull request) Do not remove `event.original` in main ingest pipeline.	8.13.0 or higher
1.24.0	Enhancement (View pull request) Add "preserve_original_event" tag to documents with `event.kind` set to "pipeline_error".	8.13.0 or higher
1.23.0	Enhancement (View pull request) Support the ThreatStream API	8.13.0 or higher
1.22.3	Bug fix (View pull request) Fix labels.is_ioc_transform_source values	8.13.0 or higher
1.22.2	Bug fix (View pull request) Add missing fields in transform	8.13.0 or higher
1.22.1	Bug fix (View pull request) Fix ECS date mapping on threat fields.	8.13.0 or higher
1.22.0	Enhancement (View pull request) Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.	8.13.0 or higher
1.21.0	Enhancement (View pull request) Add destination index alias and fix docs.	8.12.0 or higher
1.20.0	Enhancement (View pull request) Set sensitive values as secret.	8.12.0 or higher
1.19.2	Enhancement (View pull request) Changed owners	8.8.0 or higher
1.19.1	Bug fix (View pull request) Fix IOC expiration duration character casting.	8.8.0 or higher
1.19.0	Enhancement (View pull request) ECS version updated to 8.11.0.	8.8.0 or higher
1.18.0	Enhancement (View pull request) ECS version updated to 8.10.0.	8.8.0 or higher
1.17.0	Enhancement (View pull request) Add DLM policy. Add owner.type to package manifest. Update format_version to 3.0.0 Enhancement (View pull request) Add tags.yml file so that integration’s dashboards and saved searches are tagged with "Security Solution" and displayed in the Security Solution UI.	8.8.0 or higher
1.16.1	Bug fix (View pull request) Remove dotted YAML keys.	—
1.16.0	Enhancement (View pull request) Update package-spec to 2.10.0.	8.8.0 or higher
1.15.1	Bug fix (View pull request) Fix destination fields mapping. Note - Since transform’s destination index version is changed, users must manually delete the old index. See https://docs.elastic.co/integrations/ti_anomali#logs for details on destination index versioning.	8.8.0 or higher
1.15.0	Enhancement (View pull request) Update package to ECS 8.9.0.	8.8.0 or higher
1.14.1	Bug fix (View pull request) Change sort order field in latest transform	8.8.0 or higher
1.14.0	Enhancement (View pull request) Ensure event.kind is correctly set for pipeline errors.	8.8.0 or higher
1.13.0	Enhancement (View pull request) Update package to ECS 8.8.0.	8.8.0 or higher
1.12.0	Enhancement (View pull request) Add IOC field to transform source to easily filter destination indices with unexpired indicators	8.8.0 or higher
1.11.0	Enhancement (View pull request) Support for IoC Expiration	8.8.0 or higher
1.10.0	Enhancement (View pull request) Update package to ECS 8.7.0.	8.0.0 or higher
1.9.0	Enhancement (View pull request) Update package to ECS 8.6.0.	8.0.0 or higher
1.8.0	Enhancement (View pull request) Update package to ECS 8.5.0.	8.0.0 or higher
1.7.0	Enhancement (View pull request) Deprecate EOL Limo datastream. See https://www.anomali.com/resources/limo.	8.0.0 or higher
1.6.0	Enhancement (View pull request) Update package to ECS 8.4.0	8.0.0 or higher
1.5.1	Bug fix (View pull request) Fix proxy URL documentation rendering.	8.0.0 or higher
1.5.0	Enhancement (View pull request) Update categories to include `threat_intel`.	8.0.0 or higher
1.4.0	Enhancement (View pull request) Update package to ECS 8.3.0.	8.0.0 or higher
1.3.3	Enhancement (View pull request) added links to Anomoli documentation in readme	8.0.0 or higher
1.3.2	Bug fix (View pull request) Fix threatstream	8.0.0 or higher
1.3.1	Enhancement (View pull request) Update package descriptions	8.0.0 or higher
1.3.0	Enhancement (View pull request) Update to ECS 8.2	8.0.0 or higher
1.2.3	Enhancement (View pull request) Add mapping for event.created	8.0.0 or higher
1.2.2	Enhancement (View pull request) Add documentation for multi-fields	8.0.0 or higher
1.2.1	Bug fix (View pull request) Adding first interval to Anomali Limo policy UI	8.0.0 or higher
1.2.0	Enhancement (View pull request) Update to ECS 8.0	8.0.0 or higher
1.1.3	Bug fix (View pull request) Regenerate test files using the new GeoIP database	8.0.0 or higher
1.1.2	Bug fix (View pull request) Change test public IPs to the supported subset	—
1.1.1	Enhancement (View pull request) Fixing typo in base-fields.yml	—
1.1.0	Enhancement (View pull request) Adding dashboards and threat.feed ECS fields	—
1.0.2	Enhancement (View pull request) Bump minimum version	8.0.0 or higher
1.0.1	Enhancement (View pull request) Update title and description.	—
1.0.0	Enhancement (View pull request) Initial release	—

« Alienvault OTX Integration Collective Intelligence Framework v3 Integration »