AlienVault OTX

Ingest threat intelligence indicators from AlienVault Open Threat Exchange (OTX) with Elastic Agent.


Version	1.25.2 (View all)
Compatible Kibana version(s)	8.13.0 or higher
Supported Serverless project types What's this?	Security Observability
Subscription level What's this?	Basic
Level of support What's this?	Elastic

This integration is for Alienvault OTX. It retrieves indicators for all pulses subscribed to a specific user account on OTX

Configuration

To use this package, it is required to have an account on Alienvault OTX. Once an account has been created, and at least 1 pulse has been subscribed to, the API key can be retrieved from your user profile dashboard. In the top right corner there should be an OTX KEY.

Logs

Threat

Retrieves all the related indicators over time, related to your pulse subscriptions on OTX.

Exported fields

Field	Description	Type
@timestamp	Event timestamp.	date
cloud.image.id	Image ID for the cloud instance.	keyword
data_stream.dataset	Data stream dataset name.	constant_keyword
data_stream.namespace	Data stream namespace.	constant_keyword
data_stream.type	Data stream type.	constant_keyword
event.dataset	Event dataset	constant_keyword
event.module	Event module	constant_keyword
host.containerized	If the host is a container.	boolean
host.os.build	OS build information.	keyword
host.os.codename	OS codename, if any.	keyword
input.type	Type of Filebeat input.	keyword
log.flags	Flags for the log file.	keyword
log.offset	Offset of the entry in the log file.	long
otx.content	Extra text or descriptive content related to the indicator.	keyword
otx.description	A description of the indicator.	keyword
otx.id	The ID of the indicator.	keyword
otx.indicator	The value of the indicator, for example if the type is domain, this would be the value.	keyword
otx.title	Title describing the indicator.	keyword
otx.type	The indicator type, can for example be "domain, email, FileHash-SHA256".	keyword
threat.feed.dashboard_id	Dashboard ID used for Kibana CTI UI	constant_keyword
threat.feed.name	Display friendly feed name	constant_keyword
threat.indicator.file.hash.pehash	The file's pehash, if available.	keyword
threat.indicator.first_seen	The date and time when intelligence source first reported sighting this indicator.	date
threat.indicator.last_seen	The date and time when intelligence source last reported sighting this indicator.	date
threat.indicator.modified_at	The date and time when intelligence source last modified information for this indicator.	date

An example event for threat looks as following:

{
    "@timestamp": "2024-03-08T02:55:33.690Z",
    "agent": {
        "ephemeral_id": "8edc1f21-05cd-4fa5-aadc-66e64f44856a",
        "id": "f29e7d89-991e-4f0a-838f-9c2eb93d876e",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.12.1"
    },
    "data_stream": {
        "dataset": "ti_otx.threat",
        "namespace": "ep",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "f29e7d89-991e-4f0a-838f-9c2eb93d876e",
        "snapshot": false,
        "version": "8.12.1"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "threat"
        ],
        "created": "2024-03-08T02:55:33.690Z",
        "dataset": "ti_otx.threat",
        "ingested": "2024-03-08T02:55:45Z",
        "kind": "enrichment",
        "original": "{\"count\":40359,\"next\":\"https://otx.alienvault.com/api/v1/indicators/export?types=domain%2CIPv4%2Chostname%2Curl%2CFileHash-SHA256\\u0026modified_since=2020-11-29T01%3A10%3A00+00%3A00\\u0026page=2\",\"previous\":null,\"results\":{\"content\":\"\",\"description\":null,\"id\":1251,\"indicator\":\"info.3000uc.com\",\"title\":null,\"type\":\"hostname\"}}",
        "type": [
            "indicator"
        ]
    },
    "input": {
        "type": "httpjson"
    },
    "otx": {},
    "tags": [
        "preserve_original_event",
        "forwarded",
        "otx-threat"
    ],
    "threat": {
        "indicator": {
            "type": "domain-name",
            "url": {
                "domain": "info.3000uc.com"
            }
        }
    }
}

Pulses Subscribed (Recommended)

Retrieves all indicators from subscribed pulses on OTX from API /api/v1/pulses/subscribed using Filebeat's CEL input. The following subscriptions are included by this API:

All pulses by users you are subscribed to
All pulses you are directly subscribed to
All pulses you have created yourself
All pulses from groups you are a member of

Indicators of Comprosie (IoC) Expiration

Pulses Subscribed datastream also supports IoC expiration by using latest transform. Below are the steps on how it is handled:

All the indicators are retrieved into source indices named logs-ti_otx.pulses_subscribed-* using CEL input and processed via ingest pipelines. These indicators have a property named expiration which is either a null value or a timestamp such as "2023-09-07T00:00:00". When the value is null or if the timestamp value is less than current timestamp now(), the indicator is not expired, and hence is still active.
A latest transform is continuosly run on source indices. The purpose of this transform is to:
- Move only the active indicators from source indices into destination indices named logs-ti_otx_latest.pulses_subscribed-<NUMBER> where NUMBER indicates index version.
- Delete expired indicators based on the expiration timestamp value.
All the active indicators can be retrieved using destination index alias logs-ti_otx_latest.pulses_subscribed which points to the latest destination index version.

Note: Do not use the source indices logs-ti_otx.pulses_subscribed-*, because when the indicators expire, the source indices will contain duplicates. Always use the destination index alias: logs-ti_otx_latest.pulses_subscribed to query all active indicators.

Exported fields

Field	Description	Type
@timestamp	Event timestamp.	date
data_stream.dataset	Data stream dataset name.	constant_keyword
data_stream.namespace	Data stream namespace.	constant_keyword
data_stream.type	Data stream type.	constant_keyword
event.dataset	Event dataset	constant_keyword
event.module	Event module	constant_keyword
input.type	Type of Filebeat input.	keyword
labels.is_ioc_transform_source	Field indicating if its the transform source for supporting IOC expiration. This field is dropped from destination indices to facilitate easier filtering of indicators.	constant_keyword
log.flags	Flags for the log file.	keyword
log.offset	Offset of the entry in the log file.	long
otx.content		keyword
otx.count		integer
otx.created		date
otx.description		keyword
otx.expiration		date
otx.id	The ID of the indicator.	keyword
otx.indicator		keyword
otx.is_active		integer
otx.prefetch_pulse_ids		boolean
otx.pulse.adversary		keyword
otx.pulse.attack_ids		keyword
otx.pulse.author_name		keyword
otx.pulse.created		date
otx.pulse.description		keyword
otx.pulse.extract_source		keyword
otx.pulse.id		keyword
otx.pulse.industries		keyword
otx.pulse.malware_families		keyword
otx.pulse.modified		date
otx.pulse.more_indicators		boolean
otx.pulse.name		keyword
otx.pulse.public		integer
otx.pulse.references		keyword
otx.pulse.revision		integer
otx.pulse.targeted_countries		keyword
otx.pulse.tlp		keyword
otx.role		keyword
otx.t		double
otx.t2		double
otx.t3		double
otx.title		keyword
threat.feed.dashboard_id	Dashboard ID used for Kibana CTI UI	constant_keyword
threat.feed.name	Display friendly feed name	constant_keyword
threat.indicator.file.hash.pehash	The file's pehash, if available.	keyword
threat.indicator.first_seen	The date and time when intelligence source first reported sighting this indicator.	date
threat.indicator.last_seen	The date and time when intelligence source last reported sighting this indicator.	date
threat.indicator.modified_at	The date and time when intelligence source last modified information for this indicator.	date

An example event for pulses_subscribed looks as following:

{
    "@timestamp": "2023-08-08T05:05:15.000Z",
    "agent": {
        "ephemeral_id": "c12b4f72-265e-41f0-96e0-103c81de7908",
        "id": "8299ae35-ee0e-4107-9acb-1b6acfdda1fb",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.13.0"
    },
    "data_stream": {
        "dataset": "ti_otx.pulses_subscribed",
        "namespace": "32586",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "8299ae35-ee0e-4107-9acb-1b6acfdda1fb",
        "snapshot": false,
        "version": "8.13.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "threat"
        ],
        "dataset": "ti_otx.pulses_subscribed",
        "ingested": "2024-08-02T06:03:28Z",
        "kind": "enrichment",
        "original": "{\"content\":\"\",\"count\":2,\"created\":\"2023-08-08T05:05:15\",\"description\":\"\",\"expiration\":null,\"id\":3454375108,\"indicator\":\"pinup-casino-tr.site\",\"is_active\":1,\"prefetch_pulse_ids\":false,\"pulse_raw\":\"{\\\"adversary\\\":\\\"\\\",\\\"attack_ids\\\":[\\\"T1531\\\",\\\"T1059\\\",\\\"T1566\\\"],\\\"author_name\\\":\\\"SampleUser\\\",\\\"created\\\":\\\"2023-08-22T09:43:18.855000\\\",\\\"description\\\":\\\"\\\",\\\"extract_source\\\":[],\\\"id\\\":\\\"64e38336d783f91d6948a7b1\\\",\\\"industries\\\":[],\\\"malware_families\\\":[\\\"WHIRLPOOL\\\"],\\\"modified\\\":\\\"2023-08-22T09:43:18.855000\\\",\\\"more_indicators\\\":false,\\\"name\\\":\\\"Sample Pulse\\\",\\\"public\\\":1,\\\"references\\\":[\\\"https://www.cisa.gov/news-events/analysis-reports/ar23-230a\\\"],\\\"revision\\\":1,\\\"tags\\\":[\\\"cisa\\\",\\\"backdoor\\\",\\\"whirlpool\\\",\\\"malware\\\"],\\\"targeted_countries\\\":[],\\\"tlp\\\":\\\"white\\\"}\",\"role\":null,\"t\":0,\"t2\":0.0050694942474365234,\"t3\":2.7960586547851562,\"title\":\"\",\"type\":\"domain\"}",
        "type": [
            "indicator"
        ]
    },
    "input": {
        "type": "cel"
    },
    "otx": {
        "count": 2,
        "created": "2023-08-08T05:05:15.000Z",
        "expiration": "2023-08-13T05:05:15.000Z",
        "id": "3454375108",
        "is_active": 1,
        "prefetch_pulse_ids": false,
        "pulse": {
            "attack_ids": [
                "T1531",
                "T1059",
                "T1566"
            ],
            "author_name": "SampleUser",
            "created": "2023-08-22T09:43:18.855Z",
            "description": "",
            "extract_source": [],
            "id": "64e38336d783f91d6948a7b1",
            "industries": [],
            "malware_families": [
                "WHIRLPOOL"
            ],
            "modified": "2023-08-22T09:43:18.855Z",
            "more_indicators": false,
            "name": "Sample Pulse",
            "public": 1,
            "references": [
                "https://www.cisa.gov/news-events/analysis-reports/ar23-230a"
            ],
            "revision": 1,
            "targeted_countries": [],
            "tlp": "white"
        },
        "t": 0,
        "t2": 0.0050694942474365234,
        "t3": 2.7960586547851562
    },
    "tags": [
        "preserve_original_event",
        "forwarded",
        "otx-pulses_subscribed",
        "cisa",
        "backdoor",
        "whirlpool",
        "malware"
    ],
    "threat": {
        "indicator": {
            "provider": "OTX",
            "type": "domain-name",
            "url": {
                "domain": "pinup-casino-tr.site"
            }
        }
    }
}

Changelog

Version	Details	Kibana version(s)
1.25.2	Bug fix View pull request Add missing fields in transform	8.13.0 or higher
1.25.1	Bug fix View pull request Fix ECS date mapping on threat fields.	8.13.0 or higher
1.25.0	Enhancement View pull request Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.	8.13.0 or higher
1.24.1	Bug fix View pull request Fix type-mapping inconsistency for `otx.id` field.	8.12.0 or higher
1.24.0	Enhancement View pull request Set sensitive values as secret.	8.12.0 or higher
1.23.2	Enhancement View pull request Changed owners	8.10.3 or higher
1.23.1	Bug fix View pull request Fix IOC expiration duration character casting.	8.10.3 or higher
1.23.0	Enhancement View pull request Append hash and IP values to related.* fields	8.10.3 or higher
1.22.0	Enhancement View pull request Limit request tracer log count to five.	8.10.3 or higher
1.21.0	Enhancement View pull request ECS version updated to 8.11.0.	8.10.3 or higher
1.20.0	Enhancement View pull request Improve 'event.original' check to avoid errors if set.	8.10.3 or higher
1.19.0	Enhancement View pull request Add Pulses Subscribed datastream to support IoC expiration Enhancement View pull request Add DLM policy. Add owner.type to package manifest.	8.10.3 or higher
1.18.0	Enhancement View pull request ECS version updated to 8.10.0.	8.7.1 or higher
1.17.0	Enhancement View pull request The format_version in the package manifest changed from 2.11.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added 'owner.type: elastic' to package manifest.	8.7.1 or higher
1.16.0	Enhancement View pull request Add tags.yml file so that integration's dashboards and saved searches are tagged with "Security Solution" and displayed in the Security Solution UI.	8.7.1 or higher
1.15.0	Enhancement View pull request Update package-spec to 2.10.0.	8.7.1 or higher
1.14.0	Enhancement View pull request Update package to ECS 8.9.0.	8.7.1 or higher
1.13.0	Enhancement View pull request Document duration units.	8.7.1 or higher
1.12.0	Enhancement View pull request Document valid duration units.	8.7.1 or higher
1.11.0	Enhancement View pull request Ensure event.kind is correctly set for pipeline errors.	8.7.1 or higher
1.10.0	Enhancement View pull request Update package to ECS 8.8.0.	8.7.1 or higher
1.9.0	Enhancement View pull request Add a new flag to enable request tracing	8.7.1 or higher
1.8.0	Enhancement View pull request Update package to ECS 8.7.0.	8.0.0 or higher
1.7.1	Enhancement View pull request Honor `preserve_original_event` setting.	8.0.0 or higher
1.7.0	Enhancement View pull request Update package to ECS 8.6.0.	8.0.0 or higher
1.6.1	Enhancement View pull request Add support to drop empty documents	8.0.0 or higher
1.6.0	Enhancement View pull request Update package to ECS 8.5.0.	8.0.0 or higher
1.5.0	Enhancement View pull request Update package to ECS 8.4.0	8.0.0 or higher
1.4.2	Bug fix View pull request Fix proxy URL documentation rendering.	8.0.0 or higher
1.4.1	Enhancement View pull request Update categories to include `threat_intel`.	8.0.0 or higher
1.4.0	Enhancement View pull request Update package to ECS 8.3.0.	8.0.0 or higher
1.3.2	Enhancement View pull request Update readme file to add documentation link	8.0.0 or higher
1.3.1	Enhancement View pull request Update package descriptions	8.0.0 or higher
1.3.0	Enhancement View pull request Update to ECS 8.2	8.0.0 or higher
1.2.2	Enhancement View pull request Add field mapping for event.created	8.0.0 or higher
1.2.1	Enhancement View pull request Add documentation for multi-fields	8.0.0 or higher
1.2.0	Enhancement View pull request Update to ECS 8.0	8.0.0 or higher
1.1.0	Enhancement View pull request Adding threat.feed fields and dashboards	8.0.0 or higher
1.0.3	Bug fix View pull request Change test public IPs to the supported subset	8.0.0 or higher
1.0.2	Enhancement View pull request Bump minimum version	8.0.0 or higher
1.0.1	Enhancement View pull request Update title and description.	—
1.0.0	Enhancement View pull request Initial release	—

On this page