AlienVault OTX

Ingest threat intelligence indicators from AlienVault Open Threat Exchange (OTX) with Elastic Agent.

What is an Elastic integration?
Version
1.25.3 (View all)
Compatible Kibana version(s)
8.13.0 or higher
Supported Serverless project types

What's this?
Security
Observability
Subscription level
What's this?
Basic
Level of support
What's this?
Elastic

This integration is for Alienvault OTX. It retrieves indicators for all pulses subscribed to a specific user account on OTX

Configuration

To use this package, it is required to have an account on Alienvault OTX. Once an account has been created, and at least 1 pulse has been subscribed to, the API key can be retrieved from your user profile dashboard. In the top right corner there should be an OTX KEY.

Logs

Threat

Retrieves all the related indicators over time, related to your pulse subscriptions on OTX.

Exported fields

FieldDescriptionType
@timestamp
Event timestamp.
date
cloud.image.id
Image ID for the cloud instance.
keyword
data_stream.dataset
Data stream dataset name.
constant_keyword
data_stream.namespace
Data stream namespace.
constant_keyword
data_stream.type
Data stream type.
constant_keyword
event.dataset
Event dataset
constant_keyword
event.module
Event module
constant_keyword
host.containerized
If the host is a container.
boolean
host.os.build
OS build information.
keyword
host.os.codename
OS codename, if any.
keyword
input.type
Type of Filebeat input.
keyword
log.flags
Flags for the log file.
keyword
log.offset
Offset of the entry in the log file.
long
otx.content
Extra text or descriptive content related to the indicator.
keyword
otx.description
A description of the indicator.
keyword
otx.id
The ID of the indicator.
keyword
otx.indicator
The value of the indicator, for example if the type is domain, this would be the value.
keyword
otx.title
Title describing the indicator.
keyword
otx.type
The indicator type, can for example be "domain, email, FileHash-SHA256".
keyword
threat.feed.dashboard_id
Dashboard ID used for Kibana CTI UI
constant_keyword
threat.feed.name
Display friendly feed name
constant_keyword
threat.indicator.file.hash.pehash
The file's pehash, if available.
keyword
threat.indicator.first_seen
The date and time when intelligence source first reported sighting this indicator.
date
threat.indicator.last_seen
The date and time when intelligence source last reported sighting this indicator.
date
threat.indicator.modified_at
The date and time when intelligence source last modified information for this indicator.
date

An example event for threat looks as following:

{
    "@timestamp": "2024-03-08T02:55:33.690Z",
    "agent": {
        "ephemeral_id": "8edc1f21-05cd-4fa5-aadc-66e64f44856a",
        "id": "f29e7d89-991e-4f0a-838f-9c2eb93d876e",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.12.1"
    },
    "data_stream": {
        "dataset": "ti_otx.threat",
        "namespace": "ep",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "f29e7d89-991e-4f0a-838f-9c2eb93d876e",
        "snapshot": false,
        "version": "8.12.1"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "threat"
        ],
        "created": "2024-03-08T02:55:33.690Z",
        "dataset": "ti_otx.threat",
        "ingested": "2024-03-08T02:55:45Z",
        "kind": "enrichment",
        "original": "{\"count\":40359,\"next\":\"https://otx.alienvault.com/api/v1/indicators/export?types=domain%2CIPv4%2Chostname%2Curl%2CFileHash-SHA256\\u0026modified_since=2020-11-29T01%3A10%3A00+00%3A00\\u0026page=2\",\"previous\":null,\"results\":{\"content\":\"\",\"description\":null,\"id\":1251,\"indicator\":\"info.3000uc.com\",\"title\":null,\"type\":\"hostname\"}}",
        "type": [
            "indicator"
        ]
    },
    "input": {
        "type": "httpjson"
    },
    "otx": {},
    "tags": [
        "preserve_original_event",
        "forwarded",
        "otx-threat"
    ],
    "threat": {
        "indicator": {
            "type": "domain-name",
            "url": {
                "domain": "info.3000uc.com"
            }
        }
    }
}

Retrieves all indicators from subscribed pulses on OTX from API /api/v1/pulses/subscribed using Filebeat's CEL input. The following subscriptions are included by this API:

  • All pulses by users you are subscribed to
  • All pulses you are directly subscribed to
  • All pulses you have created yourself
  • All pulses from groups you are a member of

Indicators of Comprosie (IoC) Expiration

Pulses Subscribed datastream also supports IoC expiration by using latest transform. Below are the steps on how it is handled:

  1. All the indicators are retrieved into source indices named logs-ti_otx.pulses_subscribed-* using CEL input and processed via ingest pipelines. These indicators have a property named expiration which is either a null value or a timestamp such as "2023-09-07T00:00:00". When the value is null or if the timestamp value is less than current timestamp now(), the indicator is not expired, and hence is still active.
  2. A latest transform is continuosly run on source indices. The purpose of this transform is to:
    • Move only the active indicators from source indices into destination indices named logs-ti_otx_latest.pulses_subscribed-<NUMBER> where NUMBER indicates index version.
    • Delete expired indicators based on the expiration timestamp value.
  3. All the active indicators can be retrieved using destination index alias logs-ti_otx_latest.pulses_subscribed which points to the latest destination index version.
  • Note: Do not use the source indices logs-ti_otx.pulses_subscribed-*, because when the indicators expire, the source indices will contain duplicates. Always use the destination index alias: logs-ti_otx_latest.pulses_subscribed to query all active indicators.

Exported fields

FieldDescriptionType
@timestamp
Event timestamp.
date
data_stream.dataset
Data stream dataset name.
constant_keyword
data_stream.namespace
Data stream namespace.
constant_keyword
data_stream.type
Data stream type.
constant_keyword
event.dataset
Event dataset
constant_keyword
event.module
Event module
constant_keyword
input.type
Type of Filebeat input.
keyword
labels.is_ioc_transform_source
Indicates whether an IOC is in the raw source data stream, or the in latest destination index.
constant_keyword
log.flags
Flags for the log file.
keyword
log.offset
Offset of the entry in the log file.
long
otx.content
keyword
otx.count
integer
otx.created
date
otx.description
keyword
otx.expiration
date
otx.id
The ID of the indicator.
keyword
otx.indicator
keyword
otx.is_active
integer
otx.prefetch_pulse_ids
boolean
otx.pulse.adversary
keyword
otx.pulse.attack_ids
keyword
otx.pulse.author_name
keyword
otx.pulse.created
date
otx.pulse.description
keyword
otx.pulse.extract_source
keyword
otx.pulse.id
keyword
otx.pulse.industries
keyword
otx.pulse.malware_families
keyword
otx.pulse.modified
date
otx.pulse.more_indicators
boolean
otx.pulse.name
keyword
otx.pulse.public
integer
otx.pulse.references
keyword
otx.pulse.revision
integer
otx.pulse.targeted_countries
keyword
otx.pulse.tlp
keyword
otx.role
keyword
otx.t
double
otx.t2
double
otx.t3
double
otx.title
keyword
threat.feed.dashboard_id
Dashboard ID used for Kibana CTI UI
constant_keyword
threat.feed.name
Display friendly feed name
constant_keyword
threat.indicator.file.hash.pehash
The file's pehash, if available.
keyword
threat.indicator.first_seen
The date and time when intelligence source first reported sighting this indicator.
date
threat.indicator.last_seen
The date and time when intelligence source last reported sighting this indicator.
date
threat.indicator.modified_at
The date and time when intelligence source last modified information for this indicator.
date

An example event for pulses_subscribed looks as following:

{
    "@timestamp": "2023-08-08T05:05:15.000Z",
    "agent": {
        "ephemeral_id": "c12b4f72-265e-41f0-96e0-103c81de7908",
        "id": "8299ae35-ee0e-4107-9acb-1b6acfdda1fb",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.13.0"
    },
    "data_stream": {
        "dataset": "ti_otx.pulses_subscribed",
        "namespace": "32586",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "8299ae35-ee0e-4107-9acb-1b6acfdda1fb",
        "snapshot": false,
        "version": "8.13.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "threat"
        ],
        "dataset": "ti_otx.pulses_subscribed",
        "ingested": "2024-08-02T06:03:28Z",
        "kind": "enrichment",
        "original": "{\"content\":\"\",\"count\":2,\"created\":\"2023-08-08T05:05:15\",\"description\":\"\",\"expiration\":null,\"id\":3454375108,\"indicator\":\"pinup-casino-tr.site\",\"is_active\":1,\"prefetch_pulse_ids\":false,\"pulse_raw\":\"{\\\"adversary\\\":\\\"\\\",\\\"attack_ids\\\":[\\\"T1531\\\",\\\"T1059\\\",\\\"T1566\\\"],\\\"author_name\\\":\\\"SampleUser\\\",\\\"created\\\":\\\"2023-08-22T09:43:18.855000\\\",\\\"description\\\":\\\"\\\",\\\"extract_source\\\":[],\\\"id\\\":\\\"64e38336d783f91d6948a7b1\\\",\\\"industries\\\":[],\\\"malware_families\\\":[\\\"WHIRLPOOL\\\"],\\\"modified\\\":\\\"2023-08-22T09:43:18.855000\\\",\\\"more_indicators\\\":false,\\\"name\\\":\\\"Sample Pulse\\\",\\\"public\\\":1,\\\"references\\\":[\\\"https://www.cisa.gov/news-events/analysis-reports/ar23-230a\\\"],\\\"revision\\\":1,\\\"tags\\\":[\\\"cisa\\\",\\\"backdoor\\\",\\\"whirlpool\\\",\\\"malware\\\"],\\\"targeted_countries\\\":[],\\\"tlp\\\":\\\"white\\\"}\",\"role\":null,\"t\":0,\"t2\":0.0050694942474365234,\"t3\":2.7960586547851562,\"title\":\"\",\"type\":\"domain\"}",
        "type": [
            "indicator"
        ]
    },
    "input": {
        "type": "cel"
    },
    "otx": {
        "count": 2,
        "created": "2023-08-08T05:05:15.000Z",
        "expiration": "2023-08-13T05:05:15.000Z",
        "id": "3454375108",
        "is_active": 1,
        "prefetch_pulse_ids": false,
        "pulse": {
            "attack_ids": [
                "T1531",
                "T1059",
                "T1566"
            ],
            "author_name": "SampleUser",
            "created": "2023-08-22T09:43:18.855Z",
            "description": "",
            "extract_source": [],
            "id": "64e38336d783f91d6948a7b1",
            "industries": [],
            "malware_families": [
                "WHIRLPOOL"
            ],
            "modified": "2023-08-22T09:43:18.855Z",
            "more_indicators": false,
            "name": "Sample Pulse",
            "public": 1,
            "references": [
                "https://www.cisa.gov/news-events/analysis-reports/ar23-230a"
            ],
            "revision": 1,
            "targeted_countries": [],
            "tlp": "white"
        },
        "t": 0,
        "t2": 0.0050694942474365234,
        "t3": 2.7960586547851562
    },
    "tags": [
        "preserve_original_event",
        "forwarded",
        "otx-pulses_subscribed",
        "cisa",
        "backdoor",
        "whirlpool",
        "malware"
    ],
    "threat": {
        "indicator": {
            "provider": "OTX",
            "type": "domain-name",
            "url": {
                "domain": "pinup-casino-tr.site"
            }
        }
    }
}

Changelog

VersionDetailsKibana version(s)

1.25.3

Bug fix View pull request
Fix labels.is_ioc_transform_source values

8.13.0 or higher

1.25.2

Bug fix View pull request
Add missing fields in transform

8.13.0 or higher

1.25.1

Bug fix View pull request
Fix ECS date mapping on threat fields.

8.13.0 or higher

1.25.0

Enhancement View pull request
Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.

8.13.0 or higher

1.24.1

Bug fix View pull request
Fix type-mapping inconsistency for otx.id field.

8.12.0 or higher

1.24.0

Enhancement View pull request
Set sensitive values as secret.

8.12.0 or higher

1.23.2

Enhancement View pull request
Changed owners

8.10.3 or higher

1.23.1

Bug fix View pull request
Fix IOC expiration duration character casting.

8.10.3 or higher

1.23.0

Enhancement View pull request
Append hash and IP values to related.* fields

8.10.3 or higher

1.22.0

Enhancement View pull request
Limit request tracer log count to five.

8.10.3 or higher

1.21.0

Enhancement View pull request
ECS version updated to 8.11.0.

8.10.3 or higher

1.20.0

Enhancement View pull request
Improve 'event.original' check to avoid errors if set.

8.10.3 or higher

1.19.0

Enhancement View pull request
Add Pulses Subscribed datastream to support IoC expiration

Enhancement View pull request
Add DLM policy. Add owner.type to package manifest.

8.10.3 or higher

1.18.0

Enhancement View pull request
ECS version updated to 8.10.0.

8.7.1 or higher

1.17.0

Enhancement View pull request
The format_version in the package manifest changed from 2.11.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added 'owner.type: elastic' to package manifest.

8.7.1 or higher

1.16.0

Enhancement View pull request
Add tags.yml file so that integration's dashboards and saved searches are tagged with "Security Solution" and displayed in the Security Solution UI.

8.7.1 or higher

1.15.0

Enhancement View pull request
Update package-spec to 2.10.0.

8.7.1 or higher

1.14.0

Enhancement View pull request
Update package to ECS 8.9.0.

8.7.1 or higher

1.13.0

Enhancement View pull request
Document duration units.

8.7.1 or higher

1.12.0

Enhancement View pull request
Document valid duration units.

8.7.1 or higher

1.11.0

Enhancement View pull request
Ensure event.kind is correctly set for pipeline errors.

8.7.1 or higher

1.10.0

Enhancement View pull request
Update package to ECS 8.8.0.

8.7.1 or higher

1.9.0

Enhancement View pull request
Add a new flag to enable request tracing

8.7.1 or higher

1.8.0

Enhancement View pull request
Update package to ECS 8.7.0.

8.0.0 or higher

1.7.1

Enhancement View pull request
Honor preserve_original_event setting.

8.0.0 or higher

1.7.0

Enhancement View pull request
Update package to ECS 8.6.0.

8.0.0 or higher

1.6.1

Enhancement View pull request
Add support to drop empty documents

8.0.0 or higher

1.6.0

Enhancement View pull request
Update package to ECS 8.5.0.

8.0.0 or higher

1.5.0

Enhancement View pull request
Update package to ECS 8.4.0

8.0.0 or higher

1.4.2

Bug fix View pull request
Fix proxy URL documentation rendering.

8.0.0 or higher

1.4.1

Enhancement View pull request
Update categories to include threat_intel.

8.0.0 or higher

1.4.0

Enhancement View pull request
Update package to ECS 8.3.0.

8.0.0 or higher

1.3.2

Enhancement View pull request
Update readme file to add documentation link

8.0.0 or higher

1.3.1

Enhancement View pull request
Update package descriptions

8.0.0 or higher

1.3.0

Enhancement View pull request
Update to ECS 8.2

8.0.0 or higher

1.2.2

Enhancement View pull request
Add field mapping for event.created

8.0.0 or higher

1.2.1

Enhancement View pull request
Add documentation for multi-fields

8.0.0 or higher

1.2.0

Enhancement View pull request
Update to ECS 8.0

8.0.0 or higher

1.1.0

Enhancement View pull request
Adding threat.feed fields and dashboards

8.0.0 or higher

1.0.3

Bug fix View pull request
Change test public IPs to the supported subset

8.0.0 or higher

1.0.2

Enhancement View pull request
Bump minimum version

8.0.0 or higher

1.0.1

Enhancement View pull request
Update title and description.

—

1.0.0

Enhancement View pull request
Initial release

—

On this page