Threat Intel module
editThreat Intel module
editThis functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.
This module ingests data from a collection of different threat intelligence
sources. The ingested data is meant to be used with
Indicator
Match rules, but is also compatible with other features like
Enrich
Processors. The related threat intel attribute that is meant to be used for
matching incoming source data is stored under the threatintel.indicator.*
fields.
The available filesets are:
edit-
abuseurl
: Supports gathering URL entities from Abuse.ch. -
abusemalware
: Supports gathering Malware/Payload entities from Abuse.ch. -
misp
: Supports gathering threat intel attributes from MISP (replaces MISP module). -
malwarebazaar
: Supports gathering Malware/Payload entities from Malware Bazaar. -
otx
: Supports gathering threat intel attributes from AlientVault OTX. -
anomali
: Supports gathering threat intel attributes from Anomali.
Read the quick start to learn how to configure and run modules.
abuseurl
fileset settings
editThis fileset contacts the abuse.ch API and fetches all new malicious URLs found the last 60 minutes.
To configure the module, please utilize the default URL unless specified as the example below:
- module: threatintel abuseurl: enabled: true var.input: httpjson var.url: https://urlhaus-api.abuse.ch/v1/urls/recent/ var.interval: 60m
-
var.paths
-
An array of glob-based paths that specify where to look for the log files. All
patterns supported by Go Glob
are also supported here. For example, you can use wildcards to fetch all files
from a predefined level of subdirectories:
/path/to/log/*/*.log
. This fetches all.log
files from the subfolders of/path/to/log
. It does not fetch log files from the/path/to/log
folder itself. If this setting is left empty, Filebeat will choose log paths based on your operating system. -
var.url
- The URL of the API endpoint to connect with.
-
var.interval
- How often the API is polled for updated information.
Abuse.ch URL Threat Intel is mapped to the following ECS fields.
URL Threat Intel Fields | ECS Fields |
---|---|
url |
threat.indicator.url.full |
date_added |
@timestamp |
host |
threatintel.indicator.ip/domain |
abusemalware
fileset settings
editThis fileset contacts the Abuse.ch API and fetches all new malicious hashes found the last 60 minutes.
To configure the module, please utilize the default URL unless specified as the example below:
- module: threatintel abusemalware: enabled: true var.input: httpjson var.url: https://urlhaus-api.abuse.ch/v1/payloads/recent/ var.interval: 60m
-
var.paths
-
An array of glob-based paths that specify where to look for the log files. All
patterns supported by Go Glob
are also supported here. For example, you can use wildcards to fetch all files
from a predefined level of subdirectories:
/path/to/log/*/*.log
. This fetches all.log
files from the subfolders of/path/to/log
. It does not fetch log files from the/path/to/log
folder itself. If this setting is left empty, Filebeat will choose log paths based on your operating system. -
var.url
- The URL of the API endpoint to connect with.
-
var.interval
- How often the API is polled for updated information.
Abuse.ch Malware Threat Intel is mapped to the following ECS fields.
Malware Threat IntelFields | ECS Fields |
---|---|
md5_hash |
threatintel.indicator.file.hash.md5 |
sha256_hash |
threatintel.indicator.file.hash.sha256 |
file_size |
threatintel.indicator.file.size |
malwarebazaar
fileset settings
editThis fileset contacts the Malware Bazaar API and fetches all new malicious hashes found the last 10 minutes.
To configure the module, please utilize the default URL unless specified as the example below:
- module: threatintel malwarebazaar: enabled: true var.input: httpjson var.url: https://mb-api.abuse.ch/api/v1/ var.interval: 10m
-
var.paths
-
An array of glob-based paths that specify where to look for the log files. All
patterns supported by Go Glob
are also supported here. For example, you can use wildcards to fetch all files
from a predefined level of subdirectories:
/path/to/log/*/*.log
. This fetches all.log
files from the subfolders of/path/to/log
. It does not fetch log files from the/path/to/log
folder itself. If this setting is left empty, Filebeat will choose log paths based on your operating system. -
var.url
- The URL of the API endpoint to connect with.
-
var.interval
- How often the API is polled for updated information.
Malware Bazaar Threat Intel is mapped to the following ECS fields.
Malware Threat IntelFields | ECS Fields |
---|---|
md5_hash |
threatintel.indicator.file.hash.md5 |
sha256_hash |
threatintel.indicator.file.hash.sha256 |
tlsh |
threatintel.indicator.file.hash.tlsh |
ssdeep |
threatintel.indicator.file.hash.ssdeep |
imphash |
threatintel.indicator.file.pe.imphash |
file_size |
threatintel.indicator.file.size |
file_name |
threatintel.indicator.file.name |
file_type_mime |
threatintel.indicator.file.mime_type |
file_type |
threatintel.indicator.file.type |
reporter |
threatintel.indicator.provider |
origin_country |
threatintel.indicator.geo.country_iso_code |
signature |
threatintel.indicator.signature |
code_sign.subject_cn |
threatintel.indicator.file.x509.subject.common_name |
code_sign.issuer_cn |
threatintel.indicator.file.x509.issuer.common_name |
code_sign.algorithm |
threatintel.indicator.file.x509.public_key_algorithm |
code_sign.valid_from |
threatintel.indicator.file.x509.not_before |
code_sign.valid_to |
threatintel.indicator.file.x509.not_after |
code_sign.serial_number |
threatintel.indicator.file.x509.serial_number |
misp
fileset settings
editThis fileset communicates with a local or remote MISP server. This replaces the older MISP module.
The fileset configuration allows to set the polling interval, how far back it should look initially, and optionally any filters used to filter the results.
- module: threatintel misp: enabled: true var.input: httpjson var.url: https://SERVER/events/restSearch var.api_token: xVfaM3DSt8QEwO2J1ix00V4ZHJs14nq5GMsHcK6Z var.first_interval: 24h var.interval: 60m
To configure the output with filters, use fields that already exist on the MISP server, and define either a single value or multiple. By adding a filter, only events that have attributes that match the filter will be returned.
The below filters are only examples, for a full list of all fields please reference the MISP fields located on the MISP server itself.
- module: threatintel misp: enabled: true var.input: httpjson var.url: https://SERVER/events/restSearch var.api_token: xVfaM3DSt8QEwO2J1ix00V4ZHJs14nq5GMsHcK6Z var.filters: - type: ["md5", "sha256", "url", "ip-src"] - threat_level: 4 var.first_interval: 24h var.interval: 60m
-
var.paths
-
An array of glob-based paths that specify where to look for the log files. All
patterns supported by Go Glob
are also supported here. For example, you can use wildcards to fetch all files
from a predefined level of subdirectories:
/path/to/log/*/*.log
. This fetches all.log
files from the subfolders of/path/to/log
. It does not fetch log files from the/path/to/log
folder itself. If this setting is left empty, Filebeat will choose log paths based on your operating system. -
var.url
- The URL of the API endpoint to connect with.
-
var.interval
- How often the API is polled for updated information.
-
var.first_interval
- How far back to search when retrieving events the first time the beat starts up. After the first interval has passed the module itself will use the timestamp from the last response as the filter when retrieving new events.
-
var.filters
- List of filters to apply when retrieving new events from the MISP server, this field is optional and defaults to all events.
MISP Threat Intel is mapped to the following ECS fields.
Malware Threat IntelFields | ECS Fields |
---|---|
misp.first_seen |
threatintel.indicator.first_seen |
misp.last_seen |
threatintel.indicator.last_seen |
misp.tag |
tag |
misp.value |
threatintel.indicator.* |
misp.value
is mapped to the appropriate field dependent on attribute type.
otx
fileset settings
editTo configure the module, please utilize the default URL unless specified as the example below:
- module: threatintel otx: enabled: true var.input: httpjson var.url: https://otx.alienvault.com/api/v1/indicators/export var.api_token: 754dcaafbcb9740dc0d119e72d5eaad699cc4a5cdbc856fc6215883842ba8142 var.first_interval: 24h var.lookback_range: 2h var.interval: 60m
To filter only on specific indicator types, this is an example of some possible filters that are supported:
- module: threatintel otx: enabled: true var.input: httpjson var.url: https://otx.alienvault.com/api/v1/indicators/export var.types: "domain,IPv4,hostname,url,FileHash-SHA256" var.first_interval: 24h var.interval: 60m
-
var.paths
-
An array of glob-based paths that specify where to look for the log files. All
patterns supported by Go Glob
are also supported here. For example, you can use wildcards to fetch all files
from a predefined level of subdirectories:
/path/to/log/*/*.log
. This fetches all.log
files from the subfolders of/path/to/log
. It does not fetch log files from the/path/to/log
folder itself. If this setting is left empty, Filebeat will choose log paths based on your operating system. -
var.url
- The URL of the API endpoint to connect with.
-
var.api_token
- The API key used to access OTX. This can be found on your OTX API homepage.
-
var.interval
- How often the API is polled for updated information.
-
var.first_interval
- How far back to search when retrieving events the first time the beat starts up. After the first interval has passed the module itself will use the timestamp from the last response as the filter when retrieving new events.
-
var.types
- A comma delimited list of indicator types to include, defaults to all. A list of possible types to filter on can be found in the AlientVault OTX documentation.
OTX Threat Intel is mapped to the following ECS fields.
Malware Threat Intel Fields | ECS Fields |
---|---|
otx.type |
threatintel.indicator.type |
otx.description |
threatintel.indicator.description |
otx.indicator |
threatintel.indicator.* |
otx.indicator
is mapped to the appropriate field dependent on attribute type.
anomali
fileset settings
editTo configure the module please fill in the credentials, for Anomali Limo (the free Taxii service) these are usually default credentials found at the Anomali Limo webpage Anomali Limo offers multiple sources called collections. Each collection has a specific ID, which then fits into the url used in this configuration. A list of different collections can be found using the credentials at Limo Collections.
The example below uses the collection of ID 41 as can be seen in the URL.
- module: threatintel anomali: enabled: true var.input: httpjson var.url: https://limo.anomali.com/api/v1/taxii2/feeds/collections/41/objects?match[type]=indicator var.username: guest var.password: guest var.interval: 60m
To filter on specific types, you can define var.types
as a comma delimited list
of object types. This defaults to "indicators".
- module: threatintel anomali: enabled: true var.input: httpjson var.url: https://limo.anomali.com/api/v1/taxii2/feeds/collections/41/objects?match[type]=indicator var.types: "indicators,other" var.username: guest var.password: guest var.interval: 60m
-
var.paths
-
An array of glob-based paths that specify where to look for the log files. All
patterns supported by Go Glob
are also supported here. For example, you can use wildcards to fetch all files
from a predefined level of subdirectories:
/path/to/log/*/*.log
. This fetches all.log
files from the subfolders of/path/to/log
. It does not fetch log files from the/path/to/log
folder itself. If this setting is left empty, Filebeat will choose log paths based on your operating system. -
var.url
- The URL of the API endpoint to connect with. Limo offers multiple collections of threat intelligence.
-
var.username
- Username used to access the API.
-
var.password
- Password used to access the API.
-
var.interval
- How often the API is polled for updated information.
-
var.types
- A comma delimited list of indicator types to include, defaults to all. A list of possible types to filter on can be found on the Stix 2.1 Object types page.
Anomali Threat Intel is mapped to the following ECS fields.
Malware Threat Intel Fields | ECS Fields |
---|---|
anomali.description |
threatintel.indicator.description |
anomali.created |
threatintel.indicator.first_seen |
anomali.modified |
threatintel.indicator.last_seen |
anomali.pattern |
threatintel.indicator.* |
anomali.labels |
tags |
anomali.pattern
is mapped to the appropriate field dependent on attribute type.
Dashboards
editThis module comes with dashboards for the threat information feeds.
Overview of the information provided, and the health of, the Threat Intel module.
Overview of the information provided by the Abuse.ch Malware feed.
Overview of the information provided by the Abuse.ch URL feed.
Overview of the information provided by the AlienVault OTX feed.
Overview of the information provided by the Anomali Limo feed.
Overview of the information provided by the MISP feed.
Fields
editFor a description of each field in the module, see the exported fields section.