IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

› ›

Threat Intel module

edit

Threat Intel module

edit

This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.

This module ingests data from a collection of different threat intelligence sources. The ingested data is meant to be used with Indicator Match rules, but is also compatible with other features like Enrich Processors. The related threat intel attribute that is meant to be used for matching incoming source data is stored under the threatintel.indicator.* fields.

The available filesets are:

edit

abuseurl: Supports gathering URL entities from Abuse.ch.
abusemalware: Supports gathering Malware/Payload entities from Abuse.ch.
misp: Supports gathering threat intel attributes from MISP (replaces MISP module).
malwarebazaar: Supports gathering Malware/Payload entities from Malware Bazaar.
otx: Supports gathering threat intel attributes from AlientVault OTX.
anomali: Supports gathering threat intel attributes from Anomali Limo.
anomalithreatstream: Supports gathering threat intel attributes from Anomali ThreatStream.
recordedfuture: Supports gathering threat intel attributes from Recorded Future.

Read the quick start to learn how to configure and run modules.

`abuseurl` fileset settings

edit

This fileset contacts the abuse.ch API and fetches all new malicious URLs found the last 60 minutes.

To configure the module, please utilize the default URL unless specified as the example below:

- module: threatintel
  abuseurl:
    enabled: true
    var.input: httpjson
    var.url: https://urlhaus-api.abuse.ch/v1/urls/recent/
    var.interval: 60m

var.paths: An array of glob-based paths that specify where to look for the log files. All patterns supported by Go Glob are also supported here. For example, you can use wildcards to fetch all files from a predefined level of subdirectories: /path/to/log/*/*.log. This fetches all .log files from the subfolders of /path/to/log. It does not fetch log files from the /path/to/log folder itself. If this setting is left empty, Filebeat will choose log paths based on your operating system.
var.url: The URL of the API endpoint to connect with.
var.interval: How often the API is polled for updated information.
var.proxy_url: Optional URL to use as HTTP proxy.

Abuse.ch URL Threat Intel is mapped to the following ECS fields.

URL Threat Intel Fields	ECS Fields
url	threatintel.indicator.url.full
date_added	@timestamp
host	threatintel.indicator.ip/domain

`abusemalware` fileset settings

edit

This fileset contacts the Abuse.ch API and fetches all new malicious hashes found the last 60 minutes.

To configure the module, please utilize the default URL unless specified as the example below:

- module: threatintel
  abusemalware:
    enabled: true
    var.input: httpjson
    var.url: https://urlhaus-api.abuse.ch/v1/payloads/recent/
    var.interval: 60m

var.paths: An array of glob-based paths that specify where to look for the log files. All patterns supported by Go Glob are also supported here. For example, you can use wildcards to fetch all files from a predefined level of subdirectories: /path/to/log/*/*.log. This fetches all .log files from the subfolders of /path/to/log. It does not fetch log files from the /path/to/log folder itself. If this setting is left empty, Filebeat will choose log paths based on your operating system.
var.url: The URL of the API endpoint to connect with.
var.interval: How often the API is polled for updated information.
var.proxy_url: Optional URL to use as HTTP proxy.

Abuse.ch Malware Threat Intel is mapped to the following ECS fields.

Malware Threat IntelFields	ECS Fields
md5_hash	threatintel.indicator.file.hash.md5
sha256_hash	threatintel.indicator.file.hash.sha256
file_size	threatintel.indicator.file.size

`malwarebazaar` fileset settings

edit

This fileset contacts the Malware Bazaar API and fetches all new malicious hashes found the last 10 minutes.

To configure the module, please utilize the default URL unless specified as the example below:

- module: threatintel
  malwarebazaar:
    enabled: true
    var.input: httpjson
    var.url: https://mb-api.abuse.ch/api/v1/
    var.interval: 10m

var.paths: An array of glob-based paths that specify where to look for the log files. All patterns supported by Go Glob are also supported here. For example, you can use wildcards to fetch all files from a predefined level of subdirectories: /path/to/log/*/*.log. This fetches all .log files from the subfolders of /path/to/log. It does not fetch log files from the /path/to/log folder itself. If this setting is left empty, Filebeat will choose log paths based on your operating system.
var.url: The URL of the API endpoint to connect with.
var.interval: How often the API is polled for updated information.
var.proxy_url: Optional URL to use as HTTP proxy.

Malware Bazaar Threat Intel is mapped to the following ECS fields.

Malware Threat IntelFields	ECS Fields
md5_hash	threatintel.indicator.file.hash.md5
sha256_hash	threatintel.indicator.file.hash.sha256
tlsh	threatintel.indicator.file.hash.tlsh
ssdeep	threatintel.indicator.file.hash.ssdeep
imphash	threatintel.indicator.file.pe.imphash
file_size	threatintel.indicator.file.size
file_name	threatintel.indicator.file.name
file_type_mime	threatintel.indicator.file.mime_type
file_type	threatintel.indicator.file.type
reporter	threatintel.indicator.provider
origin_country	threatintel.indicator.geo.country_iso_code
signature	threatintel.indicator.signature
code_sign.subject_cn	threatintel.indicator.file.x509.subject.common_name
code_sign.issuer_cn	threatintel.indicator.file.x509.issuer.common_name
code_sign.algorithm	threatintel.indicator.file.x509.public_key_algorithm
code_sign.valid_from	threatintel.indicator.file.x509.not_before
code_sign.valid_to	threatintel.indicator.file.x509.not_after
code_sign.serial_number	threatintel.indicator.file.x509.serial_number

`misp` fileset settings

edit

This fileset communicates with a local or remote MISP server. This replaces the older MISP module.

The fileset configuration allows to set the polling interval, how far back it should look initially, and optionally any filters used to filter the results.

- module: threatintel
  misp:
    enabled: true
    var.input: httpjson
    var.url: https://SERVER/events/restSearch
    var.api_token: xVfaM3DSt8QEwO2J1ix00V4ZHJs14nq5GMsHcK6Z
    var.first_interval: 24h
    var.interval: 60m

To configure the output with filters, use fields that already exist on the MISP server, and define either a single value or multiple. By adding a filter, only events that have attributes that match the filter will be returned.

The below filters are only examples, for a full list of all fields please reference the MISP fields located on the MISP server itself.

- module: threatintel
  misp:
    enabled: true
    var.input: httpjson
    var.url: https://SERVER/events/restSearch
    var.api_token: xVfaM3DSt8QEwO2J1ix00V4ZHJs14nq5GMsHcK6Z
    var.filters:
      - type: ["md5", "sha256", "url", "ip-src"]
      - threat_level: 4
    var.first_interval: 24h
    var.interval: 60m

var.paths: An array of glob-based paths that specify where to look for the log files. All patterns supported by Go Glob are also supported here. For example, you can use wildcards to fetch all files from a predefined level of subdirectories: /path/to/log/*/*.log. This fetches all .log files from the subfolders of /path/to/log. It does not fetch log files from the /path/to/log folder itself. If this setting is left empty, Filebeat will choose log paths based on your operating system.
var.url: The URL of the API endpoint to connect with.
var.interval: How often the API is polled for updated information.
var.first_interval: How far back to search when retrieving events the first time Filebeat starts up. After the first interval has passed the module itself will use the timestamp from the last response as the filter when retrieving new events.
var.filters: List of filters to apply when retrieving new events from the MISP server, this field is optional and defaults to all events.
var.proxy_url: Optional URL to use as HTTP proxy.

MISP Threat Intel is mapped to the following ECS fields.

Malware Threat IntelFields	ECS Fields
misp.first_seen	threatintel.indicator.first_seen
misp.last_seen	threatintel.indicator.last_seen
misp.tag	tag
misp.value	threatintel.indicator.*

misp.value is mapped to the appropriate field dependent on attribute type.

`otx` fileset settings

edit

To configure the module, please utilize the default URL unless specified as the example below:

- module: threatintel
  otx:
    enabled: true
    var.input: httpjson
    var.url: https://otx.alienvault.com/api/v1/indicators/export
    var.api_token: 754dcaafbcb9740dc0d119e72d5eaad699cc4a5cdbc856fc6215883842ba8142
    var.first_interval: 24h
    var.lookback_range: 2h
    var.interval: 60m

To filter only on specific indicator types, this is an example of some possible filters that are supported:

- module: threatintel
  otx:
    enabled: true
    var.input: httpjson
    var.url: https://otx.alienvault.com/api/v1/indicators/export
    var.types: "domain,IPv4,hostname,url,FileHash-SHA256"
    var.first_interval: 24h
    var.interval: 60m

var.paths: An array of glob-based paths that specify where to look for the log files. All patterns supported by Go Glob are also supported here. For example, you can use wildcards to fetch all files from a predefined level of subdirectories: /path/to/log/*/*.log. This fetches all .log files from the subfolders of /path/to/log. It does not fetch log files from the /path/to/log folder itself. If this setting is left empty, Filebeat will choose log paths based on your operating system.
var.url: The URL of the API endpoint to connect with.
var.api_token: The API key used to access OTX. This can be found on your OTX API homepage.
var.interval: How often the API is polled for updated information.
var.first_interval: How far back to search when retrieving events the first time the Filebeat starts up. After the first interval has passed the module itself will use the timestamp from the last response as the filter when retrieving new events.
var.types: A comma delimited list of indicator types to include, defaults to all. A list of possible types to filter on can be found in the AlientVault OTX documentation.
var.proxy_url: Optional URL to use as HTTP proxy.

OTX Threat Intel is mapped to the following ECS fields.

Malware Threat Intel Fields	ECS Fields
otx.type	threatintel.indicator.type
otx.description	threatintel.indicator.description
otx.indicator	threatintel.indicator.*

otx.indicator is mapped to the appropriate field dependent on attribute type.

`anomali` fileset settings

edit

To configure the module please fill in the credentials, for Anomali Limo (the free Taxii service) these are usually default credentials found at the Anomali Limo webpage Anomali Limo offers multiple sources called collections. Each collection has a specific ID, which then fits into the url used in this configuration. A list of different collections can be found using the credentials at Limo Collections.

The example below uses the collection of ID 41 as can be seen in the URL.

- module: threatintel
  anomali:
    enabled: true
    var.input: httpjson
    var.url: https://limo.anomali.com/api/v1/taxii2/feeds/collections/41/objects?match[type]=indicator
    var.username: guest
    var.password: guest
    var.interval: 60m

To filter on specific types, you can define var.types as a comma delimited list of object types. This defaults to "indicators".

- module: threatintel
  anomali:
    enabled: true
    var.input: httpjson
    var.url: https://limo.anomali.com/api/v1/taxii2/feeds/collections/41/objects?match[type]=indicator
    var.types: "indicators,other"
    var.username: guest
    var.password: guest
    var.interval: 60m

var.paths: An array of glob-based paths that specify where to look for the log files. All patterns supported by Go Glob are also supported here. For example, you can use wildcards to fetch all files from a predefined level of subdirectories: /path/to/log/*/*.log. This fetches all .log files from the subfolders of /path/to/log. It does not fetch log files from the /path/to/log folder itself. If this setting is left empty, Filebeat will choose log paths based on your operating system.
var.url: The URL of the API endpoint to connect with. Limo offers multiple collections of threat intelligence.
var.username: Username used to access the API.
var.password: Password used to access the API.
var.interval: How often the API is polled for updated information.
var.types: A comma delimited list of indicator types to include, defaults to all. A list of possible types to filter on can be found on the Stix 2.1 Object types page.
var.proxy_url: Optional URL to use as HTTP proxy.

Anomali Threat Intel is mapped to the following ECS fields.

Malware Threat Intel Fields	ECS Fields
anomali.description	threatintel.indicator.description
anomali.created	threatintel.indicator.first_seen
anomali.modified	threatintel.indicator.last_seen
anomali.pattern	threatintel.indicator.*
anomali.labels	tags

anomali.pattern is mapped to the appropriate field dependent on attribute type.

`anomalithreatstream` fileset settings

edit

To configure the ThreatStream integration you first need to define an output in the Anomali ThreatStream Integrator using the Elastic SDK provided by Anomali. It will deliver indicators via HTTP or HTTPS to a Filebeat instance running as a server.

Configure an Integrator output with the following settings:

Indicator Filter: * (or use any desired filter).
SDK Executable Command: /path/to/python /path/to/anomali-sdk/main.py. Adjust the paths to the python executable and the directory where the Elastic SDK has been unpacked.
Metadata in JSON Format: {"url": "https://filebeat:8080/", "server_certificate": "/path/to/cert.pem", "secret": "my secret"}.
- url: Use the host and port where Filebeat will be running, and http or https accordingly.
- server_certificate: If using HTTPS, absolute path to the server certificate. Otherwise don’t set this field.
- secret: A shared secret string to authenticate messages between the SDK and Filebeat.

Then configure the anomalithreatstream fileset in Filebeat accordingly:

- module: threatintel
  anomalithreatstream:
    enabled: true
    var.input: http_endpoint
    var.listen_address: 0.0.0.0 # Listen on all interfaces.
    var.listen_port: 8080
    var.secret: 'my secret'
    var.ssl_certificate: path/to/server_ssl_cert.pem
    var.ssl_key: path/to/ssl_key.pem

var.listen_address: Local address to bind the HTTP server to. Use 0.0.0.0 to accept connections from all interfaces.
var.listen_port: Port number to use for the HTTP server.
var.secret: Shared secret between the SDK and Filebeat, used to authenticate messages.
var.ssl_certificate: Path to the public SSL certificate for the HTTPS server. If unset, Filebeat will use unsecure HTTP connections.
var.ssl_key: Path to the certificate’s private key.

Anomali ThreatStream fields are mapped to the following ECS fields:

ThreatStream fields	ECS Fields
asn	threatintel.indicator.as.number
classification[1]	threatintel.indicator.marking.tlp
confidence[1]	threatintel.indicator.confidence
country	threatintel.indicator.geo.country_iso_code
date_first	threatintel.indicator.first_seen
date_last	threatintel.indicator.last_seen
detail	tags
domain	threatintel.indicator.url.domain
email	threatintel.indicator.email.address
itype[1]	threatintel.indicator.type
lat	threatintel.indicator.geo.location.lat
lon	threatintel.indicator.geo.location.lon
md5	threatintel.indicator.file.hash
org	threatintel.indicator.as.organization.name
severity[1]	event.severity
source	threatintel.indicator.provider
srcip	threatintel.indicator.ip
url	threatintel.indicator.url.original

[1]: Field is used to derive a value for the ECS field but its original value is kept under threatintel.anomalithreatstream.

`recordedfuture` fileset settings

edit

The recordedfuture fileset fetches intelligence from the Recorded Future Connect API. It supports domain, hash, ip and url data types.

To enable it you need to define the URL to fetch data from. You can construct this URL using the Recorded Future API Explorer. The URL must point to the /search endpoint and contain a suitable limit (how many records to return from a single request) and fields parameters. The entity and timestamps fields are required.

Sample configuration:

- module: threatintel
  recordedfuture:
    enabled: true
    var.input: httpjson
    var.interval: 5m
    var.first_interval: 168h
    var.url: "https://api.recordedfuture.com/v2/ip/search?limit=200&fields=entity,timestamps,risk,intelCard,location&metadata=false"
    var.api_token: "<RF_TOKEN>"

To fetch threat intelligence from multiple data types, you must define more than one instance of the module:

- module: threatintel
  recordedfuture:
    enabled: true
    var.input: httpjson
    var.interval: 5m
    var.first_interval: 168h
    var.url: "https://api.recordedfuture.com/v2/ip/search?limit=200&fields=entity,timestamps,risk,intelCard,location&metadata=false"
    var.api_token: "<RF_TOKEN>"
- module: threatintel
  recordedfuture:
    enabled: true
    var.input: httpjson
    var.interval: 1m
    var.first_interval: 168h
    var.url: "https://api.recordedfuture.com/v2/hash/search?limit=200&fields=entity,fileHashes,timestamps,risk,intelCard,location&metadata=false"
    var.api_token: "<RF_TOKEN>"

var.url: The URL of the API endpoint to connect with.
var.api_token: The API token used to access Recorded Future API.
var.interval: How often the API is polled for updated information.
var.first_interval: How far back to search when retrieving events the first time Filebeat starts up. After the first interval has passed the module itself will use the timestamp from the last response as the filter when retrieving new events.
var.proxy_url: Optional URL to use as HTTP proxy.

Recorded Future fields are mapped to the following ECS fields:

Recorded Future fields	ECS Fields
entity.name	threatintel.indicator.{url,ip,domain,file.hash}
entity.type	threatintel.indicator.type
fileHashes	threatintel.indicator.file.hash
intelCard	event.reference
location.asn	threatintel.indicator.as.number
location.location	threatintel.indicator.geo
location.organization	threatintel.indicator.as.organization.name
risk.score	event.risk_score
timestamps.firstSeen	threatintel.indicator.first_seen
timestamps.lastSeen	threatintel.indicator.last_seen