« Threat intelligence Alienvault OTX Integration »
Elastic Docs ›Elastic integrations ›Threat intelligence

AbuseCH integration

edit

AbuseCH integration

edit

Version

2.5.0 (View all)

Compatible Kibana version(s)

8.13.0 or higher

Supported Serverless project types
What’s this?

Security
Observability

Subscription level
What’s this?

Basic

Level of support
What’s this?

Elastic

This integration is for AbuseCH logs. It includes the following datasets for retrieving indicators from the AbuseCH API:

  • url dataset: Supports URL based indicators from AbuseCH API.
  • malware dataset: Supports Malware based indicators from AbuseCH API.
  • malwarebazaar dataset: Supports indicators from the MalwareBazaar from AbuseCH.
  • threatfox dataset: Supports indicators from AbuseCH Threat Fox API.

Expiration of Indicators of Compromise (IOCs)

edit

All AbuseCH datasets now support indicator expiration. For URL dataset, a full list of active indicators are ingested every interval. For other datasets namely Malware, MalwareBazaar, and ThreatFox, the indicators are expired after duration IOC Expiration Duration configured in the integration setting. An Elastic Transform is created for every source index to facilitate only active indicators be available to the end users. Each transform creates a destination index named logs-ti_abusech_latest.dest_* which only contains active and unexpired indicators. The indiator match rules and dashboards are updated to list only active indicators. Destinations indices are aliased to logs-ti_abusech_latest.<datastream_name>.

Source Datastream Destination Index Pattern Destination Alias

logs-ti_abusech.url-*

logs-ti_abusech_latest.dest_url-*

logs-ti_abusech_latest.url

logs-ti_abusech.malware-*

logs-ti_abusech_latest.dest_malware-*

logs-ti_abusech_latest.malware

logs-ti_abusech.malwarebazaar-*

logs-ti_abusech_latest.dest_malwarebazaar-*

logs-ti_abusech_latest.malwarebazaar

logs-ti_abusech.threatfox-*

logs-ti_abusech_latest.dest_threatfox-*

logs-ti_abusech_latest.threatfox
ILM Policy
edit

To facilitate IOC expiration, source datastream-backed indices .ds-logs-ti_abusech.<datastream_name>-* are allowed to contain duplicates from each polling interval. ILM policy logs-ti_abusech.<datastream_name>-default_policy is added to these source indices so it doesn’t lead to unbounded growth. This means data in these source indices will be deleted after 5 days from ingested date.

Logs

edit
URL
edit

The AbuseCH URL data_stream retrieves full list of active threat intelligence indicators every interval from the Active Indicators URL database dump https://urlhaus.abuse.ch/downloads/json/.

Exported fields
Field Description Type

@timestamp

Event timestamp.

date

abusech.url.blacklists.spamhaus_dbl

If the indicator is listed on the spamhaus blacklist.

keyword

abusech.url.blacklists.surbl

If the indicator is listed on the surbl blacklist.

keyword

abusech.url.deleted_at

The timestamp when the indicator is (will be) deleted.

date

abusech.url.id

The ID of the indicator.

keyword

abusech.url.larted

Indicates whether the malware URL has been reported to the hosting provider (true or false).

boolean

abusech.url.last_online

Last timestamp when the URL has been serving malware.

date

abusech.url.reporter

The Twitter handle of the reporter that has reported this malware URL (or anonymous).

keyword

abusech.url.tags

A list of tags associated with the queried malware URL.

keyword

abusech.url.threat

The threat corresponding to this malware URL.

keyword

abusech.url.url_status

The current status of the URL. Possible values are: online, offline and unknown.

keyword

abusech.url.urlhaus_reference

Link to URLhaus entry.

keyword

cloud.image.id

Image ID for the cloud instance.

keyword

data_stream.dataset

Data stream dataset name.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

input.type

Type of Filebeat input.

keyword

labels.interval

User-configured value for Interval setting. This is used in calculation of indicator expiration time.

keyword

labels.is_ioc_transform_source

Indicates whether an IOC is in the raw source data stream, or the in latest destination index.

constant_keyword

log.flags

Flags for the log file.

keyword

log.offset

Offset of the entry in the log file.

long

threat.feed.dashboard_id

Dashboard ID used for Kibana CTI UI

constant_keyword

threat.feed.name

Display friendly feed name

constant_keyword

threat.indicator.first_seen

The date and time when intelligence source first reported sighting this indicator.

date

threat.indicator.last_seen

The date and time when intelligence source last reported sighting this indicator.

date

threat.indicator.modified_at

The date and time when intelligence source last modified information for this indicator.

date
Malware
edit

The AbuseCH malware data_stream retrieves threat intelligence indicators from the payload API endpoint https://urlhaus-api.abuse.ch/v1/payloads/recent/.

Exported fields
Field Description Type

@timestamp

Event timestamp.

date

abusech.malware.deleted_at

The indicator expiration timestamp.

date

abusech.malware.ioc_expiration_duration

The configured expiration duration.

keyword

abusech.malware.signature

Malware family.

keyword

abusech.malware.virustotal.link

Link to the Virustotal report.

keyword

abusech.malware.virustotal.percent

AV detection in percent.

float

abusech.malware.virustotal.result

AV detection ratio.

keyword

cloud.image.id

Image ID for the cloud instance.

keyword

data_stream.dataset

Data stream dataset name.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

input.type

Type of Filebeat input.

keyword

labels.is_ioc_transform_source

Indicates whether an IOC is in the raw source data stream, or the in latest destination index.

constant_keyword

log.flags

Flags for the log file.

keyword

log.offset

Offset of the entry in the log file.

long

threat.feed.dashboard_id

Dashboard ID used for Kibana CTI UI

constant_keyword

threat.feed.name

Display friendly feed name

constant_keyword

threat.indicator.first_seen

The date and time when intelligence source first reported sighting this indicator.

date

threat.indicator.last_seen

The date and time when intelligence source last reported sighting this indicator.

date

threat.indicator.modified_at

The date and time when intelligence source last modified information for this indicator.

date
MalwareBazaar
edit

The AbuseCH malwarebazaar data_stream retrieves threat intelligence indicators from the MalwareBazaar API endpoint https://mb-api.abuse.ch/api/v1/.

Exported fields
Field Description Type

@timestamp

Event timestamp.

date

abusech.malwarebazaar.anonymous

Identifies if the sample was submitted anonymously.

long

abusech.malwarebazaar.code_sign.algorithm

Algorithm used to generate the public key.

keyword

abusech.malwarebazaar.code_sign.cscb_listed

Whether the certificate is present on the Code Signing Certificate Blocklist (CSCB).

boolean

abusech.malwarebazaar.code_sign.cscb_reason

Why the certificate is present on the Code Signing Certificate Blocklist (CSCB).

keyword

abusech.malwarebazaar.code_sign.issuer_cn

Common name (CN) of issuing certificate authority.

keyword

abusech.malwarebazaar.code_sign.serial_number

Unique serial number issued by the certificate authority.

keyword

abusech.malwarebazaar.code_sign.subject_cn

Common name (CN) of subject.

keyword

abusech.malwarebazaar.code_sign.thumbprint

Hash of certificate.

keyword

abusech.malwarebazaar.code_sign.thumbprint_algorithm

Algorithm used to create thumbprint.

keyword

abusech.malwarebazaar.code_sign.valid_from

Time at which the certificate is first considered valid.

date

abusech.malwarebazaar.code_sign.valid_to

Time at which the certificate is no longer considered valid.

keyword

abusech.malwarebazaar.deleted_at

The indicator expiration timestamp.

date

abusech.malwarebazaar.dhash_icon

In case the file is a PE executable: dhash of the samples icon.

keyword

abusech.malwarebazaar.intelligence.downloads

Number of downloads from MalwareBazaar.

long

abusech.malwarebazaar.intelligence.mail.Generic

Malware seen in generic spam traffic.

keyword

abusech.malwarebazaar.intelligence.mail.IT

Malware seen in IT spam traffic.

keyword

abusech.malwarebazaar.intelligence.uploads

Number of uploads from MalwareBazaar.

long

abusech.malwarebazaar.ioc_expiration_duration

The configured expiration duration.

keyword

cloud.image.id

Image ID for the cloud instance.

keyword

data_stream.dataset

Data stream dataset name.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

input.type

Type of Filebeat input.

keyword

labels.is_ioc_transform_source

Indicates whether an IOC is in the raw source data stream, or the in latest destination index.

constant_keyword

log.flags

Flags for the log file.

keyword

log.offset

Offset of the entry in the log file.

long

threat.feed.dashboard_id

Dashboard ID used for Kibana CTI UI

constant_keyword

threat.feed.name

Display friendly feed name

constant_keyword

threat.indicator.first_seen

The date and time when intelligence source first reported sighting this indicator.

date

threat.indicator.last_seen

The date and time when intelligence source last reported sighting this indicator.

date

threat.indicator.modified_at

The date and time when intelligence source last modified information for this indicator.

date
Threat Fox
edit

The AbuseCH threatfox data_stream retrieves threat intelligence indicators from the Threat Fox API endpoint https://threatfox-api.abuse.ch/api/v1/.

Exported fields
Field Description Type

@timestamp

Event timestamp.

date

abusech.threatfox.confidence_level

Confidence level between 0-100.

long

abusech.threatfox.deleted_at

The indicator expiration timestamp.

date

abusech.threatfox.ioc_expiration_duration

The configured expiration duration.

keyword

abusech.threatfox.malware

The malware associated with the IOC.

keyword

abusech.threatfox.tags

A list of tags associated with the queried malware sample.

keyword

abusech.threatfox.threat_type

The type of threat.

keyword

abusech.threatfox.threat_type_desc

The threat descsription.

keyword

cloud.image.id

Image ID for the cloud instance.

keyword

data_stream.dataset

Data stream dataset name.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

input.type

Type of Filebeat input.

keyword

labels.is_ioc_transform_source

Indicates whether an IOC is in the raw source data stream, or the in latest destination index.

constant_keyword

log.flags

Flags for the log file.

keyword

log.offset

Offset of the entry in the log file.

long

threat.feed.dashboard_id

Dashboard ID used for Kibana CTI UI

constant_keyword

threat.feed.name

Display friendly feed name

constant_keyword

threat.indicator.first_seen

The date and time when intelligence source first reported sighting this indicator.

date

threat.indicator.last_seen

The date and time when intelligence source last reported sighting this indicator.

date

threat.indicator.modified_at

The date and time when intelligence source last modified information for this indicator.

date

Changelog

edit
Changelog
Version Details Kibana version(s)

2.5.0

Enhancement (View pull request)
Do not remove event.original in main ingest pipeline.

8.13.0 or higher

2.4.0

Enhancement (View pull request)
Add "preserve_original_event" tag to documents with event.kind set to "pipeline_error".

8.13.0 or higher

2.3.5

Bug fix (View pull request)
Use triple-brace Mustache templating when referencing variables in ingest pipelines.

8.13.0 or higher

2.3.4

Bug fix (View pull request)
Use triple-brace Mustache templating when referencing variables in ingest pipelines.

8.13.0 or higher

2.3.3

Bug fix (View pull request)
Fix labels.is_ioc_transform_source values

8.13.0 or higher

2.3.2

Bug fix (View pull request)
Add missing fields in transform

8.13.0 or higher

2.3.1

Bug fix (View pull request)
Fix ECS date mapping on threat fields.

8.13.0 or higher

2.3.0

Enhancement (View pull request)
Improve error reporting for API request failures.

8.13.0 or higher

2.2.0

Enhancement (View pull request)
Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.

8.13.0 or higher

2.1.0

Enhancement (View pull request)
Improve error handling and reporting in malwarebazaar data stream.

8.12.0 or higher

2.0.1

Bug fix (View pull request)
Adjust field mappings for transform destination index.

8.12.0 or higher

2.0.0

Enhancement (View pull request)
Support IoC expiration

8.12.0 or higher

1.25.0

Enhancement (View pull request)
Update manifest format version to v3.0.3.

8.7.1 or higher

1.24.1

Enhancement (View pull request)
Changed owners

8.7.1 or higher

1.24.0

Enhancement (View pull request)
Limit request tracer log count to five.

8.7.1 or higher

1.23.0

Enhancement (View pull request)
ECS version updated to 8.11.0.

8.7.1 or higher

1.22.0

Enhancement (View pull request)
Improve event.original check to avoid errors if set.

8.7.1 or higher

1.21.0

Enhancement (View pull request)
ECS version updated to 8.10.0.

8.7.1 or higher

1.20.0

Enhancement (View pull request)
The format_version in the package manifest changed from 2.11.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added owner.type: elastic to package manifest.

8.7.1 or higher

1.19.0

Enhancement (View pull request)
Add tags.yml file so that integration’s dashboards and saved searches are tagged with "Security Solution" and displayed in the Security Solution UI.

8.7.1 or higher

1.18.0

Enhancement (View pull request)
Update package-spec to 2.9.0.

8.7.1 or higher

1.17.0

Enhancement (View pull request)
Update package to ECS 8.9.0.

8.7.1 or higher

1.16.0

Enhancement (View pull request)
Document duration units.

8.7.1 or higher

1.15.2

Bug fix (View pull request)
Change ThreatFox initial interval to integer and note valid values 1-7

8.7.1 or higher

1.15.1

Bug fix (View pull request)
Remove fields with empty string values

8.7.1 or higher

1.15.0

Enhancement (View pull request)
Convert visualizations to lens.

8.7.1 or higher

1.14.0

Enhancement (View pull request)
Document valid duration units.

8.7.1 or higher

1.13.0

Enhancement (View pull request)
Ensure event.kind is correctly set for pipeline errors.

8.7.1 or higher

1.12.0

Enhancement (View pull request)
Update package to ECS 8.8.0.

8.7.1 or higher

1.11.0

Enhancement (View pull request)
Add a new flag to enable request tracing

8.7.1 or higher

1.10.0

Enhancement (View pull request)
Update package to ECS 8.7.0.

8.0.0 or higher

1.9.0

Enhancement (View pull request)
Update package to ECS 8.6.0.

8.0.0 or higher

1.8.0

Enhancement (View pull request)
Update package to ECS 8.5.0.

8.0.0 or higher

1.7.1

Bug fix (View pull request)
Change abusech.malwarebazaar.code_sign to Nested field

8.0.0 or higher

1.7.0

Enhancement (View pull request)
Add Threat Fox datastream

8.0.0 or higher

1.6.0

Enhancement (View pull request)
Update package to ECS 8.4.0

8.0.0 or higher

1.5.1

Bug fix (View pull request)
Fix proxy URL documentation rendering.

8.0.0 or higher

1.5.0

Enhancement (View pull request)
Update categories to include threat_intel.

8.0.0 or higher

1.4.0

Enhancement (View pull request)
Update package to ECS 8.3.0.

8.0.0 or higher

1.3.2

Enhancement (View pull request)
Added link to AbuseCH documentation in readme

8.0.0 or higher

1.3.1

Enhancement (View pull request)
Update package descriptions

8.0.0 or higher

1.3.0

Enhancement (View pull request)
Update to ECS 8.2

8.0.0 or higher

1.2.3

Enhancement (View pull request)
Add mapping for event.created

8.0.0 or higher

1.2.2

Enhancement (View pull request)
Add documentation for multi-fields

8.0.0 or higher

1.2.1

Bug fix (View pull request)
Fix field mapping conflicts in threat.indicator.file.x509.not_before/not_after

—

1.2.0

Enhancement (View pull request)
Update to ECS 8.0

8.0.0 or higher

1.1.5

Bug fix (View pull request)
Removes extra tag from dashboards

8.0.0 or higher

1.1.4

Bug fix (View pull request)
Regenerate test files using the new GeoIP database

8.0.0 or higher

1.1.3

Bug fix (View pull request)
Change test public IPs to the supported subset

—

1.1.2

Enhancement (View pull request)
Fixing typo in base-fields.yml

—

1.1.1

Enhancement (View pull request)
Update ECS fields for threat.feed.name

—

1.1.0

Enhancement (View pull request)
Adding dashboards and adding minor tweaks to pipeline

8.0.0 or higher

1.0.4

Enhancement (View pull request)
Bump minimum version in manifest

8.0.0 or higher

1.0.3

Enhancement (View pull request)
Bump minimum version

—

1.0.2

Enhancement (View pull request)
Update title and description.

—

1.0.1

Enhancement (View pull request)
Fix invisible package icon

—

1.0.0

Enhancement (View pull request)
Initial Release

—
« Threat intelligence Alienvault OTX Integration »