« Recorded Future Integration ThreatConnect »
Elastic Docs ›Elastic integrations ›Threat intelligence

ThreatQuotient Integration

edit

ThreatQuotient Integration

edit

Version

1.30.0 (View all)

Compatible Kibana version(s)

8.13.0 or higher

Supported Serverless project types
What’s this?

Security
Observability

Subscription level
What’s this?

Basic

Level of support
What’s this?

Partner

The ThreatQuotient integration uses the available ThreatQuotient REST API to retrieve indicators and Threat Intelligence.

Logs

edit
Threat
edit

The ThreatQ integration requires you to set a valid URL, combination of Oauth2 credentials and the ID of the collection to retrieve indicators from. By default the indicators will be collected every 1 minute, and deduplication is handled by the API itself. This datastream supports expiration of indicators of compromise (IOC).

Expiration of Indicators of Compromise (IOCs)
edit

The ThreatQ’s Threat datastream supports IOC expiration. The ingested IOCs expire after certain duration. In ThreatQ feed, this can happen in 3 ways:

  • When the value of threatq.status is Expired.
  • When either of the fields threatq.expires_at or threatq.expired_at reaches current now() timestamp.
  • When the indicator is not updated in a long time leading to default expiration set by IOC Expiration Duration configuration parameter. For more details, see Handling Orphaned IOCs.

The field threatq.ioc_expiration_reason indicates which among the 3 methods stated above is the reason for indicator expiration.

An Elastic Transform is created to faciliate only active IOCs be available to the end users. This transform creates destination indices named logs-ti_threatq_latest.dest_threat-* which only contains active and unexpired IOCs. The latest destination index also has an alias named logs-ti_threatq_latest.threat. When querying for active indicators or setting up indicator match rules, only use the latest destination indices or the alias to avoid false positives from expired IOCs. Dashboards for the Threat datastream are also pointing to the latest destination indices containing active IoCs. Please read ILM Policy below which is added to avoid unbounded growth on source datastream .ds-logs-ti_threatq.threat-* indices.

Handling Orphaned IOCs
edit

Some IOCs may never expire and will continue to stay in the latest destination indices logs-ti_threatq_latest.dest_threat-*. To avoid any false positives from such orphaned IOCs, users are allowed to configure IOC Expiration Duration parameter while setting up the integration. This parameter deletes any indicator ingested into destination indices logs-ti_threatq_latest.dest_threat-* after this specified duration is reached, defaults to 90d from source’s @timestamp field. Note that IOC Expiration Duration parameter only exists to add a fail-safe default expiration in case IOCs never expire.

ILM Policy
edit

To facilitate IOC expiration, source datastream-backed indices .ds-logs-ti_threatq.threat-* are allowed to contain duplicates from each polling interval. ILM policy is added to these source indices so it doesn’t lead to unbounded growth. This means data in these source indices will be deleted after 5 days from ingested date.

Exported fields
Field Description Type

@timestamp

Event timestamp.

date

cloud.image.id

Image ID for the cloud instance.

keyword

data_stream.dataset

Data stream dataset name.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

input.type

Type of Filebeat input.

keyword

labels.is_ioc_transform_source

Indicates whether an IOC is in the raw source data stream, or the in latest destination index.

constant_keyword

log.flags

Flags for the log file.

keyword

log.offset

Offset of the entry in the log file.

long

threat.feed.dashboard_id

Dashboard ID used for Kibana CTI UI

constant_keyword

threat.indicator.first_seen

The date and time when intelligence source first reported sighting this indicator.

date

threat.indicator.last_seen

The date and time when intelligence source last reported sighting this indicator.

date

threat.indicator.modified_at

The date and time when intelligence source last modified information for this indicator.

date

threatq.adversaries

Adversaries that are linked to the object

keyword

threatq.attributes

These provide additional context about an object

flattened

threatq.created_at

Object creation time

date

threatq.expired_at

Expiration time given by the API. Either expires_at or expired_at are present in the data.

date

threatq.expires_at

Expiration time given by the API. Either expires_at or expired_at are present in the data.

date

threatq.expires_calculated_at

Expiration calculation time

date

threatq.id

Indicator ID. id, indicator_id or both could be present in the dataset.

long

threatq.indicator_id

Indicator ID. id, indicator_id or both could be present in the dataset.

long

threatq.indicator_value

Original indicator value

keyword

threatq.ioc_expiration_reason

Reason why the indicator is expired. Set inside the ingest pipeline.

keyword

threatq.ioc_expired_at

Expiration time calculated by the integration needed for the transform.

date

threatq.published_at

Object publication time

date

threatq.status

Object status within the Threat Library

keyword

threatq.updated_at

Last modification time

date
Example

An example event for threat looks as following:

{
    "@timestamp": "2019-11-15T00:00:02.000Z",
    "agent": {
        "ephemeral_id": "9f1b0b7f-5be0-463d-9551-3d66aab12b6f",
        "id": "8299ae35-ee0e-4107-9acb-1b6acfdda1fb",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.13.0"
    },
    "data_stream": {
        "dataset": "ti_threatq.threat",
        "namespace": "94389",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "8299ae35-ee0e-4107-9acb-1b6acfdda1fb",
        "snapshot": false,
        "version": "8.13.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "threat"
        ],
        "created": "2024-08-02T06:46:26.556Z",
        "dataset": "ti_threatq.threat",
        "ingested": "2024-08-02T06:46:36Z",
        "kind": "enrichment",
        "original": "{\"adversaries\":[],\"attributes\":[{\"attribute_id\":3,\"created_at\":\"2020-09-11 14:35:53\",\"id\":1877,\"indicator_id\":336,\"name\":\"Description\",\"touched_at\":\"2020-10-15 14:36:00\",\"updated_at\":\"2020-10-15 14:36:00\",\"value\":\"Malicious Host\"},{\"attribute_id\":4,\"created_at\":\"2020-09-11 14:35:53\",\"id\":1878,\"indicator_id\":336,\"name\":\"Country\",\"touched_at\":\"2020-10-15 14:36:00\",\"updated_at\":\"2020-10-15 14:36:00\",\"value\":\"MP\"}],\"class\":\"network\",\"created_at\":\"2020-09-11 14:35:51\",\"expires_calculated_at\":\"2020-10-15 14:40:03\",\"hash\":\"1ece659dcec98b1e1141160b55655c96\",\"id\":336,\"published_at\":\"2020-09-11 14:35:51\",\"score\":4,\"sources\":[{\"created_at\":\"2020-09-11 14:35:53\",\"creator_source_id\":12,\"id\":336,\"indicator_id\":336,\"indicator_status_id\":2,\"indicator_type_id\":15,\"name\":\"AlienVault OTX\",\"published_at\":\"2020-09-11 14:35:53\",\"reference_id\":1,\"source_expire_days\":\"30\",\"source_id\":12,\"source_score\":1,\"source_type\":\"connectors\",\"updated_at\":\"2020-10-15 14:36:00\"}],\"status\":{\"description\":\"Poses a threat\",\"id\":2,\"name\":\"Active\"},\"status_id\":2,\"touched_at\":\"2021-06-07 19:47:27\",\"type\":{\"class\":\"network\",\"id\":15,\"name\":\"IP Address\"},\"type_id\":15,\"updated_at\":\"2019-11-15 00:00:02\",\"value\":\"89.160.20.156\"}",
        "type": [
            "indicator"
        ]
    },
    "input": {
        "type": "httpjson"
    },
    "tags": [
        "preserve_original_event",
        "forwarded",
        "threatq-threat"
    ],
    "threat": {
        "indicator": {
            "confidence": "Low",
            "ip": "89.160.20.156",
            "type": "ipv4-addr"
        }
    },
    "threatq": {
        "attributes": {
            "country": [
                "MP"
            ],
            "description": [
                "Malicious Host"
            ]
        },
        "created_at": "2020-09-11T14:35:51.000Z",
        "expires_calculated_at": "2020-10-15T14:40:03.000Z",
        "id": 336,
        "indicator_value": "89.160.20.156",
        "ioc_expiration_reason": "Expiration set by Elastic from the integration's parameter `IOC Expiration Duration`",
        "ioc_expired_at": "2019-11-20T00:00:02.000Z",
        "published_at": "2020-09-11T14:35:51.000Z",
        "status": "Active"
    }
}

Changelog

edit
Changelog
Version Details Kibana version(s)

1.30.0

Enhancement (View pull request)
Do not remove event.original in main ingest pipeline.

8.13.0 or higher

1.29.0

Enhancement (View pull request)
Add "preserve_original_event" tag to documents with event.kind set to "pipeline_error".

8.13.0 or higher

1.28.3

Bug fix (View pull request)
Fix labels.is_ioc_transform_source values

8.13.0 or higher

1.28.2

Bug fix (View pull request)
Add missing fields in transform

8.13.0 or higher

1.28.1

Bug fix (View pull request)
Fix ECS date mapping on threat fields.

8.13.0 or higher

1.28.0

Enhancement (View pull request)
Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.

8.13.0 or higher

1.27.1

Bug fix (View pull request)
Adjust field mappings for transform destination index.

8.12.0 or higher

1.27.0

Enhancement (View pull request)
Improve handling of empty responses.

8.12.0 or higher

1.26.0

Enhancement (View pull request)
Support user configurable page size.

8.12.0 or higher

1.25.0

Enhancement (View pull request)
Set sensitive values as secret.

8.12.0 or higher

1.24.2

Enhancement (View pull request)
Changed owners

8.11.0 or higher

1.24.1

Bug fix (View pull request)
Fix IOC expiration duration character casting.

8.11.0 or higher

1.24.0

Enhancement (View pull request)
Support for IoC Expiration

8.11.0 or higher

1.23.0

Enhancement (View pull request)
Limit request tracer log count to five.

8.7.1 or higher

1.22.0

Enhancement (View pull request)
ECS version updated to 8.11.0.

8.7.1 or higher

1.21.0

Enhancement (View pull request)
Improve event.original check to avoid errors if set.

8.7.1 or higher

1.20.0

Enhancement (View pull request)
Set partner owner type.

8.7.1 or higher

1.19.0

Enhancement (View pull request)
ECS version updated to 8.10.0.

8.7.1 or higher

1.18.0

Enhancement (View pull request)
The format_version in the package manifest changed from 2.11.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added owner.type: elastic to package manifest.

8.7.1 or higher

1.17.0

Enhancement (View pull request)
Add tags.yml file so that integration’s dashboards and saved searches are tagged with "Security Solution" and displayed in the Security Solution UI.

8.7.1 or higher

1.16.0

Enhancement (View pull request)
Update package-spec to 2.10.0.

8.7.1 or higher

1.15.0

Enhancement (View pull request)
Update package to ECS 8.9.0.

8.7.1 or higher

1.14.0

Enhancement (View pull request)
Document duration units.

8.7.1 or higher

1.13.0

Enhancement (View pull request)
Document valid duration units.

8.7.1 or higher

1.12.0

Enhancement (View pull request)
Ensure event.kind is correctly set for pipeline errors.

8.7.1 or higher

1.11.0

Enhancement (View pull request)
Update package to ECS 8.8.0.

8.7.1 or higher

1.10.0

Enhancement (View pull request)
Add a new flag to enable request tracing

8.7.1 or higher

1.9.0

Enhancement (View pull request)
Update package to ECS 8.7.0.

8.0.0 or higher

1.8.1

Bug fix (View pull request)
Honor preserve_original_event setting.

8.0.0 or higher

1.8.0

Enhancement (View pull request)
Update package to ECS 8.6.0.

8.0.0 or higher

1.7.1

Bug fix (View pull request)
Remove duplicate fields.

8.0.0 or higher

1.7.0

Enhancement (View pull request)
Update package to ECS 8.5.0.

8.0.0 or higher

1.6.0

Enhancement (View pull request)
Update package to ECS 8.4.0

8.0.0 or higher

1.5.1

Bug fix (View pull request)
Fix proxy URL documentation rendering.

8.0.0 or higher

1.5.0

Enhancement (View pull request)
Update categories to include threat_intel.

8.0.0 or higher

1.4.0

Enhancement (View pull request)
Update package to ECS 8.3.0.

8.0.0 or higher

1.3.2

Enhancement (View pull request)
update readme to include link to threatQ documentation

8.0.0 or higher

1.3.1

Enhancement (View pull request)
Update package descriptions

8.0.0 or higher

1.3.0

Enhancement (View pull request)
Update to ECS 8.2

8.0.0 or higher

1.2.2

Enhancement (View pull request)
Add event.created field mapping

8.0.0 or higher

1.2.1

Enhancement (View pull request)
Add documentation for multi-fields

8.0.0 or higher

1.2.0

Enhancement (View pull request)
Update to ECS 8.0

8.0.0 or higher

1.1.0

Enhancement (View pull request)
Add threat.feed ECS fields and dashboard

8.0.0 or higher

1.0.2

Bug fix (View pull request)
Change test public IPs to the supported subset

8.0.0 or higher

1.0.1

Enhancement (View pull request)
Bumping minimum version

8.0.0 or higher

1.0.0

Enhancement (View pull request)
Initial release

—
« Recorded Future Integration ThreatConnect »