Establish robust threat intelligence with Elastic Security
As a powerful search engine, Elasticsearch provides various ways to collect and enrich data with threat intel feeds, while the Elastic Security detection engine helps security analysts to detect alerts with threat indicator matching. In this blog post, we’ll provide an introduction to threat intelligence and demonstrate how Elastic Security can help organizations establish robust cyber threat intelligence (CTI) capabilities.
CTI is contextual information obtained through research and analysis of emerging threats and the application of defensive countermeasures to enhance the protective posture of contested environments. It helps organizations to better understand past, current, and future threats.
Threat feeds that can be obtained from various open source and commercial platforms become actionable “threat intelligence” when raw feeds are processed, parsed, and interpreted by security platforms to detect and understand the statistical breakdowns of the source and targets of attack activities.
Threat intelligence in security operation centres (SOCs)
Threat information is typically delivered via data feeds that can be of different types, such as file hashes, FQDN, IP addresses, URL reputation data, CVE, etc. — generally referred to as indicators of compromise (IOC) — and such intelligence has an important role in the SOC.
IOCs are the main deliverable for such tactical threat intelligence feeds and these are particularly useful for updating signature-based defence systems to defend against known attack types. IOCs also prove useful in proactive measures such as threat hunting.
The SOC team will typically be responsible for processing threat data and using it to add additional context to internal sources of data that are malicious. This is particularly valuable to SOC analysts that are responsible for analysing big data sets across SIEM platforms. Threat intelligence integration with SIEM detection capabilities helps with the prioritisation of alerts.
Threat intelligence integration with SIEM detection capabilities helps detect threats that might currently prevail inside the network through indicator lookups. Detections with known indicators increase the detection maturity in day-to-day security operations.
Planning and collection of threat feeds
Threat feeds from various sources can be of different types and in different forms. Methods to process and collect these feeds vary depending on the source platform but are mostly supported with APIs and standard file formats such as STIX, JSON, CSV, etc. Elastic, with its rich API-driven capabilities, provides a comprehensive data collection module for threat feeds from various sources.
Filebeat Threat Intel module supports different threat feed streams, including open source feeds, and has rich flexibility in enabling and disabling feeds, filtering the types of feed, time intervals to pull the feeds and, importantly, it automatically deduplicates feeds from the same source. This blog post covers the detailed steps for ingesting threat data with the Filebeat Threat Intel module.
Analytics with threat intelligence
The data collected by threat feeds provide previously identified indicators of a potential compromise and can assist in improving the effectiveness of the security devices that can leverage this information to detect or even block these known threats.
Dynamic or static indicators (threat data) that are streamed into an Elasticsearch index can be used with multiple use cases such as:
- Detecting malicious events and traffic through indicator match rules
- Enriching incoming traffic to add additional context
Indicator match detection rules
The Detection Rules GitHub repository is the home for rules created by the community and the Elastic Intelligence and Analytics team. Elastic provides out-of-the-box prebuilt detection rules addressing different threat use cases and this library of rules is continuously growing. Additionally, Elastic also provides the user with the ability to define and create their own detection rules to match their environmental use case requirements.
Indicator match rules are one of the detection rule types used to generate alerts when network or host events are observed that match the data provided by threat feeds. The indicator match rule compares the field values of the specified event and the indicator field values in the threat intel index. When the field values are identical, an alert is generated in the detection engine.
Elastic detection provides various out-of-the-box detection rules, including threat intel detection. Threat Intel detection rule is triggered when indicators from the threat intel Filebeat module have a match against local file or network observations.
Additionally, Elastic detection also provides the ability to create your own detection rules across various rule types.
As an example, let’s create an indicator match rule to identify a Network connection to a Malicious IP. With this indicator match rule, we can detect and generate an alert for outgoing network traffic to an IP address from a threat feed. When this alert is generated, this may indicate that the host generating the connection may be compromised or engaged in other suspicious user activity that could warrant further investigation.
Figure 2: Indicator match detection rule creation
Index patterns: Elasticsearch indices where you would like this rule to run
[Multiple index patterns supported]
Custom query: Relevant data sets to be queried in the provided index pattern.
[In the above example: Query to look for any outbound network connection
event.category: network and network.direction : ("outbound" or "egress" ) ]
Indicator index pattern: Index pattern of the threat intel feeds
Indicator Index query: Relevant data sets to be queried from the threat intel index
[In the above example: Query to look for IPV4 indicator type
event.module : "threatintel" and threatintel.indicator.type : "ipv4-addr" ]
Indicator mapping: Fields to lookup between the data index and the threat intel index. In our example, we’re matching the destination IP value from the network data with the indicator IP address from the threat intel feed.
This simple example indicator match rule can provide a straightforward way to identify network activity to a potentially compromised destination.
Triaging threat intel alerts:
When the indicator lookups trigger an alert, Elastic improves the analyst experience with a flyout that provides a summary of the alert that includes the threat intelligence context.
Figure 3: Alert rendering from indicator match rule
Figure 4: Alert summary with threat intel information
Threat intel enrichment
In general, data enrichment refers to the process of appending and enhancing collected data with relevant context obtained from additional sources. Adding context from threat information during the time of data collection has a unique advantage of quickly narrowing down bad events across a large data set. For example, the collected threat feed data about domains can be used to enrich the network logs to see if there are any known threat activities associated with that particular domain connection.
The Elasticsearch enrich processor provides an efficient way to add additional context to the incoming data during ingestion. Enrich processors work with an enrich policy. The purpose of an enrich policy is to take the data in an index and make it available to the enrich processor in an efficient format, so that the enrich processor can enrich incoming documents very quickly.
Let us take the domain enrichment example;
- The first step would be to create a enrich policy, which can easily be done using the Dev Tools in Kibana or by using the Elasticsearch API
PUT /_enrich/policy/ti-domain-policy
{
"match": {
"indices": "threatintel-*",
"match_field": "threatintel.indicator.domain",
"enrich_fields": [
"threatintel.indicator.type",
"fileset.name",
"threatintel.indicator.domain",
"malicious-event",
"tags"
]
}
}
- The second step is to execute the enrich policy. Behind the scenes, the execute policy API creates a system dot index that is fully managed by Elasticsearch
POST /_enrich/policy/ti-domain-policy/_execute
Note : With continuous updates of threat feeds, Enrich policy can be executed once in a day using the above execution API to update the enrich index with new feeds
- Now that that enrich policy has been executed, a pipeline with enrich processors can be defined:
PUT /_ingest/pipeline/threatintel-enrichment
{
"processors": [
{
"enrich": {
"description": "Enrich dns domain with threat intel indicators",
"policy_name": "ti-domain-policy",
"field": "dns.question.name",
"target_field": "enrichment",
"max_matches": "1",
"ignore_missing": true
}
}
]
}
The defined pipeline creates an enrich processor that utilises the domain enrich policy which will match the dns.question.name field value from the incoming document to the domain indicator (threatintel.indicator.domain) field value in the threatintel index. Once the lookup matches, the incoming document is added with relevant threat information associated with the domain indicator.
By using the defined pipeline in the data ingestion, respective data fields are looked up with threat intel feeds and enriched with additional threat intel information when matched.
Figure 5a: Enriched network data
Figure 5b: Enriched network data
These enriched malicious events can be a good metric to evaluate the blacklisting across security devices and this assists security analysts in narrowing down the time to event resolution.
Figure 6: Threat intel enrichment metrics in Kibana canvas
Wrapping up
Cyber threat intelligence (CTI) is considered one of the essential capabilities in an organization’s security program. When used properly, CTI can enable better-informed security and business decisions, and ultimately allow organizations to take decisive action to protect their users, data, and reputation against adversaries.
It’s easy to get started with Elastic Security — we offer Quick Start training to set you up for success. Start your free 14-day trial (no credit card required) or download our products, free, for your on-prem deployment