- X-Pack Reference for 6.0-6.2 and 5.x:
- Introduction
- Setting Up X-Pack
- Breaking Changes
- X-Pack APIs
- Graphing Connections in Your Data
- Profiling your Queries and Aggregations
- Reporting from Kibana
- Securing the Elastic Stack
- Getting Started with Security
- How Security Works
- Setting Up User Authentication
- Configuring SAML Single-Sign-On on the Elastic Stack
- Configuring Role-based Access Control
- Auditing Security Events
- Encrypting Communications
- Restricting Connections with IP Filtering
- Cross Cluster Search, Tribe, Clients and Integrations
- Reference
- Monitoring the Elastic Stack
- Alerting on Cluster and Index Events
- Machine Learning in the Elastic Stack
- Troubleshooting
- Getting Help
- X-Pack security
- Can’t log in after upgrading to 6.2.4
- Some settings are not returned via the nodes settings API
- Authorization exceptions
- Users command fails due to extra arguments
- Users are frequently locked out of Active Directory
- Certificate verification fails for curl on Mac
- SSLHandshakeException causes connections to fail
- Common SSL/TLS exceptions
- Internal Server Error in Kibana
- Setup-passwords command fails due to connection failure
- X-Pack Watcher
- X-Pack monitoring
- X-Pack machine learning
- Limitations
- License Management
- Release Notes
WARNING: Version 6.2 of the Elastic Stack has passed its EOL date.
This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.
Count Functions
editCount Functions
editCount functions detect anomalies when the number of events in a bucket is anomalous.
Use non_zero_count
functions if your data is sparse and you want to ignore
cases where the bucket count is zero.
Use distinct_count
functions to determine when the number of distinct values
in one field is unusual, as opposed to the total count.
Use high-sided functions if you want to monitor unusually high event rates. Use low-sided functions if you want to look at drops in event rate.
The X-Pack machine learning features include the following count functions:
Count, High_count, Low_count
editThe count
function detects anomalies when the number of events in a bucket is
anomalous.
The high_count
function detects anomalies when the count of events in a
bucket are unusually high.
The low_count
function detects anomalies when the count of events in a
bucket are unusually low.
These functions support the following properties:
-
by_field_name
(optional) -
over_field_name
(optional) -
partition_field_name
(optional)
For more information about those properties, see Detector Configuration Objects.
Example 1: Analyzing events with the count function.
{ "function" : "count" }
This example is probably the simplest possible analysis. It identifies time buckets during which the overall count of events is higher or lower than usual.
When you use this function in a detector in your job, it models the event rate and detects when the event rate is unusual compared to its past behavior.
Example 2: Analyzing errors with the high_count function.
{ "function" : "high_count", "by_field_name" : "error_code", "over_field_name": "user" }
If you use this high_count
function in a detector in your job, it
models the event rate for each error code. It detects users that generate an
unusually high count of error codes compared to other users.
Example 3: Analyzing status codes with the low_count function.
{ "function" : "low_count", "by_field_name" : "status_code" }
In this example, the function detects when the count of events for a status code is lower than usual.
When you use this function in a detector in your job, it models the event rate for each status code and detects when a status code has an unusually low count compared to its past behavior.
Example 4: Analyzing aggregated data with the count function.
{ "summary_count_field_name" : "events_per_min", "detectors" [ { "function" : "count" } ] }
If you are analyzing an aggregated events_per_min
field, do not use a sum
function (for example, sum(events_per_min)
). Instead, use the count function
and the summary_count_field_name
property.
Non_zero_count, High_non_zero_count, Low_non_zero_count
editThe non_zero_count
function detects anomalies when the number of events in a
bucket is anomalous, but it ignores cases where the bucket count is zero. Use
this function if you know your data is sparse or has gaps and the gaps are not
important.
The high_non_zero_count
function detects anomalies when the number of events
in a bucket is unusually high and it ignores cases where the bucket count is
zero.
The low_non_zero_count
function detects anomalies when the number of events in
a bucket is unusually low and it ignores cases where the bucket count is zero.
These functions support the following properties:
-
by_field_name
(optional) -
partition_field_name
(optional)
For more information about those properties, see Detector Configuration Objects.
For example, if you have the following number of events per bucket:
1,22,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,43,31,0,0,0,0,0,0,0,0,0,0,0,0,2,1
The non_zero_count
function models only the following data:
1,22,2,43,31,2,1
Example 5: Analyzing signatures with the high_non_zero_count function.
{ "function" : "high_non_zero_count", "by_field_name" : "signaturename" }
If you use this high_non_zero_count
function in a detector in your job, it
models the count of events for the signaturename
field. It ignores any buckets
where the count is zero and detects when a signaturename
value has an
unusually high count of events compared to its past behavior.
Population analysis (using an over_field_name
property value) is not
supported for the non_zero_count
, high_non_zero_count
, and
low_non_zero_count
functions. If you want to do population analysis and your
data is sparse, use the count
functions, which are optimized for that scenario.
Distinct_count, High_distinct_count, Low_distinct_count
editThe distinct_count
function detects anomalies where the number of distinct
values in one field is unusual.
The high_distinct_count
function detects unusually high numbers of distinct
values in one field.
The low_distinct_count
function detects unusually low numbers of distinct
values in one field.
These functions support the following properties:
-
field_name
(required) -
by_field_name
(optional) -
over_field_name
(optional) -
partition_field_name
(optional)
For more information about those properties, see Detector Configuration Objects.
Example 6: Analyzing users with the distinct_count function.
{ "function" : "distinct_count", "field_name" : "user" }
This distinct_count
function detects when a system has an unusual number
of logged in users. When you use this function in a detector in your job, it
models the distinct count of users. It also detects when the distinct number of
users is unusual compared to the past.
Example 7: Analyzing ports with the high_distinct_count function.
{ "function" : "high_distinct_count", "field_name" : "dst_port", "over_field_name": "src_ip" }
This example detects instances of port scanning. When you use this function in a
detector in your job, it models the distinct count of ports. It also detects the
src_ip
values that connect to an unusually high number of different
dst_ports
values compared to other src_ip
values.
On this page