- Elasticsearch Guide: other versions:
- Elasticsearch introduction
- Getting started with Elasticsearch
- Set up Elasticsearch
- Installing Elasticsearch
- Configuring Elasticsearch
- Important Elasticsearch configuration
- Important System Configuration
- Bootstrap Checks
- Heap size check
- File descriptor check
- Memory lock check
- Maximum number of threads check
- Max file size check
- Maximum size virtual memory check
- Maximum map count check
- Client JVM check
- Use serial collector check
- System call filter check
- OnError and OnOutOfMemoryError checks
- Early-access check
- G1GC check
- All permission check
- Discovery configuration check
- Starting Elasticsearch
- Stopping Elasticsearch
- Adding nodes to your cluster
- Set up X-Pack
- Configuring X-Pack Java Clients
- Bootstrap Checks for X-Pack
- Upgrade Elasticsearch
- API conventions
- Document APIs
- Search APIs
- Aggregations
- Metrics Aggregations
- Avg Aggregation
- Weighted Avg Aggregation
- Cardinality Aggregation
- Extended Stats Aggregation
- Geo Bounds Aggregation
- Geo Centroid Aggregation
- Max Aggregation
- Min Aggregation
- Percentiles Aggregation
- Percentile Ranks Aggregation
- Scripted Metric Aggregation
- Stats Aggregation
- Sum Aggregation
- Top Hits Aggregation
- Value Count Aggregation
- Median Absolute Deviation Aggregation
- Bucket Aggregations
- Adjacency Matrix Aggregation
- Auto-interval Date Histogram Aggregation
- Children Aggregation
- Composite Aggregation
- Date Histogram Aggregation
- Date Range Aggregation
- Diversified Sampler Aggregation
- Filter Aggregation
- Filters Aggregation
- Geo Distance Aggregation
- GeoHash grid Aggregation
- GeoTile Grid Aggregation
- Global Aggregation
- Histogram Aggregation
- IP Range Aggregation
- Missing Aggregation
- Nested Aggregation
- Parent Aggregation
- Range Aggregation
- Reverse nested Aggregation
- Sampler Aggregation
- Significant Terms Aggregation
- Significant Text Aggregation
- Terms Aggregation
- Pipeline Aggregations
- Avg Bucket Aggregation
- Derivative Aggregation
- Max Bucket Aggregation
- Min Bucket Aggregation
- Sum Bucket Aggregation
- Stats Bucket Aggregation
- Extended Stats Bucket Aggregation
- Percentiles Bucket Aggregation
- Moving Average Aggregation
- Moving Function Aggregation
- Cumulative Sum Aggregation
- Bucket Script Aggregation
- Bucket Selector Aggregation
- Bucket Sort Aggregation
- Serial Differencing Aggregation
- Matrix Aggregations
- Caching heavy aggregations
- Returning only aggregation results
- Aggregation Metadata
- Returning the type of the aggregation
- Metrics Aggregations
- Indices APIs
- Create Index
- Delete Index
- Get Index
- Indices Exists
- Open / Close Index API
- Shrink Index
- Split Index
- Rollover Index
- Put Mapping
- Get Mapping
- Get Field Mapping
- Types Exists
- Index Aliases
- Update Indices Settings
- Get Settings
- Analyze
- Index Templates
- Indices Stats
- Indices Segments
- Indices Recovery
- Indices Shard Stores
- Clear Cache
- Flush
- Refresh
- Force Merge
- cat APIs
- Cluster APIs
- Query DSL
- Scripting
- Mapping
- Analysis
- Anatomy of an analyzer
- Testing analyzers
- Analyzers
- Normalizers
- Tokenizers
- Standard Tokenizer
- Letter Tokenizer
- Lowercase Tokenizer
- Whitespace Tokenizer
- UAX URL Email Tokenizer
- Classic Tokenizer
- Thai Tokenizer
- NGram Tokenizer
- Edge NGram Tokenizer
- Keyword Tokenizer
- Pattern Tokenizer
- Char Group Tokenizer
- Simple Pattern Tokenizer
- Simple Pattern Split Tokenizer
- Path Hierarchy Tokenizer
- Path Hierarchy Tokenizer Examples
- Token Filters
- ASCII Folding Token Filter
- Flatten Graph Token Filter
- Length Token Filter
- Lowercase Token Filter
- Uppercase Token Filter
- NGram Token Filter
- Edge NGram Token Filter
- Porter Stem Token Filter
- Shingle Token Filter
- Stop Token Filter
- Word Delimiter Token Filter
- Word Delimiter Graph Token Filter
- Multiplexer Token Filter
- Conditional Token Filter
- Predicate Token Filter Script
- Stemmer Token Filter
- Stemmer Override Token Filter
- Keyword Marker Token Filter
- Keyword Repeat Token Filter
- KStem Token Filter
- Snowball Token Filter
- Phonetic Token Filter
- Synonym Token Filter
- Parsing synonym files
- Synonym Graph Token Filter
- Compound Word Token Filters
- Reverse Token Filter
- Elision Token Filter
- Truncate Token Filter
- Unique Token Filter
- Pattern Capture Token Filter
- Pattern Replace Token Filter
- Trim Token Filter
- Limit Token Count Token Filter
- Hunspell Token Filter
- Common Grams Token Filter
- Normalization Token Filter
- CJK Width Token Filter
- CJK Bigram Token Filter
- Delimited Payload Token Filter
- Keep Words Token Filter
- Keep Types Token Filter
- Exclude mode settings example
- Classic Token Filter
- Apostrophe Token Filter
- Decimal Digit Token Filter
- Fingerprint Token Filter
- MinHash Token Filter
- Remove Duplicates Token Filter
- Character Filters
- Modules
- Index modules
- Ingest node
- Pipeline Definition
- Ingest APIs
- Accessing Data in Pipelines
- Conditional Execution in Pipelines
- Handling Failures in Pipelines
- Processors
- Append Processor
- Bytes Processor
- Convert Processor
- Date Processor
- Date Index Name Processor
- Dissect Processor
- Dot Expander Processor
- Drop Processor
- Fail Processor
- Foreach Processor
- GeoIP Processor
- Grok Processor
- Gsub Processor
- HTML Strip Processor
- Join Processor
- JSON Processor
- KV Processor
- Lowercase Processor
- Pipeline Processor
- Remove Processor
- Rename Processor
- Script Processor
- Set Processor
- Set Security User Processor
- Split Processor
- Sort Processor
- Trim Processor
- Uppercase Processor
- URL Decode Processor
- User Agent processor
- Managing the index lifecycle
- Getting started with index lifecycle management
- Policy phases and actions
- Set up index lifecycle management policy
- Using policies to manage index rollover
- Update policy
- Index lifecycle error handling
- Restoring snapshots of managed indices
- Start and stop index lifecycle management
- Using ILM with existing indices
- SQL access
- Overview
- Getting Started with SQL
- Conventions and Terminology
- Security
- SQL REST API
- SQL Translate API
- SQL CLI
- SQL JDBC
- SQL ODBC
- SQL Client Applications
- SQL Language
- Functions and Operators
- Comparison Operators
- Logical Operators
- Math Operators
- Cast Operators
- LIKE and RLIKE Operators
- Aggregate Functions
- Grouping Functions
- Date/Time and Interval Functions and Operators
- Full-Text Search Functions
- Mathematical Functions
- String Functions
- Type Conversion Functions
- Geo Functions
- Conditional Functions And Expressions
- System Functions
- Reserved keywords
- SQL Limitations
- Monitor a cluster
- Frozen indices
- Set up a cluster for high availability
- Roll up or transform your data
- X-Pack APIs
- Info API
- Cross-cluster replication APIs
- Explore API
- Freeze index
- Index lifecycle management API
- Licensing APIs
- Machine learning APIs
- Add events to calendar
- Add jobs to calendar
- Close jobs
- Create jobs
- Create calendar
- Create datafeeds
- Create filter
- Delete calendar
- Delete datafeeds
- Delete events from calendar
- Delete filter
- Delete forecast
- Delete jobs
- Delete jobs from calendar
- Delete model snapshots
- Delete expired data
- Find file structure
- Flush jobs
- Forecast jobs
- Get calendars
- Get buckets
- Get overall buckets
- Get categories
- Get datafeeds
- Get datafeed statistics
- Get influencers
- Get jobs
- Get job statistics
- Get machine learning info
- Get model snapshots
- Get scheduled events
- Get filters
- Get records
- Open jobs
- Post data to jobs
- Preview datafeeds
- Revert model snapshots
- Set upgrade mode
- Start datafeeds
- Stop datafeeds
- Update datafeeds
- Update filter
- Update jobs
- Update model snapshots
- Migration APIs
- Rollup APIs
- Security APIs
- Authenticate
- Change passwords
- Clear cache
- Clear roles cache
- Create API keys
- Create or update application privileges
- Create or update role mappings
- Create or update roles
- Create or update users
- Delete application privileges
- Delete role mappings
- Delete roles
- Delete users
- Disable users
- Enable users
- Get API key information
- Get application privileges
- Get role mappings
- Get roles
- Get token
- Get users
- Has privileges
- Invalidate API key
- Invalidate token
- OpenID Connect Prepare Authentication API
- OpenID Connect Authenticate API
- OpenID Connect Logout API
- SSL certificate
- Transform APIs
- Unfreeze index
- Watcher APIs
- Definitions
- Secure a cluster
- Overview
- Configuring security
- Encrypting communications in Elasticsearch
- Encrypting communications in an Elasticsearch Docker Container
- Enabling cipher suites for stronger encryption
- Separating node-to-node and client traffic
- Configuring an Active Directory realm
- Configuring a file realm
- Configuring an LDAP realm
- Configuring a native realm
- Configuring a PKI realm
- Configuring a SAML realm
- Configuring a Kerberos realm
- Security files
- FIPS 140-2
- How security works
- User authentication
- Built-in users
- Internal users
- Token-based authentication services
- Realms
- Realm chains
- Active Directory user authentication
- File-based user authentication
- LDAP user authentication
- Native user authentication
- OpenID Connect authentication
- PKI user authentication
- SAML authentication
- Kerberos authentication
- Integrating with other authentication systems
- Enabling anonymous access
- Controlling the user cache
- Configuring SAML single-sign-on on the Elastic Stack
- Configuring single sign-on to the Elastic Stack using OpenID Connect
- User authorization
- Auditing security events
- Encrypting communications
- Restricting connections with IP filtering
- Cross cluster search, clients, and integrations
- Tutorial: Getting started with security
- Tutorial: Encrypting communications
- Troubleshooting
- Some settings are not returned via the nodes settings API
- Authorization exceptions
- Users command fails due to extra arguments
- Users are frequently locked out of Active Directory
- Certificate verification fails for curl on Mac
- SSLHandshakeException causes connections to fail
- Common SSL/TLS exceptions
- Common Kerberos exceptions
- Common SAML issues
- Internal Server Error in Kibana
- Setup-passwords command fails due to connection failure
- Failures due to relocation of the configuration files
- Limitations
- Alerting on cluster and index events
- Command line tools
- How To
- Testing
- Glossary of terms
- Release highlights
- Breaking changes
- Release notes
- Elasticsearch version 7.2.1
- Elasticsearch version 7.2.0
- Elasticsearch version 7.1.1
- Elasticsearch version 7.1.0
- Elasticsearch version 7.0.0
- Elasticsearch version 7.0.0-rc2
- Elasticsearch version 7.0.0-rc1
- Elasticsearch version 7.0.0-beta1
- Elasticsearch version 7.0.0-alpha2
- Elasticsearch version 7.0.0-alpha1
Date Histogram Aggregation
editDate Histogram Aggregation
editThis multi-bucket aggregation is similar to the normal
histogram, but it can
only be used with date values. Because dates are represented internally in
Elasticsearch as long values, it is possible, but not as accurate, to use the
normal histogram
on dates as well. The main difference in the two APIs is
that here the interval can be specified using date/time expressions. Time-based
data requires special support because time-based intervals are not always a
fixed length.
Calendar and Fixed intervals
editWhen configuring a date histogram aggregation, the interval can be specified in two manners: calendar-aware time intervals, and fixed time intervals.
Calendar-aware intervals understand that daylight savings changes the length of specific days, months have different amounts of days, and leap seconds can be tacked onto a particular year.
Fixed intervals are, by contrast, always multiples of SI units and do not change based on calendaring context.
Combined interval
field is deprecated
[7.2]
Deprecated in 7.2. interval
field is deprecated
Historically both calendar and fixed
intervals were configured in a single interval
field, which led to confusing
semantics. Specifying 1d
would be assumed as a calendar-aware time,
whereas 2d
would be interpreted as fixed time. To get "one day" of fixed time,
the user would need to specify the next smaller unit (in this case, 24h
).
This combined behavior was often unknown to users, and even when knowledgeable about the behavior it was difficult to use and confusing.
This behavior has been deprecated in favor of two new, explicit fields: calendar_interval
and fixed_interval
.
By forcing a choice between calendar and intervals up front, the semantics of the interval
are clear to the user immediately and there is no ambiguity. The old interval
field
will be removed in the future.
Calendar Intervals
editCalendar-aware intervals are configured with the calendar_interval
parameter.
Calendar intervals can only be specified in "singular" quantities of the unit
(1d
, 1M
, etc). Multiples, such as 2d
, are not supported and will throw an exception.
The accepted units for calendar intervals are:
-
minute (
m
,1m
) - All minutes begin at 00 seconds.
One minute is the interval between 00 seconds of the first minute and 00 seconds of the following minute in the specified timezone, compensating for any intervening leap seconds, so that the number of minutes and seconds past the hour is the same at the start and end.
-
hour (
h
,1h
) - All hours begin at 00 minutes and 00 seconds.
One hour (1h) is the interval between 00:00 minutes of the first hour and 00:00 minutes of the following hour in the specified timezone, compensating for any intervening leap seconds, so that the number of minutes and seconds past the hour is the same at the start and end.
-
day (
d
,1d
) - All days begin at the earliest possible time, which is usually 00:00:00 (midnight).
One day (1d) is the interval between the start of the day and the start of of the following day in the specified timezone, compensating for any intervening time changes.
-
week (
w
,1w
) - One week is the interval between the start day_of_week:hour:minute:second and the same day of the week and time of the following week in the specified timezone.
-
month (
M
,1M
) - One month is the interval between the start day of the month and time of day and the same day of the month and time of the following month in the specified timezone, so that the day of the month and time of day are the same at the start and end.
-
quarter (
q
,1q
) -
One quarter (1q) is the interval between the start day of the month and
time of day and the same day of the month and time of day three months later,
so that the day of the month and time of day are the same at the start and end.
-
year (
y
,1y
) -
One year (1y) is the interval between the start day of the month and time of
day and the same day of the month and time of day the following year in the
specified timezone, so that the date and time are the same at the start and end.
Calendar Interval Examples
editAs an example, here is an aggregation requesting bucket intervals of a month in calendar time:
POST /sales/_search?size=0 { "aggs" : { "sales_over_time" : { "date_histogram" : { "field" : "date", "calendar_interval" : "month" } } } }
If you attempt to use multiples of calendar units, the aggregation will fail because only singular calendar units are supported:
POST /sales/_search?size=0 { "aggs" : { "sales_over_time" : { "date_histogram" : { "field" : "date", "calendar_interval" : "2d" } } } }
{ "error" : { "root_cause" : [...], "type" : "x_content_parse_exception", "reason" : "[1:82] [date_histogram] failed to parse field [calendar_interval]", "caused_by" : { "type" : "illegal_argument_exception", "reason" : "The supplied interval [2d] could not be parsed as a calendar interval.", "stack_trace" : "java.lang.IllegalArgumentException: The supplied interval [2d] could not be parsed as a calendar interval." } } }
Fixed Intervals
editFixed intervals are configured with the fixed_interval
parameter.
In contrast to calendar-aware intervals, fixed intervals are a fixed number of SI units and never deviate, regardless of where they fall on the calendar. One second is always composed of 1000ms. This allows fixed intervals to be specified in any multiple of the supported units.
However, it means fixed intervals cannot express other units such as months, since the duration of a month is not a fixed quantity. Attempting to specify a calendar interval like month or quarter will throw an exception.
The accepted units for fixed intervals are:
- milliseconds (ms)
- seconds (s)
- Defined as 1000 milliseconds each
- minutes (m)
- All minutes begin at 00 seconds.
Defined as 60 seconds each (60,000 milliseconds)
- hours (h)
- All hours begin at 00 minutes and 00 seconds. Defined as 60 minutes each (3,600,000 milliseconds)
- days (d)
- All days begin at the earliest possible time, which is usually 00:00:00 (midnight).
Defined as 24 hours (86,400,000 milliseconds)
Fixed Interval Examples
editIf we try to recreate the "month" calendar_interval
from earlier, we can approximate that with
30 fixed days:
POST /sales/_search?size=0 { "aggs" : { "sales_over_time" : { "date_histogram" : { "field" : "date", "fixed_interval" : "30d" } } } }
But if we try to use a calendar unit that is not supported, such as weeks, we’ll get an exception:
POST /sales/_search?size=0 { "aggs" : { "sales_over_time" : { "date_histogram" : { "field" : "date", "fixed_interval" : "2w" } } } }
{ "error" : { "root_cause" : [...], "type" : "x_content_parse_exception", "reason" : "[1:82] [date_histogram] failed to parse field [fixed_interval]", "caused_by" : { "type" : "illegal_argument_exception", "reason" : "failed to parse setting [date_histogram.fixedInterval] with value [2w] as a time value: unit is missing or unrecognized", "stack_trace" : "java.lang.IllegalArgumentException: failed to parse setting [date_histogram.fixedInterval] with value [2w] as a time value: unit is missing or unrecognized" } } }
Notes
editIn all cases, when the specified end time does not exist, the actual end time is the closest available time after the specified end.
Widely distributed applications must also consider vagaries such as countries that start and stop daylight savings time at 12:01 A.M., so end up with one minute of Sunday followed by an additional 59 minutes of Saturday once a year, and countries that decide to move across the international date line. Situations like that can make irregular timezone offsets seem easy.
As always, rigorous testing, especially around time-change events, will ensure that your time interval specification is what you intend it to be.
WARNING: To avoid unexpected results, all connected servers and clients must sync to a reliable network time service.
fractional time values are not supported, but you can address this by
shifting to another time unit (e.g., 1.5h
could instead be specified as 90m
).
You can also specify time values using abbreviations supported by time units parsing.
Keys
editInternally, a date is represented as a 64 bit number representing a timestamp
in milliseconds-since-the-epoch (01/01/1970 midnight UTC). These timestamps are
returned as the key
name of the bucket. The key_as_string
is the same
timestamp converted to a formatted
date string using the format
parameter specification:
If you don’t specify format
, the first date
format specified in the field mapping is used.
POST /sales/_search?size=0 { "aggs" : { "sales_over_time" : { "date_histogram" : { "field" : "date", "calendar_interval" : "1M", "format" : "yyyy-MM-dd" } } } }
Supports expressive date format pattern |
Response:
{ ... "aggregations": { "sales_over_time": { "buckets": [ { "key_as_string": "2015-01-01", "key": 1420070400000, "doc_count": 3 }, { "key_as_string": "2015-02-01", "key": 1422748800000, "doc_count": 2 }, { "key_as_string": "2015-03-01", "key": 1425168000000, "doc_count": 2 } ] } } }
Timezone
editDate-times are stored in Elasticsearch in UTC. By default, all bucketing and
rounding is also done in UTC. Use the time_zone
parameter to indicate
that bucketing should use a different timezone.
You can specify timezones as either an ISO 8601 UTC offset (e.g. +01:00
or
-08:00
) or as a timezone ID as specified in the IANA timezone database,
such as`America/Los_Angeles`.
Consider the following example:
PUT my_index/_doc/1?refresh { "date": "2015-10-01T00:30:00Z" } PUT my_index/_doc/2?refresh { "date": "2015-10-01T01:30:00Z" } GET my_index/_search?size=0 { "aggs": { "by_day": { "date_histogram": { "field": "date", "calendar_interval": "day" } } } }
If you don’t specify a timezone, UTC is used. This would result in both of these documents being placed into the same day bucket, which starts at midnight UTC on 1 October 2015:
{ ... "aggregations": { "by_day": { "buckets": [ { "key_as_string": "2015-10-01T00:00:00.000Z", "key": 1443657600000, "doc_count": 2 } ] } } }
If you specify a time_zone
of -01:00
, midnight in that timezone is one hour
before midnight UTC:
GET my_index/_search?size=0 { "aggs": { "by_day": { "date_histogram": { "field": "date", "calendar_interval": "day", "time_zone": "-01:00" } } } }
Now the first document falls into the bucket for 30 September 2015, while the second document falls into the bucket for 1 October 2015:
{ ... "aggregations": { "by_day": { "buckets": [ { "key_as_string": "2015-09-30T00:00:00.000-01:00", "key": 1443574800000, "doc_count": 1 }, { "key_as_string": "2015-10-01T00:00:00.000-01:00", "key": 1443661200000, "doc_count": 1 } ] } } }
When using time zones that follow DST (daylight savings time) changes,
buckets close to the moment when those changes happen can have slightly different
sizes than you would expect from the used interval
.
For example, consider a DST start in the CET
time zone: on 27 March 2016 at 2am,
clocks were turned forward 1 hour to 3am local time. If you use day
as interval
,
the bucket covering that day will only hold data for 23 hours instead of the usual
24 hours for other buckets. The same is true for shorter intervals, like 12h,
where you’ll have only a 11h bucket on the morning of 27 March when the DST shift
happens.
Offset
editUse the offset
parameter to change the start value of each bucket by the
specified positive (+
) or negative offset (-
) duration, such as 1h
for
an hour, or 1d
for a day. See Time units for more possible time
duration options.
For example, when using an interval of day
, each bucket runs from midnight
to midnight. Setting the offset
parameter to +6h
changes each bucket
to run from 6am to 6am:
PUT my_index/_doc/1?refresh { "date": "2015-10-01T05:30:00Z" } PUT my_index/_doc/2?refresh { "date": "2015-10-01T06:30:00Z" } GET my_index/_search?size=0 { "aggs": { "by_day": { "date_histogram": { "field": "date", "calendar_interval": "day", "offset": "+6h" } } } }
Instead of a single bucket starting at midnight, the above request groups the documents into buckets starting at 6am:
{ ... "aggregations": { "by_day": { "buckets": [ { "key_as_string": "2015-09-30T06:00:00.000Z", "key": 1443592800000, "doc_count": 1 }, { "key_as_string": "2015-10-01T06:00:00.000Z", "key": 1443679200000, "doc_count": 1 } ] } } }
The start offset
of each bucket is calculated after time_zone
adjustments have been made.
Keyed Response
editSetting the keyed
flag to true
associates a unique string key with each
bucket and returns the ranges as a hash rather than an array:
POST /sales/_search?size=0 { "aggs" : { "sales_over_time" : { "date_histogram" : { "field" : "date", "calendar_interval" : "1M", "format" : "yyyy-MM-dd", "keyed": true } } } }
Response:
{ ... "aggregations": { "sales_over_time": { "buckets": { "2015-01-01": { "key_as_string": "2015-01-01", "key": 1420070400000, "doc_count": 3 }, "2015-02-01": { "key_as_string": "2015-02-01", "key": 1422748800000, "doc_count": 2 }, "2015-03-01": { "key_as_string": "2015-03-01", "key": 1425168000000, "doc_count": 2 } } } } }
Scripts
editAs with the normal histogram,
both document-level scripts and
value-level scripts are supported. You can control the order of the returned
buckets using the order
settings and filter the returned buckets based on a min_doc_count
setting
(by default all buckets between the first
bucket that matches documents and the last one are returned). This histogram
also supports the extended_bounds
setting, which enables extending the bounds of the histogram beyond the data
itself. For more information, see
Extended Bounds
.
Missing value
editThe missing
parameter defines how to treat documents that are missing a value.
By default, they are ignored, but it is also possible to treat them as if they
have a value.
Order
editBy default the returned buckets are sorted by their key
ascending, but you can
control the order using
the order
setting. This setting supports the same order
functionality as
Terms Aggregation
.
Using a script to aggregate by day of the week
editWhen you need to aggregate the results by day of the week, use a script that returns the day of the week:
POST /sales/_search?size=0 { "aggs": { "dayOfWeek": { "terms": { "script": { "lang": "painless", "source": "doc['date'].value.dayOfWeekEnum.value" } } } } }
Response:
{ ... "aggregations": { "dayOfWeek": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "7", "doc_count": 4 }, { "key": "4", "doc_count": 3 } ] } } }
The response will contain all the buckets having the relative day of the week as key : 1 for Monday, 2 for Tuesday… 7 for Sunday.
On this page