Date histogram aggregation

When configuring a date histogram aggregation, the interval can be specified in two manners: calendar-aware time intervals, and fixed time intervals.

Calendar-aware intervals understand that daylight savings changes the length of specific days, months have different amounts of days, and leap seconds can be tacked onto a particular year.

Fixed intervals are, by contrast, always multiples of SI units and do not change based on calendaring context.

Calendar-aware intervals are configured with the calendar_interval parameter. You can specify calendar intervals using the unit name, such as month, or as a single unit quantity, such as 1M. For example, day and 1d are equivalent. Multiple quantities, such as 2d, are not supported.

The accepted calendar intervals are:

response = client.search(
  index: 'sales',
  size: 0,
  body: {
    aggregations: {
      sales_over_time: {
        date_histogram: {
          field: 'date',
          calendar_interval: 'month'
        }
      }
    }
  }
)
puts response

res, err := es.Search(
	es.Search.WithIndex("sales"),
	es.Search.WithBody(strings.NewReader(`{
	  "aggs": {
	    "sales_over_time": {
	      "date_histogram": {
	        "field": "date",
	        "calendar_interval": "month"
	      }
	    }
	  }
	}`)),
	es.Search.WithSize(0),
	es.Search.WithPretty(),
)
fmt.Println(res, err)

POST /sales/_search?size=0
{
  "aggs": {
    "sales_over_time": {
      "date_histogram": {
        "field": "date",
        "calendar_interval": "month"
      }
    }
  }
}

response = client.search(
  index: 'sales',
  size: 0,
  body: {
    aggregations: {
      sales_over_time: {
        date_histogram: {
          field: 'date',
          calendar_interval: '2d'
        }
      }
    }
  }
)
puts response

res, err := es.Search(
	es.Search.WithIndex("sales"),
	es.Search.WithBody(strings.NewReader(`{
	  "aggs": {
	    "sales_over_time": {
	      "date_histogram": {
	        "field": "date",
	        "calendar_interval": "2d"
	      }
	    }
	  }
	}`)),
	es.Search.WithSize(0),
	es.Search.WithPretty(),
)
fmt.Println(res, err)

POST /sales/_search?size=0
{
  "aggs": {
    "sales_over_time": {
      "date_histogram": {
        "field": "date",
        "calendar_interval": "2d"
      }
    }
  }
}

{
  "error" : {
    "root_cause" : [...],
    "type" : "x_content_parse_exception",
    "reason" : "[1:82] [date_histogram] failed to parse field [calendar_interval]",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "The supplied interval [2d] could not be parsed as a calendar interval.",
      "stack_trace" : "java.lang.IllegalArgumentException: The supplied interval [2d] could not be parsed as a calendar interval."
    }
  }
}

Fixed intervals are configured with the fixed_interval parameter.

In contrast to calendar-aware intervals, fixed intervals are a fixed number of SI units and never deviate, regardless of where they fall on the calendar. One second is always composed of 1000ms. This allows fixed intervals to be specified in any multiple of the supported units.

However, it means fixed intervals cannot express other units such as months, since the duration of a month is not a fixed quantity. Attempting to specify a calendar interval like month or quarter will throw an exception.

The accepted units for fixed intervals are:

response = client.search(
  index: 'sales',
  size: 0,
  body: {
    aggregations: {
      sales_over_time: {
        date_histogram: {
          field: 'date',
          fixed_interval: '30d'
        }
      }
    }
  }
)
puts response

res, err := es.Search(
	es.Search.WithIndex("sales"),
	es.Search.WithBody(strings.NewReader(`{
	  "aggs": {
	    "sales_over_time": {
	      "date_histogram": {
	        "field": "date",
	        "fixed_interval": "30d"
	      }
	    }
	  }
	}`)),
	es.Search.WithSize(0),
	es.Search.WithPretty(),
)
fmt.Println(res, err)

POST /sales/_search?size=0
{
  "aggs": {
    "sales_over_time": {
      "date_histogram": {
        "field": "date",
        "fixed_interval": "30d"
      }
    }
  }
}

response = client.search(
  index: 'sales',
  size: 0,
  body: {
    aggregations: {
      sales_over_time: {
        date_histogram: {
          field: 'date',
          fixed_interval: '2w'
        }
      }
    }
  }
)
puts response

res, err := es.Search(
	es.Search.WithIndex("sales"),
	es.Search.WithBody(strings.NewReader(`{
	  "aggs": {
	    "sales_over_time": {
	      "date_histogram": {
	        "field": "date",
	        "fixed_interval": "2w"
	      }
	    }
	  }
	}`)),
	es.Search.WithSize(0),
	es.Search.WithPretty(),
)
fmt.Println(res, err)

POST /sales/_search?size=0
{
  "aggs": {
    "sales_over_time": {
      "date_histogram": {
        "field": "date",
        "fixed_interval": "2w"
      }
    }
  }
}

{
  "error" : {
    "root_cause" : [...],
    "type" : "x_content_parse_exception",
    "reason" : "[1:82] [date_histogram] failed to parse field [fixed_interval]",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "failed to parse setting [date_histogram.fixedInterval] with value [2w] as a time value: unit is missing or unrecognized",
      "stack_trace" : "java.lang.IllegalArgumentException: failed to parse setting [date_histogram.fixedInterval] with value [2w] as a time value: unit is missing or unrecognized"
    }
  }
}

In all cases, when the specified end time does not exist, the actual end time is the closest available time after the specified end.

Widely distributed applications must also consider vagaries such as countries that start and stop daylight savings time at 12:01 A.M., so end up with one minute of Sunday followed by an additional 59 minutes of Saturday once a year, and countries that decide to move across the international date line. Situations like that can make irregular time zone offsets seem easy.

As always, rigorous testing, especially around time-change events, will ensure that your time interval specification is what you intend it to be.

Internally, a date is represented as a 64 bit number representing a timestamp in milliseconds-since-the-epoch (01/01/1970 midnight UTC). These timestamps are returned as the key name of the bucket. The key_as_string is the same timestamp converted to a formatted date string using the format parameter specification:

response = client.search(
  index: 'sales',
  size: 0,
  body: {
    aggregations: {
      sales_over_time: {
        date_histogram: {
          field: 'date',
          calendar_interval: '1M',
          format: 'yyyy-MM-dd'
        }
      }
    }
  }
)
puts response

res, err := es.Search(
	es.Search.WithIndex("sales"),
	es.Search.WithBody(strings.NewReader(`{
	  "aggs": {
	    "sales_over_time": {
	      "date_histogram": {
	        "field": "date",
	        "calendar_interval": "1M",
	        "format": "yyyy-MM-dd"
	      }
	    }
	  }
	}`)),
	es.Search.WithSize(0),
	es.Search.WithPretty(),
)
fmt.Println(res, err)

POST /sales/_search?size=0
{
  "aggs": {
    "sales_over_time": {
      "date_histogram": {
        "field": "date",
        "calendar_interval": "1M",
        "format": "yyyy-MM-dd" 
      }
    }
  }
}

Response:

{
  ...
  "aggregations": {
    "sales_over_time": {
      "buckets": [
        {
          "key_as_string": "2015-01-01",
          "key": 1420070400000,
          "doc_count": 3
        },
        {
          "key_as_string": "2015-02-01",
          "key": 1422748800000,
          "doc_count": 2
        },
        {
          "key_as_string": "2015-03-01",
          "key": 1425168000000,
          "doc_count": 2
        }
      ]
    }
  }
}

Elasticsearch stores date-times in Coordinated Universal Time (UTC). By default, all bucketing and rounding is also done in UTC. Use the time_zone parameter to indicate that bucketing should use a different time zone.

For example, if the interval is a calendar day and the time zone is America/New_York then 2020-01-03T01:00:01Z is : # Converted to 2020-01-02T18:00:01 # Rounded down to 2020-01-02T00:00:00 # Then converted back to UTC to produce 2020-01-02T05:00:00:00Z # Finally, when the bucket is turned into a string key it is printed in America/New_York so it’ll display as "2020-01-02T00:00:00".

It looks like:

bucket_key = localToUtc(Math.floor(utcToLocal(value) / interval) * interval))

You can specify time zones as an ISO 8601 UTC offset (e.g. +01:00 or -08:00) or as an IANA time zone ID, such as America/Los_Angeles.

Consider the following example:

response = client.index(
  index: 'my-index-000001',
  id: 1,
  refresh: true,
  body: {
    date: '2015-10-01T00:30:00Z'
  }
)
puts response

response = client.index(
  index: 'my-index-000001',
  id: 2,
  refresh: true,
  body: {
    date: '2015-10-01T01:30:00Z'
  }
)
puts response

response = client.index(
  index: 'my-index-000001',
  size: 0,
  body: {
    aggregations: {
      by_day: {
        date_histogram: {
          field: 'date',
          calendar_interval: 'day'
        }
      }
    }
  }
)
puts response

{
	res, err := es.Index(
		"my-index-000001",
		strings.NewReader(`{
	  "date": "2015-10-01T00:30:00Z"
	}`),
		es.Index.WithDocumentID("1"),
		es.Index.WithRefresh("true"),
		es.Index.WithPretty(),
	)
	fmt.Println(res, err)
}

{
	res, err := es.Index(
		"my-index-000001",
		strings.NewReader(`{
	  "date": "2015-10-01T01:30:00Z"
	}`),
		es.Index.WithDocumentID("2"),
		es.Index.WithRefresh("true"),
		es.Index.WithPretty(),
	)
	fmt.Println(res, err)
}

{
	res, err := es.Search(
		es.Search.WithIndex("my-index-000001"),
		es.Search.WithBody(strings.NewReader(`{
	  "aggs": {
	    "by_day": {
	      "date_histogram": {
	        "field": "date",
	        "calendar_interval": "day"
	      }
	    }
	  }
	}`)),
		es.Search.WithSize(0),
		es.Search.WithPretty(),
	)
	fmt.Println(res, err)
}

PUT my-index-000001/_doc/1?refresh
{
  "date": "2015-10-01T00:30:00Z"
}

PUT my-index-000001/_doc/2?refresh
{
  "date": "2015-10-01T01:30:00Z"
}

GET my-index-000001/_search?size=0
{
  "aggs": {
    "by_day": {
      "date_histogram": {
        "field":     "date",
        "calendar_interval":  "day"
      }
    }
  }
}

If you don’t specify a time zone, UTC is used. This would result in both of these documents being placed into the same day bucket, which starts at midnight UTC on 1 October 2015:

{
  ...
  "aggregations": {
    "by_day": {
      "buckets": [
        {
          "key_as_string": "2015-10-01T00:00:00.000Z",
          "key":           1443657600000,
          "doc_count":     2
        }
      ]
    }
  }
}

If you specify a time_zone of -01:00, midnight in that time zone is one hour before midnight UTC:

response = client.search(
  index: 'my-index-000001',
  size: 0,
  body: {
    aggregations: {
      by_day: {
        date_histogram: {
          field: 'date',
          calendar_interval: 'day',
          time_zone: '-01:00'
        }
      }
    }
  }
)
puts response

res, err := es.Search(
	es.Search.WithIndex("my-index-000001"),
	es.Search.WithBody(strings.NewReader(`{
	  "aggs": {
	    "by_day": {
	      "date_histogram": {
	        "field": "date",
	        "calendar_interval": "day",
	        "time_zone": "-01:00"
	      }
	    }
	  }
	}`)),
	es.Search.WithSize(0),
	es.Search.WithPretty(),
)
fmt.Println(res, err)

GET my-index-000001/_search?size=0
{
  "aggs": {
    "by_day": {
      "date_histogram": {
        "field":     "date",
        "calendar_interval":  "day",
        "time_zone": "-01:00"
      }
    }
  }
}

Now the first document falls into the bucket for 30 September 2015, while the second document falls into the bucket for 1 October 2015:

{
  ...
  "aggregations": {
    "by_day": {
      "buckets": [
        {
          "key_as_string": "2015-09-30T00:00:00.000-01:00", 
          "key": 1443574800000,
          "doc_count": 1
        },
        {
          "key_as_string": "2015-10-01T00:00:00.000-01:00", 
          "key": 1443661200000,
          "doc_count": 1
        }
      ]
    }
  }
}

Use the offset parameter to change the start value of each bucket by the specified positive (+) or negative offset (-) duration, such as 1h for an hour, or 1d for a day. See Time units for more possible time duration options.

For example, when using an interval of day, each bucket runs from midnight to midnight. Setting the offset parameter to +6h changes each bucket to run from 6am to 6am:

response = client.index(
  index: 'my-index-000001',
  id: 1,
  refresh: true,
  body: {
    date: '2015-10-01T05:30:00Z'
  }
)
puts response

response = client.index(
  index: 'my-index-000001',
  id: 2,
  refresh: true,
  body: {
    date: '2015-10-01T06:30:00Z'
  }
)
puts response

response = client.index(
  index: 'my-index-000001',
  size: 0,
  body: {
    aggregations: {
      by_day: {
        date_histogram: {
          field: 'date',
          calendar_interval: 'day',
          offset: '+6h'
        }
      }
    }
  }
)
puts response

{
	res, err := es.Index(
		"my-index-000001",
		strings.NewReader(`{
	  "date": "2015-10-01T05:30:00Z"
	}`),
		es.Index.WithDocumentID("1"),
		es.Index.WithRefresh("true"),
		es.Index.WithPretty(),
	)
	fmt.Println(res, err)
}

{
	res, err := es.Index(
		"my-index-000001",
		strings.NewReader(`{
	  "date": "2015-10-01T06:30:00Z"
	}`),
		es.Index.WithDocumentID("2"),
		es.Index.WithRefresh("true"),
		es.Index.WithPretty(),
	)
	fmt.Println(res, err)
}

{
	res, err := es.Search(
		es.Search.WithIndex("my-index-000001"),
		es.Search.WithBody(strings.NewReader(`{
	  "aggs": {
	    "by_day": {
	      "date_histogram": {
	        "field": "date",
	        "calendar_interval": "day",
	        "offset": "+6h"
	      }
	    }
	  }
	}`)),
		es.Search.WithSize(0),
		es.Search.WithPretty(),
	)
	fmt.Println(res, err)
}

PUT my-index-000001/_doc/1?refresh
{
  "date": "2015-10-01T05:30:00Z"
}

PUT my-index-000001/_doc/2?refresh
{
  "date": "2015-10-01T06:30:00Z"
}

GET my-index-000001/_search?size=0
{
  "aggs": {
    "by_day": {
      "date_histogram": {
        "field":     "date",
        "calendar_interval":  "day",
        "offset":    "+6h"
      }
    }
  }
}

Instead of a single bucket starting at midnight, the above request groups the documents into buckets starting at 6am:

{
  ...
  "aggregations": {
    "by_day": {
      "buckets": [
        {
          "key_as_string": "2015-09-30T06:00:00.000Z",
          "key": 1443592800000,
          "doc_count": 1
        },
        {
          "key_as_string": "2015-10-01T06:00:00.000Z",
          "key": 1443679200000,
          "doc_count": 1
        }
      ]
    }
  }
}

Setting the keyed flag to true associates a unique string key with each bucket and returns the ranges as a hash rather than an array:

response = client.search(
  index: 'sales',
  size: 0,
  body: {
    aggregations: {
      sales_over_time: {
        date_histogram: {
          field: 'date',
          calendar_interval: '1M',
          format: 'yyyy-MM-dd',
          keyed: true
        }
      }
    }
  }
)
puts response

res, err := es.Search(
	es.Search.WithIndex("sales"),
	es.Search.WithBody(strings.NewReader(`{
	  "aggs": {
	    "sales_over_time": {
	      "date_histogram": {
	        "field": "date",
	        "calendar_interval": "1M",
	        "format": "yyyy-MM-dd",
	        "keyed": true
	      }
	    }
	  }
	}`)),
	es.Search.WithSize(0),
	es.Search.WithPretty(),
)
fmt.Println(res, err)

POST /sales/_search?size=0
{
  "aggs": {
    "sales_over_time": {
      "date_histogram": {
        "field": "date",
        "calendar_interval": "1M",
        "format": "yyyy-MM-dd",
        "keyed": true
      }
    }
  }
}

Response:

{
  ...
  "aggregations": {
    "sales_over_time": {
      "buckets": {
        "2015-01-01": {
          "key_as_string": "2015-01-01",
          "key": 1420070400000,
          "doc_count": 3
        },
        "2015-02-01": {
          "key_as_string": "2015-02-01",
          "key": 1422748800000,
          "doc_count": 2
        },
        "2015-03-01": {
          "key_as_string": "2015-03-01",
          "key": 1425168000000,
          "doc_count": 2
        }
      }
    }
  }
}

If the data in your documents doesn’t exactly match what you’d like to aggregate, use a runtime field . For example, if the revenue for promoted sales should be recognized a day after the sale date:

POST /sales/_search?size=0
{
  "runtime_mappings": {
    "date.promoted_is_tomorrow": {
      "type": "date",
      "script": """
        long date = doc['date'].value.toInstant().toEpochMilli();
        if (doc['promoted'].value) {
          date += 86400;
        }
        emit(date);
      """
    }
  },
  "aggs": {
    "sales_over_time": {
      "date_histogram": {
        "field": "date.promoted_is_tomorrow",
        "calendar_interval": "1M"
      }
    }
  }
}

You can control the order of the returned buckets using the order settings and filter the returned buckets based on a min_doc_count setting (by default all buckets between the first bucket that matches documents and the last one are returned). This histogram also supports the extended_bounds setting, which enables extending the bounds of the histogram beyond the data itself, and hard_bounds that limits the histogram to specified bounds. For more information, see Extended Bounds and Hard Bounds.

response = client.search(
  index: 'sales',
  size: 0,
  body: {
    aggregations: {
      sale_date: {
        date_histogram: {
          field: 'date',
          calendar_interval: 'year',
          missing: '2000/01/01'
        }
      }
    }
  }
)
puts response

res, err := es.Search(
	es.Search.WithIndex("sales"),
	es.Search.WithBody(strings.NewReader(`{
	  "aggs": {
	    "sale_date": {
	      "date_histogram": {
	        "field": "date",
	        "calendar_interval": "year",
	        "missing": "2000/01/01"
	      }
	    }
	  }
	}`)),
	es.Search.WithSize(0),
	es.Search.WithPretty(),
)
fmt.Println(res, err)

POST /sales/_search?size=0
{
  "aggs": {
    "sale_date": {
      "date_histogram": {
        "field": "date",
        "calendar_interval": "year",
        "missing": "2000/01/01" 
      }
    }
  }
}

POST /sales/_search?size=0
{
  "runtime_mappings": {
    "date.day_of_week": {
      "type": "keyword",
      "script": "emit(doc['date'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"
    }
  },
  "aggs": {
    "day_of_week": {
      "terms": { "field": "date.day_of_week" }
    }
  }
}

{
  ...
  "aggregations": {
    "day_of_week": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "Sunday",
          "doc_count": 4
        },
        {
          "key": "Thursday",
          "doc_count": 3
        }
      ]
    }
  }
}