IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« Use Painless scripts in runtime fields How painless dispatches functions »

› ›

Using Datetime in Painless

edit

Using Datetime in Painless

edit

Datetime API

edit

Datetimes in Painless use the standard Java libraries and are available through the Painless Shared API. Most of the classes from the following Java packages are available to use in Painless scripts:

Datetime Representation

edit

Datetimes in Painless are most commonly represented as a numeric value, a string value, or a complex value.

numeric: a datetime representation as a number from a starting offset called an epoch; in Painless this is typically a long as milliseconds since an epoch of 1970-01-01 00:00:00 Zulu Time
string: a datetime representation as a sequence of characters defined by a standard format or a custom format; in Painless this is typically a String of the standard format ISO 8601
complex: a datetime representation as a complex type (object) that abstracts away internal details of how the datetime is stored and often provides utilities for modification and comparison; in Painless this is typically a ZonedDateTime

Switching between different representations of datetimes is often necessary to achieve a script’s objective(s). A typical pattern in a script is to switch a numeric or string datetime to a complex datetime, modify or compare the complex datetime, and then switch it back to a numeric or string datetime for storage or to return a result.

Datetime Parsing and Formatting

edit

Datetime parsing is a switch from a string datetime to a complex datetime, and datetime formatting is a switch from a complex datetime to a string datetime.

A DateTimeFormatter is a complex type (object) that defines the allowed sequence of characters for a string datetime. Datetime parsing and formatting often require a DateTimeFormatter. For more information about how to use a DateTimeFormatter see the Java documentation.

Datetime Parsing Examples

edit

parse from milliseconds

String milliSinceEpochString = "434931330000";
long milliSinceEpoch = Long.parseLong(milliSinceEpochString);
Instant instant = Instant.ofEpochMilli(milliSinceEpoch);
ZonedDateTime zdt = ZonedDateTime.ofInstant(instant, ZoneId.of('Z'));

parse from ISO 8601

String datetime = '1983-10-13T22:15:30Z';
ZonedDateTime zdt = ZonedDateTime.parse(datetime);

Note the parse method uses ISO 8601 by default.

parse from RFC 1123

String datetime = 'Thu, 13 Oct 1983 22:15:30 GMT';
ZonedDateTime zdt = ZonedDateTime.parse(datetime,
        DateTimeFormatter.RFC_1123_DATE_TIME);

Note the use of a built-in DateTimeFormatter.

parse from a custom format

String datetime = 'custom y 1983 m 10 d 13 22:15:30 Z';
DateTimeFormatter dtf = DateTimeFormatter.ofPattern(
        "'custom' 'y' yyyy 'm' MM 'd' dd HH:mm:ss VV");
ZonedDateTime zdt = ZonedDateTime.parse(datetime, dtf);

Note the use of a custom DateTimeFormatter.

Datetime Formatting Examples

edit

format to ISO 8601

ZonedDateTime zdt =
        ZonedDateTime.of(1983, 10, 13, 22, 15, 30, 0, ZoneId.of('Z'));
String datetime = zdt.format(DateTimeFormatter.ISO_INSTANT);

Note the use of a built-in DateTimeFormatter.

format to a custom format

ZonedDateTime zdt =
        ZonedDateTime.of(1983, 10, 13, 22, 15, 30, 0, ZoneId.of('Z'));
DateTimeFormatter dtf = DateTimeFormatter.ofPattern(
        "'date:' yyyy/MM/dd 'time:' HH:mm:ss");
String datetime = zdt.format(dtf);

Note the use of a custom DateTimeFormatter.

Datetime Conversion

edit

Datetime conversion is a switch from a numeric datetime to a complex datetime and vice versa.

Datetime Conversion Examples

edit

convert from milliseconds

long milliSinceEpoch = 434931330000L;
Instant instant = Instant.ofEpochMilli(milliSinceEpoch);
ZonedDateTime zdt = ZonedDateTime.ofInstant(instant, ZoneId.of('Z'));

convert to milliseconds

ZonedDateTime zdt =
        ZonedDateTime.of(1983, 10, 13, 22, 15, 30, 0, ZoneId.of('Z'));
long milliSinceEpoch = zdt.toInstant().toEpochMilli();

Datetime Pieces

edit

Datetime representations often contain the data to extract individual datetime pieces such as year, hour, timezone, etc. Use individual pieces of a datetime to create a complex datetime, and use a complex datetime to extract individual pieces.

Datetime Pieces Examples

edit

create a complex datetime from pieces

int year = 1983;
int month = 10;
int day = 13;
int hour = 22;
int minutes = 15;
int seconds = 30;
int nanos = 0;
ZonedDateTime zdt = ZonedDateTime.of(
        year, month, day, hour, minutes, seconds, nanos, ZoneId.of('Z'));

extract pieces from a complex datetime

ZonedDateTime zdt =
        ZonedDateTime.of(1983, 10, 13, 22, 15, 30, 100, ZoneId.of(tz));
int year = zdt.getYear();
int month = zdt.getMonthValue();
int day = zdt.getDayOfMonth();
int hour = zdt.getHour();
int minutes = zdt.getMinute();
int seconds = zdt.getSecond();
int nanos = zdt.getNano();

Datetime Modification

edit

Use either a numeric datetime or a complex datetime to do modification such as adding several seconds to a datetime or subtracting several days from a datetime. Use standard numeric operators to modify a numeric datetime. Use methods (or fields) to modify a complex datetime. Note many complex datetimes are immutable so upon modification a new complex datetime is created that requires assignment or immediate use.

Datetime Modification Examples

edit

Subtract three seconds from a numeric datetime in milliseconds

long milliSinceEpoch = 434931330000L;
milliSinceEpoch = milliSinceEpoch - 1000L*3L;

Add three days to a complex datetime

ZonedDateTime zdt =
        ZonedDateTime.of(1983, 10, 13, 22, 15, 30, 0, ZoneId.of('Z'));
ZonedDateTime updatedZdt = zdt.plusDays(3);

Subtract 125 minutes from a complex datetime

ZonedDateTime zdt =
        ZonedDateTime.of(1983, 10, 13, 22, 15, 30, 0, ZoneId.of('Z'));
ZonedDateTime updatedZdt = zdt.minusMinutes(125);

Set the year on a complex datetime

ZonedDateTime zdt =
        ZonedDateTime.of(1983, 10, 13, 22, 15, 30, 0, ZoneId.of('Z'));
ZonedDateTime updatedZdt = zdt.withYear(1976);

Datetime Difference (Elapsed Time)

edit

Use either two numeric datetimes or two complex datetimes to calculate the difference (elapsed time) between two different datetimes. Use subtraction to calculate the difference between two numeric datetimes of the same time unit such as milliseconds. For complex datetimes there is often a method or another complex type (object) available to calculate the difference. Use ChronoUnit to calculate the difference between two complex datetimes if supported.

Datetime Difference Examples

edit

Difference in milliseconds between two numeric datetimes

long startTimestamp = 434931327000L;
long endTimestamp = 434931330000L;
long differenceInMillis = endTimestamp - startTimestamp;

Difference in milliseconds between two complex datetimes

ZonedDateTime zdt1 =
        ZonedDateTime.of(1983, 10, 13, 22, 15, 30, 11000000, ZoneId.of('Z'));
ZonedDateTime zdt2 =
        ZonedDateTime.of(1983, 10, 13, 22, 15, 35, 0, ZoneId.of('Z'));
long differenceInMillis = ChronoUnit.MILLIS.between(zdt1, zdt2);

Difference in days between two complex datetimes

ZonedDateTime zdt1 =
        ZonedDateTime.of(1983, 10, 13, 22, 15, 30, 11000000, ZoneId.of('Z'));
ZonedDateTime zdt2 =
        ZonedDateTime.of(1983, 10, 17, 22, 15, 35, 0, ZoneId.of('Z'));
long differenceInDays = ChronoUnit.DAYS.between(zdt1, zdt2);

Datetime Comparison

edit

Use either two numeric datetimes or two complex datetimes to do a datetime comparison. Use standard comparison operators to compare two numeric datetimes of the same time unit such as milliseconds. For complex datetimes there is often a method or another complex type (object) available to do the comparison.

Datetime Comparison Examples

edit

Greater than comparison of two numeric datetimes in milliseconds

long timestamp1 = 434931327000L;
long timestamp2 = 434931330000L;

if (timestamp1 > timestamp2) {
   // handle condition
}

Equality comparison of two complex datetimes

ZonedDateTime zdt1 =
        ZonedDateTime.of(1983, 10, 13, 22, 15, 30, 0, ZoneId.of('Z'));
ZonedDateTime zdt2 =
        ZonedDateTime.of(1983, 10, 13, 22, 15, 30, 0, ZoneId.of('Z'));

if (zdt1.equals(zdt2)) {
    // handle condition
}

Less than comparison of two complex datetimes

ZonedDateTime zdt1 =
        ZonedDateTime.of(1983, 10, 13, 22, 15, 30, 0, ZoneId.of('Z'));
ZonedDateTime zdt2 =
        ZonedDateTime.of(1983, 10, 17, 22, 15, 35, 0, ZoneId.of('Z'));

if (zdt1.isBefore(zdt2)) {
    // handle condition
}

Greater than comparison of two complex datetimes

ZonedDateTime zdt1 =
        ZonedDateTime.of(1983, 10, 13, 22, 15, 30, 0, ZoneId.of('Z'));
ZonedDateTime zdt2 =
        ZonedDateTime.of(1983, 10, 17, 22, 15, 35, 0, ZoneId.of('Z'));

if (zdt1.isAfter(zdt2)) {
    // handle condition
}

Datetime Zone

edit

Both string datetimes and complex datetimes have a timezone with a default of UTC. Numeric datetimes do not have enough explicit information to have a timezone, so UTC is always assumed. Use methods (or fields) in conjunction with a ZoneId to change the timezone for a complex datetime. Parse a string datetime into a complex datetime to change the timezone, and then format the complex datetime back into a desired string datetime. Note many complex datetimes are immutable so upon modification a new complex datetime is created that requires assignment or immediate use.

Datetime Zone Examples

edit

Modify the timezone for a complex datetime

ZonedDateTime utc =
        ZonedDateTime.of(1983, 10, 13, 22, 15, 30, 0, ZoneId.of('Z'));
ZonedDateTime pst = utc.withZoneSameInstant(ZoneId.of('America/Los_Angeles'));

Modify the timezone for a string datetime

String gmtString = 'Thu, 13 Oct 1983 22:15:30 GMT';
ZonedDateTime gmtZdt = ZonedDateTime.parse(gmtString,
        DateTimeFormatter.RFC_1123_DATE_TIME); 
ZonedDateTime pstZdt =
        gmtZdt.withZoneSameInstant(ZoneId.of('America/Los_Angeles'));
String pstString = pstZdt.format(DateTimeFormatter.RFC_1123_DATE_TIME);

Note the use of a built-in DateTimeFormatter.

Datetime Input

edit

There are several common ways datetimes are used as input for a script determined by the Painless context. Typically, datetime input will be accessed from parameters specified by the user, from an original source document, or from an indexed document.

Datetime Input From User Parameters

edit

Use the params section during script specification to pass in a numeric datetime or string datetime as a script input. Access to user-defined parameters within a script is dependent on the Painless context, though, the parameters are most commonly accessible through an input called params.

Examples

Parse a numeric datetime from user parameters to a complex datetime

Input:

...
"script": {
    ...
    "params": {
        "input_datetime": 434931327000
    }
}
...

Script:

long inputDateTime = params['input_datetime'];
Instant instant = Instant.ofEpochMilli(inputDateTime);
ZonedDateTime zdt = ZonedDateTime.ofInstant(instant, ZoneId.of('Z'));

Parse a string datetime from user parameters to a complex datetime

Input:

...
"script": {
    ...
    "params": {
        "input_datetime": "custom y 1983 m 10 d 13 22:15:30 Z"
    }
}
...

Script:

String datetime = params['input_datetime'];
DateTimeFormatter dtf = DateTimeFormatter.ofPattern(
        "'custom' 'y' yyyy 'm' MM 'd' dd HH:mm:ss VV");
ZonedDateTime zdt = ZonedDateTime.parse(datetime, dtf);

Note the use of a custom DateTimeFormatter.

Datetime Input From a Source Document

edit

Use an original source document as a script input to access a numeric datetime or string datetime for a specific field within that document. Access to an original source document within a script is dependent on the Painless context and is not always available. An original source document is most commonly accessible through an input called ctx['_source'] or params['_source'].

Examples

Parse a numeric datetime from a sourced document to a complex datetime

Input:

{
  ...
  "input_datetime": 434931327000
  ...
}

Script:

long inputDateTime = ctx['_source']['input_datetime']; 
Instant instant = Instant.ofEpochMilli(inputDateTime);
ZonedDateTime zdt = ZonedDateTime.ofInstant(instant, ZoneId.of('Z'));

Note access to _source is dependent on the Painless context.

Parse a string datetime from a sourced document to a complex datetime
- Input:
```
{
  ...
  "input_datetime": "1983-10-13T22:15:30Z"
  ...
}
```
- Script:
```
String datetime = params['_source']['input_datetime']; 
ZonedDateTime zdt = ZonedDateTime.parse(datetime); 
```
  Note access to _source is dependent on the Painless context.
  
  Note the parse method uses ISO 8601 by default.

Datetime Input From an Indexed Document

edit

Use an indexed document as a script input to access a complex datetime for a specific field within that document where the field is mapped as a standard date or a nanosecond date. Numeric datetime fields mapped as numeric and string datetime fields mapped as keyword are accessible through an indexed document as well. Access to an indexed document within a script is dependent on the Painless context and is not always available. An indexed document is most commonly accessible through an input called doc.

Examples

Format a complex datetime from an indexed document to a string datetime
- Assumptions:
  - The field input_datetime exists in all indexes as part of the query
  - All indexed documents contain the field input_datetime
- Mappings:
```
{
  "mappings": {
    ...
    "properties": {
      ...
      "input_datetime": {
        "type": "date"
      }
      ...
    }
    ...
  }
}
```
- Script:
```
ZonedDateTime input = doc['input_datetime'].value;
String output = input.format(DateTimeFormatter.ISO_INSTANT); 
```
  Note the use of a built-in DateTimeFormatter.

Find the difference between two complex datetimes from an indexed document

Assumptions:
- The fields start and end may not exist in all indexes as part of the query
- The fields start and end may not have values in all indexed documents

Mappings:

{
  "mappings": {
    ...
    "properties": {
      ...
      "start": {
        "type": "date"
      },
      "end": {
        "type": "date"
      }
      ...
    }
    ...
  }
}

Script:

if (doc.containsKey('start') && doc.containsKey('end')) { 

    if (doc['start'].size() > 0 && doc['end'].size() > 0) { 

        ZonedDateTime start = doc['start'].value;
        ZonedDateTime end = doc['end'].value;
        long differenceInMillis = ChronoUnit.MILLIS.between(start, end);

        // handle difference in times
    } else {
        // handle fields without values
    }
} else {
    // handle index with missing fields
}

	When a query’s results span multiple indexes, some indexes may not contain a specific field. Use the `containsKey` method call on the `doc` input to ensure a field exists as part of the index for the current document.
	Some fields within a document may have no values. Use the `size` method call on a field within the `doc` input to ensure that field has at least one value for the current document.

Datetime Now

edit

Under most Painless contexts the current datetime, now, is not supported. There are two primary reasons for this. The first is that scripts are often run once per document, so each time the script is run a different now is returned. The second is that scripts are often run in a distributed fashion without a way to appropriately synchronize now. Instead, pass in a user-defined parameter with either a string datetime or numeric datetime for now. A numeric datetime is preferred as there is no need to parse it for comparison.

Datetime Now Examples

edit

Use a numeric datetime as now

Assumptions:
- The field input_datetime exists in all indexes as part of the query
- All indexed documents contain the field input_datetime

Mappings:

{
  "mappings": {
      ...
      "properties": {
        ...
        "input_datetime": {
        "type": "date"
      }
      ...
    }
    ...
  }
}

Input:

...
"script": {
    ...
    "params": {
        "now": <generated numeric datetime in milliseconds since epoch>
    }
}
...

Script:

long now = params['now'];
ZonedDateTime inputDateTime = doc['input_datetime'];
long millisDateTime = inputDateTime.toInstant().toEpochMilli();
long elapsedTime = now - millisDateTime;

Use a string datetime as now

Assumptions:
- The field input_datetime exists in all indexes as part of the query
- All indexed documents contain the field input_datetime

Mappings:

{
  "mappings": {
    ...
    "properties": {
      ...
      "input_datetime": {
        "type": "date"
      }
      ...
    }
    ...
  }
}

Input:

...
"script": {
    ...
    "params": {
        "now": "<generated string datetime in ISO-8601>"
    }
}
...

Script:

String nowString = params['now'];
ZonedDateTime nowZdt = ZonedDateTime.parse(nowString); 
long now = ZonedDateTime.toInstant().toEpochMilli();
ZonedDateTime inputDateTime = doc['input_datetime'];
long millisDateTime = zdt.toInstant().toEpochMilli();
long elapsedTime = now - millisDateTime;

Note this parses the same string datetime every time the script runs. Use a numeric datetime to avoid a significant performance hit.

Datetime Examples in Contexts

edit

Load the Example Data

edit

Run the following curl commands to load the data necessary for the context examples into an Elasticsearch cluster:

Create mappings for the sample data.

PUT /messages
{
  "mappings": {
    "properties": {
      "priority": {
        "type": "integer"
      },
      "datetime": {
        "type": "date"
      },
      "message": {
        "type": "text"
      }
    }
  }
}

Load the sample data.

POST /_bulk
{ "index" : { "_index" : "messages", "_id" : "1" } }
{ "priority": 1, "datetime": "2019-07-17T12:13:14Z", "message": "m1" }
{ "index" : { "_index" : "messages", "_id" : "2" } }
{ "priority": 1, "datetime": "2019-07-24T01:14:59Z", "message": "m2" }
{ "index" : { "_index" : "messages", "_id" : "3" } }
{ "priority": 2, "datetime": "1983-10-14T00:36:42Z", "message": "m3" }
{ "index" : { "_index" : "messages", "_id" : "4" } }
{ "priority": 3, "datetime": "1983-10-10T02:15:15Z", "message": "m4" }
{ "index" : { "_index" : "messages", "_id" : "5" } }
{ "priority": 3, "datetime": "1983-10-10T17:18:19Z", "message": "m5" }
{ "index" : { "_index" : "messages", "_id" : "6" } }
{ "priority": 1, "datetime": "2019-08-03T17:19:31Z", "message": "m6" }
{ "index" : { "_index" : "messages", "_id" : "7" } }
{ "priority": 3, "datetime": "2019-08-04T17:20:00Z", "message": "m7" }
{ "index" : { "_index" : "messages", "_id" : "8" } }
{ "priority": 2, "datetime": "2019-08-04T18:01:01Z", "message": "m8" }
{ "index" : { "_index" : "messages", "_id" : "9" } }
{ "priority": 3, "datetime": "1983-10-10T19:00:45Z", "message": "m9" }
{ "index" : { "_index" : "messages", "_id" : "10" } }
{ "priority": 2, "datetime": "2019-07-23T23:39:54Z", "message": "m10" }

Day-of-the-Week Bucket Aggregation Example

edit

The following example uses a terms aggregation as part of the bucket script aggregation context to display the number of messages from each day-of-the-week.

GET /messages/_search?pretty=true
{
  "aggs": {
    "day-of-week-count": {
      "terms": {
        "script": "return doc[\"datetime\"].value.getDayOfWeekEnum();"
      }
    }
  }
}

Morning/Evening Bucket Aggregation Example

edit

The following example uses a terms aggregation as part of the bucket script aggregation context to display the number of messages received in the morning versus the evening.

GET /messages/_search?pretty=true
{
  "aggs": {
    "am-pm-count": {
      "terms": {
        "script": "return doc[\"datetime\"].value.getHour() < 12 ? \"AM\" : \"PM\";"
      }
    }
  }
}

Age of a Message Script Field Example

edit

The following example uses a script field as part of the field context to display the elapsed time between "now" and when a message was received.

GET /_search?pretty=true
{
  "query": {
    "match_all": {}
  },
  "script_fields": {
    "message_age": {
      "script": {
        "source": "ZonedDateTime now = ZonedDateTime.ofInstant(Instant.ofEpochMilli(params[\"now\"]), ZoneId.of(\"Z\")); ZonedDateTime mdt = doc[\"datetime\"].value; String age; long years = mdt.until(now, ChronoUnit.YEARS); age = years + \"Y \"; mdt = mdt.plusYears(years); long months = mdt.until(now, ChronoUnit.MONTHS); age += months + \"M \"; mdt = mdt.plusMonths(months); long days = mdt.until(now, ChronoUnit.DAYS); age += days + \"D \"; mdt = mdt.plusDays(days); long hours = mdt.until(now, ChronoUnit.HOURS); age += hours + \"h \"; mdt = mdt.plusHours(hours); long minutes = mdt.until(now, ChronoUnit.MINUTES); age += minutes + \"m \"; mdt = mdt.plusMinutes(minutes); long seconds = mdt.until(now, ChronoUnit.SECONDS); age += hours + \"s\"; return age;",
        "params": {
          "now": 1574005645830
        }
      }
    }
  }
}

The following shows the script broken into multiple lines:

ZonedDateTime now = ZonedDateTime.ofInstant(
        Instant.ofEpochMilli(params['now']), ZoneId.of('Z')); 
ZonedDateTime mdt = doc['datetime'].value; 

String age;

long years = mdt.until(now, ChronoUnit.YEARS); 
age = years + 'Y '; 
mdt = mdt.plusYears(years); 

long months = mdt.until(now, ChronoUnit.MONTHS);
age += months + 'M ';
mdt = mdt.plusMonths(months);

long days = mdt.until(now, ChronoUnit.DAYS);
age += days + 'D ';
mdt = mdt.plusDays(days);

long hours = mdt.until(now, ChronoUnit.HOURS);
age += hours + 'h ';
mdt = mdt.plusHours(hours);

long minutes = mdt.until(now, ChronoUnit.MINUTES);
age += minutes + 'm ';
mdt = mdt.plusMinutes(minutes);

long seconds = mdt.until(now, ChronoUnit.SECONDS);
age += hours + 's';

return age;

	Parse the datetime "now" as input from the user-defined params.
	Store the datetime the message was received as a `ZonedDateTime`.
	Find the difference in years between "now" and the datetime the message was received.
	Add the difference in years later returned in the format `Y <years> ...` for the age of a message.
	Add the years so only the remainder of the months, days, etc. remain as the difference between "now" and the datetime the message was received. Repeat this pattern until the desired granularity is reached (seconds in this example).
	Return the age of the message in the format `Y <years> M <months> D <days> h <hours> m <minutes> s <seconds>`.

« Use Painless scripts in runtime fields How painless dispatches functions »