New

The executive guide to generative AI

Read more
About usPartnersSupport|Login
IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Breaking changes in 7.0 Release notes »
Elastic Docs ›Elasticsearch Guide [7.7] ›Breaking changes ›Breaking changes in 7.0

Java time migration guide

edit

Java time migration guide

edit

With 7.0, Elasticsearch switched from joda time to java time for date-related parsing, formatting, and calculations. This guide is designed to help you determine if your cluster is impacted and, if so, prepare for the upgrade.

Convert date formats

edit

To upgrade to 7.0, you’ll need to convert any joda-time date formats to their java-time equivalents.

To help track this effort, you can prefix java-time date formats with an 8 in Elasticsearch 6.8 and later versions.

For example, you can change the date format YYYY-MM-dd to 8yyyy-MM-dd to indicate the date format uses java time.

Elasticsearch treats date formats starting with the 8 prefix differently depending on the version:

6.8: Date formats with an 8 prefix are handled as java-time formats. Date formats without an 8 prefix are treated as joda-time formats. We recommend converting these joda-time formats to java-time before upgrading to 7.x.

7.x and later: For indices created in 6.x, date formats without an 8 prefix are treated as joda-time formats. For indices created in 7.x and later versions, all date formats are treated as java-time formats, regardless of whether it starts with an 8 prefix.

Impacted features

edit

The switch to java time only impacts custom date and date_nanos formats.

These formats are commonly used in:

If you don’t use custom date formats, you can skip the rest of this guide. Most custom date formats are compatible. However, several require an update.

To see if your date format is impacted, use the deprecation info API or the Kibana upgrade assistant.

Incompatible date formats

edit

Custom date formats containing the following joda-time literals should be changed before upgrading.

Y (Year of era)

Replace with y.

Example: YYYY-MM-dd should become yyyy-MM-dd.

In java time, Y is used for week-based year. Using Y in place of y could result in off-by-one errors in year calculation.

For pattern YYYY-ww and date 2019-01-01T00:00:00.000Z will give 2019-01 For pattern YYYY-ww and date 2018-12-31T00:00:00.000Z will give 2019-01 (counter-intuitive) because there is >4 days of that week in 2019

y (Year)

Replace with u.

Example: yyyy-MM-dd should become uuuu-MM-dd.

In java time, y is used for year of era. u can contain non-positive values while y cannot. y can also be associated with an era field.

C (Century of era)

Century of era is not supported in java time. There is no replacement. Instead, we recommend you preprocess your input.

x (Week year)

Replace with Y.

In java time, x means zone-offset.

Failure to properly convert x (Week year) to Y could result in data loss.

Z (Zone offset/id)

Replace with multiple X's.

Z has a similar meaning in java time. However, java time expects different numbers of literals to parse different forms.

Consider migrating to X, which gives you more control over how time is parsed. For example, the joda-time format YYYY-MM-dd'T'hh:mm:ssZZ accepts the following dates:

2010-01-01T01:02:03Z
2010-01-01T01:02:03+01
2010-01-01T01:02:03+01:02
2010-01-01T01:02:03+01:02:03

In java time, you cannot parse all these dates using a single format Instead, you must specify 3 separate formats:

2010-01-01T01:02:03Z
2010-01-01T01:02:03+01
both parsed with yyyy-MM-dd'T'hh:mm:ssX

2010-01-01T01:02:03+01:02
yyyy-MM-dd'T'hh:mm:ssXXX

2010-01-01T01:02:03+01:02:03
yyyy-MM-dd'T'hh:mm:ssXXXXX

The formats must then be delimited using ||:

yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXXX||yyyy-MM-dd'T'hh:mm:ssXXXXX

The same applies if you expect your pattern to occur without a colon (:): For example, the YYYY-MM-dd'T'hh:mm:ssZ format accepts the following date forms:

2010-01-01T01:02:03Z
2010-01-01T01:02:03+01
2010-01-01T01:02:03+0102
2010-01-01T01:02:03+010203

To accept all these forms in java time, you must use the || delimiter:

yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXX||yyyy-MM-dd'T'hh:mm:ssXXXX
d (Day)

In java time, d is still interpreted as "day" but is less flexible.

For example, the joda-time date format YYYY-MM-dd accepts 2010-01-01 or 2010-01-1.

In java time, you must use the || delimiter to provide specify each format:

yyyy-MM-dd||yyyy-MM-d

In java time, d also does not accept more than 2 digits. To accept days with more than two digits, you must include a text literal in your java-time date format. For example, to parse 2010-01-00001, you must use the following java-time date format:

yyyy-MM-'000'dd
e (Name of day)

In java time, e is still interpreted as "name of day" but does not parse short- or full-text forms.

For example, the joda-time date format EEE YYYY-MM accepts both Wed 2020-01 and Wednesday 2020-01.

To accept both of these dates in java time, you must specify each format using the || delimiter:

cccc yyyy-MM||ccc yyyy-MM

The joda-time literal E is interpreted as "day of week." The java-time literal c is interpreted as "localized day of week." E does not accept full-text day formats, such as Wednesday.

EEEE and similar text forms

Support for full-text forms depends on the locale data provided with your Java Development Kit (JDK) and other implementation details. We recommend you test formats containing these patterns carefully before upgrading.

z (Time zone text)

In java time, z outputs Z for Zulu when given a UTC timezone.

Test with your data

edit

We strongly recommend you test any date format changes using real data before deploying in production.

Update index mappings

edit

To update joda-time date formats in index mappings, you must create a new index with an updated mapping and reindex your data to it. You can however update your pipelines or templates.

The following my_index_1 index contains a mapping for the datetime field, a date field with a custom joda-time date format.

GET my_index_1/_mapping
Copy as curlTry in Elastic 
{
  "my_index_1" : {
    "mappings" : {
      "properties" : {
         "datetime": {
           "type": "date",
           "format": "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis"
         }
      }
    }
  }
}

To change the date format for the datetime field, create a separate index containing an updated mapping and date format.

For example, the following my_index_2 index changes the datetime field’s date format to 8uuuu/MM/dd HH:mm:ss||uuuu/MM/dd||epoch_millis. The 8 prefix indicates this date format uses java time.

PUT my_index_2
{
  "mappings": {
    "properties": {
      "datetime": {
        "type": "date",
        "format": "8uuuu/MM/dd HH:mm:ss||uuuu/MM/dd||epoch_millis"
      }
    }
  }
}
Copy as curlTry in Elastic 

Next, reindex data from the old index to the new index.

The following reindex API request reindexes data from my_index_1 to my_index_2.

POST _reindex
{
  "source": {
    "index": "my_index_1"
  },
  "dest": {
    "index": "my_index_2"
  }
}
Copy as curlTry in Elastic 

If you use index aliases, update them to point to the new index.

POST /_aliases
{
    "actions" : [
        { "remove" : { "index" : "my_index_1", "alias" : "my_index" } },
        { "add" : { "index" : "my_index_2", "alias" : "my_index" } }
    ]
}
Copy as curlTry in Elastic 

Update ingest pipelines

edit

If your ingest pipelines contain joda-time date formats, you can update them using the put ingest pipeline API.

PUT _ingest/pipeline/my_pipeline
{
  "description": "Pipeline for routing data to specific index",
  "processors": [
    {
      "date": {
        "field": "createdTime",
        "formats": [
         "8uuuu-w"
        ]
      },
      "date_index_name": {
        "field": "@timestamp",
        "date_rounding": "d",
        "index_name_prefix": "x-",
        "index_name_format": "8uuuu-w"
      }
    }
  ]
}
Copy as curlTry in Elastic 

Update index templates

edit

If your index templates contain joda-time date formats, you can update them using the put index template API.

PUT _template/template_1
{
  "index_patterns": [
    "te*",
    "bar*"
  ],
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "_source": {
      "enabled": false
    },
    "properties": {
      "host_name": {
        "type": "keyword"
      },
      "created_at": {
        "type": "date",
        "format": "8EEE MMM dd HH:mm:ss Z yyyy"
      }
    }
  }
}
Copy as curlTry in Elastic 

Update external tools and templates

edit

Ensure you also update any date formats in templates or tools outside of Elasticsearch. This can include tools such as Beats or Logstash.

« Breaking changes in 7.0 Release notes »