Mapping changes

The string field datatype has been replaced by the text field for full text analyzed content, and the keyword field for not-analyzed exact string values. For backwards compatibility purposes, during the 5.x series:

String mappings now have the following default mappings:

{
  "type": "text",
  "fields": {
    "keyword": {
      "type": "keyword",
      "ignore_above": 256
    }
  }
}

This allows to perform full-text search on the original field name and to sort and run aggregations on the sub keyword field.

Numeric fields are now indexed with a completely different data-structure, called BKD tree, that is expected to require less disk space and be faster for range queries than the previous way that numerics were indexed.

Term queries will return constant scores now, while they used to return higher scores for rare terms due to the contribution of the document frequency, which this new BKD structure does not record. If scoring is needed, then it is advised to map the numeric fields as `keyword`s too.

Note that this keyword mapping do not need to replace the numeric mapping. For instance if you need both sorting and scoring on your numeric field, you could map it both as a number and a keyword using fields:

PUT my_index
{
  "mappings": {
    "my_type": {
      "properties": {
        "my_number": {
          "type": "long",
          "fields": {
            "keyword": {
              "type":  "keyword"
            }
          }
        }
      }
    }
  }
}

Also the precision_step parameter is now irrelevant and will be rejected on indices that are created on or after 5.0.

Like Numeric fields the Geo point field now uses the new BKD tree structure. Since this structure is fundamentally designed for multi-dimension spatial data, the following field parameters are no longer needed or supported: geohash, geohash_prefix, geohash_precision, lat_lon. Geohashes are still supported from an API perspective, and can still be accessed using the .geohash field extension, but they are no longer used to index geo point data.

The _timestamp and _ttl fields were deprecated and are now removed. As a replacement for _timestamp, you should populate a regular date field with the current timestamp on application side. For _ttl, you should either use time-based indices when applicable, or cron a delete-by-query with a range query on a timestamp field

On all field datatypes (except for the deprecated string field), the index property now only accepts true/false instead of not_analyzed/no. The string field still accepts analyzed/not_analyzed/no.

Previously, setting a field to index:no would also disable doc-values. Now, doc-values are enabled by default on all types but text and binary, regardless of the value of the index property.

When dynamically mapping a field containing a floating point number, the field now defaults to using float instead of double. The reasoning is that floats should be more than enough for most cases but would decrease storage requirements significantly.

norms now take a boolean instead of an object. This boolean is the replacement for norms.enabled. There is no replacement for norms.loading since eager loading of norms is not useful anymore now that norms are disk-based.

Setting fielddata.format: doc_values in the mappings used to implicitly enable doc-values on a field. This no longer works: the only way to enable or disable doc-values is by using the doc_values property of mappings.

Regex filters are not supported anymore and will be dropped on upgrade.

The source transform feature has been removed. Instead, use an ingest pipeline.

To prevent mapping explosions, the following limits are applied to indices created in 5.x:

See Settings to prevent mappings explosion for more.

The join between parent and child documents no longer relies on indexed fields and therefore from 5.0.0 onwards the _parent field is no longer indexed. In order to find documents that refer to a specific parent id, the new parent_id query can be used. The GET response and hits inside the search response still include the parent id under the _parent key.

The _source mapping no longer supports the format option. It will still be accepted for indices created before the upgrade to 5.0 for backwards compatibility, but it will have no effect. Indices created on or after 5.0 will reject this option.

Core types no longer support the object notation, which was used to provide per document boosts as follows:

{
  "value": "field_value",
  "boost": 42
}

Per-field boosts on the _all are now compressed into a single byte instead of the 4 bytes used previously. While this will make the index much more space-efficient, it also means that index time boosts will be less accurately encoded.

You can no longer create indexes with _ttl or _timestamp enabled. Indexes with them enabled created before 5.0 will continue to work.

You should replace _timestamp in new indexes by adding a field to your source either in the application producing the data or with an ingest pipeline like this one:

PUT _ingest/pipeline/timestamp
{
  "description" : "Adds a timestamp field at the current time",
  "processors" : [ {
    "set" : {
      "field": "timestamp",
      "value": "{{_ingest.timestamp}}"
    }
  } ]
}

PUT newindex/type/1?pipeline=timestamp
{
  "example": "data"
}

GET newindex/type/1

Which produces

{
  "_source": {
    "example": "data",
    "timestamp": "2016-06-21T18:48:55.560+0000"
  },
  ...
}

If you have an old index created with 2.x that has _timestamp enabled then you can migrate it to a new index with the a timestamp field in the source with reindex:

POST _reindex
{
  "source": {
    "index": "oldindex"
  },
  "dest": {
    "index": "newindex"
  },
  "script": {
    "lang": "painless",
    "source": "ctx._source.timestamp = ctx._timestamp; ctx._timestamp = null"
  }
}

You can replace _ttl with time based index names (preferred) or by adding a cron job which runs a delete-by-query on a timestamp field in the source document. If you had documents like this:

POST index/type/_bulk
{"index":{"_id":1}}
{"example": "data", "timestamp": "2016-06-21T18:48:55.560+0000" }
{"index":{"_id":2}}
{"example": "data", "timestamp": "2016-04-21T18:48:55.560+0000" }

Then you could delete all of the documents from before June 1st with:

POST index/type/_delete_by_query
{
  "query": {
    "range" : {
      "timestamp" : {
        "lt" : "2016-05-01"
      }
    }
  }
}

Blank field names in mappings is not allowed after 5.0.