Elasticsearch 2.4.0 released

Contention in a lock used to ensure that the same document can’t be updated in parallel proved to be a bottleneck. A change to improve the locking strategy led to a significant increase in indexing throughput: as much as 15-20% improvement for small documents.

In Elasticsearch 2.0, field mappings underwent a major overhaul to remove ambiguity, improve reliability, to fix bugs, and to add tests. We found a problem: we use dots to denote the hierarchy of JSON objects, so simultaneously allowing dots in field names introduced ambiguity and the possibility of field conflicts.  Thus, as part of the 2.0 mappings change, dots had to be prohibited in field names.  This has left some users stuck faced with a tough choice: stay on 1.7 or change your Elasticsearch usage while simultaneously reindexing all your old data in an external process.  Neither of these choices is great.

Elasticsearch 5.0 will include a number of significant internal mapping improvements which makes mappings easier to reason about and to test. These changes allowed us to reintroduce support for dots in field names. In 5.0, indexing a field like foo.bar.baz: 5 will result in the following mappings:

{
  "foo": {
    "type": "object",
    "properties": {
      "bar": {
        "type": "object",
        "properties": {
          "baz": {
            "type": "long"
          }
        }
      }
    }
  }
}
        

Unfortunately, the whole of these mapping changes were too large to backport to the 2.x branch, which would leave 1.x users in a difficult position: how will they be able to upgrade to 5.0 if they can’t even upgrade to 2.x?

The good news is we came up with not one, but two solutions!

The first solution is provided in this release. You can disable the check which prohibits dots in field names by starting Elasticsearch as follows:

export ES_JAVA_OPTS="-Dmapper.allow_dots_in_name=true"
./bin/elasticsearch
        

(ES_JAVA_OPTS can be set in the /etc/default/elasticsearch file if you're using the Debian package, or in the /etc/sysconfig/elasticsearch file if you're using the RPM.)

This will allow you to use your existing 1.x indices in 2.4.0, and also to create new indices which use dotted field names. Elasticsearch 5.0 will be able to read these old mappings and to automatically upgrade them from foo.bar.baz to the object form described above.

WARNING: Elasticsearch 5.0 will only be able to upgrade these mappings as long as there are no conflicts between fields. Compare these two documents:

# Document 1
{
  "foo.one": 5,
  "foo.two.three": 10
}
# Document 2
{
  "foo.one": 5,
  "foo.one.two": 10
}
        

Document 1 has no conflicts - field foo can be upgraded to an object field in both cases. Document 2 has a conflict because foo.one is both a long field and an object field. This would prevent an index containing the second document from being upgraded to 5.x.

Also note that, while you will be able to upgrade indices created in 2.x directly to 5.0, indices originally created in 1.x (regardless of whether they have dots in field names or not) must be reindexed in 2.x before you can upgrade to 5.x.  The good news is, once you get to 2.4, you can use the reindex API to do this for you.

The second solution allows users to upgrade directly from 1.x to 5.0, with reindex-from-remote. Indices created in 1.x need to be reindexed before they can be read by 5.0 anyway, so reindex-from-remote allows you to import old indices directly into a 5.0 cluster from a remote 1.x cluster.  You can try this out already in our latest 5.0 alpha and participate in our Pioneer Program.

Please download Elasticsearch 2.4.0, try it out, and let us know what you think on Twitter (@elastic) or in our forum. You can report any problems on the GitHub issues page.