Script related changes

edit

Switched Default Language from Groovy to Painless

edit

The default scripting language for Elasticsearch is now Painless. Painless is a custom-built language with syntax similar to Groovy designed to be fast as well as secure. Many Groovy scripts will be identitical to Painless scripts to help make the transition between languages as simple as possible.

Documentation for Painless can be found at Painless Scripting Language

One common difference to note between Groovy and Painless is the use of parameters — all parameters in Painless must be prefixed with params. now. The following example shows the difference:

Groovy:

{
  "script_score": {
    "script": {
      "lang": "groovy",
      "source": "Math.log(_score * 2) + my_modifier",
      "params": {
        "my_modifier": 8
      }
    }
  }
}

Painless (my_modifer is prefixed with params):

{
  "script_score": {
    "script": {
      "lang": "painless",
      "source": "Math.log(_score * 2) + params.my_modifier",
      "params": {
        "my_modifier": 8
      }
    }
  }
}

The script.default_lang setting has been removed. It is no longer possible set the default scripting language. If a different language than painless is used then this should be explicitly specified on the script itself.

For scripts with no explicit language defined, that are part of already stored percolator queries, the default language can be controlled with the script.legacy.default_lang setting.

Removed 1.x script and template syntax

edit

The deprecated 1.x syntax of defining inline scripts / templates and referring to file or index base scripts / templates have been removed.

The script and params string parameters can no longer be used and instead the script object syntax must be used. This applies for the update api, script sort, script_score function, script query, scripted_metric aggregation and script_heuristic aggregation.

So this usage of inline scripts is no longer allowed:

{
  "script_score": {
    "lang": "groovy",
    "script": "Math.log(_score * 2) + my_modifier",
    "params": {
      "my_modifier": 8
    }
  }
}

and instead this syntax must be used:

{
  "script_score": {
    "script": {
      "lang": "groovy",
      "source": "Math.log(_score * 2) + my_modifier",
      "params": {
        "my_modifier": 8
      }
    }
  }
}

The script or script_file parameter can no longer be used to refer to file based scripts and templates and instead file must be used.

This usage of referring to file based scripts is no longer valid:

{
  "script_score": {
    "script": "calculate-score",
    "params": {
      "my_modifier": 8
    }
  }
}

This usage is valid:

{
  "script_score": {
    "script": {
      "lang": "groovy",
      "file": "calculate-score",
      "params": {
        "my_modifier": 8
      }
    }
  }
}

The script_id parameter can no longer be used the refer to indexed based scripts and templates and instead id must be used.

This usage of referring to indexed scripts is no longer valid:

{
  "script_score": {
    "script_id": "indexedCalculateScore",
    "params": {
      "my_modifier": 8
    }
  }
}

This usage is valid:

{
  "script_score": {
    "script": {
      "id": "indexedCalculateScore",
      "lang" : "groovy",
      "params": {
        "my_modifier": 8
      }
    }
  }
}

Template query

edit

The query field in the template query can no longer be used. This 1.x syntax can no longer be used:

{
    "query": {
        "template": {
            "query": {"match_{{template}}": {}},
            "params" : {
                "template" : "all"
            }
        }
    }
}

and instead the following syntax should be used:

{
    "query": {
        "template": {
            "source": {"match_{{template}}": {}},
            "params" : {
                "template" : "all"
            }
        }
    }
}

Search templates

edit

The top level template field in the search template api has been replaced with consistent template / script object syntax. This 1.x syntax can no longer be used:

{
    "template" : {
        "query": { "match" : { "{{my_field}}" : "{{my_value}}" } },
        "size" : "{{my_size}}"
    },
    "params" : {
        "my_field" : "foo",
        "my_value" : "bar",
        "my_size" : 5
    }
}

and instead the following syntax should be used:

{
    "source" : {
        "query": { "match" : { "{{my_field}}" : "{{my_value}}" } },
        "size" : "{{my_size}}"
    },
    "params" : {
        "my_field" : "foo",
        "my_value" : "bar",
        "my_size" : 5
    }
}

Indexed scripts and templates

edit

Indexed scripts and templates have been replaced by stored scripts which stores the scripts and templates in the cluster state instead of a dedicate .scripts index.

For the size of stored scripts there is a soft limit of 65535 bytes. If scripts exceed that size then the script.max_size_in_bytes setting can be added to elasticsearch.yml to change the soft limit to a higher value. If scripts are really large, other options like native scripts should be considered.

Previously indexed scripts in the .scripts index will not be used any more as Elasticsearch will now try to fetch the scripts from the cluster state. Upon upgrading to 5.x the .scripts index will remain to exist, so it can be used by a script to migrate the stored scripts from the .scripts index into the cluster state. The current format of the scripts and templates hasn’t been changed, only the 1.x format has been removed.

Python migration script

edit

The following Python script can be used to import your indexed scripts into the cluster state as stored scripts:

from elasticsearch import Elasticsearch,helpers

es = Elasticsearch([
	{'host': 'localhost'}
])

for doc in helpers.scan(es, index=".scripts", preserve_order=True):
	es.put_script(lang=doc['_type'], id=doc['_id'], body=doc['_source'])

This script makes use of the official Elasticsearch Python client and therefore you need to make sure that your have installed the client in your environment. For more information on this please see elasticsearch-py.

Perl migration script

edit

The following Perl script can be used to import your indexed scripts into the cluster state as stored scripts:

use Search::Elasticsearch;

my $es     = Search::Elasticsearch->new( nodes => 'localhost:9200');
my $scroll = $es->scroll_helper( index => '.scripts', sort => '_doc');

while (my $doc = $scroll->next) {
  $e->put_script(
    lang => $doc->{_type},
    id   => $doc->{_id},
    body => $doc->{_source}
  );
}

This script makes use of the official Elasticsearch Perl client and therefore you need to make sure that your have installed the client in your environment. For more information on this please see Search::Elasticsearch.

Verifying script migration

edit

After you have moved the scripts via the provided script or otherwise then you can verify with the following request if the migration has happened successfully:

GET _cluster/state?filter_path=metadata.stored_scripts

The response should include all your scripts from the .scripts index. After you have verified that all your scripts have been moved, optionally as a last step, you can delete the .scripts index as Elasticsearch no longer uses it.

Indexed scripts Java APIs

edit

All the methods related to interacting with indexed scripts have been removed. The Java API methods for interacting with stored scripts have been added under ClusterAdminClient class. The sugar methods that used to exist on the indexed scripts API methods don’t exist on the methods for stored scripts. The only way to provide scripts is by using BytesReference implementation, if a string needs to be provided the BytesArray class should be used.

Scripting engines now register only a single language

edit

Prior to 5.0.0, script engines could register multiple languages. The Javascript script engine in particular registered both "lang": "js" and "lang": "javascript". Script engines can now only register a single language. All references to "lang": "js" should be changed to "lang": "javascript" for existing users of the lang-javascript plugin.

Scripting engines now register only a single extension

edit

Prior to 5.0.0 scripting engines could register multiple extensions. The only engine doing this was the Javascript engine, which registered "js" and "javascript". It now only registers the "js" file extension for on-disk scripts.

.javascript files are no longer supported (use .js)

edit

The Javascript engine previously registered "js" and "javascript". It now only registers the "js" file extension for on-disk scripts.

Removed scripting query string parameters from update rest api

edit

The script, script_id and scripting_upsert query string parameters have been removed from the update api.

Java transport client

edit

The TemplateQueryBuilder has been moved to the lang-mustache module. Therefor when using the TemplateQueryBuilder from the Java native client the lang-mustache module should be on the classpath. Also the transport client should load the lang-mustache module as plugin:

TransportClient transportClient = TransportClient.builder()
        .settings(Settings.builder().put("node.name", "node"))
        .addPlugin(MustachePlugin.class)
        .build();
transportClient.addTransportAddress(
        new InetSocketTransportAddress(new InetSocketAddress(InetAddresses.forString("127.0.0.1"), 9300))
);

Also the helper methods in QueryBuilders class that create a TemplateQueryBuilder instance have been removed, instead the constructors on TemplateQueryBuilder should be used.

Template query

edit

The template query has been deprecated in favour of the search template api. The template query is scheduled to be removed in the next major version.

GeoPoint scripts

edit

The following helper methods have been removed from GeoPoint scripting:

  • factorDistance
  • factorDistanceWithDefault
  • factorDistance02
  • factorDistance13
  • arcDistanceInKm
  • arcDistanceInKmWithDefault
  • arcDistanceInMiles
  • arcDistanceInMilesWithDefault
  • distanceWithDefault
  • distanceInKm
  • distanceInKmWithDefault
  • distanceInMiles
  • distanceInMilesWithDefault
  • geohashDistanceInKm
  • geohashDistanceInMiles

Instead use arcDistance, arcDistanceWithDefault, planeDistance, planeDistanceWithDefault, geohashDistance, geohashDistanceWithDefault and convert from default units (meters) to desired units using the appropriate constance (e.g., multiply by 0.001 to convert to Km).

Only 15 unique scripts can be compiled per minute by default

edit

If you compile too many unique scripts within a small amount of time, Elasticsearch will reject the new dynamic scripts with a circuit_breaking_exception error. By default, up to 15 inline scripts per minute will be compiled. You can change this setting dynamically by setting script.max_compilations_per_minute.

You should watch out for this if you are hard-coding values into your scripts.

Elasticsearch recommends the usage of parameters for efficient script handling. See details here.