Breaking changes in 6.0
editBreaking changes in 6.0
editThis section discusses the changes that you need to be aware of when migrating your application to Elasticsearch 6.0.
Indices created before 6.0
editElasticsearch 6.0 can read indices created in version 5.0 or above. An Elasticsearch 6.0 node will not start in the presence of indices created in a version of Elasticsearch before 5.0.
Reindex indices from Elasticsearch 2.x or before
Indices created in Elasticsearch 2.x or before will need to be reindexed with Elasticsearch 5.x in order to be readable by Elasticsearch 6.x.
Multiple mapping types are not supported in indices created in 6.0
The ability to have multiple mapping types per index has been removed in 6.0. New indices will be restricted to a single type. This is the first step in the plan to remove mapping types altogether. Indices created in 5.x will continue to support multiple mapping types.
See Removal of mapping types for more information.
Also see:
edit- Aggregations changes
- Cat API changes
- Clients changes
- Cluster changes
- Document API changes
- Indices changes
- Ingest changes
- Java API changes
- Mapping changes
- Packaging changes
- Percolator changes
- Plugins changes
- Reindex changes
- REST changes
- Stats and info changes
- Search and Query DSL changes
- Settings changes
- Scripting changes
Aggregations changes
editDeprecated pattern
element of include/exclude for terms aggregations has been removed
editThe include
and exclude
options of terms
aggregations used to accept a
sub pattern
object which has been removed. The pattern should now be directly
put as a value of the include
and exclude
fields. For instance, the below
terms
aggregation:
POST /twitter/_search?size=0 { "aggs" : { "top_users" : { "terms" : { "field" : "user", "include": { "pattern": "foo.*" }, "exclude": { "pattern": ".*bar" } } } } }
should be replaced with:
POST /twitter/_search?size=0 { "aggs" : { "top_users" : { "terms" : { "field" : "user", "include": "foo.*", "exclude": ".*bar" } } } }
Numeric to
and from
parameters in date_range
aggregation are interpreted according to format
now
editNumeric to
and from
parameters in date_range
aggregations used to always be interpreted as epoch_millis
,
making other numeric formats like epoch_seconds
unusable for numeric input values.
Now we interpret these parameters according to the format
of the target field.
If the format
in the mappings is not compatible with the numeric input value, a compatible
format
(e.g. epoch_millis
, epoch_second
) must be specified in the date_range
aggregation, otherwise an error is thrown.
global_ordinals_hash
and global_ordinals_low_cardinality
are deprecated in the terms
aggregation
editThe execution hints global_ordinals_hash
and global_ordinals_low_cardinality
are deprecated and should be replaced
by global_ordinals
which now internally choose whether it should remap global ordinals to dense ordinals or directly use the
segment ordinals.
Analysis changes
editSynonym Token Filter
editIn 6.0, Synonym Token Filter tokenizes synonyms with whatever tokenizer and token filters appear before it in the chain.
The tokenizer
and ignore_case
parameters are deprecated
and will be ignored when used in new indices. These parameters
will continue to function as before when used in indices
created in 5.x.
Limiting the length of an analyzed text during highlighting
editHighlighting a text that was indexed without offsets or term vectors,
requires analysis of this text in memory real time during the search request.
For large texts this analysis may take substantial amount of time and memory.
To protect against this, the maximum number of characters that to be analyzed will be
limited to 1000000 in the next major Elastic version. For this version, by default the limit
is not set. A deprecation warning will be issued when an analyzed text exceeds 1000000.
The limit can be set for a particular index with the index setting
index.highlight.max_analyzed_offset
.
Cat API changes
editUnbounded queue size in cat thread pool
editPreviously if a queue size backing a thread pool was unbounded, the cat thread pool API would output an empty string in the queue_size column. This has been changed to now output -1 so that the output is always present and always numeric.
Clients changes
editJava High Level REST Client
editStarting from version 5.6.0 a new Java client has been released: the Java High Level REST Client.
This official high-level client (named like this to differentiate it from the existing low-level client) for
Elasticsearch can be used to execute search, index, delete, update and bulk operations using the same Core
Java classes as the TransportClient
uses.
This Java High Level REST Client is designed to replace the TransportClient
in a near future.
Cluster changes
editCluster name no longer allowed in path.data
editPreviously the cluster name could be used in the path.data
setting with a
warning. This is now no longer allowed. For instance, in the previous version
this was valid:
# Assuming path.data is /tmp/mydata # No longer supported: $ tree /tmp/mydata /tmp/mydata ├── <cluster_name> │ └── nodes │ └── 0 │ └── <etc> # Should be changed to: $ tree /tmp/mydata /tmp/mydata ├── nodes │ └── 0 │ └── <etc>
Document API changes
editversion type force
removed
editDocument modification operations may no longer specify the version_type
of
force
to override any previous version checks.
Adding a version
to an upsert request is no longer supported.
created
field removed in the Index API
editThe created
field has been removed in the Index API as in the index
and
create
bulk operations. operation
field should be used instead.
found
field removed in the Delete API
editThe found
field has been removed in the Delete API as in the delete
bulk
operations. operation
field should be used instead.
Geo changes
editDeprecated ShapeBuilders
helper class
editThe ShapeBuilders
class containing static methods to create shape specific builders
(e.g., ShapeBuilders.newPoint
) has been deprecated. One must now use the builder’s
constructor instead (e.g., `new MultiPointBuilder()).
Deprecated ShapeBuilder.parse
static method
editThe ShapeBuilder.parse
method has been deprecated. Use ShapeParser.parse
instead.
Indices changes
editIndex templates use index_patterns
instead of template
editPreviously templates expressed the indices that they should match using a glob
style pattern in the template
field. They should now use the index_patterns
field instead. As the name implies you can define multiple glob style patterns
in an array but for convenience defining a single pattern as a bare string is
also supported. So both of these examples are valid:
PUT _template/template_1 { "index_patterns": ["te*", "bar*"], "settings": { "number_of_shards": 1 } } PUT _template/template_2 { "index_patterns": "te*", "settings": { "number_of_shards": 1 } }
Shadow Replicas have been removed
editShadow replicas don’t see enough usage, and have been removed. This includes the following settings:
-
index.shared_filesystem
-
index.shadow_replicas
-
node.add_lock_id_to_custom_path
Open/Close index API allows wildcard expressions that match no indices by default
editThe default value of the allow_no_indices
option for the Open/Close index API
has been changed from false
to true
so it is aligned with the behaviour of the
Delete index API. As a result, Open/Close index API don’t return an error by
default when a provided wildcard expression doesn’t match any closed/open index.
Delete a document
editDelete a document from non-existing index has been modified to not create the index. However if an external versioning is used the index will be created and the document will be marked for deletion.
Indices aliases api resolves indices expressions only against indices
editThe index parameter in the update-aliases, put-alias, and delete-alias APIs no longer accepts alias names. Instead, it accepts only index names (or wildcards which will expand to matching indices).
Delete index api resolves indices expressions only against indices
editThe index parameter in the delete index API no longer accepts alias names. Instead, it accepts only index names (or wildcards which will expand to matching indices).
Support for +
has been removed in index expressions
editOmitting the +
has the same effect as specifying it, hence support for +
has been removed in index expressions.
Translog retention
editTranslog files are now kept for up to 12 hours (by default), with a maximum size of 512mb
(default), and
are no longer deleted on flush
. This is to increase the chance of doing an operation based recovery when
bringing up replicas up to speed.
Limit to the difference between max_size and min_size in NGramTokenFilter and NGramTokenizer
editTo safeguard against creating too many index terms, the difference between max_ngram and min_ngram in NGramTokenFilter and NGramTokenizer has been limited to 1. This default limit can be changed with the index setting index.max_ngram_diff. Note that if the limit is exceeded a deprecation warning is logged.
Limit to the difference between max_size and min_size in ShingleTokenFilter
editTo safeguard against creating too many tokens, the difference between max_shingle_size and min_shingle_size in ShingleTokenFilter has been limited to 3. This default limit can be changed with the index setting index.max_shingle_diff. Note that if the limit is exceeded a deprecation warning will be logged.
Ingest changes
editTimestamp meta-data field type has changed
editThe type of the "timestamp" meta-data field has changed from java.lang.String
to java.util.Date
.
The format of the string-formatted ingest.timestamp field has changed
editPreviously, since Elasticsearch 5.4.0, you needed to use ingest.new_date_format
to have the
ingest.timestamp
metadata field be formatted in such a way that ES can coerce it to a field
of type date
without further transformation. This is not necessary anymore and this setting was
removed. You can now simply set a field to {{ingest.timestamp}}
in a pipeline, and have that
field be of type date
without any mapping errors.
Java API changes
editsetSource
methods require XContentType
editPreviously the setSource
methods and other methods that accepted byte/string representations of
an object source did not require the XContentType to be specified. The auto-detection of the content
type is no longer used, so these methods now require the XContentType as an additional argument when
providing the source in bytes or as a string.
DeleteByQueryRequest
requires an explicitly set query
editIn previous versions of Elasticsearch, delete by query requests without an explicit query
were accepted, match_all was used as the default query and all documents were deleted
as a result. From version 6.0.0, a DeleteByQueryRequest
requires an explicit query be set.
InternalStats
and Stats
getCountAsString() method removed
editThe count
value in the stats aggregation represents a doc count that shouldn’t require a formatted
version. This method was deprecated in 5.4 in favour of just using
String.valueOf(getCount())
if needed
ActionRequestBuilder#execute
returns ActionFuture
rather than ListenableActionFuture
editWhen sending a request through the request builders e.g. client.prepareSearch().execute(), it used to
be possible to call addListener
against the returned ListenableActionFuture
. With this change an
ActionFuture
is returned instead, which is consistent with what the Client
methods return, hence
it is not possible to associate the future with listeners. The execute
method that accept a listener
as an argument can be used instead.
Terms.Order
and Histogram.Order
classes replace by BucketOrder
editThe terms
, histogram
, and date_histogram
aggregation code has been refactored to use common
code for ordering buckets. The BucketOrder
class must be used instead of Terms.Order
and
Histogram.Order
. The static
methods in the BucketOrder
class must be called instead of directly
accessing internal order instances, e.g. BucketOrder.count(boolean)
and BucketOrder.aggregation(String, boolean)
.
Use BucketOrder.key(boolean)
to order the terms
aggregation buckets by _term
.
getTookInMillis()
removed in BulkResponse
, SearchResponse
and TermVectorsResponse
editIn BulkResponse
, SearchResponse
and TermVectorsResponse
getTookInMiilis()
method
has been removed in favor of getTook
method. getTookInMiilis()
is easily replaced by
getTook().getMillis()
.
GetField
and SearchHitField
replaced by DocumentField
editAs GetField
and SearchHitField
have the same members, they have been unified into
DocumentField
.
Some Aggregation classes have moved packages
editThe classes for the variants of the range aggregation (geo_distance, date and ip) were moved into the org.elasticsearch.search.aggregations.bucket.range
package.
The org.elasticsearch.search.aggregations.bucket.terms.support
package was removed and the classes were moved to
org.elasticsearch.search.aggregations.bucket.terms
.
The filter aggregation classes were moved to org.elasticsearch.search.aggregations.bucket.filter
Constructor for PercentileRanksAggregationBuilder
has changed
editIt is now required to include the desired ranks as a non-null, non-empty array of doubles to the builder’s constructor,
rather than configuring them via a setter on the builder. The setter method values()
has correspondingly
been removed.
Mapping changes
editCoercion of boolean fields
editPreviously, Elasticsearch recognized the strings true
, false
, on
, off
, yes
, no
, 0
, 1
as booleans. Elasticsearch 6.0
recognizes only the strings true
and false
as booleans and will throw an error otherwise. For backwards compatibility purposes, during the 6.x
series the previous coercion rules will continue to work on pre-6.0 indices. This means that you do not need to change affected existing
mappings immediately. However, it is not possible to create new indices from existing index templates that violate the strict boolean
coercion rules.
The _all
meta field is now disabled by default
editOn new mappings, the _all
meta field that contains a copy of the text from
each field is now disabled by default. The query_string
and
simple_query_string
queries that previously used _all
to search will now
check if _all
is enabled/disabled and switch to executing the query across all
fields if _all
is disabled. _all
can no longer be configured for indices
created with Elasticsearch version 6.0 or later.
The include_in_all
mapping parameter is now disallowed
editSince the _all
field is now disabled by default and cannot be configured for
indices created with Elasticsearch 6.0 or later, the include_in_all
setting is
now disallowed for these indices' mappings.
Unrecognized match_mapping_type
options not silently ignored
editPreviously Elasticsearch would silently ignore any dynamic templates that
included a match_mapping_type
type that was unrecognized. An exception is now
thrown on an unrecognized type.
Validation of locale
on date fields
editThe locale
option of date
fields previously allowed almost any string values,
and unrecognized values would simply be ignored. Unrecognized values are now
rejected on indices created with Elasticsearch 6.0 or newer.
Packaging changes
editConfiguring custom user and group for package is no longer allowed
editPreviously someone could configure the $ES_USER
and $ES_GROUP
variables to
change which user and group Elasticsearch was run as. This is no longer
possible, the DEB and RPM packages now exclusively use the user and group
elasticsearch
. If a custom user or group is needed then a provisioning system
should use the tarball distribution instead of the provided RPM and DEB
packages.
path.conf
is no longer a configurable setting
editPrevious versions of Elasticsearch enabled setting path.conf
as a
setting. This was rather convoluted as it meant that you could start
Elasticsearch with a config file that specified via path.conf
that
Elasticsearch should use another config file. Instead, to configure a custom
config directory, use the ES_PATH_CONF
environment variable.
CONF_DIR
is no longer supported
editPrevious versions of Elasticsearch enabled using the CONF_DIR
environment
variable to specify a custom configuration directory for some configuration
files and some scripts (it was used inconsistently). Starting in Elasticsearch
6.0.0, the usage of this environment variable has been superceded by
ES_PATH_CONF
, and this new environment variable is consistently used for all
configuration files and scripts.
Default path settings are removed
editPrevious versions of Elasticsearch enabled setting default.path.data
and
default.path.logs
to set the default data path and default logs path if they
were not otherwise set in the configuration file. These settings have been
removed and now data paths and log paths can be configured via settings
only. Related, this means that the environment variables DATA_DIR
and
LOG_DIR
no longer have any effect as these were used to set
default.path.data
and default.path.logs
in the packaging scripts.
Additionally, this means that if you were using the package distributions (i.e.,
you have installed Elasticsearch from the RPM or the DEB distributions), you had
not previously explicitly configured path.data
or path.logs
, and you carry
over your elasticsearch.yml
file from 5.x, then you will need to add settings
for path.data
and path.logs
. To use the defaults that you were implicitly
using previously, you should add these lines to your elasticsearch.yml
:
path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch
(If you already had explicit values for either of these settings, you should of course preserve those). If you do not do this, Elasticsearch will refuse to start.
32-bit is no longer maintained
editWe previously attempted to ensure that Elasticsearch could be started on 32-bit JVM (although a bootstrap check prevented using a 32-bit JVM in production). We are no longer maintaining this attempt.
-server
flag no longer filtered in the Windows service
editRelated to the previous change that 32-bit is no longer supported, the default
jvm.options
file that ships with Elasticsearch no longer includes the
-server
flag. Previously, the installation of the Windows service would filter
the -server
flag because it is not compatible with the service. Since the
default jvm.options
file no longer includes -server
, the filtering of this
flag was removed from the installation of the Windows service. This means that
if you have a jvm.options
file that includes this flag you will need to remove
it.
ES_JVM_OPTIONS
is no longer supported
editThe environment variable ES_JVM_OPTIONS
that enabled a custom location for the
jvm.options
file has been removed in favor of using the environment variable
ES_PATH_CONF
. This environment variable is already used in the packaging to
support relocating the configuration files so this change merely aligns the
other configuration files with the location of the jvm.options
file.
ES_INCLUDE
is no longer supported
editThe environment variable ES_INCLUDE
could previously be used to establish the
environment used to start Elasticsearch (and various supporting scripts). This
legacy feature could be useful when there were several environment variables
useful for configuring JVM options; this functionality had previously been
replaced by Setting JVM options. Therefore, ES_INCLUDE
has been removed.
Percolator changes
editDeprecated percolator and mpercolate apis have been removed
editInstead the percolate
query should be used via either the search or msearch apis.
The percolator.map_unmapped_fields_as_string setting has been deprecated
editThe percolator.map_unmapped_fields_as_text
setting should be used instead.
Plugins changes
editMapper attachments plugin
edit- The mapper attachments plugin has been deprecated in elasticsearch 5.0 and is now removed. You can use ingest attachment plugin instead.
S3 Repository plugin
edit- The bucket an s3 repository is configured with will no longer be created automatically. It must exist before the s3 repository is created.
-
Support for specifying s3 credentials through environment variables and
system properties has been removed. Use the
elasticsearch-keystore
tool to securely store the credentials. -
Specifying region has been removed. This includes the settings
cloud.aws.region
,cloud.aws.s3.region
,repositories.s3.region
, and specifying region inside the repository settings. Instead, specify the full endpoint if a custom s3 location is needed, or rely on the default behavior which automatically locates the region of the configured bucket. -
Specifying s3 signer type has been removed, including
cloud.aws.signer
andcloud.aws.s3.signer
. -
Global repositories settings have been removed. This includes
repositories.s3.bucket
,repositories.s3.server_side_encryption
,repositories.s3.buffer_size
,repositories.s3.max_retries
,repositories.s3.use_throttle_retries
,repositories.s3.chunk_size
,repositories.s3.compress
,repositories.s3.storage_class
,repositories.s3.canned_acl
,repositories.s3.base_path
, andrepositories.s3.path_style_access
. Instead, these settings should be set directly in the settings per repository. See S3 Repository settings. -
Shared client settings have been removed. This includes
cloud.aws.access_key
,cloud.aws.secret_key
,cloud.aws.protocol
,cloud.aws.proxy.host
,cloud.aws.proxy.port
,cloud.aws.proxy.username
,cloud.aws.proxy.password
,cloud.aws.signer
,cloud.aws.read_timeout
,cloud.aws.s3.access_key
,cloud.aws.s3.secret_key
,cloud.aws.s3.protocol
,cloud.aws.s3.proxy.host
,cloud.aws.s3.proxy.port
,cloud.aws.s3.proxy.username
,cloud.aws.s3.proxy.password
,cloud.aws.s3.signer
,cloud.aws.s3.read_timeout
,repositories.s3.access_key
,repositories.s3.secret_key
,repositories.s3.endpoint
andrepositories.s3.protocol
. Instead, use the new named client settings unders3.client.CLIENT_NAME.*
. -
The
path_style_access
setting has been removed. The repository will use the appropriate path style access based on the configured endpoint.
Azure Repository plugin
edit- The container an azure repository is configured with will no longer be created automatically. It must exist before the azure repository is created.
-
Global repositories settings you are able to set in elasticsearch config file under
repositories.azure
name space have been removed. This includesrepositories.azure.account
,repositories.azure.container
,repositories.azure.base_path
,repositories.azure.location_mode
,repositories.azure.chunk_size
andrepositories.azure.compress
. You must set those settings per repository instead. Respectivelyaccount
,container
,base_path
,location_mode
,chunk_size
andcompress
. See Azure Repository settings.
GCS Repository plugin
edit-
The
service_account
setting has been removed. A service account json credential file must now be specified in the elasticsearch keystore.
EC2 Discovery plugin
edit-
Specifying ec2 signer type has been removed, including
cloud.aws.signer
andcloud.aws.ec2.signer
. -
The region setting has been removed. This includes the settings
cloud.aws.region
andcloud.aws.ec2.region
. Instead, specify the full endpoint. -
All
cloud.aws.*
andcloud.aws.ec2.*
settings have been removed. Usediscovery.ec2.*
settings instead.
Ignoring hidden foldersedit
Previous versions of Elasticsearch would skip hidden files and directories when scanning the plugins folder. This leniency has been removed.
ICU Analysis plugin
editThe icu4j library has been upgraded to 59.1, Indices created in the previous major version will need to be reindexed in order to return correct (and correctly ordered) results, and to take advantage of new characters.
Plugins should not construct Environment
instances from Settings
editPreviously, plugins could construct an Environment
instance from Settings
to
discover the path to plugin-specific config files. This will no longer work in
all situations as the Settings
object does not carry the necessary information
for the config path to be set correctly. Instead, plugins that need to know the
config path should have a single constructor that accepts a pair of Settings
and Path
instances, and construct an Environment
using the corresponding
constructor on Environment
.
Reindex changes
editsize
parameter
editThe size
parameter can no longer be explicitly set to -1
. If all documents are required then the size
parameter should not be set.
REST changes
editUnquoted JSON
editIn previous versions of Elasticsearch, JSON documents were allowed to contain unquoted field names.
This feature was removed in the 5.x series, but a backwards-compatibility layer was added via the
system property elasticsearch.json.allow_unquoted_field_names
. This backwards-compatibility layer
has been removed in Elasticsearch 6.0.0.
Duplicate Keys in JSON, CBOR, Yaml and Smile
editIn previous versions of Elasticsearch, documents were allowed to contain duplicate keys. Elasticsearch 6.0.0 enforces that all keys are unique. This applies to all content types: JSON, CBOR, Yaml and Smile.
Content-Type Auto-detection
editIn previous versions of Elasticsearch, having a proper Content-Type for the data in a request was not enforced. Elasticsearch 6.0.0 enforces that all requests with a body must have a supported Content-Type and this type will be used when parsing the data.
When using the source
query string parameter, the source_content_type
parameter must also be specified with
the media type of the source.
Boolean API parameters
editAll REST APIs parameters (both request parameters and JSON body) support providing boolean "false" as the
value false
and boolean "true" as the value true
. All other values will raise an error.
Analyze API changes
editThe deprecated request parameters and plain text in request body has been removed. Define parameters in request body.
Support custom normalizer in Analyze API
editAnalyze API can analyze normalizer and custom normalizer.
In previous versions of Elasticsearch, Analyze API is requiring a tokenizer
or analyzer
parameter.
In Elasticsearch 6.0.0, Analyze API can analyze a text as a keyword field with custom normalizer
or if char_filter
/filter
is set and tokenizer
/analyzer
is not set.
timestamp
and ttl
in index requests
edittimestamp
and ttl
are not accepted anymore as parameters of index/update
requests.
Refresh requests with one or more shard failures return HTTP 500 response instead of 200
editRefresh requests that are broadcast to multiple shards that can have one or more shards fail during the request now return a 500 response instead of a 200 response in the event there is at least one failure.
Delete by Query API requires an explicit query
editIn previous versions of Elasticsearch, delete by query requests without an explicit query were accepted, match_all was used as the default query and all documents were deleted as a result. From version 6.0.0, delete by query requests require an explicit query.
DELETE document calls now implicitly create the type
editRunning DELETE index/type/id
now implicitly creates type
with a default
mapping if it did not exist yet.
Indices information APIs
editPreviously it was possible to execute GET /_aliases,_mappings
or GET
/myindex/_settings,_alias
by separating multiple types of requests with commas
in order to retrieve multiple types of information about one or more indices.
This comma-separation for retrieving multiple pieces of information has been
removed. GET /_all
can be used to retrieve all aliases, settings, and
mappings for all indices.
In order to retrieve only the mapping for an index use:
GET /myindex/_mapping
(or _alias
for a list of aliases, or _settings
for the settings).
Requests to existing endpoints with incorrect HTTP verb now return 405 responses
editIssuing a request to an endpoint that exists, but with an incorrect HTTP verb
(such as a POST
request to /myindex/_settings
) now returns an HTTP 405
response instead of a 404. An Allow
header is added to the 405 responses
containing the allowed verbs. For example:
$ curl -v -XPOST 'localhost:9200/my_index/_settings' * Trying 127.0.0.1... * TCP_NODELAY set * Connected to localhost (127.0.0.1) port 9200 (#0) > POST /my_index/_settings HTTP/1.1 > Host: localhost:9200 > User-Agent: curl/7.51.0 > Accept: */* > < HTTP/1.1 405 Method Not Allowed < Allow: PUT,GET < content-type: application/json; charset=UTF-8 < content-length: 134 < { "error" : "Incorrect HTTP method for uri [/my_index/_settings] and method [POST], allowed: [PUT, GET]", "status" : 405 } * Curl_http_done: called premature == 0 * Connection #0 to host localhost left intact
Disallow using _cache
and _cache_key
editThe _cache
and _cache_key
options in queries have been deprecated since version 2.0.0 and
have been ignored since then, issuing a deprecation warning. These options have now been completely
removed, so using them now will throw an error.
IndexClosedException to return 400 status code
editAn IndexClosedException
is returned whenever an api that doesn’t support
closed indices (e.g. search) is called passing closed indices as parameters
and ignore_unavailable
is set to false
. The response status code returned
in such case changed from 403
to 400
Scripting changes
editGroovy, JavaScript, and Python languages removed
editThe Groovy, JavaScript, and Python scripting languages were deprecated in elasticsearch 5.0 and have now been removed. Use painless instead.
Native scripts removed
editNative scripts have been removed. Instead,
implement a ScriptEngine
.
Date fields now return dates
editdoc.some_date_field.value
now returns ReadableDateTime
s instead of
milliseconds since epoch as a long
. The same is true for
doc.some_date_field[some_number]
. Use doc.some_date_field.value.millis
to
fetch the milliseconds since epoch if you need it.
Removed access to index internal via the _index
variable
editThe _index
variable has been removed. If you used it for advanced scoring, consider writing a Similarity
plugin.
Script Settings
editAll of the existing scripting security settings have been removed. Instead
they are replaced with script.allowed_types
and script.allowed_contexts
.
lang
can no longer be specified when using a stored script as part of a request
editThe lang
variable can no longer be specified as part of a request that uses a stored
script otherwise an error will occur. Note that a request using a stored script is
different from a request that puts a stored script. The language of the script has
already been stored as part of the cluster state and an id
is sufficient to access
all of the information necessary to execute a stored script.
'lang` can no longer be used when putting, getting, or deleting a stored script
editStored scripts can no longer have the lang
parameter specified as part of the url
when performing PUT, GET, and DELETE actions on the _scripts/
path. All stored
scripts must have a unique id
as the namespace is only id
now and no longer lang
and id
.
Stored search template apis removed
editThe PUT, GET and DELETE _search/template
apis have been removed. Store search templates with the stored scripts apis instead.
For example, previously one might have stored a search template with the following:
PUT /_search/template/custom_template { "query": { "match": { "f1": "{{f1}}" } } }
And instead one would now use the following:
PUT /_scripts/custom_template { "script": { "lang": "mustache", "source": { "query": { "match": { "f1": "{{f1}}" } } } } }
Search and Query DSL changes
editChanges to queries
edit-
The
collect_payloads
parameter of thespan_near
query has been removed. Payloads will be loaded when needed. -
Queries on boolean fields now strictly parse boolean-like values. This means
only the strings
"true"
and"false"
will be parsed into their boolean counterparts. Other strings will cause an error to be thrown. -
The
in
query (a synonym for theterms
query) has been removed -
The
geo_bbox
query (a synonym for thegeo_bounding_box
query) has been removed -
The
mlt
query (a synonym for themore_like_this
query) has been removed. -
The deprecated
like_text
,ids
anddocs
parameters (all synonyms forlike
) of themore_like_this
query have been removed. Also the deprecatedmin_word_len
(a synonym formin_word_length
) andmax_word_len
(a synonym formax_word_length
) have been removed. -
The
fuzzy_match
andmatch_fuzzy
query (synonyma for thematch
query) have been removed -
The
terms
query now always returns scores equal to1
and is not subject toindices.query.bool.max_clause_count
anymore. -
The deprecated
indices
query has been removed. -
Support for empty query objects (
{ }
) has been removed from the query DSL. An error is thrown whenever an empty query object is provided. -
The deprecated
minimum_number_should_match
parameter in thebool
query has been removed, useminimum_should_match
instead. -
The
query_string
query now correctly parses the maximum number of states allowed when determinizing a regex asmax_determinized_states
instead of the typomax_determined_states
. -
The
query_string
query no longer acceptsenable_position_increment
, useenable_position_increments
instead. -
For
geo_distance
queries, sorting, and aggregations thesloppy_arc
option has been removed from thedistance_type
parameter. -
The
geo_distance_range
query, which was deprecated in 5.0, has been removed. -
The
optimize_bbox
parameter has been removed fromgeo_distance
queries. -
The
ignore_malformed
andcoerce
parameters have been removed fromgeo_bounding_box
,geo_polygon
, andgeo_distance
queries. -
The
disable_coord
parameter of thebool
andcommon_terms
queries has been removed. If provided, it will be ignored and issue a deprecation warning. -
The
template
query has been removed. This query was deprecated since 5.0 -
The
percolate
query’sdocument_type
has been deprecated. From 6.0 and later it is no longer required to specify thedocument_type
parameter. -
The
split_on_whitespace
parameter for thequery_string
query has been removed. If provided, it will be ignored and issue a deprecation warning. Thequery_string
query now splits on operator only. -
The
use_dis_max
parameter for thequery_string
query has been removed. If provided, it will be ignored and issue a deprecation warning. Thetie_breaker
parameter must be used instead. -
The
auto_generate_phrase_queries
parameter for thequery_string
query has been removed, use an explicit quoted query instead. If provided, it will be ignored and issue a deprecation warning. -
The
all_fields
parameter for thequery_string
andsimple_query_string
has been removed. Setdefault_field
to *` instead. If provided,default_field
will be automatically set to*
-
The
index
parameter in the terms filter, used to look up terms in a dedicated index is now mandatory. Previously, the index defaulted to the index the query was executed on. Now this index must be explicitly set in the request. -
The deprecated
type
andslop
parameter for thematch
query have been removed. Instead of setting thetype
, thematch_phrase
ormatch_phrase_prefix
should be used. Theslop
removed from thematch
query but is supported formatch_phrase
andmatch_phrase_prefix
. -
The deprecated
phrase_slop
parameter (a synonym for theslop
parameter) of thematch_phrase
query has been removed. -
The deprecated
query
parameter (a synonym for thefilter
parameter) of theconstant_score
query has been removed. -
The deprecated
phrase_slop
parameter (a synonym for theslop
parameter) of themulti_match
query has been removed. -
The deprecated
prefix
parameter (a synonym for thevalue
parameter) of theprefix
query has been removed. -
The deprecated
le
(a synonym forlte
) andge
(a synonym forgte
) parameter of therange
query have been removed. -
The deprecated
types
and_type
synonyms for thetype
parameter of theids
query have been removed -
The deprecated multi term rewrite parameters
constant_score_auto
,constant_score_filter
(synonyms forconstant_score
) have been removed.
Search shards API
editThe search shards API no longer accepts the type
url parameter, which didn’t
have any effect in previous versions.
Changes to the Profile API
editThe "time"
field showing human readable timing output has been replaced by the "time_in_nanos"
field which displays the elapsed time in nanoseconds. The "time"
field can be turned on by adding
"?human=true"
to the request url. It will display a rounded, human readable time value.
Scoring changes
editQuery normalization
editQuery normalization has been removed. This means that the TF-IDF similarity no longer tries to make scores comparable across queries and that boosts are now integrated into scores as simple multiplicative factors.
Other similarities are not affected as they did not normalize scores and already integrated boosts into scores as multiplicative factors.
See LUCENE-7347
for more
information.
Coordination factors
editCoordination factors have been removed from the scoring formula. This means that boolean queries no longer score based on the number of matching clauses. Instead, they always return the sum of the scores of the matching clauses.
As a consequence, use of the TF-IDF similarity is now discouraged as this was an important component of the quality of the scores that this similarity produces. BM25 is recommended instead.
See LUCENE-7347
for more
information.
Fielddata on _uid
editFielddata on _uid
is deprecated. It is possible to switch to _id
instead
but the only reason why it has not been deprecated too is because it is used
for the random_score
function. If you really need access to the id of
documents for sorting, aggregations or search scripts, the recommendation is
to duplicate the id as a field in the document.
Highlighters
editThe unified
highlighter is the new default choice for highlighter.
The offset strategy for each field is picked internally by this highlighter depending on the
type of the field (index_options
).
It is still possible to force the highlighter to fvh
or plain
types.
The postings
highlighter has been removed from Lucene and Elasticsearch.
The unified
highlighter outputs the same highlighting when index_options
is set
to offsets
.
fielddata_fields
editThe deprecated fielddata_fields
have now been removed. docvalue_fields
should be used instead.
docvalue_fields
editdocvalue_fields
now have a default upper limit of 100 fields that can be requested.
This limit can be overridden by using the index.max_docvalue_fields_search
index setting.
script_fields
editscript_fields
now have a default upper limit of 32 script fields that can be requested.
This limit can be overridden by using the index.max_script_fields
index setting.
Inner hits
editThe source inside a hit of inner hits keeps its full path with respect to the entire source. In prior versions the source field names were relative to the inner hit.
Scroll
editThe from
parameter can no longer be used in the search request body when initiating a scroll.
The parameter was already ignored in these situations, now in addition an error is thrown.
Limit on from/size in top hits and inner hits
editThe maximum number of results (from
+ size
) that is allowed to be retrieved
via inner hits and top hits has been limited to 100. The limit can be controlled
via the index.max_inner_result_window
index setting.
Scroll queries that use the request_cache are deprecated
editSetting request_cache:true
on a query that creates a scroll ('scroll=1m`)
is deprecated and the request will not use the cache internally.
In future versions we will return a 400 - Bad request
instead of just ignoring
the hint.
Scroll queries are not meant to be cached.
Limiting the number of terms that can be used in a Terms Query request
editExecuting a Terms Query with a lot of terms may degrade the cluster performance,
as each additional term demands extra processing and memory.
To safeguard against this, the maximum number of terms that can be used in a
Terms Query request has been limited to 65536. This default maximum can be changed
for a particular index with the index setting index.max_terms_count
.
Invalid _search
request body
editFor 6.x and starting in 6.3 a deprecation warning will be printed to warn against search requests that contain extra tokens after the main object. These extra tokens were ignored by the query parser before 6.3 but the next major version will not accept invalid body anymore.
Settings changes
editRemove support for elasticsearch.json and elasticsearch.yaml configuration file
editThe configuration file found in the Elasticsearch config directory could previously have
a .yml
, .yaml
or .json
extension. Only elasticsearch.yml
is now supported.
Duplicate keys in configuration file
editIn previous versions of Elasticsearch, the configuration file was allowed to contain duplicate keys. For example:
node: name: my-node node attr: rack: my-rack
In Elasticsearch 6.0.0, this is no longer permitted. Instead, this must be specified in a single key as:
node: name: my-node attr: rack: my-rack
Coercion of boolean settings
editPreviously, Elasticsearch recognized the strings true
, false
, on
, off
, yes
, no
, 0
, 1
as booleans. Elasticsearch 6.0
recognizes only true
and false
as boolean and will throw an error otherwise. For backwards compatibility purposes, during the 6.x series
index settings on pre-6.0 indices will continue to work. Note that this does not apply to node-level settings that are stored
in elasticsearch.yml
.
Snapshot settings
editThe internal setting cluster.routing.allocation.snapshot.relocation_enabled
that allowed shards with running snapshots to be reallocated to
different nodes has been removed. Enabling this setting could cause allocation issues if a shard got allocated off a node and then
reallocated back to this node while a snapshot was running.
Store throttling settings
editStore throttling has been removed. As a consequence, the
indices.store.throttle.type
and indices.store.throttle.max_bytes_per_sec
cluster settings and the index.store.throttle.type
and
index.store.throttle.max_bytes_per_sec
index settings are not
recognized anymore.
Store settings
editThe default
index.store.type
has been removed. If you were using it, we
advise that you simply remove it from your index settings and Elasticsearch
will use the best store
implementation for your operating system.
Network settings
editThe blocking TCP client, blocking TCP server, and blocking HTTP server have been removed.
As a consequence, the network.tcp.blocking_server
, network.tcp.blocking_client
,
network.tcp.blocking
,transport.tcp.blocking_client
, transport.tcp.blocking_server
,
and http.tcp.blocking_server
settings are not recognized anymore.
The previously unused settings transport.netty.max_cumulation_buffer_capacity
,
transport.netty.max_composite_buffer_components
and
http.netty.max_cumulation_buffer_capacity
have been removed.
Similarity settings
editThe base
similarity is now ignored as coords and query normalization have
been removed. If provided, this setting will be ignored and issue a
deprecation warning.
Script Settings
editAll of the existing scripting security settings have been removed. Instead
they are replaced with script.allowed_types
and script.allowed_contexts
.
Discovery Settings
editThe discovery.type
settings no longer supports the values gce
, aws
and ec2
.
Integration with these platforms should be done by setting the discovery.zen.hosts_provider
setting to
one of those values.
Stats and info changes
editRemoval of throttle_time
in the store
stats
editGiven that store throttling has been removed, the store
stats do not report
throttle_time
anymore.
FS stats no longer reports if the disk spins
editElasticsearch has defaulted to assuming that it is running on SSDs since the 2.x series of Elasticsearch. As such, Elasticsearch no longer needs to collect information from the operating system as to whether or not the underlying disks of each data path spin or not. While this functionality was no longer needed starting in the 2.x series of Elasticsearch, it was maintained in the filesystem section of the nodes stats APIs. This information has now been removed.