Elasticsearch highlights

This list summarizes the most important enhancements in Elasticsearch 7.11. For the complete list, go to Elasticsearch release highlights.

Typically, you index data into Elasticsearch to promote faster search. However, indexing can be slow and requires more disk space, and you have to reindex your data to add fields to existing documents. With runtime fields, you can add fields to documents already indexed to Elasticsearch without reindexing your data.

You access runtime fields from the search API like any other field, and Elasticsearch sees runtime fields no differently. You can define runtime fields in the index mapping or in the search request. It’s your choice, which is part of the inherent flexibility of runtime fields.

See the runtime fields documentation to read more about their benefits and how to use them.

In 7.10, we managed an 11% speed improvement in the date histogram aggregation. In 7.11, we’ve sped it up an additional 85% in some common cases - most noteably when the date_histogram is a top level aggregation and there aren’t sub-aggregations. Kibana frequently uses date_histogram in this way.

CCR is now aware of data streams. Followers now replicate full data streams, not just their backing indices.

We’ve added a new event.type category for audit records, namely the security_config_change, in the existing audit trail. Events in this category indicate that a security configuration has been set (e.g., a user or role was created or updated) or cleared (e.g., a user or role was deleted). The events are emitted by default, but can be explicitly toggled by the security_config_change handler. The record contains all the change details (e.g., the rules of the particular role mapping that has been created or updated), but all credentials are redacted. The change details are formatted as a JSON object and are part of audit record structure. They are not JSON-escaped and put in a string field.

Sample audit log output:

{"type":"audit", "timestamp":"2020-09-26T12:58:13,369+0300", "node.id":"6BXu_9j6QPK7jvmOwzxqLQ", "event.type":"transport", "event.action":"access_granted", "user.name":"elastic", "user.realm":"reserved", "user.roles":["superuser"], "origin.type":"rest", "authentication.type":"REALM", "origin.address":"[::1]:50481", "request.id":"JLr1ftaoTuODAUZl-8g4Bg", "request.name":"PutUserRequest"}
{"type":"audit", "timestamp":"2020-09-26T12:58:13,370+0300", "node.id":"6BXu_9j6QPK7jvmOwzxqLQ", "event.type":"security_config_change", "event.action":"put", "request.id":"JLr1ftaoTuODAUZl-8g4Bg", "config_change":{"put_user":{"username":"test_user2","roles":["superuser"],"full_name":"Joe Average","email":"joe.average@example.com","metadata":{"intelligence":7},"enabled":true,"password_hash":"<redacted>"}}}
{"type":"audit", "timestamp":"2020-09-26T12:58:52,954+0300", "node.id":"6BXu_9j6QPK7jvmOwzxqLQ", "event.type":"transport", "event.action":"access_granted", "user.name":"elastic", "user.realm":"reserved", "user.roles":["superuser"], "origin.type":"rest", "authentication.type":"REALM", "origin.address":"[::1]:50482", "request.id":"i2XtJLCoRheGuwUdCXjDJw", "request.name":"PutRoleRequest"}
{"type":"audit", "timestamp":"2020-09-26T12:58:52,955+0300", "node.id":"6BXu_9j6QPK7jvmOwzxqLQ", "event.type":"security_config_change", "event.action":"put", "request.id":"i2XtJLCoRheGuwUdCXjDJw", "config_change":{"put_role":{"name":"role_fls","cluster_privileges":["all"],"run_as":[],"indices_privileges":[{"names":["apm*"],"privileges":["read"],"field_security":{"grant":["granted"]},"query":"{\"term\": {\"service.name\": \"bar\"}}","allow_restricted_indices":false},{"names":["apm-all*"],"privileges":["all"],"query":"{\"term\": {\"service.name\": \"bar2\"}}","allow_restricted_indices":false}],"application_privileges":[],"metadata":{},"configurable_cluster_privileges":{}}}}

EQL’s case-insensitive : operator now supports the * and ? wildcards. The * wildcard matches zero or more characters. The ? wildcard matches exactly one character. You can also now use : as a lookup operator to compare a string value to a list of strings.

See the Wildcards and Lookup operators sections of the EQL syntax documentation.

Our benchmarks have demonstrated that when Elasticsearch is using a smaller heap size, it performs better with an alternative set of garbage collection options. Elasticsearch now ergonomically chooses different G1GC options for heap sizes smaller than, but not including, 8GB.

In 7.11, we move data frame analytics from experimental to beta.

Data frame analytics enable you to perform different analyses of your data and annotate it with the results. Possible analysis types are outlier detection, regression, and classification. Data frame analytics evolved a lot while it was an experimental feature starting from 7.3. The aim is to move the feature to GA in the near future with the least possible breaking changes.

[beta] This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features. As an alternative to the pivot type of transform, you can now choose a latest type of transform. It enables you to copy the most recent documents into a new index. You need to identify one or more fields as the unique key for grouping your data, as well as a date field that sorts the data chronologically. For example, you can use this type of transform to keep track of the latest purchase for each customer or the latest event for each host.