What’s new in 7.12
editWhat’s new in 7.12
editHere are the highlights of what’s new and improved in Elasticsearch 7.12!
For detailed information about this release, see the Release notes and Migration guide.
Other versions: 7.11 | 7.10 | 7.9 | 7.8 | 7.7 | 7.6 | 7.5 | 7.4 | 7.3 | 7.2 | 7.1 | 7.0
Frozen tier and shared snapshot caches
editWith 7.12, we’re adding a frozen phase to ILM along with a corresponding frozen data tier as experimental features. You can use the new frozen phase and frozen tier to archive rarely searched data as searchable snapshots for long-term storage and maximum cost savings.
The key advantage of the frozen tier is an experimental new option for mounting searchable snapshots: shared snapshot caches. Rather than loading a full copy of a snapshotted index into your cluster, this option uses a small local cache containing only recently searched parts of the index’s data. If a search requires data that’s not in the cache, Elasticsearch fetches the missing data from the snapshot repository. These caches have a fixed size and can significantly reduce your storage costs. ILM uses shared snapshot caches by default in the frozen phase and corresponding frozen tier.
To add a node to the frozen tier in an on-premise cluster, assign it the
data_frozen
node role and set
xpack.searchable.snapshot.shared_cache.size
in elasticsearch.yml
. See the Data tiers
and
Searchable snapshots
documentation.
Analyze snapshot repositories
editWhile there are many third-party storage systems, not all of them work well as snapshot repositories. Some systems perform poorly or behave incorrectly as a repository, especially when multiple Elasticsearch nodes access them concurrently.
To detect and avoid these problems, you can now use the repository analysis API to analyze a repository’s storage system before using it in production. The API performs a collection of read and write operations on the repository. These operations are designed to detect incorrect behavior and measure the performance characteristics of a system.
EQL: Case-insensitive in
lookups and functions
editThe in
and not in
lookup
operators now support case-insensitive variants: in~
and not in~
. You can
also make an EQL function
case-insensitive by adding ~
after the function name, such as
endsWith~(process_name, ".exe")
.
EQL: like
and regex
keywords
editIn 7.12, we added the like
and regex
keywords to EQL. You can use the like
keyword to match strings to wildcard
patterns, such as foo*
or ba?
. You can use the regex
keyword to match
strings to regular expressions. You can use both like
and regex
to match a
string against a list of patterns.
By default, both keywords are case-sensitive. For case-insensitive matching, use
like~
or regex~
. For more information, see the
Pattern comparison
keywords section of the EQL syntax documentation.
Retention policy for transforms
editContinuous transforms add new data to the destination index as new entities are
encountered. 7.12 introduces a data retention policy for transforms that
enables you to delete old data. This is especially useful for the latest
transform that has been added in 7.11. For example, if you collect the latest
host information you might want to assume hosts have been decommissioned if they
have not appeared in the logs for more than 30 days. At each checkpoint,
transformed documents whose age is greater than a configured value are deleted.
Hyperparameter importance
editData frame analytics training is influenced by settings called hyperparameters. One of the training outputs is now hyperparameter importance. This value shows which hyperparameter had the most impact on the training, and what the optimum values were for each one. You can use this information to speed up the training process as the hyperparameter optimization process can be stopped when you see that gains in accuracy flatten.
Search-time runtime fields support for transforms
editYou can now use runtime_mappings
in transforms to support search-time
runtime fields in searches of the source index. The syntax is the same as for a
standard search. After specifying them, the runtime fields can be used in the
rest of the transform configuration.