Elasticsearch 7.3.0 is here! Learn about data frames, voting-only master nodes, vector similarity functions, and more.

<p>Today we are pleased to announce the release of Elasticsearch 7.3.0, based on Lucene 8.1.0. This is the latest stable release, and is already available for deployment via our <a href="/products/elasticsearch/service">Elasticsearch Service</a>.
</p><p>
	Latest stable release in 7.x:
</p><ul>
	<li><a href="/downloads/elasticsearch">Download Elasticsearch 7.3.0</a></li>
	<li><a href="/guide/en/elasticsearch/reference/7.3/release-notes-7.3.0.html">Elasticsearch 7.3.0 release notes</a></li>
	<li><a href="/guide/en/elasticsearch/reference/7.3/breaking-changes.html">Elasticsearch 7.3 breaking changes</a></li>
</ul><p>You can read about all the changes in the release notes linked above, but let's take a look at a few changes worth highlighting.
</p><h2>Data frames — transform and pivot your streaming data</h2><p>Data frame transforms are a core new feature in Elasticsearch that enable you to transform an existing index to a secondary, summarized index. <a href="/guide/en/elasticsearch/reference/7.3/data-frame-apis.html">Data frame transforms</a> enable you to pivot your data and create entity-centric indices that can summarize the behavior of an entity. This organizes the data into an analysis-friendly format.
</p><p>Data frame transforms were originally available in 7.2. With 7.3 they can now run either as a single batch transform or continuously incorporating new data as it is ingested.
</p><p>Imagine you are streaming audit logs events from the many different hosts you have in your data center, and you want to analyze user behavior to find anything that looks suspicious. With data frames you can group the logs events by user, by host, and by day of the week. So for each user you have a count of interactions per server grouped by type of request. Organizing the data by entities and summarizing many events makes it easier to run different numerical analysis models and spot unusual behaviors.
</p><p>Data frames allow for new possibilities of machine learning analysis (such as outlier detection, which could be a perfect match for the above security example), but they can be useful for other types of visualisations or custom types of analysis. We are curious to see what users are up to using this feature.
</p><p>Data frame transforms is released as beta under a free Basic license.
</p><h2>Giving search a boost</h2><p>Elasticsearch is for search, and here are some exciting new search features we're releasing with 7.3.
</p><h3>Discover the least-frequent values</h3><p>We’ve added an all new <a href="/guide/en/elasticsearch/reference/7.3/search-aggregations-bucket-rare-terms-aggregation.html"><code>rare_terms</code> aggregation</a>, which uses a resource efficient algorithm with predictable results. It is an aggregation designed to identify the long-tail of keywords, such as terms that have low doc counts. From a technical perspective, rare terms aggregation works by maintaining a map of terms with a counter associated with each value. The counter is incremented each time the term is identified. If the counter surpasses a predefined threshold, the term is removed from the map and inserted into a <a href="https://en.wikipedia.org/wiki/Cuckoo_filter">cuckoo filter</a>. If a future term is found in the cuckoo filter, we assume it was previously removed from the map and is "common." This aggregation is designed to be more memory efficient than the alternative, which is setting a terms aggregation to <code>size: MAX_LONG</code>, or ordering a terms aggregation by count ascending, <a href="/guide/en/elasticsearch/reference/7.3/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-order">in which errors are unbounded</a>.
</p><p>There are multiple use cases for rare terms aggregation; for example, <a href="/products/siem">SIEM</a> users are frequently interested in rare occurrences, which are sometimes suspected to be a manifestation of a security event.
</p><h3>Built-in vector similarity functions for document script scoring</h3><p>There are many popular algorithms that represent records by vectors (such as <a href="https://en.wikipedia.org/wiki/Word2vec">word2vec</a> and <a href="https://en.wikipedia.org/wiki/Convolutional_neural_network">convolutional neural networks</a>), which allow relating to vector similarity as a measurement for record similarity. In this release we added <a href="/guide/en/elasticsearch/reference/7.3/query-dsl-script-score-query.html#vector-functions">two predefined functions</a> to use for calculating vector similarity between a given query vector and document vectors:
</p><ul>
	<li>Cosine similarity</li>
	<li>Dot product similarity</li>
</ul><p>These are the two types of distance functions most frequently used in vector comparisons. We released these as Painless script functions, so users have full flexibility to use them in relevance ranking in combination with other fields. Users can use these Painless functions for scoring through the <a href="/guide/en/elasticsearch/reference/7.3/query-dsl-script-score-query.html"><code>script_score</code></a> query. We plan to release additional vector similarity functions such as Euclidean distance and Manhattan distance in future releases, because each of these vector similarity functions proved superior for specific scenarios.
</p><p>Some of our users are using Elasticsearch as the data source for their machine learning algorithms, and have been requesting such functionality. We are thrilled to see what novel use the community will now find for Elasticsearch with the introduction of this capability.
</p><p>This experimental functionality is released under a free Basic license.
</p><h3>Improved intervals query</h3><p>In 7.0 we introduced the <a href="/guide/en/elasticsearch/reference/7.3/query-dsl-intervals-query.html">intervals query</a>. This query is used when users want to find records in which words or phrases are within a certain distance from each other. It provides advanced search options in an easy-to-define syntax and yields accurate results.
</p><p>The intervals query is great for use cases such as legal and patent searches. Version 7.3 includes two important additions to intervals queries:
</p><ul>
	<li>The <a href="/guide/en/elasticsearch/reference/7.3/query-dsl-intervals-query.html#intervals-wildcard"><code>wildcard</code> rule</a> within an intervals query allows for relating to a set of terms defined with a wildcard (* and ?), as well as choosing the analyzer that will be used.</li>
	<li>The <a href="/guide/en/elasticsearch/reference/7.3/query-dsl-intervals-query.html#intervals-prefix"><code>prefix</code> rule</a> allows for defining intervals between terms that start with specific letters, with the option to <a href="/guide/en/elasticsearch/reference/7.3/index-prefixes.html">index specifically for prefix </a>or to limit the query expansion to 128 alternative terms.</li>
</ul><p>Now that intervals queries are nearing feature parity with span queries, users can switch to intervals queries for the likes of legal and patent search.
</p><h3>Efficiently handle records with large number of dynamic fields</h3><p>The new <a href="/guide/en/elasticsearch/reference/7.3/flattened.html">flattened object field</a>  will allow an entire flat JSON object to be indexed into a single field. This comes in handy for situations in which documents include a very large number of fields, such as HTTP headers or image metadata. Sub-fields of flattened objects behave almost exactly like keyword fields, so only basic queries and aggregations are allowed (no support for numeric range queries, full-text search, or highlighting). Prior to 7.3, records with a large number of fields had to be indexed into separate fields, which can greatly increase the number of mappings, making mappings more difficult to manage and increasing the size of the cluster state.
</p><p>Support for flattened object types is released under a free Basic license.
</p><h3>Update index synonyms list without index downtime</h3><p>With this feature, synonym filters used by <a href="/guide/en/elasticsearch/reference/7.3/search-analyzer.html">search analyzers</a> can be updated quickly and flexibly. For example, e-commerce businesses can add new synonyms flexibly to verify that new products are associated with users’ queries, and do not return empty result sets. With the new <a href="/guide/en/elasticsearch/reference/7.3/indices-reload-analyzers.html">Reload Search Analyzers</a> API, updating synonym filters only requires placing the file on the node and calling the Reload Search Analyzers API, which loads the search analyzers without restarting the index shards. This enables a user to update the synonyms per index without search and indexing downtime (index close and reopen).
</p><p>This experimental feature is released under a free Basic license.
</p><h2>Hold on, we have more cool stuff</h2><p>
	7.3 isn't just about improved search functionality, it also includes plenty of other new niceties.
</p><h3>A new type of master node just registered to vote</h3><p>Elasticsearch 7.0 introduced a <a href="/blog/a-new-era-for-cluster-coordination-in-elasticsearch">new cluster coordination layer</a>, with a number of improvements, including faster master elections, removal of the minimum_master_nodes setting, and use of <a href="/elasticon/conf/2018/sf/reliable-by-design-applying-formal-methods-to-distributed-systems">formal methods for design validation</a>. There was another important benefit to the new cluster coordination layer in Elasticsearch 7.0: It could be used as a foundation for important improvements in Elasticsearch, such as voting-only master-eligible nodes. A voting-only master-eligible node is a node that can participate in master elections but will not act as a master in the cluster (it only votes in elections). By only voting in elections, a smaller machine can be used, and fewer hardware resources are required for the cluster. Head over to the Elasticsearch docs for <a href="/guide/en/elasticsearch/reference/7.3/modules-node.html#voting-only-node">instructions on setting up a voting-only master-eligible node</a> in Elasticsearch 7.3.
</p><p>Voting-only master-eligible nodes are available under a free Basic license.
</p><h3>Aliases can travel cross cluster</h3><p>Cross-cluster replication (CCR) was released as a <a href="/blog/elasticsearch-6-7-0-released">GA feature in Elasticsearch 6.7</a>. CCR has a variety of use cases, including <a href="/blog/cross-datacenter-replication-with-elasticsearch-cross-cluster-replication">cross-datacenter and cross-region replication</a>, replicating data to get closer to the application server and user, and maintaining a centralized reporting cluster replicated from a large number of smaller clusters. Elasticsearch 7.3 includes additional CCR functionality to ensure aliases that are manipulated on leader indices are replicated to the follower indices. Note: this process ignores <a href="/guide/en/elasticsearch/reference/7.3/indices-aliases.html#aliases-write-index">write aliases</a> because follower indices do not receive direct writes, so write aliases have no use.
</p><h3>SQL query support for frozen indices for API clients and JDBC/ODBC drivers</h3><p>This feature allows for <a href="/guide/en/elasticsearch/reference/7.3/sql-index-frozen.html">querying of frozen</a> indices through dedicated <a href="/webinars/introduction-to-elasticsearch-sql">SQL grammar extension</a>. Frozen indices are a super efficient way to keep older data that is not frequently searched, and doing that at low cost. Because users frequently will refrain from including frozen indices in their “normal” queries, when using SQL you need to explicitly request to include frozen indices. This can be done by using the FROZEN reserved word, such as <code>SELECT * FROM FROZEN myIndex LIMIT 10;</code>.
</p><h3>GUI support for snapshot restore and deletion</h3><p>Elasticsearch management UI (Kibana &gt; Management &gt; Elasticsearch) keeps evolving. In this version we enhanced the previously released <a href="/blog/elasticsearch-7-2-0-released">Snapshot Repositories section</a>, now called Snapshot and Restore, with the ability to restore from an existing snapshot. The snapshot restore wizard walks you through defining a restore task. The progress of currently running restores can be tracked in the Restore Status view. Snapshot deletion is also now available from the UI. For more information about these enhancements, see <a href="/guide/en/kibana/7.3/snapshot-repositories.html">Snapshot and Restore</a>.
</p><p>This UI functionality is released under a free Basic license.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt5830fc372cacbead/5d3a1da3ef4fb0543504c330/ES-7-3-release-snapshot--restore-blog.png" data-sys-asset-uid="blt5830fc372cacbead" alt="Snapshot and Restore UI" style="display: block; margin: auto;">
</p><h3>Discover your most unusual data using outlier detection</h3><p>The goal of <a href="/guide/en/elastic-stack-overview/7.3/dfa-outlier-detection.html">outlier detection</a> is to find the most unusual data points in an index. We analyze the numerical fields of each data point (document in an index) and annotate them with how unusual they are.
</p><p>We use unsupervised outlier detection, which means there is no need to provide a training data set to teach outlier detection to recognize outliers. In practice, this is achieved by using an ensemble of distance-based and density-based techniques to identify those data points that are the most different from the bulk of the data in the index. We assign to each analyzed data point an outlier score, which captures how different the entity is from other entities in the index.
</p><p>In addition to new outlier detection functionality, we are introducing the <a href="/guide/en/elasticsearch/reference/7.3/evaluate-dfanalytics.html">Evaluate API</a>, which enables users to compute a range of performance metrics such as confusion matrices, precision, recall, the <a href="https://en.wikipedia.org/wiki/Receiver_operating_characteristic">receiver-operating characteristics (ROC) curve</a>, and the area under the ROC curve. If you are running outlier detection on a source index that has already been labeled to indicate which points are truly outliers and which are normal, you can use the Evaluate API to assess the performance of the outlier detection analytics on your dataset.
</p><p>This experimental feature is released under a Platinum license.
</p><h2>Conclusion</h2><p>Please <a href="/downloads/elasticsearch">download Elasticsearch 7.3.0</a>, try it out, and let us know what you think on Twitter (<a href="https://twitter.com/elastic">@elastic</a>) or in our <a href="https://discuss.elastic.co/">Discuss forums</a>. You can report any problems on the <a href="https://github.com/elastic/elasticsearch/issues">GitHub issues page</a>.
</p>

blog-banner-release-elasticsearch.png

blog-thumb-release-elasticsearch.png

Elasticsearch 7.3.0 released

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

headshot-yaron-parasol-300x300.png

Yaron Parasol

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Communication Preferences - ONLY for /communication-preferences

<p>제출하면 귀하가 <a href="/kr/legal/elastic-cloud-account-terms" target="_self">Elastic 서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 연락처로 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">선택한 커뮤니케이션</a>을 수신하는 데 동의한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.</p>

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Serverless GDPR Text

<p>제출하면 귀하가 Elastic <a href="/kr/legal/serverless-preview-terms" target="_self">서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 세부 정보를 사용하여 Elastic이 관련 제품 및 서비스에 대해 귀하에게 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">연락할 수 있음</a>을 이해한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.</p>

<p>By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.</p>

Cloud Trial GDPR Text

<p>제출하면 <a href="/kr/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud 서비스 약관</a>에 동의하시게 됩니다. 개인 데이터는 <a href="/kr/legal/privacy-statement">Elastic의 개인정보 취급방침</a>에 따라 처리됩니다.</p>

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Newsletter GDPR Text

<p>제출하면 귀하가 Elastic <a href="/kr/legal/terms-of-use" target="_self">서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 세부 정보를 사용하여 Elastic이 관련 제품 및 서비스에 대해 귀하에게 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">연락할 수 있음</a>을 이해한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.</p>

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

이 페이지의 콘텐츠는 선택하신 언어로 제공되지 않습니다. Elastic은 다양한 언어로 콘텐츠를 제공하기 위해 최선을 다하고 있습니다.조금만 더 기다려주세요!

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

내 표준 시간대로 웨비나 시작 시간 확인하기

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

참여가 어려우신가요? 등록하시면 녹화 자료를 보내드리겠습니다. 이메일로 관련 콘텐츠도 받아보실 수 있습니다.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Read less

You'll also receive an email with related content

이메일을 통해 관련 내용을 보내드리겠습니다.

Thank you for registering. We will send you a confirmation email soon.

등록해 주셔서 감사합니다. 곧 확인 이메일을 보내드리겠습니다.

Author

Articles by Yaron Parasol

Elasticsearch 7.3.0 released

Sign up for Elastic Cloud free trial