<html><head></head><body><p>Elasticsearch is sometimes used alongside other databases. In those scenarios, it&apos;s often hard to implement solutions surrounding <a href="https://en.wikipedia.org/wiki/Two-phase_commit_protocol">two-phase commits</a> due to the lack of transaction support across all systems in play. Depending on your use case, it may or may not be necessary to verify the data exists in both data stores, where one serves as the so-called &quot;source of truth&quot; for the other.
</p><p>As Support Engineers at Elastic, we frequently see requests that ask things like the best way to structure your data for verification as well as simply how to do verification efficiently. This blog post will walk through a few examples that we have seen and helped to create, ranging from simple to advanced, that verify Elasticsearch contains the necessary data from a database like PostgreSQL.
</p><h2>Modeling Data for Verification</h2><p>The way that your data is stored, both in and out of Elasticsearch, can make a big difference in the difficulty of verification. The first thing that you need to decide is how much needs to be verified:
</p><ul>
	<li>Is the mere existence of a document enough?</li>
	<li>Or does an internal process require you to verify entire documents?</li>
</ul><p>That decision has ramifications on the amount of effort required to perform verification.
</p><h2>Evolution of Verifying Existence</h2><p>Fortunately this is not a deep, philosophical question. Instead, it&apos;s simply the question: &quot;do all of the documents exist in Elasticsearch?&quot;
</p><p>Elasticsearch offers a lot of ways to verify existence, as long as you fully understand what is happening. Remember that <a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/near-real-time.html">Elasticsearch <em>searches</em> in near real time</a>, but it can get documents directly in real time. This means that, right after indexing a document, it may not be visible to searches, but it will be available to any direct get request.
</p><h3>One By One</h3><p>For small scale deployments, the simplest approach is to perform <code>HEAD</code> requests against individual documents, then confirm that the HTTP response code is not 404 (page &#x2014; or document in this case &#x2014; not found).
</p><pre class="prettyprint">HEAD /my_index/my_type/my_id1
</pre><p>An example response would be just headers for success:
</p><pre class="prettyprint">HTTP/1.1 200 OK
Content-Type: text/plain; charset=UTF-8
Content-Length: 0
</pre><p>And for non-existence, it looks practically the same, minus the response code:
</p><pre class="prettyprint">HTTP/1.1 404 Not Found
Content-Type: text/plain; charset=UTF-8
Content-Length: 0
</pre><p>This requires that you perform <code>N</code> requests against an index expected to have <code>N</code> documents. As you might expect, that does not scale very well.
</p><h3>Batch Processing</h3><p>The next thing that you might choose to do is to perform these in batch requests using Multi GET API: <code>_mget</code>.
</p><pre class="prettyprint">GET /my_index/my_type/_mget
{
  &quot;ids&quot;: [ &quot;my_id1&quot;, &quot;my_id2&quot; ]
}
</pre><p>Another equivalent approach to this is to simply search for the IDs and return the expected number.
</p><pre class="prettyprint">GET /my_index/_refresh
GET /my_index/my_type/_search
{
  &quot;size&quot;: 2,
  &quot;query&quot;: {
    &quot;ids&quot;: {
      &quot;values&quot;: [ &quot;my_id1&quot;, &quot;my_id2&quot; ]
    }
  }
}
</pre><p>First, the <code>_refresh</code> endpoint is invoked to ensure that everything indexed is searchable. This removes the &quot;near real time search&quot; aspect of searching. Under normal searching conditions, you should not be doing that, but it makes total sense here! Presumably your verification processing happens all at once, so calling <code>_refresh</code> <em>once</em> at the start of a job is sufficient for the rest of it as long as you&apos;re not checking for new data that comes in after the process starts.
</p><p>That offers a superior request handling, but it does not avoid the background work, which remains unchanged, and it means that each document is going to be returned as well, which means added overhead.
</p><h3>Search, Then Batch</h3><p>From batch processing, people generally branch off to try to beat the problem by subdividing it. First, they will often perform a search to determine what is missing and only then begin digging for the missing document:
</p><pre class="prettyprint">GET /my_index/_refresh
GET /my_index/my_type/_search
{
  &quot;size&quot;: 0,
  &quot;query&quot;: {
    &quot;ids&quot;: {
      &quot;values&quot;: [ &quot;my_id1&quot;, &quot;my_id2&quot; ]
    }
  }
}
</pre><p>If you check the response&apos;s <code>hits.total</code> value, then it should match the number that you expect. In this simple example, it should be <code>2</code>. If it&apos;s not 2, then you need to switch gears and either:
</p><ol>
	<li>Search for all of the IDs, and actually return them by specifying an appropriate size, then find the missing needle(s) in the haystack.</li>
	<li>Fall back to the <code>_mget</code> example above and do the same thing: determine what ID is missing.</li>
</ol><h3>There Must Be A Better Way</h3><p>The long-winded or two-step processing can be a bit messy at scale, and it really implies that you&apos;re asking an expensive question, but also that you&apos;re perhaps asking the wrong question.
</p><p>What if you flipped the query or question on its head and, instead of effectively querying for &quot;find everything that exists&quot;, you asked &quot;what is missing?&quot; It is hard to query for &quot;what is not here&quot; and, in fact, it&apos;s downright impossible because the data does not exist. However, we can use aggregations to answer the question <em>if</em> you can structure data conveniently.
</p><p>Most verification use cases come from an SQL world, where integer-based keys are extremely common for primary keys and foreign keys. Even if you do not use those numeric IDs as your Elasticsearch <code>_id</code>, you should be indexing those as integer-based values with <a href="/blog/support-in-the-wild-my-biggest-elasticsearch-problem-at-scale">doc values enabled</a> (on by default in ES 2.x and later). For example:
</p><pre class="prettyprint">POST /my_index/my_type/my_id1
{
  &quot;id&quot;: 1,
  &#x2026;
}
</pre><p>The name of the field is irrelevant, but do note that it&apos;s not <code>_id</code>, which is a reserved metadata field in Elasticsearch. Once you start indexing that value, then finding <em>missing</em> data becomes very simple with histograms:
</p><pre class="prettyprint">GET /my_index/my_type/_search
{
  &quot;size&quot;: 0,
  &quot;aggs&quot;: {
    &quot;find_missing_ids&quot;: {
      &quot;histogram&quot;: {
        &quot;field&quot;: &quot;id&quot;,
        &quot;interval&quot;: 1,
        &quot;min_doc_count&quot;: 0
      },
      &quot;aggs&quot;: {
        &quot;remove_existing_bucket_selector&quot;: {
          &quot;bucket_selector&quot;: {
            &quot;buckets_path&quot;: {
              &quot;count&quot;: &quot;_count&quot;
            },
            &quot;script&quot;: {
              &quot;inline&quot;: &quot;count == 0&quot;,
              &quot;lang&quot;: &quot;expression&quot;
            }
          }
        }
      }
    }
  }
}
</pre><p>This does two things:
</p><ol>
	<li>It performs a histogram across the numeric <code>id</code> field with a step of <code>1</code> between each value, meaning that for all <code>id</code>s, it will find every integer-based value between them (e.g., 1 - 5 would be a histogram of 1, 2, 3, 4, 5).</li>
	<li>It removes any histogram bucket that actually contains data by using a bucket selector, which only exists in Elasticsearch 2.0 and later.</li>
</ol><p>By removing what exists, we&apos;re left with what does not exist which might look something like this:
</p><pre class="prettyprint">{
  &quot;took&quot;: 4,
  &quot;timed_out&quot;: false,
  &quot;_shards&quot;: {
    &quot;total&quot;: 5,
    &quot;successful&quot;: 5,
    &quot;failed&quot;: 0
  },
  &quot;hits&quot;: {
    &quot;total&quot;: 5,
    &quot;max_score&quot;: 0,
    &quot;hits&quot;: []
  },
  &quot;aggregations&quot;: {
    &quot;find_missing_ids&quot;: {
      &quot;buckets&quot;: [
        {
          &quot;key&quot;: 4,
          &quot;doc_count&quot;: 0
        }
      ]
    }
  }
}
</pre><p>From a tiny index with 5 documents, known thanks to <code>hits.total</code>, this shows that you are missing the document whose <code>id</code> is 4.
</p><p>That&apos;s all there is to it: if you can create a histogram from the unique key of the document, then you can create one that only shows missing data. You cannot do this with strings, which means things like <code>&quot;my_id&quot;</code> and UUIDs (e.g., a GUID) cannot use this approach with aggregations! It comes back to structuring your data for this type of check.
</p><h4>Mind The Gap</h4><p>For those not very familiar with sequential identifiers, it is not uncommon for systems (e.g., SQL databases) to prefetch batches of identifiers. Within those batches, it&apos;s possible for some values to get skipped for many reasons, such as a failed transaction or even for simpler reasons like the record being deleted explicitly afterward.
</p><p>In such cases, you need to be aware that those will appear as missing documents according to the histogram. To avoid them, as the client, you can tell Elasticsearch to explicitly ignore those values by adding a secondary bucket selector that eliminates those <code>id</code>s explicitly, or ignore them on the client side.
</p><h2>Verifying Entire Documents</h2><p>The verification of entire documents is a different beast from simple existence checking. In this scenario, entire documents need to be verified to ensure that the data contained is the expected data. From the Elasticsearch standpoint, it&apos;s actually easier, but the work on your side is more difficult.
</p><h3>Scrolling Through Data</h3><p>The simplest way to perform this check is to scroll through it. Fortunately, this does not mean piping all of your data into an HTML table and using your mouse wheel to double check everything. No, it means using the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html"><code>_scroll</code> API to iterate across all of your data</a> (<code>_doc</code>, used below,&#xA0;is described in that link). Like every search API, it also works on the same near real time principles noted above.
</p><pre class="prettyprint">GET /my_index/my_type/_search?scroll=1m
{
  &quot;sort&quot;: [
    &quot;_doc&quot;
  ]
}
</pre><p>The scroll time is the amount of time that you, as client software, need to process that batch before requesting the next batch. Make sure that it&apos;s long enough to be able to loop through the subsequent response and read through the linked documentation above!
</p><p>By scrolling through the data, you can go document-by-document and verify all of the data is correct per your requirements.
</p><h3>Taking Control of Versioning</h3><p>Elasticsearch supports <a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/optimistic-concurrency-control.html">optimistic concurrency control</a>, which is another way to say versioning.
</p><p>You can take complete control of versioning by supplying your own version number. If you do this, then you can possibly side-step full document verification by verifying version numbers. To enable version numbers in search responses, you must specify the version flag:
</p><pre class="prettyprint">GET /my_index/my_type/_search
{
  &quot;version&quot;: true
}
</pre><h2>Side-Stepping Verification</h2><p>It is possible that you can side-step verification altogether, if you can trust that the data was only provided by a trusted user and any ingestion failures were properly handled (e.g., if Elasticsearch was down when document 15123 should have been indexed, then something still needs to add it). In that scenario, then verification can become redundant.
</p><p><a href="https://www.elastic.co/guide/en/x-pack/current/index.html">X-Pack Security</a> in 5.0 and <a href="https://www.elastic.co/guide/en/shield/current/index.html">Shield</a> in earlier releases can provide the security aspect to give trusted users access and block untrusted users, but it is up to you to properly handle ingestion failure because it completely depends on what is doing the ingestion.
</p><h2>Fin</h2><p>You have done it! You made it through my long blog post.
</p><p>I hope that you enjoyed reading about data verification and how structuring your data can lead to dramatically simplified approaches to doing it. Thinking about this kind of approach, you can start to see how other problems can be solved by thinking about the query or data differently, alongside rich features like aggregations or X-Pack Security.
</p><p>We are constantly looking for creative ways to solve interesting problems, and this is no exception. As always, we encourage our users to discuss these issues on our <a href="https://discuss.elastic.co/c/logstash">forum</a>, and open GitHub issues for any issues that you may come across and we are also available on <a href="https://twitter.com/elastic">Twitter</a> (<a href="https://twitter.com/pickypg">@pickypg</a> for me) and <a href="https://webchat.freenode.net/#elasticsearch">IRC</a> to help you out!
</p></body></html>

field-half-green-half-brown.jpg

shoes-blue-yellow-grass.jpg

Elasticsearch: Verifying Data Integrity with External Data Stores

<html><head></head><body><p>As a Support Engineer at Elastic, I come across a lot of different issues from our customers, ranging from development questions surrounding Kibana to helping a user to understand why their Elasticsearch cluster had issues. In this blog post, I want to describe the number one problem that I run into with users running Elasticsearch as they begin to scale their workloads. Without fail, when a user approaches me with issues running Elasticsearch, whether it is a customer or not, I point them to this issue.
</p><h3>Java Heap Pressure</h3><p>Elasticsearch has so many wildly different use cases that I could not write a reasonably short blog post describing what can and cannot consume memory. However, there is one thing that constantly stands out above all of the other concerns that you might have while running an Elasticsearch cluster at scale.
</p><p>For the users that I help, <a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/fielddata.html"><strong>fielddata</strong></a> is the problem that is the most likely to cause their cluster&apos;s instability. Fielddata is the bane of my existence and it&apos;s the most frequent cause of the highest severity issues that I handle with our customers.
</p><h3>Understanding Fielddata</h3><p>The inverted index is the magic that makes Elasticsearch queries so fast. This data structure holds a sorted list of all the unique terms that appear in a field, and each term points to the list of documents that contain that term:
</p><pre class="prettyprint">Term:    Docs: 1  2  3  4  5
----------------------------
brown          X     X  X 
fox                     X  X
quick             X     X
----------------------------
</pre><p>Search asks the question: What documents contain term <code>brown</code> in the
	<code>foo</code> field? The inverted index is the perfect structure to answer this question: look up the term of interest in the sorted list and you immediately know which documents match your query.
</p><p>Sorting or aggregations, however, need to be able to answer this question: What terms does Document 1 contain in the <code>foo</code> field? To answer this, we need a data structure that is the opposite of the inverted index:
</p><pre class="prettyprint">Docs:   Terms:
----------------------------
1       [ brown ]
2       [ quick ]
3       [ brown ]
4       [ brown, fox, quick ]
5       [ fox ]
----------------------------
</pre><p>This is the purpose of fielddata. Fielddata can be generated at query time by reading the inverted index, inverting the term &lt;-&gt; doc data structure, and storing the results in memory. The two major downsides of this approach should be obvious:
</p><ol>
	<li>Loading fielddata can be slow, especially with big segments.</li>
	<li>It consumes a lot of valuable heap space.</li>
</ol><p>Because loading fielddata is costly, we try to do it as seldom as possible. Once loaded, we keep it in memory for as long as possible.
</p><p>By default, fielddata is loaded on demand, which means that you will not see it until you are using it. Also, by being loaded per segment, it means that new segments that get created will slowly add to your overall memory usage until the field&apos;s fielddata is evicted from memory. Eviction happens in only a few ways:
</p><ol>
	<li>Deleting the index or indices that contains it.</li>
	<li>Closing the index or indices that contains it.</li>
	<li>Segment fielddata is removed when segments are removed (e.g., background merging).
	<ul>
		<li>This usually just means that the problem is moving rather than going away.</li>
	</ul></li>
	<li>Restarting the node containing the fielddata.</li>
	<li>Clearing the relevant fielddata cache.</li>
	<li>Automatically evicting the fielddata to make room for other fielddata.
	<ul>
		<li>This will not happen with default settings.</li>
	</ul></li>
</ol><p>
	While the first two ways will cause the memory to be evicted, they&apos;re not useful in terms of solving the problem because they make the index unusable. Segment merging is happening in the background and it is not a way to clear fielddata. The fourth and fifth ways are unlikely to be a long term solution because they do not prevent fielddata from being reloaded.
</p><p>The sixth option, evicting fielddata when the cache is full, leads to different issues: one request triggers fielddata loading for one field and the next request triggers loading for another, causing the first field to be evicted. This causes memory thrashing and slow garbage collections, and your users suffer from very slow queries while they wait for their fielddata to be loaded.
</p><p>Simply put, once fielddata becomes a problem, then it stays a problem.
</p><h3>Why Fielddata is Bad</h3><p>At small scales, you can generally get away with fielddata usage without even realizing that you are using it. In highly controlled environments, you may even enjoy that specific fields are being loaded into memory for theoretically faster access.
</p><p>However, almost without fail, you are bound to run into a problem with it eventually. Whether it&apos;s because someone ran a test request on the production system without thinking that it would be a problem (it&apos;s just one query, right?), your queries changed to match new data, or you just finally reached a scale where it no longer works: you will eventually run into memory pressure that does not go away.
</p><p>As I noted earlier, fielddata does not go away on its own. In Elasticsearch 1.3 and later, we allow up to 60% of your Java heap&apos;s memory to be consumed by fielddata per node. We control this via the <a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html#circuit-breaker">Fielddata Circuit Breaker</a>, which checks incoming requests for potential fielddata usage and then <em>blocks</em> them if they require more memory than is currently available. Any circuit breaker&apos;s purpose is to prevent any bad requests, which means that it never gets the chance to cause a problem (e.g., allocate even more fielddata), but it&apos;s important to note that it will not clear any existing fielddata.
</p><p>For example, if a node has 10 GBs of Java heap, then 60% of that is going to be 6 GBs. If a new request requires 1 GB of fielddata to be loaded for that node that is already using 4 GBs of the heap for fielddata, then it will allow it because 4 GB, plus 1 GB, is less than 6 GB. However, if the next request needed 2 GB for yet another field&apos;s fielddata, then the entire request would be rejected because the fielddata is exhausted (<code>5 GB + 2 GB = 7 GB</code>, which is clearly greater than 6 GB).
</p><p>Note: for versions prior to Elasticsearch 1.3, we allowed an unlimited amount of your Java heap to be consumed by fielddata.
</p><h3>Finding Your Fielddata</h3><p>Fortunately, it&apos;s not all bad news. Not only do we have a solution to the problem, but we also provide a way to find and understand <em>your</em> problem with it.
</p><pre class="prettyprint">$ curl -XGET &apos;localhost:9200/_cat/fielddata?v&amp;fields=*&apos;
</pre><p>This will provide a list of each node with its fielddata usage. For instance, at startup, my local node is using absolutely no fielddata:
</p><pre class="prettyprint">id                     host            ip          node  total 
iExRFXn1Qw23iRzhwor-Wg Chriss-MBP.home 192.168.1.2 WallE    0b
</pre><p>
	To see it change, it&apos;s as simple as sorting, scripting, or aggregating any field. So let&apos;s do all three!
</p><pre class="prettyprint">$ curl -XGET localhost:9200/test-index/_search -d &apos;{
  &quot;query&quot;: {
    &quot;filtered&quot;: {
      &quot;filter&quot;: {
        &quot;script&quot;: {
          &quot;script&quot;: &quot;doc[\&quot;percentage\&quot;].value &gt; 0.5&quot;
        }
      }
    }
  },
  &quot;aggs&quot;: {
    &quot;max_number&quot;: {
      &quot;max&quot;: {
        &quot;field&quot;: &quot;number&quot;
      }
    }
  },
  &quot;sort&quot;: [
    {
      &quot;@timestamp&quot;: {
        &quot;order&quot;: &quot;desc&quot;
      }
    }
  ]
}&apos;
</pre><p><br><img src="https://api.contentstack.io/v2/assets/575e4c8d58208ba076e28083/download?uid=blt99a4dc58fe05e36e?uid=blt99a4dc58fe05e36e" data-sys-asset-uid="blt2669af8a14c49161" style="max&#xAD;-width:100%; margin-left: auto; margin-right: auto; display: block;">
</p><p>Although order is irrelevant for this, the first field that will be impacted will be the <code>percentage</code> field that is accessed inside of the scripted filter. The second field used will be the
	<code>number</code> field from the aggregation. Finally, the last field is the
	<code>@timestamp</code> field used to sort the filtered results. Taking another look at the
	<code>_cat/fielddata</code> command above confirms this:
</p><pre class="prettyprint">id                     host            ip          node   total number @timestamp percentage
iExRFXn1Qw23iRzhwor-Wg Chriss-MBP.home 192.168.1.2 WallE 49.9kb 24.8kb     24.8kb       192b
</pre><h3>Use Doc Values</h3><p>The solution to this fielddata problem is to avoid it altogether. Fortunately, you can avoid the use of fielddata by <em>manually</em> mapping all of your fields to use <a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/doc-values.html"><strong>doc values</strong></a>. Without repeating too much from the guide, doc values offload this burden by writing the fielddata to disk at index time, thereby allowing Elasticsearch to load the values outside of your Java heap as they are needed.
</p><p><img src="https://api.contentstack.io/v2/assets/575e4c8d58208ba076e28092/download?uid=bltb1debf53257bb9ca?uid=bltb1debf53257bb9ca" data-sys-asset-uid="blt1d4631b848bcd4d2" style="max&#xAD;-width:100%; margin-left: auto; margin-right: auto; display: block;">
</p><p>By taking the burden out of your heap, you get fast access to the on-disk fielddata through the file system cache, which gives in-memory performance without the cost of garbage collections coming into play. This also frees up a lot of headroom for the Elasticsearch heap so that more operations (e.g., bulk indexing and concurrent searches) can use the heap without placing the node under memory pressure, which leads to garbage collection that will slow it down.
</p><p>You might be asking: How do I enable these doc values? It&apos;s as simple as adding this to every field&apos;s mapping, <em>except</em> analyzed string fields:
</p><pre class="prettyprint">&quot;doc_values&quot; : true
</pre><p>Unfortunately, you <em>must</em> do this before you index any data. This means that, for any existing index that is not already using doc values, you cannot flip the switch to enable them. You would have to reindex.
</p><h3>Updating an Active Cluster</h3><p>Naturally, up to this point, you may be wondering how to remove fielddata from your cluster. The answer depends on your data.
</p><h4>Time Based Indices</h4><p>If you are using time based indices (e.g., <code>logstash-2015.07.18</code>), such as with logging use cases, then you should update your template(s) to use doc values so that future indices (e.g., tomorrow&apos;s) get created with doc values. From there, the problem should take care of itself as indices that use fielddata will age themselves out.
</p><p>If you maintain indices for significant periods of time (e.g., many months), then it may be worth looking into reindexing older indices to take advantage of doc values.
</p><h4>Other Indices</h4><p>For non-time based indices (e.g., <code>my-index</code>), the problem gets more complicated because it depends entirely on how you use the indices. However, to avoid using fielddata, then you will have to reindex the data. In Elasticsearch: The Definitive Guide, <a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/index-aliases.html">we discuss using aliases for this purpose</a> as a mechanism for transitioning between two indices without impacting clients.
</p><h3>Potential Downsides to Doc Values</h3><p>Unfortunately, nothing is perfect. From my perspective, the only important downside to doc values is that they cannot be used with analyzed strings. But by default, all strings are mapped as analyzed strings though! That is the <em>only</em> exception to the use of doc values.
</p><p>Looking back at the Understanding Fielddata section of this post, it is critical to think about why you might use fielddata with an analyzed string. For regular, unstructured search, you will not use any fielddata. With that in mind, the only time that you should catch yourself using fielddata for analyzed strings is with the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-significantterms-aggregation.html">significant terms aggregation</a>. All other uses of fielddata should be avoided by using a <code>not_analyzed</code> version of the string because sorting, aggregating, and scripting against an analyzed string is not going to give you the results you expect.
</p><p>We are currently discussing ways to simplify this issue with the release of Elasticsearch 2.0, such as <a href="https://github.com/elastic/elasticsearch/issues/11901">issue #11901</a> and <a href="https://github.com/elastic/elasticsearch/issues/12394">issue #12394</a>, and it is worth taking a look at these issues to see some common pitfalls so that you can avoid them yourself in the mean time! The second issue shows a very good way to take advantage of both analyzed and not analyzed strings by using multifields, thereby allowing you to use each field when it is appropriate.
</p><p>There are other minor issues that I want to list for completeness:
</p><ul>
	<li>Prior to Elasticsearch 1.4, we did not actively push doc values because they did not get as much attention as they deserved. However, with Elasticsearch 1.4 and later, you should be using them.
	<ul>
		<li>As a support engineer, my honest opinion is that you should be running a minimum of Elasticsearch 1.6 or later, so this is not an issue.</li>
	</ul></li>
	<li>Technically, doc values write extra data when documents are indexed.
	<ul>
		<li>In practice, this does not degrade performance at all. The overall benefits of doc values significantly outweigh the cost.</li>
	</ul></li>
	<li>Minor note: Facets do not use doc values, which means that Kibana 3, which relies on facets, does not use doc values.
	<ul>
		<li>You should upgrade to Kibana 4 to take advantage of doc values, as well as support for aggregations, which allows for more powerful visualizations and dashboards.</li>
		<li>Facets were deprecated in Elasticsearch 1.0 and they were replaced with aggregations. They will be completely removed in Elasticsearch 2.0.</li>
	</ul></li>
</ul><h3>Fixing It For Good</h3><p>We&apos;re not there yet, but Elasticsearch 2.0 is on the horizon and we expect to release the first beta very soon. I cannot wait for this release because it will do away with this issue for any new index by applying <code>&quot;doc_values&quot; : true</code> by default for every possible field (read: everything that is not an analyzed string!).
</p><p>Elasticsearch 2.0 contains a lot of changes that I am excited to see. Changing doc values to default to <code>true</code> is just one major change that I am looking forward too, but continuing the analyzed string discussion from above, there are numerous changes that will simplify your deployment and your life (and mine too!). From my support perspective, such changes range from further improvements to shard recovery, to diff-based cluster state communication. Feel free to read about some of the other changes in <a href="https://www.elastic.co/blog/elasticsearch-2.0.0.beta1-coming-soon">the Elasticsearch 2.0.0.beta1 announcement post here</a> and keep an eye out for the release of beta1 in the very near future.
</p><h3>Hope That Helps</h3><p>It&apos;s literally my job to help people to be successful running our products and I have seen a lot. This is truly the number one concern that I have with users running Elasticsearch at scale and I hope that you can catch it before it becomes a problem for you, or fix it now that you are aware of it!
</p><p>Stay tuned to this blog for future announcements regarding Elasticsearch 2.0. And as always, feel free to <a href="https://twitter.com/elastic">tweet us <strong>@elastic</strong></a>,&#xA0;visit <a href="https://discuss.elastic.co/">our forums</a>, and check out our webinars such as&#xA0;<a href="https://www.elastic.co/webinars/performance-and-scale">Configuring Your Elasticsearch Cluster for Performance &amp; Scale</a>.
</p></body></html>

Support in the Wild: My Biggest Elasticsearch Problem at Scale

<article>
  <p>With the release of Elasticsearch 1.3.8 and 1.4.3, we have, by default, disabled the ability for Groovy scripts to be sent dynamically with a request or an indexed script, but Groovy remains the default scripting language. This blog post is here to show you how to continue to use scripts for unsandboxed languages, albeit a little less dynamically.</p>
  <p>This blog post exists to help you to understand what that means and, more importantly, how to continue using scripts to get important tasks done safely.</p>
  <div class="section">
    <div class="titlepage">
      <div>
        <div>
          <h2 class="title"><a href="#_what_is_a_dynamic_script" id="_what_is_a_dynamic_script"></a>What is a Dynamic Script?</h2>
        </div>
      </div>
    </div>
    <p>For anyone not familiar with Elasticsearch, you can submit scripts as a part of many different requests: searches, aggregations, updates, upserts, and deletes by query. In short, you can effectively add scripting behavior to extend beyond the normal behavior for your own use cases.</p>
    <p>For example, this is a dynamic script that returns documents that have the same value for field1 and field2 + shift:</p><pre class="prettyprint">GET /_search<br>{<br>  "query":{<br>    "filtered":{<br>      "filter":{<br>        "script":{<br>          "script":"doc['field1'].value == (doc['field2'].value + shift)",<br>          "lang":"groovy",<br>          "params":{<br>            "shift":3<br>          }<br>        }<br>      }<br>    }<br>  }<br>}</pre>
    <p>You can change the language, which may naturally require the syntax to change and add limitations (e.g., <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html#_lucene_expressions_scripts" target="_top" class="ulink">Lucene Expressions</a> instead of Groovy scripts), by changing the lang parameter to the chosen language.</p>
    <div class="section">
      <div class="titlepage">
        <div>
          <div>
            <h3 class="title"><a href="#_why_is_that_dynamic" id="_why_is_that_dynamic"></a>Why is that Dynamic?</h3>
          </div>
        </div>
      </div>
      <p>The above is an example of a dynamic script because the actual script portion needs to be interpreted and compiled on the server side <span class="emphasis"><em>dynamically</em></span>. Dynamic scripts are any script that is sent to the data nodes via an Elasticsearch API, which includes <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html#_indexed_scripts" target="_top" class="ulink">indexed scripts</a>.</p>
      <p>In other words, if the script is not stored with <span class="emphasis"><em>all</em></span> of the data nodes, then it is treated as a dynamic script.</p>
    </div>
  </div>
  <div class="section">
    <div class="titlepage">
      <div>
        <div>
          <h2 class="title"><a href="#_what_to_expect_with_dynamic_scripting_disabled" id="_what_to_expect_with_dynamic_scripting_disabled"></a>What to Expect with Dynamic Scripting Disabled</h2>
        </div>
      </div>
    </div>
    <p>Due to the changes in the latest release, dynamic scripting is now disabled by default for Groovy. With that and given the earlier example script, this is what you should expect to see if you submitted that request (reduced for brevity and emphasis):</p><pre class="prettyprint">{<br>   "error":"SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[8FJ02MofSnqVvOQ10BXxhQ][test][0]: SearchParseException[[test][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{...}]]]; nested: ScriptException[dynamic scripting for [groovy] disabled]...",<br>   "status":400<br>}</pre>
    <p>The key part of the error message is “ScriptException[dynamic scripting for [groovy] disabled]”.</p>
  </div>
  <div class="section">
    <div class="titlepage">
      <div>
        <div>
          <h2 class="title"><a href="#_how_to_continue_to_use_scripting" id="_how_to_continue_to_use_scripting"></a>How to Continue to use Scripting</h2>
        </div>
      </div>
    </div>
    <p>There are three ways to submit a script to Elasticsearch. The two dynamic ways are per-request scripting (shown) or using an indexed script. Using an indexed script works by storing Groovy scripts in Elasticsearch itself, and using them on demand (this works really well, but it still allows untrusted users to add their own scripts given open access to your cluster!). These work in much the same way as a stored procedure works in a relational database: you prewrite your script and invoke it by name later as part of a request.</p><pre class="prettyprint">GET /_search<br>{<br>  "query":{<br>    "filtered":{<br>      "filter":{<br>        "script":{<br>          "script_id":"your_custom_script",<br>          "lang":"groovy",<br>          "params":{<br>            "shift":3<br>          }<br>        }<br>      }<br>    }<br>  }<br>}</pre>
    <p>Notice that not much changed, except that the inner script became script_id and its value changed to the name of the prewritten script.</p>
    <p>The non-dynamic way to provide scripts to Elasticsearch is to write your script to a file on disk, instead of indexing them, and then store it with the configuration for each script that you plan to use. This is how you can continue to use unsandboxed scripts with dynamic scripting disabled, for any scripting language.</p>
    <p>In the first example, the Groovy script was literally doc['field1'].value == doc['field2'].value + shift. You can write that script into your file with a “.groovy” extension:</p><pre class="programlisting prettyprint lang-json">doc['field1'].value == (doc['field2'].value + shift)</pre>
    <p>If this file were named your_custom_script.groovy and placed in the config/scripts directory of <span class="emphasis"><em>all</em></span> of your Elasticsearch data nodes, then Elasticsearch will pick up the script within 60 seconds (<a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html#_automatic_script_reloading" target="_top" class="ulink">configurable by changing watcher.interval</a> in your elasticsearch.yml) and pre-compile it for use with future requests. You do need to ensure that the file is readable by the user running Elasticsearch! After writing this to disk, your configuration directory should look something like this:</p><pre class="literallayout">config/<br />  elasticsearch.yml<br />  logging.yml<br />  scripts/<br />    your_custom_script.groovy</pre>
    <p>This is intentionally not as dynamic as submitting the script with each request or indexing the script, but it does still allow for dynamically adding scripts in a <span class="emphasis"><em>trusted</em></span> environment.</p>
  </div>
  <div class="chapter">
    <div class="titlepage">
      <div>
        <div>
          <h1 class="title"><a href="#_using_a_script_written_to_disk" id="_using_a_script_written_to_disk"></a>Using a script written to disk</h1>
        </div>
      </div>
    </div>
    <p>Your script will not be usable until the script is loaded, at which point you will see something like this in your log file(s):</p><pre class="literallayout">[2015-02-11 11:14:47,066][INFO ][script                   ] [Sergei Kravinoff] compiling script file [/path/to/elasticsearch-1.4.3/config/scripts/your_custom_script.groovy]</pre>
    <p>Once your script has been loaded by <span class="emphasis"><em>all</em></span> of your Elasticsearch data nodes, then you can begin to use it. To use it, you supply the name of your script as the file (not script_id!):</p><pre class="prettyprint">GET /_search<br>{<br>  "query":{<br>    "filtered":{<br>      "filter":{<br>        "script":{<br>          "file":"your_custom_script",<br>          "lang":"groovy",<br>          "params":{<br>            "shift":3<br>          }<br>        }<br>      }<br>    }<br>  }<br>}</pre>
    <p>Note: the language is optional because Groovy is the default language. If you use a different scripting language or if you change the default scripting language (e.g., to Lucene Expressions), then the language must be supplied to help it to find the right script.&nbsp;As a best practice, we recommend that applications include the lang parameter. This will insulate you from any future changes in the default scripting language.</p>
  </div>
  <div class="chapter">
    <div class="titlepage">
      <div>
        <div>
          <h1 class="title"><a href="#_questions" id="_questions"></a>Questions?</h1>
        </div>
      </div>
    </div>
    <p>If you have any questions, then please do not hesitate to reach out to us on Twitter (<a href="https://twitter.com/elasticsearch" target="_top" class="ulink">@elasticsearch</a>). You can report any problems on the <a href="https://github.com/elasticsearch/elasticsearch/issues" target="_top" class="ulink">GitHub issues page</a>.</p>
  </div>
  <aside id="post_tags"></aside>
</article>

Running Groovy Scripts without Dynamic Scripting

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Support Engineer

Chris Earle

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

O conteúdo desta página não está disponível no idioma selecionado. A Elastic está trabalhando para garantir que o conteúdo esteja disponível em vários idiomas. Agradecemos a compreensão.

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Communication Preferences - ONLY for /communication-preferences

<p>Ao enviar a confirmação, você reconhece que leu e concorda com nossos <a href="/pt/legal/elastic-cloud-account-terms" target="_self">Termos de Serviço</a> e também concorda em receber <a href="/pt/legal/privacy-statement#how-we-use-the-information" target="_self">comunicações especiais</a> por meio dos dados de contato que forneceu acima. Consulte a <a href="/pt/legal/privacy-statement/" target="_self">Declaração de Privacidade da Elastic</a> para saber mais detalhes ou para cancelar a qualquer momento.</p>

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Serverless GDPR Text

<p>Ao enviar a confirmação, você reconhece que leu e concorda com nossos <a href="/pt/legal/serverless-preview-terms" target="_self">Termos de Serviço</a> e que a Elastic pode <a href="/pt/legal/privacy-statement#how-we-use-the-information" target="_self">entrar em contato com você</a> para divulgar nossos produtos e serviços relacionados usando os dados que você forneceu acima. Consulte a <a href="/pt/legal/privacy-statement/" target="_self">Declaração de Privacidade da Elastic</a> para saber mais detalhes ou para cancelar a qualquer momento.</p>

<p>By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.</p>

Cloud Trial GDPR Text

<p>Ao enviar, você concorda com os <a href="/pt/agreements/global/cloud-services-monthly" rel="nofollow">Termos de Serviço do Elastic Cloud</a>. Seus dados pessoais serão tratados de acordo com a <a href="/pt/legal/privacy-statement">Declaração de Privacidade da Elastic</a>.
</p>

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Newsletter GDPR Text

<p>Ao enviar a confirmação, você reconhece que leu e concorda com nossos <a href="/pt/legal/terms-of-use" target="_self">Termos de Serviço</a> e que a Elastic pode <a href="/pt/legal/privacy-statement#how-we-use-the-information" target="_self">entrar em contato com você</a> para divulgar nossos produtos e serviços relacionados usando os dados que você forneceu acima. Consulte a <a href="/pt/legal/privacy-statement/" target="_self">Declaração de Privacidade da Elastic</a> para saber mais detalhes ou para cancelar a qualquer momento.</p>

Explore similar demos

Explore outras demonstrações semelhantes

Overview

Load More

Speakers

Close

See more insights

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

Consultar a hora de início do webinar no meu fuso horário

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Não vai conseguir participar? Inscreva-se e nós lhe enviaremos a gravação. Você também receberá um email com conteúdo relacionado.

Author

Articles by Chris Earle

Support Engineer, Elastic

Elasticsearch: Verifying Data Integrity with External Data Stores

Support in the Wild: My Biggest Elasticsearch Problem at Scale

Running Groovy Scripts without Dynamic Scripting