Elasticsearch can help you analyze images for facial recognition. Learn more about how to detect and encode facial information — and find matches in search — using Python.

Have you ever tried to search for objects in images? Elasticsearch can help you store, analyze, and search for objects in images or videos.
In this quick tutorial, we’ll show you how to build a system for facial recognition with Python. Learn more about how to detect and encode facial information — and find matches in search. 
 <h2>A quick walkthrough of the basics</h2>Need a refresher? Let's briefly review some fundamentals about a few basic concepts.
 <h3>Facial recognition</h3>Facial recognition is the process of identifying a person using their face — for example, for the purpose of implementing an authentication mechanism (like unlocking your smartphone). It captures, analyzes, and compares patterns based on the person's facial details. This process can be divided into three phases:&nbsp;
<ul>
	<li aria-level="1">Face detection: Identify human faces in digital images</li>
	<li aria-level="1">Face data encoding: Convert facial features into digital representation</li>
	<li aria-level="1">Face match: Search and compare facial features</li>
</ul>We’ll walk you through each phase in our example.
 <h3>128-dimension vector</h3>Facial features can be transformed into a set of digital information in order to be stored and analysed.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt4f879e5217b5bb57/606cabbc2c290626b4382280/facial-recognition-blog-vector.png" data-sys-asset-uid="blt4f879e5217b5bb57" alt="facial-recognition-blog-vector.png" "="" width="317" height="469" style="width: 317px; height: 469px; display: block; margin: auto;">
<h3>Vector data type</h3>Elasticsearch offers the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/dense-vector.html"><code>dense_vector</code></a> data type to store dense vectors of float values. The maximum number of dimensions in a vector shouldn’t exceed 2048, which is enough to store the face features representation.
Now, let's implement all these concepts.
 <h2>Everything you need</h2>To detect faces and encode the information, you’ll need the following:
<ul>
	<li aria-level="1"><a href="https://www.python.org/downloads/">Python</a>: We’ll be using Python 3 in this example, which <a href="https://www.python.org/downloads/">you can find here</a>. Installation and download steps will depend on your operating system.</li>
	<li aria-level="1">An Elasticsearch cluster: You can spin up a cluster in a <a href="https://cloud.elastic.co/registration">free trial of Elastic Cloud</a>.&nbsp;</li>
	<li aria-level="1"><a href="https://pypi.org/project/face-recognition/">Face-recognition library</a>: A simple face recognition Python library.</li>
	<li aria-level="1"><a href="https://elasticsearch-py.readthedocs.io/en/v7.10.1/">Python Elasticsearch client</a>: The official Python client for Elasticsearch.</li>
</ul>Please note we’ve tested the following instructions on Ubuntu 20.04 LTS and Ubuntu 18.04 LTS. Some changes may need to be made depending on your operating system.
 <h2>Install Python and Python libraries</h2>Ubuntu 20 and other versions of Debian Linux are delivered with the installation of Python 3. If this isn’t the case with your system, you can download and install Python <a href="https://www.python.org/downloads/">using this link</a>.
To confirm that your version is the latest, you can run the following command:&nbsp;
<pre class="prettyprint">sudo apt update 
sudo apt upgrade
</pre>Confirm the Python version is 3.x:
<pre class="prettyprint">python3 -V
</pre>Install pip3 to manage the Python libraries:
<pre class="prettyprint">sudo apt install -y python3-pip
</pre>Install cmake needed for the <code>face_recognition</code> library:
<pre class="prettyprint">pip3 install CMake
</pre>Add the cmake bin folder to your $PATH directory:
<pre class="prettyprint">export PATH=$CMake_bin_folder:$PATH
</pre>At the end, install the following libraries before starting scripting our main program:
<pre class="prettyprint">pip3 install dlib 
pip3 install numpy 
pip3 install face_recognition 
pip3 install elasticsearch
</pre> <h2>Detect and encode facial information from an image</h2>Using the <code>face_recognition</code> library, we can detect the faces from an image and transform the facial features into a 128-dimension vector.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltba1ba92ea79adfda/606cabdbf46b594bd59f6b02/facial-recognition-blog-vector-facial-features.png" data-sys-asset-uid="bltba1ba92ea79adfda" alt="facial-recognition-blog-vector-facial-features.png" "="" width="381" height="239" style="width: 381px; height: 239px; display: block; margin: auto;">
Create a getVectorFromPicture.py file:
<pre class="prettyprint">touch getVectorFromPicture.py
</pre>Append the file with following script:
<pre class="prettyprint">import face_recognition 
import numpy as np 
import sys 
image = face_recognition.load_image_file("$PATH_TO_IMAGE") 
# detect the faces from the images 
face_locations = face_recognition.face_locations(image) 
# encode the 128-dimension face encoding for each face in the image 
face_encodings = face_recognition.face_encodings(image, face_locations) 
# Display the 128-dimension for each face detected 
for face_encoding in face_encodings: 
 print("Face found ==&gt; ", face_encoding.tolist())
</pre>Let’s execute <code>getVectorFromPicture.py</code> to get the facial features representation for images of Elastic’s founders. The <code>$PATH_TO_IMAGE</code> variable needs to be changed in the script to set the image file name.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt2fcb4d8cc1d9008a/606cac60a9cf2022b93564bf/facial-recognition-blog-elastic-founders.png" data-sys-asset-uid="blt2fcb4d8cc1d9008a" alt="facial-recognition-blog-elastic-founders.png" width="565" height="352" style="width: 565px; height: 352px; display: block; margin: auto;">
We can now store the facial features representation into Elasticsearch.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt09790e6298edb432/606cac86d8743426b393db5b/facial-recognition-blog-features-representation.png" data-sys-asset-uid="blt09790e6298edb432" alt="facial-recognition-blog-features-representation.png" width="341" height="230" style="width: 341px; height: 230px; display: block; margin: auto;">
Let’s first create the index with mapping that contains a field with <code>dense_vector</code> type:
<pre class="prettyprint"># Store the face 128-dimension in Elasticsearch 
## Create the mapping 
curl -XPUT "http://localhost:9200/faces" -H 'Content-Type: application/json' -d' 
{ 
 "mappings" : { 
 "properties" : { 
 "face_name" : { 
 "type" : "keyword" 
 }, 
 "face_encoding" : { 
 "type" : "dense_vector", 
 "dims" : 128 
 } 
 } 
 } 
}'
</pre>We need to create one document for each face representation, which can be done using the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html">Index API</a>:
<pre class="prettyprint">## Index the face feature representation
curl -XPOST "http://localhost:9200/faces/_doc" -H 'Content-Type: application/json' -d'
{
 "face_name": "name",
 "face_encoding": [
 -0.14664565,
 0.07806452,
 0.03944433,
 ...
 ...
 ...
 -0.03167224,
 -0.13942884
 ]
}'
</pre>
<h2>Match the faces</h2>Let's say we have indexed four documents in Elasticsearch that contain each facial representation of Elastic’s founders. We can now use another image of our founders to match the individual images.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt654b494db42a931c/606cad11efddb74bd62b8576/facial-recognition-blog-elastic-founders-match.png" data-sys-asset-uid="blt654b494db42a931c" alt="facial-recognition-blog-elastic-founders-match.png" width="591" height="394" style="width: 591px; height: 394px; display: block; margin: auto;">
Create a recognizeFaces.py file:&nbsp;
<pre class="prettyprint">touch recognizeFaces.py
</pre>In this script, we’ll extract the vectors for each face detected from the input image, and we use the vectors for each face to build the query to send to Elasticsearch:
Import the libraries:
<pre class="prettyprint">import face_recognition 
import numpy as np 
from elasticsearch import Elasticsearch 
import sys
</pre>Add the following section to connect to Elasticsearch:
<pre class="prettyprint"># Connect to Elasticsearch cluster 
from elasticsearch import Elasticsearch 
es = Elasticsearch( 
cloud_id="cluster-1:dXMa5Fx...", 
 http_auth=("elastic", "&lt;password&gt;"), 
)
</pre>We will use the <code>cosineSimilarity</code> function to calculate the measure of cosine similarity between a given query vectors and document vectors stored into Elasticsearch.
<pre class="prettyprint">i=0 
for face_encoding in face_encodings: 
 i += 1 
 print("Face",i) 
 response = es.search( 
 index="faces", 
 body={ 
 "size": 1, 
 "_source": "face_name", 
 "query": { 
 "script_score": { 
 "query" : { 
 "match_all": {} 
 }, 
 "script": { 
 "source": "cosineSimilarity(params.query_vector, 'face_encoding')", 
 "params": { 
 "query_vector":face_encoding.tolist() 
 } 
 } 
 } 
 } 
 } 
 )
</pre>Let’s assume the score under 0.93 is considered an unknown face:
<pre class="prettyprint"> for hit in response['hits']['hits']: 
 #double score=float(hit['_score']) 
 if (float(hit['_score']) &gt; 0.93): 
 print("==&gt; This face match with ", hit['_source']['face_name'], ",the score is" ,hit['_score']) 
 else: 
 print("==&gt; Unknown face")
</pre>Let's execute our script:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc9d46c6ceaba0074/606cad301609c821964d9038/facial-recognition-blog-execute-script.gif" data-sys-asset-uid="bltc9d46c6ceaba0074" alt="facial-recognition-blog-execute-script.gif" style="display: block; margin: auto;">
The script was able to detect all faces with score match higher than 0.93.
 <h2>Go a step further with advanced search</h2>Facial recognition and search can be combined for advanced use cases. You can use Elasticsearch to build more sophisticated queries like <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-queries.html"><code>geo-queries</code></a>, <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html"><code>query-dsl-bool-query</code></a>, and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html"><code>search-aggregations</code></a>.
As an example, the following query applies <code>cosineSimilarity</code> search on a specific location within a 200-km radius:
<pre class="prettyprint">GET /_search 
{ 
 "query": { 
 "script_score": { 
 "query": { 
 "bool": { 
 "must": { 
 "match_all": {} 
 }, 
 "filter": { 
 "geo_distance": { 
 "distance": "200km", 
 "pin.location": { 
 "lat": 40, 
 "lon": -70 
 } 
 } 
 } 
 } 
 }, 
 "script": { 
 "source": "cosineSimilarity(params.query_vector, 'face_encoding')", 
 "params": { 
 "query_vector":[ 
 -0.14664565,
 0.07806452,
 0.03944433,
 ...
 ...
 ...
 -0.03167224,
 -0.13942884
 ] 
 } 
 } 
 } 
 } 
}
</pre>Combining <code>cosineSimilarity</code> with other Elasticsearch queries give you unlimited possibilities to implement more complex use cases.
 <h2>Conclusion</h2>Facial recognition can be relevant in many use cases, and you may already use it in your daily life. The concepts described above can be generalized to any object detection in images or videos, so you may extend your use case to a very large utilization.&nbsp;
Elasticsearch can help simplify your advanced requirements. Try it out with a <a href="https://cloud.elastic.co/registration">free 14-day trial of Elastic Cloud</a>, our official managed Elasticsearch offering, and let us know what you think in our <a href="https://discuss.elastic.co/c/elastic-stack/elasticsearch/6">Discuss forums</a>.

blog-banner-elasticsearch-service.png

blog-thumb-release-elasticsearch.png

How to build a facial recognition system using Elasticsearch and Python

Elasticsearch can be optimized for any use case. In this post, learn how to size and benchmark your Elastic Stack deployment for logs and metrics observability.

With Elasticsearch, it's easy to hit the ground running. When I built my first Elasticsearch cluster, it was ready for indexing and search within a matter of minutes. And while I was pleasantly surprised at how quickly I was able to deploy it, my mind was already racing towards next steps. But then I remembered I needed to slow down (we all need that reminder sometimes!) and answer a few questions before I got ahead of myself. Questions like:<ul><li>How confident am I that this will work in production?</li><li>What is the throughput capacity of my cluster?</li><li>How is it performing?</li><li>Do I have enough available resources in my cluster?</li><li>Is it the right size?</li></ul>If these sound familiar, great! These are questions all of us should be thinking about as we deploy new products into our ecosystems. In this post, we'll tackle performance Elasticsearch benchmarking and sizing questions like the above. We’ll go beyond “it depends” to equip you with a set of methods and recommendations to help you size your Elasticsearch cluster and benchmark your environment. As sizing exercises are specific to each use case, we will focus on <a href="https://www.elastic.co/log-monitoring" target="_self">logging</a> and <a href="https://www.elastic.co/infrastructure-monitoring" target="_self">metrics</a> in this post.<h2>Think like a service provider</h2>When we define the architecture of any system, we need to have a clear vision about the use case and the features that we offer, which is why it’s important to think as a service provider — where the quality of our service is the main concern. In addition, the architecture can be influenced by the constraints that we may have, like the hardware available, the global strategy of our company and many other constraints that are important to consider in our sizing exercise.Note that with the <a href="https://www.elastic.co/elasticsearch/service" target="_self">Elasticsearch Service on Elastic Cloud</a>, we take care of a lot of the maintenance and data tiering that we describe below. We also provide a predefined observability template along with a checkbox to enable the shipment of logs and metrics to a dedicated monitoring cluster. Feel free to spin up a <a href="https://www.elastic.co/cloud/" target="_self">free trial cluster</a> as you follow along.<h2>Computing resource&nbsp;basics</h2>Performance is contingent on how you're using Elasticsearch, as well as what you're running it on. Let's review some fundamentals around computing resources. For each search or indexing operation the following resources are involved:<h3>Storage: Where data persists</h3><ul><li>SSDs are recommended whenever possible, in particular for nodes running search and index operations. Due to the higher cost of SSD storage, a <a href="https://www.elastic.co/blog/implementing-hot-warm-cold-in-elasticsearch-with-index-lifecycle-management" target="_self">hot-warm architecture</a> is recommended to reduce expenses.</li><li>When operating on bare metal, local disk is king!</li><li>Elasticsearch does not need redundant storage (RAID 1/5/10 is not necessary), logging and metrics use cases typically have at least one replica shard, which is the minimum to ensure fault tolerance while minimizing the number of writes.</li></ul><h3>Memory: Where data is buffered</h3><ul><li>JVM Heap:Stores metadata about the cluster, indices, shards, segments, and fielddata. This is ideally set to 50% of available RAM.</li><li>OS Cache:Elasticsearch will use the remainder of available memory to cache data, improving performance dramatically by avoiding disk reads during full-text search, aggregations on doc values, and sorts.</li></ul><h3>Compute: Where data is processed</h3>Elasticsearch nodes have thread pools and thread queues that use the available compute resources. The quantity and performance of CPU cores governs the average speed and peak throughput of data operations in Elasticsearch.<h3>Network: Where data is transferred</h3>The network performance — both bandwidth and latency — can have an impact on the inter-node communication and inter-cluster features like <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.9/modules-cross-cluster-search.html" target="_self">cross-cluster search</a> and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.9/xpack-ccr.html" target="_self">cross-cluster replication</a>.<h2>Sizing by data volume</h2>For metrics and logging use cases, we typically manage a huge amount of data, so it makes sense to use the data volume to initially size our Elasticsearch cluster. At the beginning of this exercise we need to ask some questions to better understand the data that needs to be managed in our cluster.<ul><li>How much raw data (GB) we will index per day?</li><li>How many days we will retain the data?</li><li>How many days in the hot zone?</li><li>How many days in the warm zone?</li><li>How many replica shards will you enforce?</li></ul>In general we add 5% or 10% for margin of error and 15% to stay under the disk watermarks. We also recommend adding a node for hardware failure.<h3>Let’s do the math</h3><ul><li>Total Data (GB) = Raw data (GB) per day * Number of days retained * (Number of replicas + 1) * Indexing/Compression Factor</li><li>Total Storage (GB) = Total data (GB) * (1 + 0.15 disk Watermark threshold + 0.1 Margin of error)</li><li>Total Data Nodes = ROUNDUP(Total storage (GB) / Memory per data node / Memory:Data ratio) In case of large deployment it's safer to add a node for failover capacity.</li></ul>1.2 is an average ratio we have observed with throughout deployments.To get the value that relates to your data, index logs and metrics and divide the index size (without replica) by the raw volume size.<h2>Example: Sizing a small cluster</h2>You might be pulling logs and metrics from some applications, databases, web servers, the network, and other supporting services . Let's assume this pulls in 1GB per day and you need to keep the data 9 months.You can use 8GB memory per node for this small deployment. Let’s do the math:<ul><li>Total Data (GB)&nbsp;= 1GB x (9 x 30 days) x 2 = 540GB</li><li>Total Storage (GB)= 540GB x (1+0.15+0.1) = 675GB</li><li>Total Data Nodes = 675GB disk / 8GB RAM /30 ratio = 3 nodes</li></ul>Let’s see how simple it is to build this deployment on <a href="https://www.elastic.co/cloud" target="_self">Elastic Cloud</a>:<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt839edb187d5c4ecb/5f9ad378878c0d548cdc2608/blog-logs-metrics-sizing-small-cloud.png" data-sys-asset-uid="blt839edb187d5c4ecb" alt="Sizing your small cluster in Elastic Cloud"/><h3>Example: Sizing a large deployment</h3>Your small deployment got successful and more partners want to use your Elasticsearch service, you may need resizing your cluster to much the new requirements.Let’s do the math with the following inputs:<ul><li>You receive 100GB per day and we need to keep this data for 30 days in the hot zone and 12 months in the warm zone.</li><li>We have 64GB of memory per node with 30GB allocated for heap and the remaining for OS cache.</li><li>The typical memory:data ratio for the hot zone used is 1:30 and for the warm zone is 1:160.</li></ul>If we receive 100GB per day and we have to keep this data for 30 days, this gives us:<ul><li>Total Data (GB) in the hot zone = (100GB x 30 days * 2) = 6000GB</li><li>Total Storage (GB) in the hot zone = 6000GB x (1+0.15+0.1) = 7500GB</li><li>Total Data Nodes in the hot zone = ROUNDUP(7500 / 64 / 30) + 1 = 5 nodes</li><li>Total Data (GB) in the warm zone = (100GB x 365 days * 2) = 73000GB</li><li>Total Storage (GB) in the warm zone = 73000GB x (1+0.15+0.1) = 91250GB</li><li>Total Data Nodes in the warm zone = ROUNDUP(91250 / 64 / 160) + 1 = 10 nodes</li></ul>Let’s see how simple it is to build this deployment on <a href="https://www.elastic.co/cloud" target="_self">Elastic Cloud</a>:<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt47b6081e39428a2a/5f9ad3946c73a0751fc7a376/blog-logs-metrics-sizing-large-cloud.jpg" data-sys-asset-uid="blt47b6081e39428a2a" alt="Sizing your large cluster in Elastic Cloud"/><h2>Benchmarking</h2>Now that we have our cluster(s) sized appropriately, we need to confirm that our math holds up in real world conditions. To be more confident before moving to production, we will want to do benchmark testing to confirm the expected performance, and the targeted SLA.For this benchmark, we will use the same tool our Elasticsearch engineers use <a href="https://github.com/elastic/rally" target="_self">Rally</a>. This tool is simple to deploy and execute, and completely configurable so you can test multiple scenarios. Learn more about <a href="https://www.elastic.co/blog/rally-1-0-0-released-benchmark-elasticsearch-like-we-do" target="_self">how we use Rally at Elastic</a>.To simplify the result analysis, we will split the benchmark into two sections, indexing and search requests.<h2>Indexing benchmark</h2>For the indexing benchmarks we are trying to answers the following questions:<ul><li>What is the maximum indexing throughput for my clusters?</li><li>What is the data volume that I can index per day?</li><li>Is my cluster oversized or undersized ?</li></ul>For the purpose of this benchmark we will use a 3 nodes cluster with the following configuration for each nodes:<ul><li>8 vCPU</li><li>Standard persistent disk (HDD)</li><li>32GB/16 heap</li></ul><h3>Indexing benchmark #1:</h3>The data set used for this benchmark is <a href="https://github.com/elastic/rally-tracks/tree/master/metricbeat" target="_self">Metricbeat</a> data with the following specifications:<ul><li>1,079,600 documents</li><li>Data volume: 1.2GB</li><li>AVG document size: 1.17 KB</li></ul>The indexing performance will depend also on the performance of the indexing layer, in our case Rally. We'll execute multiple benchmark runs to figure out the optimal bulk size and the optimal thread count in our case.&nbsp;We’ll start with 1 Rally client to find the optimal batch size. Starting from 100, then increasing by double in subsequent runs, which shows that we have an optimal batch size of 12k (around 13.7 MB) and we can reach 13K events indexed per second.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltf7fbc948de75b747/5f9ad3cf1f9166620ed86cf8/blog-logs-metrics-sizing-batch-1.png" data-sys-asset-uid="bltf7fbc948de75b747"/>Next, using a similar method, we find that the optimal number of clients is 16 allowing us to reach 62K events indexed per second.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt8e20acb021757a45/5f9ad3e36c73a0751fc7a380/blog-logs-metrics-sizing-clients-1.png" data-sys-asset-uid="blt8e20acb021757a45"/>Our cluster can handle a maximum indexing throughput of 62K events per second. To go further we would need to add a new node.I was curious to see how many indexing requests one node can handle, So I’ve executed the same track on a one node cluster and a two node cluster to see the difference.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc1f831d98ba4a354/5f9ad4001a81c1644e996ea7/blog-logs-metrics-sizing-clusters-1.jpg" data-sys-asset-uid="bltc1f831d98ba4a354"/><h4>Conclusion: For my test environment, the maximum indexing throughput is...</h4><ul><li>With 1 node and 1 shard we got 22K events per second.</li><li>With 2 nodes and 2 shards we got 43k events per second.</li><li>With 3 nodes and 3 shards we got 62k events per second.</li></ul>Any additional indexing request will be put in the queue, and when the queue is full the node will send rejection responses.Our dataset impacts the cluster performance, which is why it’s important to execute the tracks with your own data.<h3>Indexing benchmark #2:</h3>For the next step, I have executed the same tracks with the <a href="https://github.com/elastic/rally-tracks/tree/master/http_logs" target="_self">HTTP server log data</a> using the following configuration:<ul><li>Volumes: 31.1GB</li><li>Document: 247,249,096</li><li>Avg document size: 0.8 KB</li></ul>Optimal bulk&nbsp;size is 16K documents.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt28f6345ba1491485/5f9ad457fe49b57a83a15327/blog-logs-metrics-sizing-batch-2.png" data-sys-asset-uid="blt28f6345ba1491485" alt="blog-logs-metrics-sizing-batch-2.png"/>Yielding an optimal number of 32 clients.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd3fcf8723b3f4c14/5f9ad48b83a0a3620dd3b218/blog-logs-metrics-sizing-clients-2.png" data-sys-asset-uid="bltd3fcf8723b3f4c14"/>And the maximum indexing throughput for the http server log data is 220K events per second.<h2>Search benchmark</h2>To benchmark our search performance, we will consider using 20 clients with a targeted throughput of 1000 OPS.For the search, we will execute three benchmarks:<h3>1. Service time for queries</h3>We will compare the service time (90 percentiles) for a set of queries.<h4>Metricbeat data set</h4><ul><li>auto-date-historgram</li><li>auto-data-histogram-with-tz</li><li>date-histogram</li><li>Date-histogram-with-tz</li></ul>We can observe that auto-data-histogram-with-tz query have the higher service time.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt7d612d0591299c2f/5f9ad4a3fe49b57a83a15331/blog-logs-metrics-sizing-metricbeat.png" data-sys-asset-uid="blt7d612d0591299c2f"/><h4>HTTP server log data set</h4><ul><li>Default</li><li>Term</li><li>Range</li><li>Hourly_agg</li><li>Desc_sort_timestamp</li><li>Asc_sort_timestamp</li></ul>We can observe that desc_sort_timestamp and desc_sort_timestamp queries have the higher service time.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt44d39fb4b1ba5c50/5f9ad4c267a7c354926033c5/blog-logs-metrics-sizing-http.png" data-sys-asset-uid="blt44d39fb4b1ba5c50"/><h3>2. Service time for parallel queries</h3>Let's see the 90 percentile service time increased if I execute the queries in <a href="https://esrally.readthedocs.io/en/0.5.1/track.html" target="_self">parallel</a>.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt5358122b351cdd0d/5f9ad50a2425cd7a8af696cc/blog-logs-metrics-sizing-parallel.png" data-sys-asset-uid="blt5358122b351cdd0d"/><h3>3. Index rate and service time with parallel indexing</h3>We'll execute a <a href="https://esrally.readthedocs.io/en/0.5.1/track.html" target="_self">parallel</a> indexing task and the searches to see the indexing rate and the service time for our queries.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltdebe9d92ea2aa969/5f9ad52883a0a3620dd3b222/blog-logs-metrics-sizing-indexing.png" data-sys-asset-uid="bltdebe9d92ea2aa969"/>Let's see the 90 percentile service time for the queries increased when executing in parallel with indexing operations.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt949a10ab19056b1f/5f9ad53883a0a3620dd3b226/blog-logs-metrics-sizing-index-rate.png" data-sys-asset-uid="blt949a10ab19056b1f"/><h3>Reading our results</h3>With 32 clients for indexing and 20 users for search we have:<ul><li>Indexing throughput 173K, less than 220K noted previously.</li><li>Search throughput 1000 events per second.</li></ul><h2>Conclusion</h2>The sizing exercise equips you with a set of methods to calculate the number of nodes that you need based on data volume. In order to best plan for the future performance of your cluster, you will also need to benchmark your infrastructure — something we've already taken care of in <a href="https://www.elastic.co/cloud" target="_self">Elastic Cloud</a>. Since the performance will depend also on your use case, we recommend using your data and your queries to be closest to the reality. More details about how to define a custom workload in the <a href="https://esrally.readthedocs.io/en/stable/adding_tracks.html" target="_self">Rally documentation</a>.After the benchmark exercise, you should have a better understanding of your infrastructure performance and you can <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.9/tune-for-indexing-speed.html" target="_self">continue to fine tune Elasticsearch to improve your indexing speed</a>. Additionally, <a href="https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right" target="_self">learn how to use Rally to get your Elasticsearch right.</a>

blog-banner-charts-monitor.png

blog-thumb-charts-monitor.png

Benchmarking and sizing your Elasticsearch cluster for logs and metrics

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Yacine Younes

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/elastic-cloud-account-terms" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">ausgewählte Nachrichten und Informationen</a> an die oben von Ihnen bereitgestellten Kontaktangaben zu senden. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/serverless-preview-terms" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, bezüglich unserer zugehörigen Produkte und Dienstleistungen und unter Nutzung der von Ihnen oben bereitgestellten Kontaktangaben <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">mit Ihnen in Kontakt zu treten</a>. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Beim Absenden akzeptieren Sie die <a href="/de/agreements/global/cloud-services-monthly" rel="nofollow">Nutzungsbedingungen von Elastic Cloud</a>. Die Verarbeitung Ihrer personenbezogenen Daten unterliegt der <a href="/de/legal/privacy-statement">Datenschutzerklärung von Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/terms-of-use" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, bezüglich unserer zugehörigen Produkte und Dienstleistungen und unter Nutzung der von Ihnen oben bereitgestellten Kontaktangaben <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">mit Ihnen in Kontakt zu treten</a>. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

Der Inhalt dieser Seite ist in der ausgewählten Sprache nicht verfügbar. Wir bei Elastic arbeiten daran, die bereitgestellten Inhalte in verschiedenen Sprachen anzubieten. Bis dahin bitten wir Sie um etwas Geduld und hoffen auf Ihr Verständnis!

Author

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Articles by

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Sie können nicht dabei sein? Registrieren Sie sich, damit wir Ihnen die Aufzeichnung zukommen lassen können. Außerdem erhalten Sie eine E‑Mail mit Begleitmaterial.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Learn More

Read less

You'll also receive an email with related content

Wir schicken Ihnen zudem relevante Informationen, die von Interesse sein könnten.

Thank you for registering. We will send you a confirmation email soon.

Vielen Dank für Ihre Registrierung. Sie erhalten in Kürze eine Bestätigungs-E‑Mail von uns.

Author

Articles by Yacine Younes

How to build a facial recognition system using Elasticsearch and Python

Benchmarking and sizing your Elasticsearch cluster for logs and metrics

Sign up for Elastic Cloud free trial