High CPU usage
editHigh CPU usage
editElasticsearch uses thread pools to manage CPU resources for concurrent operations. High CPU usage typically means one or more thread pools are running low.
If a thread pool is depleted, Elasticsearch will reject requests
related to the thread pool. For example, if the search
thread pool is
depleted, Elasticsearch will reject search requests until more threads are available.
Diagnose high CPU usage
editCheck CPU usage
From your deployment menu, click Performance. The page’s CPU Usage chart shows your deployment’s CPU usage as a percentage.
High CPU usage can also deplete your CPU credits. CPU credits let Elasticsearch Service provide smaller clusters with a performance boost when needed. The CPU credits chart shows your remaining CPU credits, measured in seconds of CPU time.
You can also use the cat nodes API to get the current CPU usage for each node.
response = client.cat.nodes( v: true, s: 'cpu:desc' ) puts response
GET _cat/nodes?v=true&s=cpu:desc
The response’s cpu
column contains the current CPU usage as a percentage. The
name
column contains the node’s name.
Use the cat nodes API to get the current CPU usage for each node.
response = client.cat.nodes( v: true, s: 'cpu:desc' ) puts response
GET _cat/nodes?v=true&s=cpu:desc
The response’s cpu
column contains the current CPU usage as a percentage. The
name
column contains the node’s name.
Check hot threads
If a node has high CPU usage, use the nodes hot threads API to check for resource-intensive threads running on the node.
response = client.nodes.hot_threads( node_id: 'my-node,my-other-node' ) puts response
GET _nodes/my-node,my-other-node/hot_threads
This API returns a breakdown of any hot threads in plain text.
Reduce CPU usage
editThe following tips outline the most common causes of high CPU usage and their solutions.
Scale your cluster
Heavy indexing and search loads can deplete smaller thread pools. To better handle heavy workloads, add more nodes to your cluster or upgrade your existing nodes to increase capacity.
Spread out bulk requests
While more efficient than individual requests, large bulk indexing or multi-search requests still require CPU resources. If possible, submit smaller requests and allow more time between them.
Cancel long-running searches
Long-running searches can block threads in the search
thread pool. To check
for these searches, use the task management API.
response = client.tasks.list( actions: '*search', detailed: true ) puts response
GET _tasks?actions=*search&detailed
The response’s description
contains the search request and its queries.
running_time_in_nanos
shows how long the search has been running.
{ "nodes" : { "oTUltX4IQMOUUVeiohTt8A" : { "name" : "my-node", "transport_address" : "127.0.0.1:9300", "host" : "127.0.0.1", "ip" : "127.0.0.1:9300", "tasks" : { "oTUltX4IQMOUUVeiohTt8A:464" : { "node" : "oTUltX4IQMOUUVeiohTt8A", "id" : 464, "type" : "transport", "action" : "indices:data/read/search", "description" : "indices[my-index], search_type[QUERY_THEN_FETCH], source[{\"query\":...}]", "start_time_in_millis" : 4081771730000, "running_time_in_nanos" : 13991383, "cancellable" : true } } } } }
To cancel a search and free up resources, use the API’s _cancel
endpoint.
response = client.tasks.cancel( task_id: 'oTUltX4IQMOUUVeiohTt8A:464' ) puts response
POST _tasks/oTUltX4IQMOUUVeiohTt8A:464/_cancel
For additional tips on how to track and avoid resource-intensive searches, see Avoid expensive searches.