IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Retrieve selected fields from a search Search multiple data streams and indices »
Elastic Docs ›Elasticsearch Guide [8.9] ›Search your data

Search across clusters

edit

Search across clusters

edit

Cross-cluster search lets you run a single search request against one or more remote clusters. For example, you can use a cross-cluster search to filter and analyze log data stored on clusters in different data centers.

Supported APIs

edit

The following APIs support cross-cluster search:

  • Search
  • Async search
  • Multi search
  • Search template
  • Multi search template
  • Field capabilities
  • Painless execute API
  • [preview] This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. EQL search
  • [preview] This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. SQL search
  • [preview] This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. Vector tile search

Prerequisites

edit

  • If you use sniff mode, the local coordinating node must be able to connect to seed and gateway nodes on the remote cluster.

    We recommend using gateway nodes capable of serving as coordinating nodes. The seed nodes can be a subset of these gateway nodes.

Cross-cluster search examples

edit

Remote cluster setup

edit

The following cluster update settings API request adds three remote clusters: cluster_one, cluster_two, and cluster_three.

PUT _cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "cluster_one": {
          "seeds": [
            "127.0.0.1:9300"
          ]
        },
        "cluster_two": {
          "seeds": [
            "127.0.0.1:9301"
          ]
        },
        "cluster_three": {
          "seeds": [
            "127.0.0.1:9302"
          ]
        }
      }
    }
  }
}

Search a single remote cluster

edit

In the search request, you specify data streams and indices on a remote cluster as <remote_cluster_name>:<target>.

The following search API request searches the my-index-000001 index on a single remote cluster, cluster_one.

response = client.search(
  index: 'cluster_one:my-index-000001',
  body: {
    query: {
      match: {
        "user.id": 'kimchy'
      }
    },
    _source: [
      'user.id',
      'message',
      'http.response.status_code'
    ]
  }
)
puts response
GET /cluster_one:my-index-000001/_search
{
  "query": {
    "match": {
      "user.id": "kimchy"
    }
  },
  "_source": ["user.id", "message", "http.response.status_code"]
}

The API returns the following response:

{
  "took": 150,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0,
    "skipped": 0
  },
  "_clusters": {
    "total": 1,
    "successful": 1,
    "skipped": 0
  },
  "hits": {
    "total" : {
        "value": 1,
        "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "cluster_one:my-index-000001", 
        "_id": "0",
        "_score": 1,
        "_source": {
          "user": {
            "id": "kimchy"
          },
          "message": "GET /search HTTP/1.1 200 1070000",
          "http": {
            "response":
              {
                "status_code": 200
              }
          }
        }
      }
    ]
  }
}

The search response body includes the name of the remote cluster in the _index parameter.

Search multiple remote clusters

edit

The following search API request searches the my-index-000001 index on three clusters:

  • Your local cluster
  • Two remote clusters, cluster_one and cluster_two 
response = client.search(
  index: 'my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001',
  body: {
    query: {
      match: {
        "user.id": 'kimchy'
      }
    },
    _source: [
      'user.id',
      'message',
      'http.response.status_code'
    ]
  }
)
puts response
GET /my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001/_search
{
  "query": {
    "match": {
      "user.id": "kimchy"
    }
  },
  "_source": ["user.id", "message", "http.response.status_code"]
}

The API returns the following response:

{
  "took": 150,
  "timed_out": false,
  "num_reduce_phases": 4,
  "_shards": {
    "total": 3,
    "successful": 3,
    "failed": 0,
    "skipped": 0
  },
  "_clusters": {
    "total": 3,
    "successful": 3,
    "skipped": 0
  },
  "hits": {
    "total" : {
        "value": 3,
        "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "my-index-000001", 
        "_id": "0",
        "_score": 2,
        "_source": {
          "user": {
            "id": "kimchy"
          },
          "message": "GET /search HTTP/1.1 200 1070000",
          "http": {
            "response":
              {
                "status_code": 200
              }
          }
        }
      },
      {
        "_index": "cluster_one:my-index-000001", 
        "_id": "0",
        "_score": 1,
        "_source": {
          "user": {
            "id": "kimchy"
          },
          "message": "GET /search HTTP/1.1 200 1070000",
          "http": {
            "response":
              {
                "status_code": 200
              }
          }
        }
      },
      {
        "_index": "cluster_two:my-index-000001", 
        "_id": "0",
        "_score": 1,
        "_source": {
          "user": {
            "id": "kimchy"
          },
          "message": "GET /search HTTP/1.1 200 1070000",
          "http": {
            "response":
              {
                "status_code": 200
              }
          }
        }
      }
    ]
  }
}

This document’s _index parameter doesn’t include a cluster name. This means the document came from the local cluster.

This document came from cluster_one.

This document came from cluster_two.

Using async search for cross-cluster search with ccs_minimize_roundtrips=true

edit

Remote clusters can be queried asynchronously using the async search API. Async searches accept a ccs_minimize_roundtrips parameter that defaults to false. See Minimize network roundtrips to learn more about this option.

The following request does an asynchronous search of the my-index-000001 index using ccs_minimize_roundtrips=true against three clusters:

  • The local cluster, with 8 shards
  • Two remote clusters, cluster_one and cluster_two, with 10 shards each 
POST /my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001/_async_search?ccs_minimize_roundtrips=true
{
  "query": {
    "match": {
      "user.id": "kimchy"
    }
  },
  "_source": ["user.id", "message", "http.response.status_code"]
}

The API returns the following response:

{
  "id": "FklQYndoTDJ2VEFlMEVBTzFJMGhJVFEaLVlKYndBWWZSMUdicUc4WVlEaFl4ZzoxNTU=", 
  "is_partial": true,
  "is_running": true,
  "start_time_in_millis": 1685563581380,
  "expiration_time_in_millis": 1685995581380,
  "response": {
    "took": 1020,
    "timed_out": false,
    "num_reduce_phases": 0,
    "_shards": {
      "total": 8,     
      "successful": 0,
      "failed": 0,
      "skipped": 0
    },
    "_clusters": {    
      "total" : 3,
      "successful" : 0,
      "skipped": 0
    },
    "hits": {
      "total" : {
          "value": 0,
          "relation": "eq"
      },
      "max_score": null,
      "hits": []
    }
  }
}

The async search id.

When ccs_minimize_roundtrips = true and searches on the remote clusters are still running, this section indicates the number of shards in scope for the local cluster only. This will be updated to include the total number of shards across all clusters only when the search is completed.

The _clusters section indicates that 3 clusters are in scope for the search and all are currently running (since successful and skipped both equal 0).

If you query the get async search endpoint while the query is still running, you will see an update in the _clusters and _shards section of the response when the local search has finished.

GET /_async_search/FklQYndoTDJ2VEFlMEVBTzFJMGhJVFEaLVlKYndBWWZSMUdicUc4WVlEaFl4ZzoxNTU=

Response:

{
  "id": "FklQYndoTDJ2VEFlMEVBTzFJMGhJVFEaLVlKYndBWWZSMUdicUc4WVlEaFl4ZzoxNTU=",
  "is_partial": true,
  "is_running": true,
  "start_time_in_millis": 1685564911108,
  "expiration_time_in_millis": 1685996911108,
  "response": {
    "took": 11164,
    "timed_out": false,
    "terminated_early": false,
    "_shards": {
      "total": 8,
      "successful": 8,  
      "skipped": 0,
      "failed": 0
    },
    "_clusters": {
      "total": 3,
      "successful": 1,  
      "skipped": 0
    },
    "hits": {
      "total": {
        "value": 167,  
        "relation": "eq"
      },
      "max_score": null,
      "hits": []
    }
  }
}

All the local cluster shards have completed.

The local cluster search has completed, so the "successful" clusters entry is set to 1. The _clusters response section will not be updated for the remote clusters until all remote searches have finished (either successfully or been skipped).

Number of hits from the local cluster search. Final hits are not shown until searches on all clusters have been completed and merged.

After searches on all the clusters have completed, when you query the get async search endpoint, you will see the final status of the _clusters and _shards section as well as the hits.

GET /_async_search/FklQYndoTDJ2VEFlMEVBTzFJMGhJVFEaLVlKYndBWWZSMUdicUc4WVlEaFl4ZzoxNTU=

Response:

{
  "id": "FklQYndoTDJ2VEFlMEVBTzFJMGhJVFEaLVlKYndBWWZSMUdicUc4WVlEaFl4ZzoxNTU=",
  "is_partial": false,
  "is_running": false,
  "start_time_in_millis": 1685564911108,
  "expiration_time_in_millis": 1685996911108,
  "response": {
    "took": 27619,
    "timed_out": false,
    "num_reduce_phases": 4,
    "_shards": {
      "total": 28,
      "successful": 28,  
      "skipped": 0,
      "failed": 0
    },
    "_clusters": {
      "total": 3,
      "successful": 3,   
      "skipped": 0
    },
    "hits": {
      "total": {
        "value": 1067,
        "relation": "eq"
      },
      "max_score": 1.8293576,
      "hits": [...list of hits here...]
    }
  }
}

The _shards section is now updated to show that 28 total shards were searched across all clusters and that all were successful.

The _clusters section shows that searches on all 3 clusters were successful.

Using async search for cross-cluster search with ccs_minimize_roundtrips=false

edit

The _shards and _clusters section of the response behave differently when ccs_minimize_roundtrips is false in asynchronous searches.

Key differences are:

  1. The _shards section total count will be accurate immediately as the total number of shards is gathered from all clusters before the search starts.
  2. The _shards section will be incrementally updated as searches on individual shards complete, so you will get a more accurate accounting of progress during a long-running search compared to when minimize roundtrips is used.
  3. The _cluster section starts off in its final state, showing which clusters were successful or skipped based on gathering shard information before the actual search phase against each shard begins.

Example using the same set up as in the previous section (ccs_minimize_roundtrips=true):

GET /my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001/_async_search?ccs_minimize_roundtrips=false
{
  "query": {
    "match": {
      "user.id": "kimchy"
    }
  },
  "_source": ["user.id", "message", "http.response.status_code"]
}

The API returns the following response if the query takes longer than the wait_for_completion_timeout duration (see Async search).

{
  "id": "FklQYndoTDJ2VEFlMEVBTzFJMGhJVFEaLVlKYndBWWZSMUdicUc4WVlEaFl4ZzoxNTU=",
  "is_partial": true,
  "is_running": true,
  "start_time_in_millis": 1685563581380,
  "expiration_time_in_millis": 1685995581380,
  "response": {
    "took": 1020,
    "timed_out": false,
    "num_reduce_phases": 0,
    "_shards": {
      "total": 28,     
      "successful": 0,
      "failed": 0,
      "skipped": 0
    },
    "_clusters": {
      "total" : 3,
      "successful": 3,  
      "skipped": 0
    },
    "hits": {
      "total" : {
          "value": 0,
          "relation": "eq"
      },
      "max_score": null,
      "hits": []
    }
  }
}

All shards from all clusters in scope for the search are listed here. Watch this section for updates to monitor search progress.

The _clusters section shows that shard information was successfully gathered from all 3 clusters and that all will be searched (none are being skipped).

Optional remote clusters

edit

By default, a cross-cluster search fails if a remote cluster in the request returns an error or is unavailable. Use the skip_unavailable cluster setting to mark a specific remote cluster as optional for cross-cluster search.

If skip_unavailable is true, a cross-cluster search:

  • Skips the remote cluster if its nodes are unavailable during the search. The response’s _cluster.skipped value contains a count of any skipped clusters.
  • Ignores errors returned by the remote cluster, such as errors related to unavailable shards or indices. This can include errors related to search parameters such as allow_no_indices and ignore_unavailable.
  • Ignores the allow_partial_search_results parameter and the related search.default_allow_partial_results cluster setting when searching the remote cluster. This means searches on the remote cluster may return partial results.

The following cluster update settings API request changes cluster_two's skip_unavailable setting to true.

PUT _cluster/settings
{
  "persistent": {
    "cluster.remote.cluster_two.skip_unavailable": true
  }
}

If cluster_two is disconnected or unavailable during a cross-cluster search, Elasticsearch won’t include matching documents from that cluster in the final results.

How cross-cluster search handles network delays

edit

Because cross-cluster search involves sending requests to remote clusters, any network delays can impact search speed. To avoid slow searches, cross-cluster search offers two options for handling network delays:

Minimize network roundtrips

By default, Elasticsearch reduces the number of network roundtrips between remote clusters. This reduces the impact of network delays on search speed. However, Elasticsearch can’t reduce network roundtrips for large search requests, such as those including a scroll or inner hits.

See Minimize network roundtrips to learn how this option works.

Don’t minimize network roundtrips

For search requests that include a scroll or inner hits, Elasticsearch sends multiple outgoing and ingoing requests to each remote cluster. You can also choose this option by setting the ccs_minimize_roundtrips parameter to false. While typically slower, this approach may work well for networks with low latency.

See Don’t minimize network roundtrips to learn how this option works.

The vector tile search API always minimizes network roundtrips and doesn’t include the ccs_minimize_roundtrips parameter.

The Approximate kNN search doesn’t support minimizing network roundtrips, and sets the parameter ccs_minimize_roundtrips to false.

Minimize network roundtrips

edit

Here’s how cross-cluster search works when you minimize network roundtrips.

  1. You send a cross-cluster search request to your local cluster. A coordinating node in that cluster receives and parses the request.

    ccs min roundtrip client request

  2. The coordinating node sends a single search request to each cluster, including the local cluster. Each cluster performs the search request independently, applying its own cluster-level settings to the request.

    ccs min roundtrip cluster search

  3. Each remote cluster sends its search results back to the coordinating node.

    ccs min roundtrip cluster results

  4. After collecting results from each cluster, the coordinating node returns the final results in the cross-cluster search response.

    ccs min roundtrip client response

Don’t minimize network roundtrips

edit

Here’s how cross-cluster search works when you don’t minimize network roundtrips.

  1. You send a cross-cluster search request to your local cluster. A coordinating node in that cluster receives and parses the request.

    ccs min roundtrip client request

  2. The coordinating node sends a "search shards" transport layer request to each remote cluster to have them to perform a "can match" search to determine which shards on each cluster should be searched.

    ccs min roundtrip cluster search

  3. Each remote cluster sends its response back to the coordinating node. This response contains information about the indices and shards the cross-cluster search request will be executed on.

    ccs min roundtrip cluster results

  4. The coordinating node sends a search request to each shard, including those in its own cluster. Each shard performs the search request independently.

    When network roundtrips aren’t minimized, the search is executed as if all data were in the coordinating node’s cluster. We recommend updating cluster-level settings that limit searches, such as action.search.shard_count.limit, pre_filter_shard_size, and max_concurrent_shard_requests, to account for this. If these limits are too low, the search may be rejected.

    ccs dont min roundtrip shard search

  5. Each shard sends its search results back to the coordinating node.

    ccs dont min roundtrip shard results

  6. After collecting results from each cluster, the coordinating node returns the final results in the cross-cluster search response.

    ccs min roundtrip client response

Supported cross-cluster search configurations

edit

In 8.0+, Elastic supports searches from a local cluster to a remote cluster running:

  • The previous minor version.
  • The same version.
  • A newer minor version in the same major version.

Elastic also supports searches from a local cluster running the last minor version of a major version to a remote cluster running any minor version in the following major version. For example, a local 7.17 cluster can search any remote 8.x cluster.

Remote cluster version

Local cluster version

6.8

7.1–7.16

7.17

8.0

8.1

8.2

8.3

8.4

8.5

8.6

8.7

8.8

8.9

6.8

Yes

Yes

Yes

No

No

No

No

No

No

No

No

No

No

7.1–7.16

Yes

Yes

Yes

No

No

No

No

No

No

No

No

No

No

7.17

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

8.0

No

No

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

8.1

No

No

No

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

8.2

No

No

No

No

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

8.3

No

No

No

No

No

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

8.4

No

No

No

No

No

No

Yes

Yes

Yes

Yes

Yes

Yes

Yes

8.5

No

No

No

No

No

No

No

Yes

Yes

Yes

Yes

Yes

Yes

8.6

No

No

No

No

No

No

No

No

Yes

Yes

Yes

Yes

Yes

8.7

No

No

No

No

No

No

No

No

No

Yes

Yes

Yes

Yes

8.8

No

No

No

No

No

No

No

No

No

No

Yes

Yes

Yes

8.9

No

No

No

No

No

No

No

No

No

No

No

Yes

Yes

For the EQL search API, the local and remote clusters must use the same Elasticsearch version if they have versions prior to 7.17.7 (included) or prior to 8.5.1 (included).

For example, a local 8.0 cluster can search a remote 7.17 or any remote 8.x cluster. However, a search from a local 8.0 cluster to a remote 7.16 or 6.8 cluster is not supported.

Only features that exist across all searched clusters are supported. Using a feature with a remote cluster where the feature is not supported will result in undefined behavior.

A cross-cluster search using an unsupported configuration may still work. However, such searches aren’t tested by Elastic, and their behavior isn’t guaranteed.

Ensure cross-cluster search support

edit

The simplest way to ensure your clusters support cross-cluster search is to keep each cluster on the same version of Elasticsearch. If you need to maintain clusters with different versions, you can:

  • Maintain a dedicated cluster for cross-cluster search. Keep this cluster on the earliest version needed to search the other clusters. For example, if you have 7.17 and 8.x clusters, you can maintain a dedicated 7.17 cluster to use as the local cluster for cross-cluster search.
  • Keep each cluster no more than one minor version apart. This lets you use any cluster as the local cluster when running a cross-cluster search.

Cross-cluster search during an upgrade

edit

You can still search a remote cluster while performing a rolling upgrade on the local cluster. However, the local coordinating node’s "upgrade from" and "upgrade to" version must be compatible with the remote cluster’s gateway node.

Running multiple versions of Elasticsearch in the same cluster beyond the duration of an upgrade is not supported.

For more information about upgrades, see Upgrading Elasticsearch.

« Retrieve selected fields from a search Search multiple data streams and indices »