Cross-cluster search with application data
editCross-cluster search with application data
editElastic APM utilizes Elasticsearch’s cross-cluster search functionality. Cross-cluster search lets you run a single search request against one or more remote clusters — making it easy to search APM data across multiple sources. This means you can also have deployments per data type, making sizing and scaling more predictable, and allowing for better performance while managing multiple observability use cases.
Step 1. Set up remote clusters.
If you’re using the Hosted Elasticsearch Service, see Enable cross-cluster search.
You can add remote clusters directly in Kibana, under Management → Elasticsearch → Remote clusters. All you need is a name for the remote cluster and the seed node(s). Remember the names of your remote clusters, you’ll need them in step two. See managing remote clusters for detailed information on the setup process.
Alternatively, you can configure remote clusters
in Elasticsearch’s elasticsearch.yml file.
Step 2. Edit the default Applications UI data views.
Applications UI data views determine which clusters and indices to display data from.
Data views follow this convention: <cluster-name>:<index-pattern>.
To display data from all remote clusters and the local cluster,
duplicate and prepend the defaults with *:.
For example, the default data view for Error indices is logs-apm*,apm*.
To add all remote clusters, change this to *:logs-apm*,*:apm*,logs-apm*,apm*
You can also specify certain clusters to display data from, for example,
cluster-one:logs-apm*,cluster-one:apm*,logs-apm*,apm*.
There are two ways to edit the default data view:
-
In the Applications UI — Navigate to Applications → Settings → Indices, and change all
xpack.apm.indices.*values to include remote clusters. -
In
kibana.yml— Update thexpack.apm.indices.*configuration values to include remote clusters.
Exclude data tiers from search
In a cross-cluster search (CCS) environment, it’s possible for different clusters to serve different data tiers in responses. If one of the requested clusters responds slowly, it can cause a timeout at the proxy after 320 seconds. This results in 502 Bad Gateway server error responses presented as failure toast messages in the UI, and no data loaded.
To prevent this, you can exclude data tiers that might slow down responses from search: the data_frozen and data_cold tiers. To exclude data tiers from search in the APM UI:
- In Kibana, go to Stack Management → Advanced Settings.
- In the Observability section, update the Excluded data tiers from search option with a list of data tiers.