Overview
Elasticsearch offers two types of index templates: legacy and composable. Composable templates introduced in Elasticsearch 7.8 that are set to replace legacy templates, both can still be used in Elasticsearch 8.
This article explores the differences between these templates and how they interact. In particular, we will focus on how you can detect which template will be used when you are creating an index. Let's get started by looking at how to create the different types of index templates.
Index templates in Elasticsearch
Legacy templates can be created using the following API:
PUT _template/t1
{
"order": 1,
"index_patterns": [...],
"mappings": {...},
"settings": {...},
"alias": {...}
}Composable templates can be created using this API:
PUT _index_template/ct1
{
"priority": 1,
"index_patterns": [...],
"template": {
"mappings": {...},
"settings": {...},
"alias": {...}
}
}Component templates are a third type, which are typically used for managing multiple templates with similar structures. For example, if you need to create hundreds of templates with similar structures, you can create a component template with the common settings, mappings, and aliases, and then include it in your index templates. Component templates can be created using this API:
PUT _component_template/template_1
{
"template": {
"mappings": {...},
"settings": {...},
"alias": {...}
}
}Important!
When both legacy and composable templates exist and they match with the same index pattern, the legacy template will be ignored. If two composable templates point to the same index pattern, the template with the highest priority will be used. If two legacy templates point to the same index pattern, the templates are merged, with higher-order templates overriding lower-order ones. If the order is the same, the templates are sorted by name and merged accordingly.
Determining which template an index will use when it is created
To determine which template an index will use upon creation, you can use the _simulate_index API. This API will return the template that will be used, along with any overlapping templates. However, if no composable templates are present, the API will return an empty body. In that case, you can create a dummy index and check the logs of the elected master node to determine which template will be used.
What happens if you have both legacy templates and composable templates?
As noted above, if you have both legacy and composable templates, the legacy template will be ignored as if it did not exist.
PUT _template/t1
{
"index_patterns": [
"test_index-*"
],
"mappings": {
"properties": {
"field_1": {
"type": "integer"
},
"field_2": {
"type": "integer"
}
}
}
}In such a case, you would get a warning message like the following when you run the command:
legacy template [t1] has index patterns [test_index-] matching patterns from existing composable templates [ct1] with patterns (ct1 => [test_index-]); this template [t1] may be ignored in favor of a composable template at index creation time
PUT _index_template/ct1
{
"index_patterns": [
"test_index-*"
],
"template": {
"mappings": {
"properties": {
"field_1": {
"type": "integer"
}
}
},
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
}
}
}If a newly created composable template matches an existing legacy template with the same or includes an index pattern you will get a warning message like the following:
index template [ct1] has index patterns [test_index-] matching patterns from existing older templates [t1] with patterns (t1 => [test_index-]); this template [ct1] will take precedence during new index creation
POST _index_template/_simulate_index/test_index-1
#response:
{
"template": {
"settings": {
"index": {
"number_of_shards": "1",
"number_of_replicas": "0",
"routing": {
"allocation": {
"include": {
"_tier_preference": "data_content"
}
}
}
}
},
"mappings": {
"properties": {
"field_1": {
"type": "integer"
}
}
},
"aliases": {}
},
"overlapping": [
{
"name": "t1",
"index_patterns": [
"test_index-*"
]
}
]
}Use this command if you want to test it:
PUT test_index-1
GET test_index-1Notes from real life scenario
Conflicts can be annoying, and they can crash the application. Imagine that you have logstash-dev-*, logstash-prd-*, logstash-stg-* legacy templates that all working fine. If someone adds a single composable template that include index pattern like a logstash-* all legacy templates will be ignored, the fields types can be change and finally it can break the application. Because of that, it’s recommended to switch from legacy to composable templates if you are using Elasticsearch 7 and onwards.
Another good point to keep in mind is that if you run the Logstash in Elasticsearch 8 or higher, Logstash will add it's template as composable template by default. Because manage_template is set to true by default and Logstash template_api is set tocomposable for Elasticsearch 8 and onwards. It will create a Logstash composable template with logstash-* index pattern if the composable template does not exist. Yes, it will ignore all legacy templates covering logstash-* and overlap them.
Template Overlapping
1. What happens if you have two composable templates that point to the same index pattern?
As previously mentioned, if you have two composable templates that point to the same index pattern, the composable template with the highest priority will take precedence.
PUT _index_template/ct1
{
"priority": 0,
"index_patterns": [
"test_index-*"
],
"template": {
"mappings": {
"properties": {
"field_1": {
"type": "integer"
}
}
},
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
}
}
}PUT _index_template/ct2
{
"priority": 1,
"index_patterns": [
"test_index-*"
],
"template": {
"mappings": {
"properties": {
"field_1": {
"type": "keyword"
},
"field_2": {
"type": "integer"
}
}
},
"settings": {
"number_of_shards": 2,
"number_of_replicas": 0
}
}
}POST _index_template/_simulate_index/test_index-1
#response:
{
"template": {
"settings": {
"index": {
"number_of_shards": "2",
"number_of_replicas": "0",
"routing": {
"allocation": {
"include": {
"_tier_preference": "data_content"
}
}
}
}
},
"mappings": {
"properties": {
"field_1": {
"type": "keyword"
},
"field_2": {
"type": "integer"
}
}
},
"aliases": {}
},
"overlapping": [
{
"name": "ct1",
"index_patterns": [
"test_index-*"
]
}
]
}In this example, you have two templates—ct1 and ct2—both targeting the same index pattern test_index-. However, ct2 has a higher priority (1) than ct1 (0). Therefore, when you create an index that matches the pattern test_index-, the settings and mappings defined in ct2 will be applied before ct1. If there are the same settings in the ct1 and ct2 templates, the ct2 template will overwrite.
2. What happens if you have two legacy templates that point to the same index pattern?
As highlighted above, if you have multiple templates that point to the same index pattern, the templates with lower-order values are merged first. Templates with higher-order values are merged later, overriding templates with lower values.
If two legacy templates have the same order value, they will be sorted by name. For example, in a case with [t2, t1], t1 would be merged first, t2 would be merged later, and t2 would override t1 if there are any same mapping/settings/aliases.
PUT _template/t1
{
"index_patterns": ["test_index-*"],
"mappings": {
"properties": {
"field_1": {
"type": "integer"
}
}
},
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
}
}
PUT _template/t2
{
"index_patterns": [
"test_index-*"
],
"mappings": {
"properties": {
"field_1": {
"type": "geo_point"
},
"field_2": {
"type": "long"
}
}
},
"settings": {
"number_of_shards": 2,
"number_of_replicas": 0
}
}POST _index_template/_simulate_index/test_index-1
#response
{}Unfortunately, if you don't have composable templates, this API call responds with an empty body. So how you can check that?
The answer is to create a dummy index and check the Elasticsearch elected-master logs.
PUT test_index-test2023-11-14 14:14:27 {"@timestamp":"2023-11-14T11:14:27.535Z", "log.level": "WARN", "data_stream.dataset":"deprecation.elasticsearch","data_stream.namespace":"default","data_stream.type":"logs","elasticsearch.event.category":"templates","event.code":"index_template_multiple_match","message":"index [test_index-1] matches multiple legacy templates [t1, t2], composable templates will only match a single template" , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"deprecation.elasticsearch","process.thread.name":"elasticsearch[elasticsearch][masterService#updateTask][T#3]","log.logger":"org.elasticsearch.deprecation.cluster.metadata.MetadataCreateIndexService","trace.id":"85e0a432ec11e2f2d3c7883f510376ac","elasticsearch.cluster.uuid":"Jc-a46VUSjOwuxWmbnSDZQ","elasticsearch.node.id":"MTX1x5-OTlWhiGa9lwUJPw","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"elasticsearch-cluster1"}
2023-11-14 14:14:27 {"@timestamp":"2023-11-14T11:14:27.605Z", "log.level": "INFO", "message":"[test_index-1] creating index, cause [api], templates [t2, t1], shards [2]/[0]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch][masterService#updateTask][T#3]","log.logger":"org.elasticsearch.cluster.metadata.MetadataCreateIndexService","trace.id":"85e0a432ec11e2f2d3c7883f510376ac","elasticsearch.cluster.uuid":"Jc-a46VUSjOwuxWmbnSDZQ","elasticsearch.node.id":"MTX1x5-OTlWhiGa9lwUJPw","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"elasticsearch-cluster1"}From the logs, we can see that "[test_index-1] creating index, cause [api], templates [t2, t1]".
GET _cat/templates/t*?v
name index_patterns order version composed_of
t2 [test_index-*] 0
t1 [test_index-*] 0As you can see, both legacy templates t1 and t2 have the same order; so, which one will override the other?
In this case, Elasticsearch will sort the legacy index templates according to their names and apply them. Both templates will be applied, and the first one in the list, which is t2 in this example, will override the template.
Bonus: What happens if you have two legacy templates that point to the same index pattern with same field name but inappropriate type?
Attempting to merge attributes within the legacy template, regardless of the order, is likely to fail since field definitions should remain atomic. This issue is a primary motivator for introducing the new composable templates. See the below example. We thank Philipp Krenn for adding these comments to the article.
PUT _template/test1
{
"order": 3,
"index_patterns": [
"test-*"
],
"mappings": {
"properties": {
"my_field": {
"type": "integer",
"ignore_malformed": true
}
}
}
}PUT _template/test2
{
"order": 2,
"index_patterns": [
"test-*"
],
"mappings": {
"properties": {
"my_field": {
"type": "keyword",
"ignore_above": 1024
}
}
}
}PUT test-1/_doc/1
{
"my_field": "a string..."
}#response:
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "unknown parameter [ignore_above] on mapper [my_field] of type [integer]"
}
],
"type": "mapper_parsing_exception",
"reason": "Failed to parse mapping: unknown parameter [ignore_above] on mapper [my_field] of type [integer]",
"caused_by": {
"type": "mapper_parsing_exception",
"reason": "unknown parameter [ignore_above] on mapper [my_field] of type [integer]"
}
},
"status": 400
}Notes and good things to know
- Using legacy templates in the same order can cause a lot of confusion. That’s why it's recommended to add order to the template.
- Templates with lower-order values are merged first. Templates with higher order values are merged later, overriding templates with lower values.
- You can't create two composable templates with the same priority.
{
"type": "illegal_argument_exception",
"reason": "index template [ct2] has index patterns [test_index-*] matching patterns from existing templates [ct1] with patterns (ct1 => [test_index-*]) that have the same priority [0], multiple index templates may not match during index creation, please use a different priority"
}Conclusion
In conclusion, understanding how Elasticsearch's index templates work is crucial for effective index management. By knowing how to determine which template an index will use upon creation, you can ensure that your indices are created with the correct settings, mappings, and aliases.
Resources
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-template.html https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates-v1.html https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-simulate-index.html
Elasticsearch is packed with new features to help you build the best search solutions for your use case. Start a free trial now.
Want to get Elastic certified? Find out when the next Elasticsearch Engineer training is running!
Related content
November 13, 2024
Faster integration tests with real Elasticsearch
Learn how to make your automated integration tests for Elasticsearch faster using various techniques for data initialization and performance improvement.
November 6, 2023
A conversational search experience for retail: Elasticsearch Relevance Engine with Google Cloud’s generative AI
This blog presents a new search experience for retailers using generative AI with Vertex AI and Elasticsearch.
November 12, 2024
How to use Hybrid search for an e-commerce product catalog
Learn how to use Hybrid search to build an e commerce product catalog, using faceting, promotions, personalization and behavioral analytics.
November 7, 2024
Building a recipe search with Elasticsearch
Learn how to build recipe search based on semantic search with Elasticsearch.
November 5, 2024
How to ingest data from AWS S3 into Elastic Cloud - Part 3: Elastic S3 Connector
Learn about different options to ingest data from AWS S3 into Elastic Cloud. This time we will focus on Elastic S3 Connector.