IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Keyword type family Numeric field types »
Elastic Docs ›Elasticsearch Guide [8.14] ›Mapping ›Field data types

Nested field type

edit

Nested field type

edit

The nested type is a specialised version of the object data type that allows arrays of objects to be indexed in a way that they can be queried independently of each other.

When ingesting key-value pairs with a large, arbitrary set of keys, you might consider modeling each key-value pair as its own nested document with key and value fields. Instead, consider using the flattened data type, which maps an entire object as a single field and allows for simple searches over its contents. Nested documents and queries are typically expensive, so using the flattened data type for this use case is a better option.

Nested fields have incomplete support in Kibana. While they are visible and searchable in Discover, they cannot be used to build visualizations in Lens.

How arrays of objects are flattened

edit

Elasticsearch has no concept of inner objects. Therefore, it flattens object hierarchies into a simple list of field names and values. For instance, consider the following document:

resp = client.index(
    index="my-index-000001",
    id="1",
    body={
        "group": "fans",
        "user": [
            {"first": "John", "last": "Smith"},
            {"first": "Alice", "last": "White"},
        ],
    },
)
print(resp)
response = client.index(
  index: 'my-index-000001',
  id: 1,
  body: {
    group: 'fans',
    user: [
      {
        first: 'John',
        last: 'Smith'
      },
      {
        first: 'Alice',
        last: 'White'
      }
    ]
  }
)
puts response
res, err := es.Index(
	"my-index-000001",
	strings.NewReader(`{
	  "group": "fans",
	  "user": [
	    {
	      "first": "John",
	      "last": "Smith"
	    },
	    {
	      "first": "Alice",
	      "last": "White"
	    }
	  ]
	}`),
	es.Index.WithDocumentID("1"),
	es.Index.WithPretty(),
)
fmt.Println(res, err)
PUT my-index-000001/_doc/1
{
  "group" : "fans",
  "user" : [ 
    {
      "first" : "John",
      "last" :  "Smith"
    },
    {
      "first" : "Alice",
      "last" :  "White"
    }
  ]
}

The user field is dynamically added as a field of type object.

The previous document would be transformed internally into a document that looks more like this:

{
  "group" :        "fans",
  "user.first" : [ "alice", "john" ],
  "user.last" :  [ "smith", "white" ]
}

The user.first and user.last fields are flattened into multi-value fields, and the association between alice and white is lost. This document would incorrectly match a query for alice AND smith:

resp = client.search(
    index="my-index-000001",
    body={
        "query": {
            "bool": {
                "must": [
                    {"match": {"user.first": "Alice"}},
                    {"match": {"user.last": "Smith"}},
                ]
            }
        }
    },
)
print(resp)
response = client.search(
  index: 'my-index-000001',
  body: {
    query: {
      bool: {
        must: [
          {
            match: {
              'user.first' => 'Alice'
            }
          },
          {
            match: {
              'user.last' => 'Smith'
            }
          }
        ]
      }
    }
  }
)
puts response
res, err := es.Search(
	es.Search.WithIndex("my-index-000001"),
	es.Search.WithBody(strings.NewReader(`{
	  "query": {
	    "bool": {
	      "must": [
	        {
	          "match": {
	            "user.first": "Alice"
	          }
	        },
	        {
	          "match": {
	            "user.last": "Smith"
	          }
	        }
	      ]
	    }
	  }
	}`)),
	es.Search.WithPretty(),
)
fmt.Println(res, err)
GET my-index-000001/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "user.first": "Alice" }},
        { "match": { "user.last":  "Smith" }}
      ]
    }
  }
}

Using nested fields for arrays of objects

edit

If you need to index arrays of objects and to maintain the independence of each object in the array, use the nested data type instead of the object data type.

Internally, nested objects index each object in the array as a separate hidden document, meaning that each nested object can be queried independently of the others with the nested query:

resp = client.indices.create(
    index="my-index-000001",
    body={"mappings": {"properties": {"user": {"type": "nested"}}}},
)
print(resp)

resp = client.index(
    index="my-index-000001",
    id="1",
    body={
        "group": "fans",
        "user": [
            {"first": "John", "last": "Smith"},
            {"first": "Alice", "last": "White"},
        ],
    },
)
print(resp)

resp = client.search(
    index="my-index-000001",
    body={
        "query": {
            "nested": {
                "path": "user",
                "query": {
                    "bool": {
                        "must": [
                            {"match": {"user.first": "Alice"}},
                            {"match": {"user.last": "Smith"}},
                        ]
                    }
                },
            }
        }
    },
)
print(resp)

resp = client.search(
    index="my-index-000001",
    body={
        "query": {
            "nested": {
                "path": "user",
                "query": {
                    "bool": {
                        "must": [
                            {"match": {"user.first": "Alice"}},
                            {"match": {"user.last": "White"}},
                        ]
                    }
                },
                "inner_hits": {
                    "highlight": {"fields": {"user.first": {}}}
                },
            }
        }
    },
)
print(resp)
response = client.indices.create(
  index: 'my-index-000001',
  body: {
    mappings: {
      properties: {
        user: {
          type: 'nested'
        }
      }
    }
  }
)
puts response

response = client.index(
  index: 'my-index-000001',
  id: 1,
  body: {
    group: 'fans',
    user: [
      {
        first: 'John',
        last: 'Smith'
      },
      {
        first: 'Alice',
        last: 'White'
      }
    ]
  }
)
puts response

response = client.search(
  index: 'my-index-000001',
  body: {
    query: {
      nested: {
        path: 'user',
        query: {
          bool: {
            must: [
              {
                match: {
                  'user.first' => 'Alice'
                }
              },
              {
                match: {
                  'user.last' => 'Smith'
                }
              }
            ]
          }
        }
      }
    }
  }
)
puts response

response = client.search(
  index: 'my-index-000001',
  body: {
    query: {
      nested: {
        path: 'user',
        query: {
          bool: {
            must: [
              {
                match: {
                  'user.first' => 'Alice'
                }
              },
              {
                match: {
                  'user.last' => 'White'
                }
              }
            ]
          }
        },
        inner_hits: {
          highlight: {
            fields: {
              'user.first' => {}
            }
          }
        }
      }
    }
  }
)
puts response
{
	res, err := es.Indices.Create(
		"my-index-000001",
		es.Indices.Create.WithBody(strings.NewReader(`{
	  "mappings": {
	    "properties": {
	      "user": {
	        "type": "nested"
	      }
	    }
	  }
	}`)),
	)
	fmt.Println(res, err)
}

{
	res, err := es.Index(
		"my-index-000001",
		strings.NewReader(`{
	  "group": "fans",
	  "user": [
	    {
	      "first": "John",
	      "last": "Smith"
	    },
	    {
	      "first": "Alice",
	      "last": "White"
	    }
	  ]
	}`),
		es.Index.WithDocumentID("1"),
		es.Index.WithPretty(),
	)
	fmt.Println(res, err)
}

{
	res, err := es.Search(
		es.Search.WithIndex("my-index-000001"),
		es.Search.WithBody(strings.NewReader(`{
	  "query": {
	    "nested": {
	      "path": "user",
	      "query": {
	        "bool": {
	          "must": [
	            {
	              "match": {
	                "user.first": "Alice"
	              }
	            },
	            {
	              "match": {
	                "user.last": "Smith"
	              }
	            }
	          ]
	        }
	      }
	    }
	  }
	}`)),
		es.Search.WithPretty(),
	)
	fmt.Println(res, err)
}

{
	res, err := es.Search(
		es.Search.WithIndex("my-index-000001"),
		es.Search.WithBody(strings.NewReader(`{
	  "query": {
	    "nested": {
	      "path": "user",
	      "query": {
	        "bool": {
	          "must": [
	            {
	              "match": {
	                "user.first": "Alice"
	              }
	            },
	            {
	              "match": {
	                "user.last": "White"
	              }
	            }
	          ]
	        }
	      },
	      "inner_hits": {
	        "highlight": {
	          "fields": {
	            "user.first": {}
	          }
	        }
	      }
	    }
	  }
	}`)),
		es.Search.WithPretty(),
	)
	fmt.Println(res, err)
}
PUT my-index-000001
{
  "mappings": {
    "properties": {
      "user": {
        "type": "nested" 
      }
    }
  }
}

PUT my-index-000001/_doc/1
{
  "group" : "fans",
  "user" : [
    {
      "first" : "John",
      "last" :  "Smith"
    },
    {
      "first" : "Alice",
      "last" :  "White"
    }
  ]
}

GET my-index-000001/_search
{
  "query": {
    "nested": {
      "path": "user",
      "query": {
        "bool": {
          "must": [
            { "match": { "user.first": "Alice" }},
            { "match": { "user.last":  "Smith" }} 
          ]
        }
      }
    }
  }
}

GET my-index-000001/_search
{
  "query": {
    "nested": {
      "path": "user",
      "query": {
        "bool": {
          "must": [
            { "match": { "user.first": "Alice" }},
            { "match": { "user.last":  "White" }} 
          ]
        }
      },
      "inner_hits": { 
        "highlight": {
          "fields": {
            "user.first": {}
          }
        }
      }
    }
  }
}

The user field is mapped as type nested instead of type object.

This query doesn’t match because Alice and Smith are not in the same nested object.

This query matches because Alice and White are in the same nested object.

inner_hits allow us to highlight the matching nested documents.

Interacting with nested documents

edit

Nested documents can be:

Because nested documents are indexed as separate documents, they can only be accessed within the scope of the nested query, the nested/reverse_nested aggregations, or nested inner hits.

For instance, if a string field within a nested document has index_options set to offsets to allow use of the postings during the highlighting, these offsets will not be available during the main highlighting phase. Instead, highlighting needs to be performed via nested inner hits. The same consideration applies when loading fields during a search through docvalue_fields or stored_fields.

Parameters for nested fields

edit

The following parameters are accepted by nested fields:

dynamic
(Optional, string) Whether or not new properties should be added dynamically to an existing nested object. Accepts true (default), false and strict.
properties
(Optional, object) The fields within the nested object, which can be of any data type, including nested. New properties may be added to an existing nested object.
include_in_parent
(Optional, Boolean) If true, all fields in the nested object are also added to the parent document as standard (flat) fields. Defaults to false.
include_in_root
(Optional, Boolean) If true, all fields in the nested object are also added to the root document as standard (flat) fields. Defaults to false.

Limits on nested mappings and objects

edit

As described earlier, each nested object is indexed as a separate Lucene document. Continuing with the previous example, if we indexed a single document containing 100 user objects, then 101 Lucene documents would be created: one for the parent document, and one for each nested object. Because of the expense associated with nested mappings, Elasticsearch puts settings in place to guard against performance problems:

index.mapping.nested_fields.limit
The maximum number of distinct nested mappings in an index. The nested type should only be used in special cases, when arrays of objects need to be queried independently of each other. To safeguard against poorly designed mappings, this setting limits the number of unique nested types per index. Default is 50.

In the previous example, the user mapping would count as only 1 towards this limit.

index.mapping.nested_objects.limit
The maximum number of nested JSON objects that a single document can contain across all nested types. This limit helps to prevent out of memory errors when a document contains too many nested objects. Default is 10000.

To illustrate how this setting works, consider adding another nested type called comments to the previous example mapping. For each document, the combined number of user and comment objects it contains must be below the limit.

See Settings to prevent mapping explosion regarding additional settings for preventing mappings explosion.

« Keyword type family Numeric field types »