Nested datatype
editNested datatype
editThe nested
type is a specialised version of the object
datatype
that allows arrays of objects to be indexed in a way that they can be queried
independently of each other.
When ingesting key-value pairs with a large, arbitrary set of keys, you might consider modeling each key-value pair as its own nested document with key
and value
fields. Instead, consider using the flattened datatype, which maps an entire object as a single field and allows for simple searches over its contents.
Nested documents and queries are typically expensive, so using the flattened
datatype for this use case is a better option.
How arrays of objects are flattened
editElasticsearch has no concept of inner objects. Therefore, it flattens object hierarchies into a simple list of field names and values. For instance, consider the following document:
PUT my_index/_doc/1 { "group" : "fans", "user" : [ { "first" : "John", "last" : "Smith" }, { "first" : "Alice", "last" : "White" } ] }
The previous document would be transformed internally into a document that looks more like this:
{ "group" : "fans", "user.first" : [ "alice", "john" ], "user.last" : [ "smith", "white" ] }
The user.first
and user.last
fields are flattened into multi-value fields,
and the association between alice
and white
is lost. This document would
incorrectly match a query for alice AND smith
:
GET my_index/_search { "query": { "bool": { "must": [ { "match": { "user.first": "Alice" }}, { "match": { "user.last": "Smith" }} ] } } }
Using nested
fields for arrays of objects
editIf you need to index arrays of objects and to maintain the independence of
each object in the array, use the nested
datatype instead of the
object
datatype.
Internally, nested objects index each object in
the array as a separate hidden document, meaning that each nested object can be
queried independently of the others with the nested
query:
PUT my_index { "mappings": { "properties": { "user": { "type": "nested" } } } } PUT my_index/_doc/1 { "group" : "fans", "user" : [ { "first" : "John", "last" : "Smith" }, { "first" : "Alice", "last" : "White" } ] } GET my_index/_search { "query": { "nested": { "path": "user", "query": { "bool": { "must": [ { "match": { "user.first": "Alice" }}, { "match": { "user.last": "Smith" }} ] } } } } } GET my_index/_search { "query": { "nested": { "path": "user", "query": { "bool": { "must": [ { "match": { "user.first": "Alice" }}, { "match": { "user.last": "White" }} ] } }, "inner_hits": { "highlight": { "fields": { "user.first": {} } } } } } }
Interacting with nested
documents
editNested documents can be:
-
queried with the
nested
query. -
analyzed with the
nested
andreverse_nested
aggregations. - sorted with nested sorting.
- retrieved and highlighted with nested inner hits.
Because nested documents are indexed as separate documents, they can only be
accessed within the scope of the nested
query, the
nested
/reverse_nested
aggregations, or nested inner hits.
For instance, if a string field within a nested document has
index_options
set to offsets
to allow use of the postings
during the highlighting, these offsets will not be available during the main highlighting
phase. Instead, highlighting needs to be performed via
nested inner hits. The same consideration applies when loading
fields during a search through docvalue_fields
or stored_fields
.
Parameters for nested
fields
editThe following parameters are accepted by nested
fields:
-
dynamic
-
(Optional, string)
Whether or not new
properties
should be added dynamically to an existing nested object. Acceptstrue
(default),false
andstrict
. -
properties
-
(Optional, object)
The fields within the nested object, which can be of any
datatype, including
nested
. New properties may be added to an existing nested object. -
include_in_parent
-
(Optional, boolean)
If
true
, all fields in the nested object are also added to the parent document as standard (flat) fields. Defaults tofalse
. -
include_in_root
-
(Optional, boolean)
If
true
, all fields in the nested object are also added to the root document as standard (flat) fields. Defaults tofalse
.
Limits on nested
mappings and objects
editAs described earlier, each nested object is indexed as a separate Lucene document.
Continuing with the previous example, if we indexed a single document containing 100 user
objects,
then 101 Lucene documents would be created: one for the parent document, and one for each
nested object. Because of the expense associated with nested
mappings, Elasticsearch puts
settings in place to guard against performance problems:
-
index.mapping.nested_fields.limit
-
The maximum number of distinct
nested
mappings in an index. Thenested
type should only be used in special cases, when arrays of objects need to be queried independently of each other. To safeguard against poorly designed mappings, this setting limits the number of uniquenested
types per index. Default is50
.
In the previous example, the user
mapping would count as only 1 towards this limit.
-
index.mapping.nested_objects.limit
-
The maximum number of nested JSON objects that a single document can contain across all
nested
types. This limit helps to prevent out of memory errors when a document contains too many nested objects. Default is10000
.
To illustrate how this setting works, consider adding another nested
type called comments
to the previous example mapping. For each document, the combined number of user
and comment
objects it contains must be below the limit.
See Settings to prevent mappings explosion regarding additional settings for preventing mappings explosion.