« _routing field _tier field »

› › ›

_source field

`_source` field

The _source field contains the original JSON document body that was passed at index time. The _source field itself is not indexed (and thus is not searchable), but it is stored so that it can be returned when executing fetch requests, like get or search.

If disk usage is important to you, then consider the following options:

Using synthetic _source, which reconstructs source content at the time of retrieval instead of storing it on disk. This shrinks disk usage, at the cost of slower access to _source in Get and Search queries.
Disabling the _source field completely. This shrinks disk usage but disables features that rely on _source.

Synthetic `_source`

edit

Synthetic _source is Generally Available only for TSDB indices (indices that have index.mode set to time_series). For other indices, synthetic _source is in technical preview. Features in technical preview may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.

Though very handy to have around, the source field takes up a significant amount of space on disk. Instead of storing source documents on disk exactly as you send them, Elasticsearch can reconstruct source content on the fly upon retrieval. Enable this by using the value synthetic for the index setting index.mapping.source.mode:

resp = client.indices.create(
    index="idx",
    settings={
        "index": {
            "mapping": {
                "source": {
                    "mode": "synthetic"
                }
            }
        }
    },
)
print(resp)

const response = await client.indices.create({
  index: "idx",
  settings: {
    index: {
      mapping: {
        source: {
          mode: "synthetic",
        },
      },
    },
  },
});
console.log(response);

PUT idx
{
  "settings": {
    "index": {
      "mapping": {
        "source": {
          "mode": "synthetic"
        }
      }
    }
  }
}

While this on the fly reconstruction is generally slower than saving the source documents verbatim and loading them at query time, it saves a lot of storage space. Additional latency can be avoided by not loading _source field in queries when it is not needed.

Supported fields

edit

Synthetic _source is supported by all field types. Depending on implementation details, field types have different properties when used with synthetic _source.

Most field types construct synthetic _source using existing data, most commonly doc_values and stored fields. For these field types, no additional space is needed to store the contents of _source field. Due to the storage layout of doc_values, the generated _source field undergoes modifications compared to the original document.

For all other field types, the original value of the field is stored as is, in the same way as the _source field in non-synthetic mode. In this case there are no modifications and field data in _source is the same as in the original document. Similarly, malformed values of fields that use ignore_malformed or ignore_above need to be stored as is. This approach is less storage efficient since data needed for _source reconstruction is stored in addition to other data required to index the field (like doc_values).

Synthetic `_source` restrictions

edit

Some field types have additional restrictions. These restrictions are documented in the synthetic _source section of the field type’s documentation.

Synthetic `_source` modifications

edit

When synthetic _source is enabled, retrieved documents undergo some modifications compared to the original JSON.

Arrays moved to leaf fields

edit

Synthetic _source arrays are moved to leaves. For example:

resp = client.index(
    index="idx",
    id="1",
    document={
        "foo": [
            {
                "bar": 1
            },
            {
                "bar": 2
            }
        ]
    },
)
print(resp)

response = client.index(
  index: 'idx',
  id: 1,
  body: {
    foo: [
      {
        bar: 1
      },
      {
        bar: 2
      }
    ]
  }
)
puts response

const response = await client.index({
  index: "idx",
  id: 1,
  document: {
    foo: [
      {
        bar: 1,
      },
      {
        bar: 2,
      },
    ],
  },
});
console.log(response);

PUT idx/_doc/1
{
  "foo": [
    {
      "bar": 1
    },
    {
      "bar": 2
    }
  ]
}

Will become:

{
  "foo": {
    "bar": [1, 2]
  }
}

This can cause some arrays to vanish:

resp = client.index(
    index="idx",
    id="1",
    document={
        "foo": [
            {
                "bar": 1
            },
            {
                "baz": 2
            }
        ]
    },
)
print(resp)

response = client.index(
  index: 'idx',
  id: 1,
  body: {
    foo: [
      {
        bar: 1
      },
      {
        baz: 2
      }
    ]
  }
)
puts response

const response = await client.index({
  index: "idx",
  id: 1,
  document: {
    foo: [
      {
        bar: 1,
      },
      {
        baz: 2,
      },
    ],
  },
});
console.log(response);

PUT idx/_doc/1
{
  "foo": [
    {
      "bar": 1
    },
    {
      "baz": 2
    }
  ]
}

Will become:

{
  "foo": {
    "bar": 1,
    "baz": 2
  }
}

Fields named as they are mapped

edit

Synthetic source names fields as they are named in the mapping. When used with dynamic mapping, fields with dots (.) in their names are, by default, interpreted as multiple objects, while dots in field names are preserved within objects that have subobjects disabled. For example:

resp = client.index(
    index="idx",
    id="1",
    document={
        "foo.bar.baz": 1
    },
)
print(resp)

const response = await client.index({
  index: "idx",
  id: 1,
  document: {
    "foo.bar.baz": 1,
  },
});
console.log(response);

PUT idx/_doc/1
{
  "foo.bar.baz": 1
}

Will become:

{
  "foo": {
    "bar": {
      "baz": 1
    }
  }
}

This impacts how source contents can be referenced in scripts. For instance, referencing a script in its original source form will return null:

"script": { "source": """  emit(params._source['foo.bar.baz'])  """ }

Instead, source references need to be in line with the mapping structure:

"script": { "source": """  emit(params._source['foo']['bar']['baz'])  """ }

or simply

"script": { "source": """  emit(params._source.foo.bar.baz)  """ }

The following field APIs are preferable as, in addition to being agnostic to the mapping structure, they make use of docvalues if available and fall back to synthetic source only when needed. This reduces source synthesizing, a slow and costly operation.

"script": { "source": """  emit(field('foo.bar.baz').get(null))   """ }
"script": { "source": """  emit($('foo.bar.baz', null))   """ }

Alphabetical sorting

edit

Synthetic _source fields are sorted alphabetically. The JSON RFC defines objects as "an unordered collection of zero or more name/value pairs" so applications shouldn’t care but without synthetic _source the original ordering is preserved and some applications may, counter to the spec, do something with that ordering.

Representation of ranges

edit

Range field values (e.g. long_range) are always represented as inclusive on both sides with bounds adjusted accordingly. See examples.

Reduced precision of `geo_point` values

edit

Values of geo_point fields are represented in synthetic _source with reduced precision. See examples.

Minimizing source modifications

edit

It is possible to avoid synthetic source modifications for a particular object or field, at extra storage cost. This is controlled through param synthetic_source_keep with the following option:

none: synthetic source diverges from the original source as described above (default).
arrays: arrays of the corresponding field or object preserve the original element ordering and duplicate elements. The synthetic source fragment for such arrays is not guaranteed to match the original source exactly, e.g. array [1, 2, [5], [[4, [3]]], 5] may appear as-is or in an equivalent format like [1, 2, 5, 4, 3, 5]. The exact format may change in the future, in an effort to reduce the storage overhead of this option.
all: the source for both singleton instances and arrays of the corresponding field or object gets recorded. When applied to objects, the source of all sub-objects and sub-fields gets captured. Furthermore, the original source of arrays gets captured and appears in synthetic source with no modifications.

For instance:

resp = client.indices.create(
    index="idx_keep",
    settings={
        "index": {
            "mapping": {
                "source": {
                    "mode": "synthetic"
                }
            }
        }
    },
    mappings={
        "properties": {
            "path": {
                "type": "object",
                "synthetic_source_keep": "all"
            },
            "ids": {
                "type": "integer",
                "synthetic_source_keep": "arrays"
            }
        }
    },
)
print(resp)

const response = await client.indices.create({
  index: "idx_keep",
  settings: {
    index: {
      mapping: {
        source: {
          mode: "synthetic",
        },
      },
    },
  },
  mappings: {
    properties: {
      path: {
        type: "object",
        synthetic_source_keep: "all",
      },
      ids: {
        type: "integer",
        synthetic_source_keep: "arrays",
      },
    },
  },
});
console.log(response);

PUT idx_keep
{
  "settings": {
    "index": {
      "mapping": {
        "source": {
          "mode": "synthetic"
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "path": {
        "type": "object",
        "synthetic_source_keep": "all"
      },
      "ids": {
        "type": "integer",
        "synthetic_source_keep": "arrays"
      }
    }
  }
}

resp = client.index(
    index="idx_keep",
    id="1",
    document={
        "path": {
            "to": [
                {
                    "foo": [
                        3,
                        2,
                        1
                    ]
                },
                {
                    "foo": [
                        30,
                        20,
                        10
                    ]
                }
            ],
            "bar": "baz"
        },
        "ids": [
            200,
            100,
            300,
            100
        ]
    },
)
print(resp)

const response = await client.index({
  index: "idx_keep",
  id: 1,
  document: {
    path: {
      to: [
        {
          foo: [3, 2, 1],
        },
        {
          foo: [30, 20, 10],
        },
      ],
      bar: "baz",
    },
    ids: [200, 100, 300, 100],
  },
});
console.log(response);

PUT idx_keep/_doc/1
{
  "path": {
    "to": [
      { "foo": [3, 2, 1] },
      { "foo": [30, 20, 10] }
    ],
    "bar": "baz"
  },
  "ids": [ 200, 100, 300, 100 ]
}

returns the original source, with no array deduplication and sorting:

{
  "path": {
    "to": [
      { "foo": [3, 2, 1] },
      { "foo": [30, 20, 10] }
    ],
    "bar": "baz"
  },
  "ids": [ 200, 100, 300, 100 ]
}

The option for capturing the source of arrays can be applied at index level, by setting index.mapping.synthetic_source_keep to arrays. This applies to all objects and fields in the index, except for the ones with explicit overrides of synthetic_source_keep set to none. In this case, the storage overhead grows with the number and sizes of arrays present in source of each document, naturally.

Field types that support synthetic source with no storage overhead

edit

The following field types support synthetic source using data from doc_values or <stored-fields, stored fields>>, and require no additional storage space to construct the _source field.

If you enable the ignore_malformed or ignore_above settings, then additional storage is required to store ignored field values for these types.

Disabling the `_source` field

edit

Though very handy to have around, the source field does incur storage overhead within the index. For this reason, it can be disabled as follows:

resp = client.indices.create(
    index="my-index-000001",
    mappings={
        "_source": {
            "enabled": False
        }
    },
)
print(resp)

response = client.indices.create(
  index: 'my-index-000001',
  body: {
    mappings: {
      _source: {
        enabled: false
      }
    }
  }
)
puts response

const response = await client.indices.create({
  index: "my-index-000001",
  mappings: {
    _source: {
      enabled: false,
    },
  },
});
console.log(response);

PUT my-index-000001
{
  "mappings": {
    "_source": {
      "enabled": false
    }
  }
}

Think before disabling the _source field

Users often disable the _source field without thinking about the consequences, and then live to regret it. If the _source field isn’t available then a number of features are not supported:

The update, update_by_query, and reindex APIs.
In the Kibana Discover application, field data will not be displayed.
On the fly highlighting.
The ability to reindex from one Elasticsearch index to another, either to change mappings or analysis, or to upgrade an index to a new major version.
The ability to debug queries or aggregations by viewing the original document used at index time.
Potentially in the future, the ability to repair index corruption automatically.

If disk space is a concern, rather increase the compression level instead of disabling the _source.

Including / Excluding fields from `_source`

edit

An expert-only feature is the ability to prune the contents of the _source field after the document has been indexed, but before the _source field is stored.

Removing fields from the _source has similar downsides to disabling _source, especially the fact that you cannot reindex documents from one Elasticsearch index to another. Consider using source filtering instead.

The includes/excludes parameters (which also accept wildcards) can be used as follows:

resp = client.indices.create(
    index="logs",
    mappings={
        "_source": {
            "includes": [
                "*.count",
                "meta.*"
            ],
            "excludes": [
                "meta.description",
                "meta.other.*"
            ]
        }
    },
)
print(resp)

resp1 = client.index(
    index="logs",
    id="1",
    document={
        "requests": {
            "count": 10,
            "foo": "bar"
        },
        "meta": {
            "name": "Some metric",
            "description": "Some metric description",
            "other": {
                "foo": "one",
                "baz": "two"
            }
        }
    },
)
print(resp1)

resp2 = client.search(
    index="logs",
    query={
        "match": {
            "meta.other.foo": "one"
        }
    },
)
print(resp2)

response = client.indices.create(
  index: 'logs',
  body: {
    mappings: {
      _source: {
        includes: [
          '*.count',
          'meta.*'
        ],
        excludes: [
          'meta.description',
          'meta.other.*'
        ]
      }
    }
  }
)
puts response

response = client.index(
  index: 'logs',
  id: 1,
  body: {
    requests: {
      count: 10,
      foo: 'bar'
    },
    meta: {
      name: 'Some metric',
      description: 'Some metric description',
      other: {
        foo: 'one',
        baz: 'two'
      }
    }
  }
)
puts response

response = client.search(
  index: 'logs',
  body: {
    query: {
      match: {
        'meta.other.foo' => 'one'
      }
    }
  }
)
puts response

const response = await client.indices.create({
  index: "logs",
  mappings: {
    _source: {
      includes: ["*.count", "meta.*"],
      excludes: ["meta.description", "meta.other.*"],
    },
  },
});
console.log(response);

const response1 = await client.index({
  index: "logs",
  id: 1,
  document: {
    requests: {
      count: 10,
      foo: "bar",
    },
    meta: {
      name: "Some metric",
      description: "Some metric description",
      other: {
        foo: "one",
        baz: "two",
      },
    },
  },
});
console.log(response1);

const response2 = await client.search({
  index: "logs",
  query: {
    match: {
      "meta.other.foo": "one",
    },
  },
});
console.log(response2);

PUT logs
{
  "mappings": {
    "_source": {
      "includes": [
        "*.count",
        "meta.*"
      ],
      "excludes": [
        "meta.description",
        "meta.other.*"
      ]
    }
  }
}

PUT logs/_doc/1
{
  "requests": {
    "count": 10,
    "foo": "bar" 
  },
  "meta": {
    "name": "Some metric",
    "description": "Some metric description", 
    "other": {
      "foo": "one", 
      "baz": "two" 
    }
  }
}

GET logs/_search
{
  "query": {
    "match": {
      "meta.other.foo": "one" 
    }
  }
}

	These fields will be removed from the stored `_source` field.
	We can still search on this field, even though it is not in the stored `_source`.

« _routing field _tier field »

_source field

_source field

Synthetic _source

Supported fields

Synthetic _source restrictions

Synthetic _source modifications

Arrays moved to leaf fields

Fields named as they are mapped

Alphabetical sorting

Representation of ranges

Reduced precision of geo_point values

Minimizing source modifications

Field types that support synthetic source with no storage overhead

Disabling the _source field

Including / Excluding fields from _source

`_source` field

Synthetic `_source`

Synthetic `_source` restrictions

Synthetic `_source` modifications

Reduced precision of `geo_point` values

Disabling the `_source` field

Including / Excluding fields from `_source`