_source field
edit_source
field
editThe _source
field contains the original JSON document body that was passed
at index time. The _source
field itself is not indexed (and thus is not
searchable), but it is stored so that it can be returned when executing
fetch requests, like get or search.
If disk usage is important to you then have a look at
synthetic _source
which shrinks disk usage at the cost of
only supporting a subset of mappings and slower fetches or (not recommended)
disabling the _source
field which also shrinks disk
usage but disables many features.
Synthetic _source
[preview]
This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.
editThough very handy to have around, the source field takes up a significant amount
of space on disk. Instead of storing source documents on disk exactly as you
send them, Elasticsearch can reconstruct source content on the fly upon retrieval.
Enable this by setting mode: synthetic
in _source
:
PUT idx { "mappings": { "_source": { "mode": "synthetic" } } }
While this on the fly reconstruction is generally slower than saving the source documents verbatim and loading them at query time, it saves a lot of storage space. There are a couple of restrictions to be aware of:
-
When you retrieve synthetic
_source
content it undergoes minor modifications compared to the original JSON. -
The
params._source
is unavailable in scripts. Instead use thedoc
API or thefield
. -
Synthetic
_source
can be used with indices that contain only these field types:
Runtime fields cannot, at this stage, use synthetic _source
.
Synthetic _source
modifications
editWhen synthetic _source
is enabled, retrieved documents undergo some
modifications compared to the original JSON.
Arrays moved to leaf fields
editSynthetic _source
arrays are moved to leaves. For example:
PUT idx/_doc/1 { "foo": [ { "bar": 1 }, { "bar": 2 } ] }
Will become:
{ "foo": { "bar": [1, 2] } }
Fields named as they are mapped
editSynthetic source names fields as they are named in the mapping. When used
with dynamic mapping, fields with dots (.
) in their names are, by
default, interpreted as multiple objects, while dots in field names are
preserved within objects that have subobjects
disabled. For example:
PUT idx/_doc/1 { "foo.bar.baz": 1 }
Will become:
{ "foo": { "bar": { "baz": 1 } } }
Alphabetical sorting
editSynthetic _source
fields are sorted alphabetically. The
JSON RFC defines objects as
"an unordered collection of zero or more name/value pairs" so applications
shouldn’t care but without synthetic _source
the original ordering is
preserved and some applications may, counter to the spec, do something with
that ordering.
Disabling the _source
field
editThough very handy to have around, the source field does incur storage overhead within the index. For this reason, it can be disabled as follows:
PUT my-index-000001 { "mappings": { "_source": { "enabled": false } } }
Think before disabling the _source
field
Users often disable the _source
field without thinking about the
consequences, and then live to regret it. If the _source
field isn’t
available then a number of features are not supported:
-
The
update
,update_by_query
, andreindex
APIs. - On the fly highlighting.
- The ability to reindex from one Elasticsearch index to another, either to change mappings or analysis, or to upgrade an index to a new major version.
- The ability to debug queries or aggregations by viewing the original document used at index time.
- Potentially in the future, the ability to repair index corruption automatically.
If disk space is a concern, rather increase the
compression level instead of disabling the _source
.
Including / Excluding fields from _source
editAn expert-only feature is the ability to prune the contents of the _source
field after the document has been indexed, but before the _source
field is
stored.
Removing fields from the _source
has similar downsides to disabling
_source
, especially the fact that you cannot reindex documents from one
Elasticsearch index to another. Consider using
source filtering instead.
The includes
/excludes
parameters (which also accept wildcards) can be used
as follows:
PUT logs { "mappings": { "_source": { "includes": [ "*.count", "meta.*" ], "excludes": [ "meta.description", "meta.other.*" ] } } } PUT logs/_doc/1 { "requests": { "count": 10, "foo": "bar" }, "meta": { "name": "Some metric", "description": "Some metric description", "other": { "foo": "one", "baz": "two" } } } GET logs/_search { "query": { "match": { "meta.other.foo": "one" } } }