IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« _routing field _tier field »
Elastic Docs ›Elasticsearch Guide [8.11] ›Mapping ›Metadata fields

_source field

edit

_source field

edit

The _source field contains the original JSON document body that was passed at index time. The _source field itself is not indexed (and thus is not searchable), but it is stored so that it can be returned when executing fetch requests, like get or search.

If disk usage is important to you then have a look at synthetic _source which shrinks disk usage at the cost of only supporting a subset of mappings and slower fetches or (not recommended) disabling the _source field which also shrinks disk usage but disables many features.

Synthetic _source

edit

Synthetic _source is Generally Available only for TSDB indices (indices that have index.mode set to time_series). For other indices synthetic _source is in technical preview. Features in technical preview may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.

Though very handy to have around, the source field takes up a significant amount of space on disk. Instead of storing source documents on disk exactly as you send them, Elasticsearch can reconstruct source content on the fly upon retrieval. Enable this by setting mode: synthetic in _source:

response = client.indices.create(
  index: 'idx',
  body: {
    mappings: {
      _source: {
        mode: 'synthetic'
      }
    }
  }
)
puts response
PUT idx
{
  "mappings": {
    "_source": {
      "mode": "synthetic"
    }
  }
}

While this on the fly reconstruction is generally slower than saving the source documents verbatim and loading them at query time, it saves a lot of storage space.

Synthetic _source restrictions

edit

There are a couple of restrictions to be aware of:

Synthetic _source modifications

edit

When synthetic _source is enabled, retrieved documents undergo some modifications compared to the original JSON.

Arrays moved to leaf fields
edit

Synthetic _source arrays are moved to leaves. For example:

response = client.index(
  index: 'idx',
  id: 1,
  body: {
    foo: [
      {
        bar: 1
      },
      {
        bar: 2
      }
    ]
  }
)
puts response
PUT idx/_doc/1
{
  "foo": [
    {
      "bar": 1
    },
    {
      "bar": 2
    }
  ]
}

Will become:

{
  "foo": {
    "bar": [1, 2]
  }
}

This can cause some arrays to vanish:

response = client.index(
  index: 'idx',
  id: 1,
  body: {
    foo: [
      {
        bar: 1
      },
      {
        baz: 2
      }
    ]
  }
)
puts response
PUT idx/_doc/1
{
  "foo": [
    {
      "bar": 1
    },
    {
      "baz": 2
    }
  ]
}

Will become:

{
  "foo": {
    "bar": 1,
    "baz": 2
  }
}
Fields named as they are mapped
edit

Synthetic source names fields as they are named in the mapping. When used with dynamic mapping, fields with dots (.) in their names are, by default, interpreted as multiple objects, while dots in field names are preserved within objects that have subobjects disabled. For example:

PUT idx/_doc/1
{
  "foo.bar.baz": 1
}

Will become:

{
  "foo": {
    "bar": {
      "baz": 1
    }
  }
}
Alphabetical sorting
edit

Synthetic _source fields are sorted alphabetically. The JSON RFC defines objects as "an unordered collection of zero or more name/value pairs" so applications shouldn’t care but without synthetic _source the original ordering is preserved and some applications may, counter to the spec, do something with that ordering.

Disabling the _source field

edit

Though very handy to have around, the source field does incur storage overhead within the index. For this reason, it can be disabled as follows:

response = client.indices.create(
  index: 'my-index-000001',
  body: {
    mappings: {
      _source: {
        enabled: false
      }
    }
  }
)
puts response
PUT my-index-000001
{
  "mappings": {
    "_source": {
      "enabled": false
    }
  }
}

Think before disabling the _source field

Users often disable the _source field without thinking about the consequences, and then live to regret it. If the _source field isn’t available then a number of features are not supported:

  • The update, update_by_query, and reindex APIs.
  • On the fly highlighting.
  • The ability to reindex from one Elasticsearch index to another, either to change mappings or analysis, or to upgrade an index to a new major version.
  • The ability to debug queries or aggregations by viewing the original document used at index time.
  • Potentially in the future, the ability to repair index corruption automatically.

If disk space is a concern, rather increase the compression level instead of disabling the _source.

Including / Excluding fields from _source

edit

An expert-only feature is the ability to prune the contents of the _source field after the document has been indexed, but before the _source field is stored.

Removing fields from the _source has similar downsides to disabling _source, especially the fact that you cannot reindex documents from one Elasticsearch index to another. Consider using source filtering instead.

The includes/excludes parameters (which also accept wildcards) can be used as follows:

response = client.indices.create(
  index: 'logs',
  body: {
    mappings: {
      _source: {
        includes: [
          '*.count',
          'meta.*'
        ],
        excludes: [
          'meta.description',
          'meta.other.*'
        ]
      }
    }
  }
)
puts response

response = client.index(
  index: 'logs',
  id: 1,
  body: {
    requests: {
      count: 10,
      foo: 'bar'
    },
    meta: {
      name: 'Some metric',
      description: 'Some metric description',
      other: {
        foo: 'one',
        baz: 'two'
      }
    }
  }
)
puts response

response = client.search(
  index: 'logs',
  body: {
    query: {
      match: {
        'meta.other.foo' => 'one'
      }
    }
  }
)
puts response
PUT logs
{
  "mappings": {
    "_source": {
      "includes": [
        "*.count",
        "meta.*"
      ],
      "excludes": [
        "meta.description",
        "meta.other.*"
      ]
    }
  }
}

PUT logs/_doc/1
{
  "requests": {
    "count": 10,
    "foo": "bar" 
  },
  "meta": {
    "name": "Some metric",
    "description": "Some metric description", 
    "other": {
      "foo": "one", 
      "baz": "two" 
    }
  }
}

GET logs/_search
{
  "query": {
    "match": {
      "meta.other.foo": "one" 
    }
  }
}

These fields will be removed from the stored _source field.

We can still search on this field, even though it is not in the stored _source.
« _routing field _tier field »