IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Meta-Fields _field_names field »
Elastic Docs ›Elasticsearch Guide [6.5] ›Mapping ›Meta-Fields

_all field

edit

_all field

edit

Deprecated in 6.0.0.

_all may no longer be enabled for indices created in 6.0+, use a custom field and the mapping copy_to parameter

The _all field is a special catch-all field which concatenates the values of all of the other fields into one big string, using space as a delimiter, which is then analyzed and indexed, but not stored. This means that it can be searched, but not retrieved.

The _all field allows you to search for values in documents without knowing which field contains the value. This makes it a useful option when getting started with a new dataset. For instance:

PUT /my_index
{
  "mapping": {
    "user": {
      "_all": {
        "enabled": true   
      }
    }
  }
}

PUT /my_index/user/1      
{
  "first_name":    "John",
  "last_name":     "Smith",
  "date_of_birth": "1970-10-24"
}

GET /my_index/_search
{
  "query": {
    "match": {
      "_all": "john smith 1970"
    }
  }
}

Enabling the _all field

The _all field will contain the terms: [ "john", "smith", "1970", "10", "24" ]

All values treated as strings

The date_of_birth field in the above example is recognised as a date field and so will index a single term representing 1970-10-24 00:00:00 UTC. The _all field, however, treats all values as strings, so the date value is indexed as the three string terms: "1970", "24", "10".

It is important to note that the _all field combines the original values from each field as a string. It does not combine the terms from each field.

The _all field is just a text field, and accepts the same parameters that other string fields accept, including analyzer, term_vectors, index_options, and store.

The _all field can be useful, especially when exploring new data using simple filtering. However, by concatenating field values into one big string, the _all field loses the distinction between short fields (more relevant) and long fields (less relevant). For use cases where search relevance is important, it is better to query individual fields specifically.

The _all field is not free: it requires extra CPU cycles and uses more disk space. For this reason, it is disabled by default. If needed, it can be enabled.

Using the _all field in queries

edit

The query_string and simple_query_string queries query the _all field by default if it is enabled, unless another field is specified:

GET _search
{
  "query": {
    "query_string": {
      "query": "john smith new york"
    }
  }
}

The same goes for the ?q= parameter in URI search requests (which is rewritten to a query_string query internally):

GET _search?q=john+smith+new+york

Other queries, such as the match and term queries require you to specify the _all field explicitly, as per the first example.

Enabling the _all field

edit

The _all field can be enabled per-type by setting enabled to true:

PUT my_index
{
  "mappings": {
    "type_1": { 
      "properties": {...}
    },
    "type_2": { 
      "_all": {
        "enabled": true
      },
      "properties": {...}
    }
  }
}

The _all field in type_1 is disabled.

The _all field in type_2 is enabled.

If the _all field is enabled, then URI search requests and the query_string and simple_query_string queries can automatically use it for queries (see Using the _all field in queries). You can configure them to use a different field with the index.query.default_field setting:

PUT my_index
{
  "mappings": {
    "my_type": {
      "properties": {
        "content": {
          "type": "text"
        }
      }
    }
  },
  "settings": {
    "index.query.default_field": "content" 
  }
}

The query_string query will default to querying the content field in this index.

Index boosting and the _all field

edit

Individual fields can be boosted at index time, with the boost parameter. The _all field takes these boosts into account:

PUT myindex
{
  "mappings": {
    "mytype": {
      "_all": {"enabled": true},
      "properties": {
        "title": { 
          "type": "text",
          "boost": 2
        },
        "content": { 
          "type": "text"
        }
      }
    }
  }
}

When querying the _all field, words that originated in the title field are twice as relevant as words that originated in the content field.

Using index-time boosting with the _all field has a significant impact on query performance. Usually the better solution is to query fields individually, with optional query time boosting.

Custom _all fields

edit

While there is only a single _all field per index, the copy_to parameter allows the creation of multiple custom _all fields. For instance, first_name and last_name fields can be combined together into the full_name field:

PUT myindex
{
  "mappings": {
    "mytype": {
      "properties": {
        "first_name": {
          "type":    "text",
          "copy_to": "full_name" 
        },
        "last_name": {
          "type":    "text",
          "copy_to": "full_name" 
        },
        "full_name": {
          "type":    "text"
        }
      }
    }
  }
}

PUT myindex/mytype/1
{
  "first_name": "John",
  "last_name": "Smith"
}

GET myindex/_search
{
  "query": {
    "match": {
      "full_name": "John Smith"
    }
  }
}

The first_name and last_name values are copied to the full_name field.

Highlighting and the _all field

edit

A field can only be used for highlighting if the original string value is available, either from the _source field or as a stored field.

The _all field is not present in the _source field and it is not stored or enabled by default, and so cannot be highlighted. There are two options. Either store the _all field or highlight the original fields.

Store the _all field

edit

If store is set to true, then the original field value is retrievable and can be highlighted:

PUT myindex
{
  "mappings": {
    "mytype": {
      "_all": {
        "enabled": true,
        "store": true
      }
    }
  }
}

PUT myindex/mytype/1
{
  "first_name": "John",
  "last_name": "Smith"
}

GET _search
{
  "query": {
    "match": {
      "_all": "John Smith"
    }
  },
  "highlight": {
    "fields": {
      "_all": {}
    }
  }
}

Of course, enabling and storing the _all field will use significantly more disk space and, because it is a combination of other fields, it may result in odd highlighting results.

The _all field also accepts the term_vector and index_options parameters, allowing highlighting to use it.

Highlight original fields

edit

You can query the _all field, but use the original fields for highlighting as follows:

PUT myindex
{
  "mappings": {
    "mytype": {
      "_all": {"enabled": true}
    }
  }
}

PUT myindex/mytype/1
{
  "first_name": "John",
  "last_name": "Smith"
}

GET _search
{
  "query": {
    "match": {
      "_all": "John Smith" 
    }
  },
  "highlight": {
    "fields": {
      "*_name": { 
        "require_field_match": false  
      }
    }
  }
}

The query inspects the _all field to find matching documents.

Highlighting is performed on the two name fields, which are available from the _source.

The query wasn’t run against the name fields, so set require_field_match to false.
« Meta-Fields _field_names field »