WARNING: Version 2.3 of Elasticsearch has passed its EOL date.

This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.

« Meta-Fields _field_names field »
Elastic Docs ›Elasticsearch Guide [2.3] ›Mapping ›Meta-Fields

_all field

edit

_all field

edit

The _all field is a special catch-all field which concatenates the values of all of the other fields into one big string, using space as a delimiter, which is then analyzed and indexed, but not stored. This means that it can be searched, but not retrieved.

The _all field allows you to search for values in documents without knowing which field contains the value. This makes it a useful option when getting started with a new dataset. For instance:

PUT my_index/user/1 
{
  "first_name":    "John",
  "last_name":     "Smith",
  "date_of_birth": "1970-10-24"
}

GET my_index/_search
{
  "query": {
    "match": {
      "_all": "john smith 1970"
    }
  }
}

The _all field will contain the terms: [ "john", "smith", "1970", "10", "24" ]

All values treated as strings

The date_of_birth field in the above example is recognised as a date field and so will index a single term representing 1970-10-24 00:00:00 UTC. The _all field, however, treats all values as strings, so the date value is indexed as the three string terms: "1970", "24", "10".

It is important to note that the _all field combines the original values from each field as a string. It does not combine the terms from each field.

The _all field is just a string field, and accepts the same parameters that other string fields accept, including analyzer, term_vectors, index_options, and store.

The _all field can be useful, especially when exploring new data using simple filtering. However, by concatenating field values into one big string, the _all field loses the distinction between short fields (more relevant) and long fields (less relevant). For use cases where search relevance is important, it is better to query individual fields specifically.

The _all field is not free: it requires extra CPU cycles and uses more disk space. If not needed, it can be completely disabled or customised on a per-field basis.

Using the _all field in queries

edit

The query_string and simple_query_string queries query the _all field by default, unless another field is specified:

GET _search
{
  "query": {
    "query_string": {
      "query": "john smith 1970"
    }
  }
}

The same goes for the ?q= parameter in URI search requests (which is rewritten to a query_string query internally):

GET _search?q=john+smith+1970

Other queries, such as the match and term queries require you to specify the _all field explicitly, as per the first example.

Disabling the _all field

edit

The _all field can be completely disabled per-type by setting enabled to false:

PUT my_index
{
  "mappings": {
    "type_1": { 
      "properties": {...}
    },
    "type_2": { 
      "_all": {
        "enabled": false
      },
      "properties": {...}
    }
  }
}

The _all field in type_1 is enabled.

The _all field in type_2 is completely disabled.

If the _all field is disabled, then URI search requests and the query_string and simple_query_string queries will not be able to use it for queries (see Using the _all field in queries). You can configure them to use a different field with the index.query.default_field setting:

PUT my_index
{
  "mappings": {
    "my_type": {
      "_all": {
        "enabled": false 
      },
      "properties": {
        "content": {
          "type": "string"
        }
      }
    }
  },
  "settings": {
    "index.query.default_field": "content" 
  },
}

The _all field is disabled for the my_type type.

The query_string query will default to querying the content field in this index.

Excluding fields from _all

edit

Individual fields can be included or excluded from the _all field with the include_in_all setting.

Index boosting and the _all field

edit

Individual fields can be boosted at index time, with the boost parameter. The _all field takes these boosts into account:

PUT myindex
{
  "mappings": {
    "mytype": {
      "properties": {
        "title": { 
          "type": "string",
          "boost": 2
        },
        "content": { 
          "type": "string"
        }
      }
    }
  }
}

When querying the _all field, words that originated in the title field are twice as relevant as words that originated in the content field.

Using index-time boosting with the _all field has a significant impact on query performance. Usually the better solution is to query fields individually, with optional query time boosting.

Custom _all fields

edit

While there is only a single _all field per index, the copy_to parameter allows the creation of multiple custom _all fields. For instance, first_name and last_name fields can be combined together into the full_name field:

PUT myindex
{
  "mappings": {
    "mytype": {
      "properties": {
        "first_name": {
          "type":    "string",
          "copy_to": "full_name" 
        },
        "last_name": {
          "type":    "string",
          "copy_to": "full_name" 
        },
        "full_name": {
          "type":    "string"
        }
      }
    }
  }
}

PUT myindex/mytype/1
{
  "first_name": "John",
  "last_name": "Smith"
}

GET myindex/_search
{
  "query": {
    "match": {
      "full_name": "John Smith"
    }
  }
}

The first_name and last_name values are copied to the full_name field.

Highlighting and the _all field

edit

A field can only be used for highlighting if the original string value is available, either from the _source field or as a stored field.

The _all field is not present in the _source field and it is not stored by default, and so cannot be highlighted. There are two options. Either store the _all field or highlight the original fields.

Store the _all field

edit

If store is set to true, then the original field value is retrievable and can be highlighted:

PUT myindex
{
  "mappings": {
    "mytype": {
      "_all": {
        "store": true
      }
    }
  }
}

PUT myindex/mytype/1
{
  "first_name": "John",
  "last_name": "Smith"
}

GET _search
{
  "query": {
    "match": {
      "_all": "John Smith"
    }
  },
  "highlight": {
    "fields": {
      "_all": {}
    }
  }
}

Of course, storing the _all field will use significantly more disk space and, because it is a combination of other fields, it may result in odd highlighting results.

The _all field also accepts the term_vector and index_options parameters, allowing the use of the fast vector highlighter and the postings highlighter.

Highlight original fields

edit

You can query the _all field, but use the original fields for highlighting as follows:

PUT myindex
{
  "mappings": {
    "mytype": {
      "_all": {}
    }
  }
}

PUT myindex/mytype/1
{
  "first_name": "John",
  "last_name": "Smith"
}

GET _search
{
  "query": {
    "match": {
      "_all": "John Smith" 
    }
  },
  "highlight": {
    "fields": {
      "*_name": { 
        "require_field_match": "false"  
      }
    }
  }
}

The query inspects the _all field to find matching documents.

Highlighting is performed on the two name fields, which are available from the _source.

The query wasn’t run against the name fields, so set require_field_match to false.
« Meta-Fields _field_names field »