Terms aggregation

By default, the terms aggregation returns the top ten terms with the most documents. Use the size parameter to return more terms, up to the search.max_buckets limit.

If your data contains 100 or 1000 unique terms, you can increase the size of the terms aggregation to return them all. If you have more unique terms and you need them all, use the composite aggregation instead.

Larger values of size use more memory to compute and, push the whole aggregation close to the max_buckets limit. You’ll know you’ve gone too large if the request fails with a message about max_buckets.

To get more accurate results, the terms agg fetches more than the top size terms from each shard. It fetches the top shard_size terms, which defaults to size * 1.5 + 10.

This is to handle the case when one term has many documents on one shard but is just below the size threshold on all other shards. If each shard only returned size terms, the aggregation would return an partial doc count for the term. So terms returns more terms in an attempt to catch the missing terms. This helps, but it’s still quite possible to return a partial doc count for a term. It just takes a term with more disparate per-shard doc counts.

You can increase shard_size to better account for these disparate doc counts and improve the accuracy of the selection of top terms. It is much cheaper to increase the shard_size than to increase the size. However, it still takes more bytes over the wire and waiting in memory on the coordinating node.

Even with a larger shard_size value, doc_count values for a terms aggregation may be approximate. As a result, any sub-aggregations on the terms aggregation may also be approximate.

sum_other_doc_count is the number of documents that didn’t make it into the the top size terms. If this is greater than 0, you can be sure that the terms agg had to throw away some buckets, either because they didn’t fit into size on the coordinating node or they didn’t fit into shard_size on the data node.

If you set the show_term_doc_count_error parameter to true, the terms aggregation will include doc_count_error_upper_bound, which is an upper bound to the error on the doc_count returned by each shard. It’s the sum of the size of the largest bucket on each shard that didn’t fit into shard_size.

In more concrete terms, imagine there is one bucket that is very large on one shard and just outside the shard_size on all the other shards. In that case, the terms agg will return the bucket because it is large, but it’ll be missing data from many documents on the shards where the term fell below the shard_size threshold. doc_count_error_upper_bound is the maximum number of those missing documents.

GET /_search
{
  "aggs": {
    "products": {
      "terms": {
        "field": "product",
        "size": 5,
        "show_term_doc_count_error": true
      }
    }
  }
}

These errors can only be calculated in this way when the terms are ordered by descending document count. When the aggregation is ordered by the terms values themselves (either ascending or descending) there is no error in the document count since if a shard does not return a particular term which appears in the results from another shard, it must not have that term in its index. When the aggregation is either sorted by a sub aggregation or in order of ascending document count, the error in the document counts cannot be determined and is given a value of -1 to indicate this.

By default, the terms aggregation orders terms by descending document _count. Use the order parameter to specify a different sort order.

If you really must, this is how you sort on doc count ascending:

GET /_search
{
  "aggs": {
    "genres": {
      "terms": {
        "field": "genre",
        "order": { "_count": "asc" }
      }
    }
  }
}

Ordering the buckets alphabetically by their terms in an ascending manner:

GET /_search
{
  "aggs": {
    "genres": {
      "terms": {
        "field": "genre",
        "order": { "_key": "asc" }
      }
    }
  }
}

Ordering the buckets by single value metrics sub-aggregation (identified by the aggregation name):

GET /_search
{
  "aggs": {
    "genres": {
      "terms": {
        "field": "genre",
        "order": { "max_play_count": "desc" }
      },
      "aggs": {
        "max_play_count": { "max": { "field": "play_count" } }
      }
    }
  }
}

Ordering the buckets by multi value metrics sub-aggregation (identified by the aggregation name):

GET /_search
{
  "aggs": {
    "genres": {
      "terms": {
        "field": "genre",
        "order": { "playback_stats.max": "desc" }
      },
      "aggs": {
        "playback_stats": { "stats": { "field": "play_count" } }
      }
    }
  }
}

It is also possible to order the buckets based on a "deeper" aggregation in the hierarchy. This is supported as long as the aggregations path are of a single-bucket type, where the last aggregation in the path may either be a single-bucket one or a metrics one. If it’s a single-bucket type, the order will be defined by the number of docs in the bucket (i.e. doc_count), in case it’s a metrics one, the same rules as above apply (where the path must indicate the metric name to sort by in case of a multi-value metrics aggregation, and in case of a single-value metrics aggregation the sort will be applied on that value).

The path must be defined in the following form:

AGG_SEPARATOR       =  '>' ;
METRIC_SEPARATOR    =  '.' ;
AGG_NAME            =  <the name of the aggregation> ;
METRIC              =  <the name of the metric (in case of multi-value metrics aggregation)> ;
PATH                =  <AGG_NAME> [ <AGG_SEPARATOR>, <AGG_NAME> ]* [ <METRIC_SEPARATOR>, <METRIC> ] ;

GET /_search
{
  "aggs": {
    "countries": {
      "terms": {
        "field": "artist.country",
        "order": { "rock>playback_stats.avg": "desc" }
      },
      "aggs": {
        "rock": {
          "filter": { "term": { "genre": "rock" } },
          "aggs": {
            "playback_stats": { "stats": { "field": "play_count" } }
          }
        }
      }
    }
  }
}

The above will sort the artist’s countries buckets based on the average play count among the rock songs.

Multiple criteria can be used to order the buckets by providing an array of order criteria such as the following:

GET /_search
{
  "aggs": {
    "countries": {
      "terms": {
        "field": "artist.country",
        "order": [ { "rock>playback_stats.avg": "desc" }, { "_count": "desc" } ]
      },
      "aggs": {
        "rock": {
          "filter": { "term": { "genre": "rock" } },
          "aggs": {
            "playback_stats": { "stats": { "field": "play_count" } }
          }
        }
      }
    }
  }
}

The above will sort the artist’s countries buckets based on the average play count among the rock songs and then by their doc_count in descending order.

It is possible to only return terms that match more than a configured number of hits using the min_doc_count option:

GET /_search
{
  "aggs": {
    "tags": {
      "terms": {
        "field": "tags",
        "min_doc_count": 10
      }
    }
  }
}

The above aggregation would only return tags which have been found in 10 hits or more. Default value is 1.

Terms are collected and ordered on a shard level and merged with the terms collected from other shards in a second step. However, the shard does not have the information about the global document count available. The decision if a term is added to a candidate list depends only on the order computed on the shard using local shard frequencies. The min_doc_count criterion is only applied after merging local terms statistics of all shards. In a way the decision to add the term as a candidate is made without being very certain about if the term will actually reach the required min_doc_count. This might cause many (globally) high frequent terms to be missing in the final result if low frequent terms populated the candidate lists. To avoid this, the shard_size parameter can be increased to allow more candidate terms on the shards. However, this increases memory consumption and network traffic.

Use a runtime field if the data in your documents doesn’t exactly match what you’d like to aggregate. If, for example, "anthologies" need to be in a special category then you could run this:

GET /_search
{
  "size": 0,
  "runtime_mappings": {
    "normalized_genre": {
      "type": "keyword",
      "script": """
        String genre = doc['genre'].value;
        if (doc['product'].value.startsWith('Anthology')) {
          emit(genre + ' anthology');
        } else {
          emit(genre);
        }
      """
    }
  },
  "aggs": {
    "genres": {
      "terms": {
        "field": "normalized_genre"
      }
    }
  }
}

Which will look like:

{
  "aggregations": {
    "genres": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "electronic",
          "doc_count": 4
        },
        {
          "key": "rock",
          "doc_count": 3
        },
        {
          "key": "electronic anthology",
          "doc_count": 2
        },
        {
          "key": "jazz",
          "doc_count": 2
        }
      ]
    }
  },
  ...
}

This is a little slower because the runtime field has to access two fields instead of one and because there are some optimizations that work on non-runtime keyword fields that we have to give up for for runtime keyword fields. If you need the speed, you can index the normalized_genre field.

It is possible to filter the values for which buckets will be created. This can be done using the include and exclude parameters which are based on regular expression strings or arrays of exact values. Additionally, include clauses can filter using partition expressions.

GET /_search
{
  "aggs": {
    "tags": {
      "terms": {
        "field": "tags",
        "include": ".*sport.*",
        "exclude": "water_.*"
      }
    }
  }
}

GET /_search
{
  "aggs": {
    "JapaneseCars": {
      "terms": {
        "field": "make",
        "include": [ "mazda", "honda" ]
      }
    },
    "ActiveCarManufacturers": {
      "terms": {
        "field": "make",
        "exclude": [ "rover", "jensen" ]
      }
    }
  }
}

GET /_search
{
   "size": 0,
   "aggs": {
      "expired_sessions": {
         "terms": {
            "field": "account_id",
            "include": {
               "partition": 0,
               "num_partitions": 20
            },
            "size": 10000,
            "order": {
               "last_access": "asc"
            }
         },
         "aggs": {
            "last_access": {
               "max": {
                  "field": "access_date"
               }
            }
         }
      }
   }
}

The terms aggregation does not support collecting terms from multiple fields in the same document. The reason is that the terms agg doesn’t collect the string term values themselves, but rather uses global ordinals to produce a list of all of the unique values in the field. Global ordinals results in an important performance boost which would not be possible across multiple fields.

There are three approaches that you can use to perform a terms agg across multiple fields:

Deferring calculation of child aggregations

For fields with many unique terms and a small number of required results it can be more efficient to delay the calculation of child aggregations until the top parent-level aggs have been pruned. Ordinarily, all branches of the aggregation tree are expanded in one depth-first pass and only then any pruning occurs. In some scenarios this can be very wasteful and can hit memory constraints. An example problem scenario is querying a movie database for the 10 most popular actors and their 5 most common co-stars:

GET /_search
{
  "aggs": {
    "actors": {
      "terms": {
        "field": "actors",
        "size": 10
      },
      "aggs": {
        "costars": {
          "terms": {
            "field": "actors",
            "size": 5
          }
        }
      }
    }
  }
}

Even though the number of actors may be comparatively small and we want only 50 result buckets there is a combinatorial explosion of buckets during calculation - a single actor can produce n² buckets where n is the number of actors. The sane option would be to first determine the 10 most popular actors and only then examine the top co-stars for these 10 actors. This alternative strategy is what we call the breadth_first collection mode as opposed to the depth_first mode.

GET /_search
{
  "aggs": {
    "actors": {
      "terms": {
        "field": "actors",
        "size": 10,
        "collect_mode": "breadth_first" 
      },
      "aggs": {
        "costars": {
          "terms": {
            "field": "actors",
            "size": 5
          }
        }
      }
    }
  }
}

When using breadth_first mode the set of documents that fall into the uppermost buckets are cached for subsequent replay so there is a memory overhead in doing this which is linear with the number of matching documents. Note that the order parameter can still be used to refer to data from a child aggregation when using the breadth_first setting - the parent aggregation understands that this child aggregation will need to be called first before any of the other child aggregations.

There are different mechanisms by which terms aggregations can be executed:

Elasticsearch tries to have sensible defaults so this is something that generally doesn’t need to be configured.

global_ordinals is the default option for keyword field, it uses global ordinals to allocates buckets dynamically so memory usage is linear to the number of values of the documents that are part of the aggregation scope.

map should only be considered when very few documents match a query. Otherwise the ordinals-based execution mode is significantly faster. By default, map is only used when running an aggregation on scripts, since they don’t have ordinals.

GET /_search
{
  "aggs": {
    "tags": {
      "terms": {
        "field": "tags",
        "execution_hint": "map" 
      }
    }
  }
}

Please note that Elasticsearch will ignore this execution hint if it is not applicable and that there is no backward compatibility guarantee on these hints.

The missing parameter defines how documents that are missing a value should be treated. By default they will be ignored but it is also possible to treat them as if they had a value.

GET /_search
{
  "aggs": {
    "tags": {
      "terms": {
        "field": "tags",
        "missing": "N/A" 
      }
    }
  }
}