- Elasticsearch Guide: other versions:
- Getting Started
- Set up Elasticsearch
- Installing Elasticsearch
- Configuring Elasticsearch
- Important Elasticsearch configuration
- Important System Configuration
- Bootstrap Checks
- Heap size check
- File descriptor check
- Memory lock check
- Maximum number of threads check
- Maximum size virtual memory check
- Max file size check
- Maximum map count check
- Client JVM check
- Use serial collector check
- System call filter check
- OnError and OnOutOfMemoryError checks
- Early-access check
- G1GC check
- All permission check
- Stopping Elasticsearch
- Upgrade Elasticsearch
- Set up X-Pack
- Breaking changes
- Breaking changes in 6.0
- Aggregations changes
- Analysis changes
- Cat API changes
- Clients changes
- Cluster changes
- Document API changes
- Geo changes
- Indices changes
- Ingest changes
- Java API changes
- Mapping changes
- Packaging changes
- Percolator changes
- Plugins changes
- Reindex changes
- REST changes
- Scripting changes
- Search and Query DSL changes
- Settings changes
- Stats and info changes
- Breaking changes in 6.1
- Breaking changes in 6.2
- Breaking changes in 6.0
- X-Pack Breaking Changes
- API Conventions
- Document APIs
- Search APIs
- Aggregations
- Metrics Aggregations
- Avg Aggregation
- Cardinality Aggregation
- Extended Stats Aggregation
- Geo Bounds Aggregation
- Geo Centroid Aggregation
- Max Aggregation
- Min Aggregation
- Percentiles Aggregation
- Percentile Ranks Aggregation
- Scripted Metric Aggregation
- Stats Aggregation
- Sum Aggregation
- Top Hits Aggregation
- Value Count Aggregation
- Bucket Aggregations
- Adjacency Matrix Aggregation
- Children Aggregation
- Composite Aggregation
- Date Histogram Aggregation
- Date Range Aggregation
- Diversified Sampler Aggregation
- Filter Aggregation
- Filters Aggregation
- Geo Distance Aggregation
- GeoHash grid Aggregation
- Global Aggregation
- Histogram Aggregation
- IP Range Aggregation
- Missing Aggregation
- Nested Aggregation
- Range Aggregation
- Reverse nested Aggregation
- Sampler Aggregation
- Significant Terms Aggregation
- Significant Text Aggregation
- Terms Aggregation
- Pipeline Aggregations
- Avg Bucket Aggregation
- Derivative Aggregation
- Max Bucket Aggregation
- Min Bucket Aggregation
- Sum Bucket Aggregation
- Stats Bucket Aggregation
- Extended Stats Bucket Aggregation
- Percentiles Bucket Aggregation
- Moving Average Aggregation
- Cumulative Sum Aggregation
- Bucket Script Aggregation
- Bucket Selector Aggregation
- Bucket Sort Aggregation
- Serial Differencing Aggregation
- Matrix Aggregations
- Caching heavy aggregations
- Returning only aggregation results
- Aggregation Metadata
- Returning the type of the aggregation
- Metrics Aggregations
- Indices APIs
- Create Index
- Delete Index
- Get Index
- Indices Exists
- Open / Close Index API
- Shrink Index
- Split Index
- Rollover Index
- Put Mapping
- Get Mapping
- Get Field Mapping
- Types Exists
- Index Aliases
- Update Indices Settings
- Get Settings
- Analyze
- Index Templates
- Indices Stats
- Indices Segments
- Indices Recovery
- Indices Shard Stores
- Clear Cache
- Flush
- Refresh
- Force Merge
- cat APIs
- Cluster APIs
- Query DSL
- Mapping
- Analysis
- Anatomy of an analyzer
- Testing analyzers
- Analyzers
- Normalizers
- Tokenizers
- Standard Tokenizer
- Letter Tokenizer
- Lowercase Tokenizer
- Whitespace Tokenizer
- UAX URL Email Tokenizer
- Classic Tokenizer
- Thai Tokenizer
- NGram Tokenizer
- Edge NGram Tokenizer
- Keyword Tokenizer
- Pattern Tokenizer
- Simple Pattern Tokenizer
- Simple Pattern Split Tokenizer
- Path Hierarchy Tokenizer
- Path Hierarchy Tokenizer Examples
- Token Filters
- Standard Token Filter
- ASCII Folding Token Filter
- Flatten Graph Token Filter
- Length Token Filter
- Lowercase Token Filter
- Uppercase Token Filter
- NGram Token Filter
- Edge NGram Token Filter
- Porter Stem Token Filter
- Shingle Token Filter
- Stop Token Filter
- Word Delimiter Token Filter
- Word Delimiter Graph Token Filter
- Stemmer Token Filter
- Stemmer Override Token Filter
- Keyword Marker Token Filter
- Keyword Repeat Token Filter
- KStem Token Filter
- Snowball Token Filter
- Phonetic Token Filter
- Synonym Token Filter
- Synonym Graph Token Filter
- Compound Word Token Filters
- Reverse Token Filter
- Elision Token Filter
- Truncate Token Filter
- Unique Token Filter
- Pattern Capture Token Filter
- Pattern Replace Token Filter
- Trim Token Filter
- Limit Token Count Token Filter
- Hunspell Token Filter
- Common Grams Token Filter
- Normalization Token Filter
- CJK Width Token Filter
- CJK Bigram Token Filter
- Delimited Payload Token Filter
- Keep Words Token Filter
- Keep Types Token Filter
- Classic Token Filter
- Apostrophe Token Filter
- Decimal Digit Token Filter
- Fingerprint Token Filter
- Minhash Token Filter
- Character Filters
- Modules
- Index Modules
- Ingest Node
- Pipeline Definition
- Ingest APIs
- Accessing Data in Pipelines
- Handling Failures in Pipelines
- Processors
- Append Processor
- Convert Processor
- Date Processor
- Date Index Name Processor
- Fail Processor
- Foreach Processor
- Grok Processor
- Gsub Processor
- Join Processor
- JSON Processor
- KV Processor
- Lowercase Processor
- Remove Processor
- Rename Processor
- Script Processor
- Set Processor
- Split Processor
- Sort Processor
- Trim Processor
- Uppercase Processor
- Dot Expander Processor
- URL Decode Processor
- Monitoring Elasticsearch
- X-Pack APIs
- Info API
- Explore API
- Licensing APIs
- Migration APIs
- Machine Learning APIs
- Add Events to Calendar
- Add Jobs to Calendar
- Close Jobs
- Create Calendar
- Create Datafeeds
- Create Jobs
- Delete Calendar
- Delete Datafeeds
- Delete Events from Calendar
- Delete Jobs
- Delete Jobs from Calendar
- Delete Model Snapshots
- Flush Jobs
- Forecast Jobs
- Get Calendars
- Get Buckets
- Get Overall Buckets
- Get Categories
- Get Datafeeds
- Get Datafeed Statistics
- Get Influencers
- Get Jobs
- Get Job Statistics
- Get Model Snapshots
- Get Scheduled Events
- Get Records
- Open Jobs
- Post Data to Jobs
- Preview Datafeeds
- Revert Model Snapshots
- Start Datafeeds
- Stop Datafeeds
- Update Datafeeds
- Update Jobs
- Update Model Snapshots
- Security APIs
- Watcher APIs
- Definitions
- X-Pack Commands
- How To
- Testing
- Glossary of terms
- Elasticsearch Release Notes
- Elasticsearch version 6.2.4
- Elasticsearch version 6.2.3
- Elasticsearch version 6.2.2
- Elasticsearch version 6.2.1
- Elasticsearch version 6.2.0
- Elasticsearch version 6.1.4
- Elasticsearch version 6.1.3
- Elasticsearch version 6.1.2
- Elasticsearch version 6.1.1
- Elasticsearch version 6.1.0
- Elasticsearch version 6.0.1
- Elasticsearch version 6.0.0
- Elasticsearch version 6.0.0-rc2
- Elasticsearch version 6.0.0-rc1
- Elasticsearch version 6.0.0-beta2
- Elasticsearch version 6.0.0-beta1
- Elasticsearch version 6.0.0-alpha2
- Elasticsearch version 6.0.0-alpha1
- Elasticsearch version 6.0.0-alpha1 (Changes previously released in 5.x)
- X-Pack Release Notes
- Elasticsearch X-Pack version 6.2.4
- Elasticsearch X-Pack version 6.2.3
- Elasticsearch X-Pack version 6.2.2
- Elasticsearch X-Pack version 6.2.1
- Elasticsearch X-Pack version 6.2.0
- Elasticsearch X-Pack version 6.1.4
- Elasticsearch X-Pack version 6.1.3
- Elasticsearch X-Pack version 6.1.2
- Elasticsearch X-Pack version 6.1.1
- Elasticsearch X-Pack version 6.1.0
- Elasticsearch X-Pack version 6.0.1
- Elasticsearch X-Pack version 6.0.0
- Elasticsearch X-Pack version 6.0.0-rc2
- Elasticsearch X-Pack version 6.0.0-rc1
- Elasticsearch X-Pack version 6.0.0-beta2
- Elasticsearch X-Pack version 6.0.0-beta1
- Elasticsearch X-Pack version 6.0.0-alpha2
- Elasticsearch X-Pack version 6.0.0-alpha1
WARNING: Version 6.2 of Elasticsearch has passed its EOL date.
This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.
Extended Stats Aggregation
editExtended Stats Aggregation
editA multi-value
metrics aggregation that computes stats over numeric values extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script.
The extended_stats
aggregations is an extended version of the stats
aggregation, where additional metrics are added such as sum_of_squares
, variance
, std_deviation
and std_deviation_bounds
.
Assuming the data consists of documents representing exams grades (between 0 and 100) of students
GET /exams/_search { "size": 0, "aggs" : { "grades_stats" : { "extended_stats" : { "field" : "grade" } } } }
The above aggregation computes the grades statistics over all documents. The aggregation type is extended_stats
and the field
setting defines the numeric field of the documents the stats will be computed on. The above will return the following:
{ ... "aggregations": { "grades_stats": { "count": 2, "min": 50.0, "max": 100.0, "avg": 75.0, "sum": 150.0, "sum_of_squares": 12500.0, "variance": 625.0, "std_deviation": 25.0, "std_deviation_bounds": { "upper": 125.0, "lower": 25.0 } } } }
The name of the aggregation (grades_stats
above) also serves as the key by which the aggregation result can be retrieved from the returned response.
Standard Deviation Bounds
editBy default, the extended_stats
metric will return an object called std_deviation_bounds
, which provides an interval of plus/minus two standard
deviations from the mean. This can be a useful way to visualize variance of your data. If you want a different boundary, for example
three standard deviations, you can set sigma
in the request:
GET /exams/_search { "size": 0, "aggs" : { "grades_stats" : { "extended_stats" : { "field" : "grade", "sigma" : 3 } } } }
sigma
can be any non-negative double, meaning you can request non-integer values such as 1.5
. A value of 0
is valid, but will simply
return the average for both upper
and lower
bounds.
Standard Deviation and Bounds require normality
The standard deviation and its bounds are displayed by default, but they are not always applicable to all data-sets. Your data must be normally distributed for the metrics to make sense. The statistics behind standard deviations assumes normally distributed data, so if your data is skewed heavily left or right, the value returned will be misleading.
Script
editComputing the grades stats based on a script:
GET /exams/_search { "size": 0, "aggs" : { "grades_stats" : { "extended_stats" : { "script" : { "source" : "doc['grade'].value", "lang" : "painless" } } } } }
This will interpret the script
parameter as an inline
script with the painless
script language and no script parameters. To use a stored script use the following syntax:
GET /exams/_search { "size": 0, "aggs" : { "grades_stats" : { "extended_stats" : { "script" : { "id": "my_script", "params": { "field": "grade" } } } } } }
Value Script
editIt turned out that the exam was way above the level of the students and a grade correction needs to be applied. We can use value script to get the new stats:
GET /exams/_search { "size": 0, "aggs" : { "grades_stats" : { "extended_stats" : { "field" : "grade", "script" : { "lang" : "painless", "source": "_value * params.correction", "params" : { "correction" : 1.2 } } } } } }
Missing value
editThe missing
parameter defines how documents that are missing a value should be treated.
By default they will be ignored but it is also possible to treat them as if they
had a value.