Breaking changes

• Allocation: Enable disk-space allocation decider by default #6200
• Analysis: Improvements to StemmerTokenFilter #6452
• Internal:
  • Make transport action name available in TransportAction base class #6860
  • Thread pool rejection status code should be 429 #6629
  • Remove unnecessary intermediate interfaces #6517
  • Remove unsafe unaligned memory access - illegal on SPARC #6962
• Mappings: Update mapping on master in async manner #6648
• REST API: Replace error code 503 with 429 when appropriate to tell client to back off #6627
• Security: Disable JSONP by default #6795

New features

• Aggregations:
  • Created infrastructure for changing easily the significance terms heuristic #6561
  • Added the percentiles_rank aggregation #6386
  • Deferred aggregations prevent combinatorial explosion #6128
  • Added top_hits aggregation #6124
  • Support bounding box aggregation on geo_shape/geo_point data types #5634
• Scripting:
  • Add script engine for Lucene expressions #6819
  • Add Groovy as a scripting language, add groovy sandboxing #6233
  • Allow search templates stored in an index to be retrieved and used at search time #5921
  • Allow to run scripts/templates stored in .scripts index #5484
• Mappings: Add transform to document before index #6599
• Suggester: Phrase suggest option to limit suggestions to existing phrases #3482

Enhancements

• Resiliency:
  • Resend failed shard messages when receiving a cluster state still referring to the failed shards #6881
  • Send shard exists requests if shard exists locally but is not allocated to the node #6870
  • Don't attempt to start or fail shard if no master node can be found #6841
  • Improve handling of failed primary replica handling #6825
  • During discovery, verify connect when sending a rejoin cluster request #6779
  • During relocation, process pending mapping update in phase 2 #6762
  • Set a default of 5m to recover_after_time when any of the expected*Nodes is set #6742
  • Start master/node fault detection pinging immediately during discovery #6706
  • Before deleting shard verify that another node holds an active shard instance #6692
  • Ensure index.version.created is consistent #6660
  • Cancel recovery if shard on the target node closes during recovery operation #6645
  • Raise proper failure if not fully reading translog entry #6562
  • Wait till node is part of cluster state for join process #6480
  • Have a dedicated join timeout that is higher than ping.timeout for join #6342
  • Before deleting a local unused shard copy, verify we're connected to the node it's supposed to be on #6191
  • Use Lucene built-in checksumming #5924
• Indexing:
  • Set default translog flush_threshold_ops to unlimited, to flush by byte size by default and not penalize tiny documents #6783
  • Don't acquire dirtyLock on autoid for create #6584
  • Clear versionMap on refresh not flush #6379
  • If versionMap is too large we should trigger refresh #6378
  • Reuse Lucene's TermsEnum for faster _uid/version lookup during indexing #6298
  • Don't use AllTokenStream if no fields were boosted #6187
• Analysis:
  • Improve Hunspell error messages #6850
  • Share numeric date analyzer instances between mappings #6843
  • Added missing pre built analysis components #6770
  • PatternAnalyzer should use PatternTokenFilter instead #6717
  • More resource efficient analysis wrapping usage #6714
  • Added additional Analyzers, Tokenizers, and TokenFilters from Lucene #6693
• Administration & Monitoring:
  • Improve pending API to include current executing class #6744
  • Stats: Expose IndexWriter and versionMap RAM usage #6483
  • Improve indices stats options #6390
  • Improve cluster update settings api #6244
  • Remove field names in stats url #6054
  • Index template API: Unified PUT/POST behaviour in relation to create parameter #6429
• Snapshot/Restore:
  • Restore an index without restoring its aliases #6457
  • Add ability to restore partial snapshots #5742
• Aggregations:
  • Extend allowed characters in aggregation name #6702
  • Add from support to top_hits aggregator #6299
  • Moved BucketsAggregator#docCounts field to IntArray #6529
  • GlobalOrdinalsStringTermsAggregator is inefficient for high-cardinality fields #6518
  • Remove ordinals execution hint. #6499
  • Delegation of nextReader calls #6477
• Fielddata:
  • Make BytesValues.WithOrdinals more similar to Lucene's SortedSetDocValues #6524
  • Don't expose hashes in Fielddata anymore. #6500
• Allocation:
  • Allow primaries that have never been allocated to be allocated if under the low watermark #6209
  • Change high/low watermark defaults for disk based allocation decider #6201
  • Disk-aware allocation decider should allow initial primary allocation if under the high watermark #6196
• Mapping:
  • Improve performance for many new fields introduction in mapping #6707
  • Better logic on sending mapping update new type introduction #6669
  • Wait for mapping updates during local recovery #6666
• Scripting:
  • Add a transformer to translate constant BigDecimal to double #6609
  • Add Groovy sandboxing for GString-based method invocation #6596
  • Fix optional default script loading #6582
  • Exposed _uid, _id and _type fields as stored fields (_fields notation) #6406
• Search:
  • Wrap filter only once in ApplyAcceptedDocsFilter #6873
  • Remove Queries#optimizeQuery - already handled in BooleanQuery #6743
  • Speed up exists and missing filters on high-cardinality fields #5659
  • Parent/Child: Support min_children and max_children on has_child query/filter #6019
  • Geo: Added caching support to geohash filter #6478
• Highlighting:
  • Make the HighlightQuery class public #6446
  • Highlight fields in request order #6178
  • Plain highlighter does not honor _analyzer #5497
• More Like This Query:
  • Ensure selection of best terms is indeed O(n) #6657
  • creates only one MLT query per field for all queried items #6404
  • Values of a multi-value fields are compared at the same level #6310
  • replaced exclude with include to avoid double ... #6275
  • allow for both like_text and docs/ids to be specified. #6246
• Plugins:
  • Enables plugins to define default logging configuration for their needs. #6805
  • bin/plugin tests for missing plugin name when passing --url #6013
• Build:
  • Check for tabs and nocommits in the code on validate #6474
  • Generate source jars for tests #6445
  • Packaging: Remove java-6 directories from debian init script #6350
  • Startup: Reset locale to C in bin/elasticsearch #6047
  • Windows: Modify command window title (windows) #6752
• REST API:
  • Copy the headers from REST requests to the corresponding TransportRequest(s) #6513
  • Ensure 503 responses lead to retry on another node #4066
  • Security: Make JSONP responses optional. #6164
• Internal:
  • Make XContentBuilder implement Releasable #6869
  • Don't close/reopen IndexWriter when changing RAM buffer size #6856
  • Added more utility methods to Settings #6840
  • Cleanup of the transport request/response messages #6834
  • Add local node to cluster state #6811
  • Control whether MapperService docMapper iterator should contain DEFAULT_MAPPING #6793
  • Upgrade to Jackson 2.4.1.1 #6789
  • Don't replace indices within ActionRequest and check blocks against concrete indices #6777
  • Improve Settings#get lookup for camel case support #6765
  • Improve large bytes request handling by detecting content composite buffer #6756
  • Call callback on actual mapping processed #6748
  • Remove intern calls on FieldMapper#Names for better performance #6747
  • Clean shard bulk mapping update to only use type name #6695
  • Disable explicit GC by default #6637
  • Make a hybrid directory default using mmapfs/niofs #6636
  • Make sure we don't reuse arrays when sending an error back #6631
  • Wrap RateLimiter rather than copy RateLimitedIndexOutput #6625
  • Upgrade to Lucene 4.9 #6623
  • Re-shade MVEL as a dependency #6570
  • Refactored AckedClusterStateUpdateTask & co. to remove code repetitions in subclasses #6559
  • Better default size for global index to alias map #6504
  • Suppress Unsafe is an internal proprietary API... compilation warnings #6423
  • use ConcurrentHashMapV8 for lower memory overhead #6400
  • Base64 decode parsing detects more errors #6334
  • Upgrade to netty 3.9.1 #6331
  • Added plugins to .gitignore #6324
  • Change the default type of the page recycler to CONCURRENT instead of SOFT_CONCURRENT #6320
  • Entirely cut over to TopDocs#merge for merging shard docs in the reduce phase #6197
  • Add support for Byte and BytesRef to the XContentBuilder #6127
  • Make Node and Client interfaces Closeable #4355
  • Settings: Throw error when incorrect setting applied to auto_expand_replicas #5752
  • TransportClient: Improve logging, fix minor issue #6376
  • Java API: Client intermediate interfaces removal follow-up #6563

Bug fixes

• Resiliency:
  • Increment Store refcount on RecoveryTarget #6844
  • Improve handling of failed primary replica handling #6816
  • Do not ignore ConnectTransportException for shard replication operations #6813
  • Recovering replicas might get stuck in initializing state #6808
  • During recovery, only send mapping updates to master if needed #6772
  • If the node initialisation fails, make sure the node environment is closed correctly #6715
  • Handle ConnectionTransportException during a Master/Node fault detection ping during discovery #6686
  • Recovery from local gateway should re-introduce new mappings #6659
  • Also send Refresh and Flush actions to relocation targets #6545
  • Honor time delay when retrying recoveries #6226
• Aggregations:
  • Histogram Aggregation key bug #6655
  • Fix JSON response for significant terms #6535
  • Fix reducing of range aggregations. #6484
  • Fix cardinality aggregation when doc values field is empty #6413
  • ReverseNestedAggregator does not compute parent documents correctly #6278
  • ClassCastException when sibling aggregations have the same name #6255
  • date_histogram aggregation breaks on date fields with multiple formats #6239
  • DateHistogramBuilder uses wrong data type for pre_offset and post_offset #5586
• Administration & Monitoring:
  • Admin: Recovery API should also report ongoing relocation recoveries #6585
  • Cat API: Fix NullPointerException in recovery API #6190
  • Index Templates API: GET templates doesn't honor the flat_settings parameter. #6671
  • Snapshot/Restore: Allow deleting of interrupted snapshot #6383
• Indexing:
  • Translog: Better support for partial buffer reads/writes in translog infrastructure #6576
  • Force refresh when versionMap is using too much RAM #6443
  • Lower the translog flush triggers to workaround #6377
  • Clear versionMap on refresh not flush #6363
  • DocumentMissingException is uncaught if thrown during retry of update request #6355
  • Geo: Valid polygon crossing dateline fails to parse #6179
• Search:
  • The query_string cache should returned cloned Query instances. #6733
  • Replace empty bool queries with match_all to prevent NullPointerException #6722
  • Match query with operator and, cutoff_frequency and stacked tokens #6573
  • Search template not replacing parameter after initial failure in parameter substitution #6318
  • Nested: queries/filters/aggregations expect FixedBitSets, yet it isn't the case with NoneFilterCache #6279
• Query DSL:
  • Fix MatchQueryParser not parsing fuzzy_transpositions #6300
  • Function score without function throws NPE #6292
  • Improved explanation for match_phrase_prefix #2449
• Mapping:
  • MapperParsingException when create default mapping with 'include_in_all' nested #6304
  • Fix possibility of losing meta configuration on field mapping update #5053
  • Allow _version to use disk as a doc values format. #6523
• Analysis:
  • stem_exclusion as array not working in language analyzers #6237
  • Default analyzer includes stopwords #5974
• Bulk API:
  • Fix return of wrong request type on failed updates #6630
  • Bulk request which try and fail to create multiple indices may never return #6436
  • BulkProcessor does not call afterBulk when bulk throws eg NoNodeAvailableException #5038
• Percolator:
  • Fix memory leak when percolating with nested documents #6578
  • Fix handling of nested documents #6540
  • Allow significant terms and geo hash grid aggregations in the percolator #6037
• Plugins:
  • bin/plugin removes itself #6745
  • Removing plugin does not fail when plugin dir is read only #6546
  • Fix github download link when using specific version #6321
• Java API:
  • Fix source excludes setting if no includes were provided #6632
  • BulkRequest#add(Iterable) to support UpdateRequests #6551
• Internal:
  • Fix possible NPE during shutdown for requests using timeouts #6849
  • Prevent NPE if engine is closed while version map is checked #6786
  • Fixes Logger class for BackgroundIndexer #6781
  • Check for index blocks against concrete indices on master operations #6694
  • IndexingMemoryController should only update buffer settings of fully recovered shards #6667
  • Fix possible race condition in checksum name generator #6662
  • Lucene: Use XNativeFSLockFactory instead of the buggy Lucene 4.8.1 version #6424
  • Disable circuit breaking for BigArrays #6381
  • Guava doesn't explicitly remove entries when clearing the entire cache #6296
  • Filter cache size limit not honored for 32GB or over #6268
  • Threadpool Info: Allow to serialize negative thread pool sizes #6486
  • Core: The ignore_unavailable option should also ignore indices that are closed #6475
  • Remove indicesLifecycle.Listener from IndexingMemoryController #6892
  • Store: delete unallocated shards under a cluster state task #6902
  • Routing: Restore shard routing #6393
  • Scripting: Wrap groovy script exceptions in a serializable Exception object #6628

Tests

• Resiliency: Add basic Backwards Compatibility Tests #6497
• Add test case verifying that dynamically enabling/disabling merge IO throttling works #6842
• Added ability to provide settings for external nodes in backwards compatibility tests #6809
• Add assertBusy helper test method #6753
• Reset all cluster if a test hit a failure #6734
• Take compatibility version into account for XContentType randomization #6691
• Prevent usage of system properties in the InternalTestCluster #6663
• Randomize netty worker and connection parameters #6635
• Change es.node.mode default for tests to local #6624
• Verify all threads created by node and client have the node name #6516
• Fix Test - Cluster naming #6510
• Add test for accessing _score in scripts #6509
• Randomly disable the filter cache #6280
• Allow to disable plugin loading from classpath #6242
• Add dummy docs injection to indexRandom #6235