- Elasticsearch Guide: other versions:
- Getting Started
- Set up Elasticsearch
- Installing Elasticsearch
- Configuring Elasticsearch
- Important Elasticsearch configuration
- Important System Configuration
- Bootstrap Checks
- Heap size check
- File descriptor check
- Memory lock check
- Maximum number of threads check
- Max file size check
- Maximum size virtual memory check
- Maximum map count check
- Client JVM check
- Use serial collector check
- System call filter check
- OnError and OnOutOfMemoryError checks
- Early-access check
- G1GC check
- All permission check
- Starting Elasticsearch
- Stopping Elasticsearch
- Adding nodes to your cluster
- Installing X-Pack
- Set up X-Pack
- Configuring X-Pack Java Clients
- X-Pack Settings
- Bootstrap Checks for X-Pack
- Upgrade Elasticsearch
- API Conventions
- Document APIs
- Search APIs
- Aggregations
- Metrics Aggregations
- Avg Aggregation
- Weighted Avg Aggregation
- Cardinality Aggregation
- Extended Stats Aggregation
- Geo Bounds Aggregation
- Geo Centroid Aggregation
- Max Aggregation
- Min Aggregation
- Percentiles Aggregation
- Percentile Ranks Aggregation
- Scripted Metric Aggregation
- Stats Aggregation
- Sum Aggregation
- Top Hits Aggregation
- Value Count Aggregation
- Bucket Aggregations
- Adjacency Matrix Aggregation
- Auto-interval Date Histogram Aggregation
- Intervals
- Children Aggregation
- Composite Aggregation
- Date Histogram Aggregation
- Date Range Aggregation
- Diversified Sampler Aggregation
- Filter Aggregation
- Filters Aggregation
- Geo Distance Aggregation
- GeoHash grid Aggregation
- Global Aggregation
- Histogram Aggregation
- IP Range Aggregation
- Missing Aggregation
- Nested Aggregation
- Range Aggregation
- Reverse nested Aggregation
- Sampler Aggregation
- Significant Terms Aggregation
- Significant Text Aggregation
- Terms Aggregation
- Pipeline Aggregations
- Avg Bucket Aggregation
- Derivative Aggregation
- Max Bucket Aggregation
- Min Bucket Aggregation
- Sum Bucket Aggregation
- Stats Bucket Aggregation
- Extended Stats Bucket Aggregation
- Percentiles Bucket Aggregation
- Moving Average Aggregation
- Moving Function Aggregation
- Cumulative Sum Aggregation
- Bucket Script Aggregation
- Bucket Selector Aggregation
- Bucket Sort Aggregation
- Serial Differencing Aggregation
- Matrix Aggregations
- Caching heavy aggregations
- Returning only aggregation results
- Aggregation Metadata
- Returning the type of the aggregation
- Metrics Aggregations
- Indices APIs
- Create Index
- Delete Index
- Get Index
- Indices Exists
- Open / Close Index API
- Shrink Index
- Split Index
- Rollover Index
- Put Mapping
- Get Mapping
- Get Field Mapping
- Types Exists
- Index Aliases
- Update Indices Settings
- Get Settings
- Analyze
- Index Templates
- Indices Stats
- Indices Segments
- Indices Recovery
- Indices Shard Stores
- Clear Cache
- Flush
- Refresh
- Force Merge
- cat APIs
- Cluster APIs
- Query DSL
- Mapping
- Analysis
- Anatomy of an analyzer
- Testing analyzers
- Analyzers
- Normalizers
- Tokenizers
- Standard Tokenizer
- Letter Tokenizer
- Lowercase Tokenizer
- Whitespace Tokenizer
- UAX URL Email Tokenizer
- Classic Tokenizer
- Thai Tokenizer
- NGram Tokenizer
- Edge NGram Tokenizer
- Keyword Tokenizer
- Pattern Tokenizer
- Char Group Tokenizer
- Simple Pattern Tokenizer
- Simple Pattern Split Tokenizer
- Path Hierarchy Tokenizer
- Path Hierarchy Tokenizer Examples
- Token Filters
- Standard Token Filter
- ASCII Folding Token Filter
- Flatten Graph Token Filter
- Length Token Filter
- Lowercase Token Filter
- Uppercase Token Filter
- NGram Token Filter
- Edge NGram Token Filter
- Porter Stem Token Filter
- Shingle Token Filter
- Stop Token Filter
- Word Delimiter Token Filter
- Word Delimiter Graph Token Filter
- Multiplexer Token Filter
- Conditional Token Filter
- Predicate Token Filter Script
- Stemmer Token Filter
- Stemmer Override Token Filter
- Keyword Marker Token Filter
- Keyword Repeat Token Filter
- KStem Token Filter
- Snowball Token Filter
- Phonetic Token Filter
- Synonym Token Filter
- Synonym Graph Token Filter
- Compound Word Token Filters
- Reverse Token Filter
- Elision Token Filter
- Truncate Token Filter
- Unique Token Filter
- Pattern Capture Token Filter
- Pattern Replace Token Filter
- Trim Token Filter
- Limit Token Count Token Filter
- Hunspell Token Filter
- Common Grams Token Filter
- Normalization Token Filter
- CJK Width Token Filter
- CJK Bigram Token Filter
- Delimited Payload Token Filter
- Keep Words Token Filter
- Keep Types Token Filter
- Exclude mode settings example
- Classic Token Filter
- Apostrophe Token Filter
- Decimal Digit Token Filter
- Fingerprint Token Filter
- Minhash Token Filter
- Remove Duplicates Token Filter
- Character Filters
- Modules
- Index Modules
- Ingest Node
- Pipeline Definition
- Ingest APIs
- Accessing Data in Pipelines
- Conditional Execution in Pipelines
- Handling Failures in Pipelines
- Processors
- Append Processor
- Bytes Processor
- Convert Processor
- Date Processor
- Date Index Name Processor
- Dissect Processor
- Drop Processor
- Dot Expander Processor
- Fail Processor
- Foreach Processor
- Grok Processor
- Gsub Processor
- Join Processor
- JSON Processor
- KV Processor
- Lowercase Processor
- Pipeline Processor
- Remove Processor
- Rename Processor
- Script Processor
- Set Processor
- Set Security User Processor
- Split Processor
- Sort Processor
- Trim Processor
- Uppercase Processor
- URL Decode Processor
- SQL Access
- Monitor a cluster
- Rolling up historical data
- Set up a cluster for high availability
- Secure a cluster
- Overview
- Configuring security
- Encrypting communications in Elasticsearch
- Encrypting communications in an Elasticsearch Docker Container
- Enabling cipher suites for stronger encryption
- Separating node-to-node and client traffic
- Configuring an Active Directory realm
- Configuring a file realm
- Configuring an LDAP realm
- Configuring a native realm
- Configuring a PKI realm
- Configuring a SAML realm
- Configuring a Kerberos realm
- FIPS 140-2
- Security settings
- Security files
- Auditing settings
- How security works
- User authentication
- Built-in users
- Internal users
- Realms
- Realm chains
- Active Directory user authentication
- File-based user authentication
- LDAP user authentication
- Native user authentication
- PKI user authentication
- SAML authentication
- Kerberos authentication
- Integrating with other authentication systems
- Enabling anonymous access
- Controlling the user cache
- Configuring SAML single-sign-on on the Elastic Stack
- User authorization
- Auditing security events
- Encrypting communications
- Restricting connections with IP filtering
- Cross cluster search, tribe, clients, and integrations
- Tutorial: Getting started with security
- Tutorial: Encrypting communications
- Troubleshooting
- Can’t log in after upgrading to 6.5.4
- Some settings are not returned via the nodes settings API
- Authorization exceptions
- Users command fails due to extra arguments
- Users are frequently locked out of Active Directory
- Certificate verification fails for curl on Mac
- SSLHandshakeException causes connections to fail
- Common SSL/TLS exceptions
- Common Kerberos exceptions
- Common SAML issues
- Internal Server Error in Kibana
- Setup-passwords command fails due to connection failure
- Failures due to relocation of the configuration files
- Limitations
- Alerting on Cluster and Index Events
- Command line tools
- How To
- Testing
- Glossary of terms
- X-Pack APIs
- Info API
- Cross-cluster replication APIs
- Explore API
- Licensing APIs
- Migration APIs
- Machine learning APIs
- Add events to calendar
- Add jobs to calendar
- Close jobs
- Create calendar
- Create datafeeds
- Create filter
- Create jobs
- Delete calendar
- Delete datafeeds
- Delete events from calendar
- Delete filter
- Delete forecast
- Delete jobs
- Delete jobs from calendar
- Delete model snapshots
- Find file structure
- Flush jobs
- Forecast jobs
- Get calendars
- Get buckets
- Get overall buckets
- Get categories
- Get datafeeds
- Get datafeed statistics
- Get influencers
- Get jobs
- Get job statistics
- Get machine learning info
- Get model snapshots
- Get scheduled events
- Get filters
- Get records
- Open jobs
- Post data to jobs
- Preview datafeeds
- Revert model snapshots
- Start datafeeds
- Stop datafeeds
- Update datafeeds
- Update filter
- Update jobs
- Update model snapshots
- Rollup APIs
- Security APIs
- Authenticate
- Change passwords
- Clear cache
- Clear roles cache
- Create or update application privileges
- Create or update role mappings
- Create or update roles
- Create or update users
- Delete application privileges
- Delete role mappings
- Delete roles
- Delete users
- Disable users
- Enable users
- Get application privileges
- Get role mappings
- Get roles
- Get token
- Get users
- Has privileges
- Invalidate token
- SSL certificate
- Watcher APIs
- Definitions
- Release Highlights
- Breaking changes
- Release Notes
- Elasticsearch version 6.5.4
- Elasticsearch version 6.5.3
- Elasticsearch version 6.5.2
- Elasticsearch version 6.5.1
- Elasticsearch version 6.5.0
- Elasticsearch version 6.4.3
- Elasticsearch version 6.4.2
- Elasticsearch version 6.4.1
- Elasticsearch version 6.4.0
- Elasticsearch version 6.3.2
- Elasticsearch version 6.3.1
- Elasticsearch version 6.3.0
- Elasticsearch version 6.2.4
- Elasticsearch version 6.2.3
- Elasticsearch version 6.2.2
- Elasticsearch version 6.2.1
- Elasticsearch version 6.2.0
- Elasticsearch version 6.1.4
- Elasticsearch version 6.1.3
- Elasticsearch version 6.1.2
- Elasticsearch version 6.1.1
- Elasticsearch version 6.1.0
- Elasticsearch version 6.0.1
- Elasticsearch version 6.0.0
- Elasticsearch version 6.0.0-rc2
- Elasticsearch version 6.0.0-rc1
- Elasticsearch version 6.0.0-beta2
- Elasticsearch version 6.0.0-beta1
- Elasticsearch version 6.0.0-alpha2
- Elasticsearch version 6.0.0-alpha1
- Elasticsearch version 6.0.0-alpha1 (Changes previously released in 5.x)
Breaking changes in 6.3
editBreaking changes in 6.3
editThis section discusses the changes that you need to be aware of when migrating your application to Elasticsearch 6.3.
See also Release Highlights and Release Notes.
API changes
editSuggest stats metrics deprecated
editThe suggest stats were previously folded into the search on the indices stats
API. As such, the suggest
metric on the indices stats API has been a synonym
for the search
metric. In 6.3.0, the suggest
metric is deprecated in favor
of using search
.
Similarly, the suggest
index metric on the indices
metric on the nodes stats
API has provided a response containing only an empty object since 5.0.0. In
6.3.0 this metric has been deprecated.
Packaging changes
editStarting with version 6.3, all of the X-Pack features ship with the default distribution of Elasticsearch. You no longer need to install X-Pack separately.
Deprecation of the x-pack
configuration directory
editIn 6.2 and earlier releases, the
security configuration files, such as the
log4j2.properties
, role_mapping.yml
, roles.yml
, users
, and users_roles
files, were located in an x-pack
directory within the
config directory.
Starting in 6.3, the security configuration files are located directly in the config directory.
If you upgraded to 6.3, your old security configuration files still
exist in an x-pack
folder. That path is deprecated, however, and you should
move your files out of that folder. There is a known problem where the files in
the old path are disregarded in favor of the files in the new path:
#33688. For more information, see
Failures due to relocation of the configuration files.
Settings such as files.role_mapping
now default to
ES_PATH_CONF/role_mapping.yml
. If you do not want to use the default locations,
you must update the settings appropriately. See Security settings.
Plugins changes
editGCS Repository plugin
editThe repository settings application_name
, connect_timeout
and read_timeout
have been deprecated and must now be specified in the client settings instead.
See Google Cloud Storage Client Settings.
Ingest Geoip Plugin
editIn earlier versions, database files have been stored as gzip compressed files
with the extension .gz
to save disk space. As a consequence, database files
had to be loaded in memory. Now the default database files that are stored
uncompressed as .mmdb
files which allows to memory-map them and save heap
memory. Any custom database files must also be stored uncompressed. Consequently,
the database_file
property in any ingest pipelines that use the Geoip Processor
must refer to the uncompressed database files as well.
Using the plugin installer without a TTY
editThe Elasticsearch plugin installer (elasticsearch-plugin install
) would
previously silently accept additional security permissions required by a plugin
if standard input was closed or there was no TTY attached (e.g., docker exec
<container ID> elasticsearch-plugin install
). This silent accepting of
additional security permissions has been removed. Now, a user must deliberately
accept these permissions either by keeping standard input open and attaching a
TTY (i.e., using interactive mode to accept the permissions), or by passing the
--batch
flag.
Implementing custom realms with SPI instead of XPackExtension
editThe legacy XPackExtension
extension mechanism has been removed and replaced
with an SPI based extension mechanism that is installed and built as an
elasticsearch plugin. For more information about using SPI loaded security extensions in custom realms, see Integrating with other authentication systems.
Settings changes
editConcurrency level of analyze requests
editPreviously, analyze requests would run on the same thread pool as indexing
requests. The index
thread pool has been deprecated as it is no longer needed
since indexing requests are internally converted to bulk requests and run on the
bulk
thread pool. This leaves analyze requests without a home so we added a
small thread pool called the analyze
thread pool. This thread pool defaults to
having one thread and a queue depth of sixteen requests. This means that
previously analyze requests had a level of concurrency equal to the size of the
index
thread pool and now they have a level of concurrency of one. For most
users we think this is fine as analyze requests are useful for debugging and so
probably do not see high concurrency. If you are impacted by this you can
increase the size of the thread pool by using the thread_pool.analyze.size
setting.
Renaming the bulk thread pool
editThe bulk
thread pool has been renamed to the write
thread pool. This change
was made to reflect the fact that this thread pool is used to execute all write
operations: single-document index/delete/update requests, as well as bulk
requests. The settings thread_pool.bulk.size
and thread_pool.bulk.queue_size
are still supported as fallback settings although you should transition to
thread_pool.write.size
and thread_pool.write.queue_size
, respectively. The
fallback settings will be removed in 7.0.0. Additionally, this means that some
APIs (e.g., the node stats API) will now display the name of this thread pool as
write
. If this change impacts you (e.g., for monitoring that you have in
place) you can start Elasticsearch with the JVM option
-Des.thread_pool.write.use_bulk_as_display_name=true
to have Elasticsearch
continue to display the name of this thread pool as bulk
. Elasticsearch will
stop observing this system property in 7.0.0.
Enabling monitoring
editBy default when you install X-Pack, monitoring is enabled but data collection
is disabled. To enable data collection, use the new
xpack.monitoring.collection.enabled
setting. You can update this setting by
using the Cluster Update Settings API. For more
information, see Monitoring settings.
On this page
- API changes
- Suggest stats metrics deprecated
- Packaging changes
- Deprecation of the
x-pack
configuration directory - Plugins changes
- GCS Repository plugin
- Ingest Geoip Plugin
- Using the plugin installer without a TTY
- Implementing custom realms with SPI instead of XPackExtension
- Settings changes
- Concurrency level of analyze requests
- Renaming the bulk thread pool
- Enabling monitoring