- Elasticsearch Guide: other versions:
- Elasticsearch introduction
- Getting started with Elasticsearch
- Set up Elasticsearch
- Installing Elasticsearch
- Configuring Elasticsearch
- Important Elasticsearch configuration
- Important System Configuration
- Bootstrap Checks
- Heap size check
- File descriptor check
- Memory lock check
- Maximum number of threads check
- Max file size check
- Maximum size virtual memory check
- Maximum map count check
- Client JVM check
- Use serial collector check
- System call filter check
- OnError and OnOutOfMemoryError checks
- Early-access check
- G1GC check
- All permission check
- Discovery configuration check
- Starting Elasticsearch
- Stopping Elasticsearch
- Adding nodes to your cluster
- Full-cluster restart and rolling restart
- Set up X-Pack
- Configuring X-Pack Java Clients
- Bootstrap Checks for X-Pack
- Upgrade Elasticsearch
- Aggregations
- Metrics Aggregations
- Avg Aggregation
- Weighted Avg Aggregation
- Cardinality Aggregation
- Extended Stats Aggregation
- Geo Bounds Aggregation
- Geo Centroid Aggregation
- Max Aggregation
- Min Aggregation
- Percentiles Aggregation
- Percentile Ranks Aggregation
- Scripted Metric Aggregation
- Stats Aggregation
- Sum Aggregation
- Top Hits Aggregation
- Value Count Aggregation
- Median Absolute Deviation Aggregation
- Bucket Aggregations
- Adjacency Matrix Aggregation
- Auto-interval Date Histogram Aggregation
- Children Aggregation
- Composite Aggregation
- Date histogram aggregation
- Date Range Aggregation
- Diversified Sampler Aggregation
- Filter Aggregation
- Filters Aggregation
- Geo Distance Aggregation
- GeoHash grid Aggregation
- GeoTile Grid Aggregation
- Global Aggregation
- Histogram Aggregation
- IP Range Aggregation
- Missing Aggregation
- Nested Aggregation
- Parent Aggregation
- Range Aggregation
- Rare Terms Aggregation
- Reverse nested Aggregation
- Sampler Aggregation
- Significant Terms Aggregation
- Significant Text Aggregation
- Terms Aggregation
- Subtleties of bucketing range fields
- Pipeline Aggregations
- Avg Bucket Aggregation
- Derivative Aggregation
- Max Bucket Aggregation
- Min Bucket Aggregation
- Sum Bucket Aggregation
- Stats Bucket Aggregation
- Extended Stats Bucket Aggregation
- Percentiles Bucket Aggregation
- Moving Average Aggregation
- Moving Function Aggregation
- Cumulative Sum Aggregation
- Cumulative Cardinality Aggregation
- Bucket Script Aggregation
- Bucket Selector Aggregation
- Bucket Sort Aggregation
- Serial Differencing Aggregation
- Matrix Aggregations
- Caching heavy aggregations
- Returning only aggregation results
- Aggregation Metadata
- Returning the type of the aggregation
- Metrics Aggregations
- Query DSL
- Search across clusters
- Scripting
- Mapping
- Analysis
- Anatomy of an analyzer
- Testing analyzers
- Analyzers
- Normalizers
- Tokenizers
- Char Group Tokenizer
- Classic Tokenizer
- Edge n-gram tokenizer
- Keyword Tokenizer
- Letter Tokenizer
- Lowercase Tokenizer
- N-gram tokenizer
- Path Hierarchy Tokenizer
- Path Hierarchy Tokenizer Examples
- Pattern Tokenizer
- Simple Pattern Tokenizer
- Simple Pattern Split Tokenizer
- Standard Tokenizer
- Thai Tokenizer
- UAX URL Email Tokenizer
- Whitespace Tokenizer
- Token Filters
- Apostrophe
- ASCII folding
- CJK bigram
- CJK width
- Classic
- Common grams
- Conditional
- Decimal digit
- Delimited payload
- Dictionary decompounder
- Edge n-gram
- Elision
- Fingerprint
- Flatten Graph Token Filter
- Hunspell Token Filter
- Hyphenation decompounder
- Keep types
- Keep words
- Keyword Marker Token Filter
- Keyword Repeat Token Filter
- KStem Token Filter
- Length Token Filter
- Limit Token Count Token Filter
- Lowercase Token Filter
- MinHash Token Filter
- Multiplexer Token Filter
- N-gram
- Normalization Token Filter
- Pattern Capture Token Filter
- Pattern Replace Token Filter
- Phonetic Token Filter
- Porter Stem Token Filter
- Predicate Token Filter Script
- Remove Duplicates Token Filter
- Reverse Token Filter
- Shingle Token Filter
- Snowball Token Filter
- Stemmer Token Filter
- Stemmer Override Token Filter
- Stop Token Filter
- Synonym Token Filter
- Synonym Graph Token Filter
- Trim Token Filter
- Truncate Token Filter
- Unique Token Filter
- Uppercase Token Filter
- Word Delimiter Token Filter
- Word Delimiter Graph Token Filter
- Character Filters
- Modules
- Index modules
- Ingest node
- Pipeline Definition
- Accessing Data in Pipelines
- Conditional Execution in Pipelines
- Handling Failures in Pipelines
- Processors
- Append Processor
- Bytes Processor
- Circle Processor
- Convert Processor
- Date Processor
- Date Index Name Processor
- Dissect Processor
- Dot Expander Processor
- Drop Processor
- Fail Processor
- Foreach Processor
- GeoIP Processor
- Grok Processor
- Gsub Processor
- HTML Strip Processor
- Join Processor
- JSON Processor
- KV Processor
- Lowercase Processor
- Pipeline Processor
- Remove Processor
- Rename Processor
- Script Processor
- Set Processor
- Set Security User Processor
- Split Processor
- Sort Processor
- Trim Processor
- Uppercase Processor
- URL Decode Processor
- User Agent processor
- Managing the index lifecycle
- Getting started with index lifecycle management
- Policy phases and actions
- Set up index lifecycle management policy
- Using policies to manage index rollover
- Update policy
- Index lifecycle error handling
- Restoring snapshots of managed indices
- Start and stop index lifecycle management
- Using ILM with existing indices
- Getting started with snapshot lifecycle management
- SQL access
- Overview
- Getting Started with SQL
- Conventions and Terminology
- Security
- SQL REST API
- SQL Translate API
- SQL CLI
- SQL JDBC
- SQL ODBC
- SQL Client Applications
- SQL Language
- Functions and Operators
- Comparison Operators
- Logical Operators
- Math Operators
- Cast Operators
- LIKE and RLIKE Operators
- Aggregate Functions
- Grouping Functions
- Date/Time and Interval Functions and Operators
- Full-Text Search Functions
- Mathematical Functions
- String Functions
- Type Conversion Functions
- Geo Functions
- Conditional Functions And Expressions
- System Functions
- Reserved keywords
- SQL Limitations
- Monitor a cluster
- Frozen indices
- Roll up or transform your data
- Set up a cluster for high availability
- Secure a cluster
- Overview
- Configuring security
- User authentication
- Built-in users
- Internal users
- Token-based authentication services
- Realms
- Realm chains
- Active Directory user authentication
- File-based user authentication
- LDAP user authentication
- Native user authentication
- OpenID Connect authentication
- PKI user authentication
- SAML authentication
- Kerberos authentication
- Integrating with other authentication systems
- Enabling anonymous access
- Controlling the user cache
- Configuring SAML single-sign-on on the Elastic Stack
- Configuring single sign-on to the Elastic Stack using OpenID Connect
- User authorization
- Built-in roles
- Defining roles
- Security privileges
- Document level security
- Field level security
- Granting privileges for indices and aliases
- Mapping users and groups to roles
- Setting up field and document level security
- Submitting requests on behalf of other users
- Configuring authorization delegation
- Customizing roles and authorization
- Enabling audit logging
- Encrypting communications
- Restricting connections with IP filtering
- Cross cluster search, clients, and integrations
- Tutorial: Getting started with security
- Tutorial: Encrypting communications
- Troubleshooting
- Some settings are not returned via the nodes settings API
- Authorization exceptions
- Users command fails due to extra arguments
- Users are frequently locked out of Active Directory
- Certificate verification fails for curl on Mac
- SSLHandshakeException causes connections to fail
- Common SSL/TLS exceptions
- Common Kerberos exceptions
- Common SAML issues
- Internal Server Error in Kibana
- Setup-passwords command fails due to connection failure
- Failures due to relocation of the configuration files
- Limitations
- Alerting on cluster and index events
- Command line tools
- How To
- Testing
- Glossary of terms
- REST APIs
- API conventions
- cat APIs
- Cluster APIs
- Cross-cluster replication APIs
- Document APIs
- Explore API
- Index APIs
- Add index alias
- Analyze
- Clear cache
- Clone index
- Close index
- Create index
- Delete index
- Delete index alias
- Delete index template
- Flush
- Force merge
- Freeze index
- Get field mapping
- Get index
- Get index alias
- Get index settings
- Get index template
- Get mapping
- Index alias exists
- Index exists
- Index recovery
- Index segments
- Index shard stores
- Index stats
- Index template exists
- Open index
- Put index template
- Put mapping
- Refresh
- Rollover index
- Shrink index
- Split index
- Synced flush
- Type exists
- Unfreeze index
- Update index alias
- Update index settings
- Index lifecycle management API
- Ingest APIs
- Info API
- Licensing APIs
- Machine learning anomaly detection APIs
- Add events to calendar
- Add jobs to calendar
- Close jobs
- Create jobs
- Create calendar
- Create datafeeds
- Create filter
- Delete calendar
- Delete datafeeds
- Delete events from calendar
- Delete filter
- Delete forecast
- Delete jobs
- Delete jobs from calendar
- Delete model snapshots
- Delete expired data
- Find file structure
- Flush jobs
- Forecast jobs
- Get buckets
- Get calendars
- Get categories
- Get datafeeds
- Get datafeed statistics
- Get influencers
- Get jobs
- Get job statistics
- Get machine learning info
- Get model snapshots
- Get overall buckets
- Get scheduled events
- Get filters
- Get records
- Open jobs
- Post data to jobs
- Preview datafeeds
- Revert model snapshots
- Set upgrade mode
- Start datafeeds
- Stop datafeeds
- Update datafeeds
- Update filter
- Update jobs
- Update model snapshots
- Machine learning data frame analytics APIs
- Migration APIs
- Reload search analyzers
- Rollup APIs
- Search APIs
- Security APIs
- Authenticate
- Change passwords
- Clear cache
- Clear roles cache
- Create API keys
- Create or update application privileges
- Create or update role mappings
- Create or update roles
- Create or update users
- Delegate PKI authentication
- Delete application privileges
- Delete role mappings
- Delete roles
- Delete users
- Disable users
- Enable users
- Get API key information
- Get application privileges
- Get builtin privileges
- Get role mappings
- Get roles
- Get token
- Get users
- Has privileges
- Invalidate API key
- Invalidate token
- OpenID Connect Prepare Authentication API
- OpenID Connect authenticate API
- OpenID Connect logout API
- SSL certificate
- Snapshot lifecycle management API
- Transform APIs
- Watcher APIs
- Definitions
- Release highlights
- Breaking changes
- Release notes
- Elasticsearch version 7.4.2
- Elasticsearch version 7.4.1
- Elasticsearch version 7.4.0
- Elasticsearch version 7.3.2
- Elasticsearch version 7.3.1
- Elasticsearch version 7.3.0
- Elasticsearch version 7.2.1
- Elasticsearch version 7.2.0
- Elasticsearch version 7.1.1
- Elasticsearch version 7.1.0
- Elasticsearch version 7.0.0
- Elasticsearch version 7.0.0-rc2
- Elasticsearch version 7.0.0-rc1
- Elasticsearch version 7.0.0-beta1
- Elasticsearch version 7.0.0-alpha2
- Elasticsearch version 7.0.0-alpha1
Split index API
editSplit index API
editSplits an existing index into a new index with more primary shards.
POST /twitter/_split/split-twitter-index { "settings": { "index.number_of_shards": 2 } }
Prerequisites
editBefore you can split an index:
- The index must be read-only.
- The cluster health status must be green.
You can do make an index read-only with the following request:
Description
editThe split index API allows you to split an existing index into a new index, where each original primary shard is split into two or more primary shards in the new index.
The number of times the index can be split (and the number of shards that each
original shard can be split into) is determined by the
index.number_of_routing_shards
setting. The number of routing shards
specifies the hashing space that is used internally to distribute documents
across shards with consistent hashing. For instance, a 5 shard index with
number_of_routing_shards
set to 30
(5 x 2 x 3
) could be split by a
factor of 2
or 3
. In other words, it could be split as follows:
-
5
→10
→30
(split by 2, then by 3) -
5
→15
→30
(split by 3, then by 2) -
5
→30
(split by 6)
While you can set the index.number_of_routing_shards
setting explicitly at
index creation time, the default value depends upon the number of primary
shards in the original index. The default is designed to allow you to split
by factors of 2 up to a maximum of 1024 shards. However, the original number
of primary shards must taken into account. For instance, an index created
with 5 primary shards could be split into 10, 20, 40, 80, 160, 320, or a
maximum of 640 shards (with a single split action or multiple split actions).
If the original index contains one primary shard (or a multi-shard index has been shrunk down to a single primary shard), then the index may by split into an arbitrary number of shards greater than 1. The properties of the default number of routing shards will then apply to the newly split index.
How splitting works
editA split operation:
- Creates a new target index with the same definition as the source index, but with a larger number of primary shards.
- Hard-links segments from the source index into the target index. (If the file system doesn’t support hard-linking, then all segments are copied into the new index, which is a much more time consuming process.)
- Hashes all documents again, after low level files are created, to delete documents that belong to a different shard.
- Recovers the target index as though it were a closed index which had just been re-opened.
Why doesn’t Elasticsearch support incremental resharding?
editGoing from N
shards to N+1
shards, aka. incremental resharding, is indeed a
feature that is supported by many key-value stores. Adding a new shard and
pushing new data to this new shard only is not an option: this would likely be
an indexing bottleneck, and figuring out which shard a document belongs to
given its _id
, which is necessary for get, delete and update requests, would
become quite complex. This means that we need to rebalance existing data using
a different hashing scheme.
The most common way that key-value stores do this efficiently is by using
consistent hashing. Consistent hashing only requires 1/N
-th of the keys to
be relocated when growing the number of shards from N
to N+1
. However
Elasticsearch’s unit of storage, shards, are Lucene indices. Because of their
search-oriented data structure, taking a significant portion of a Lucene index,
be it only 5% of documents, deleting them and indexing them on another shard
typically comes with a much higher cost than with a key-value store. This cost
is kept reasonable when growing the number of shards by a multiplicative factor
as described in the above section: this allows Elasticsearch to perform the
split locally, which in-turn allows to perform the split at the index level
rather than reindexing documents that need to move, as well as using hard links
for efficient file copying.
In the case of append-only data, it is possible to get more flexibility by
creating a new index and pushing new data to it, while adding an alias that
covers both the old and the new index for read operations. Assuming that the
old and new indices have respectively M
and N
shards, this has no overhead
compared to searching an index that would have M+N
shards.
Split an index
editTo split my_source_index
into a new index called my_target_index
, issue
the following request:
POST /my_source_index/_split/my_target_index { "settings": { "index.number_of_shards": 2 } }
The above request returns immediately once the target index has been added to the cluster state — it doesn’t wait for the split operation to start.
Indices can only be split if they satisfy the following requirements:
- the target index must not exist
- The source index must have fewer primary shards than the target index.
- The number of primary shards in the target index must be a multiple of the number of primary shards in the source index.
- The node handling the split process must have sufficient free disk space to accommodate a second copy of the existing index.
The _split
API is similar to the create index
API
and accepts settings
and aliases
parameters for the target index:
POST /my_source_index/_split/my_target_index { "settings": { "index.number_of_shards": 5 }, "aliases": { "my_search_indices": {} } }
The number of shards in the target index. This must be a multiple of the number of shards in the source index. |
Mappings may not be specified in the _split
request.
Monitor the split process
editThe split process can be monitored with the _cat recovery
API, or the cluster health
API can be used to wait
until all primary shards have been allocated by setting the wait_for_status
parameter to yellow
.
The _split
API returns as soon as the target index has been added to the
cluster state, before any shards have been allocated. At this point, all
shards are in the state unassigned
. If, for any reason, the target index
can’t be allocated, its primary shard will remain unassigned
until it
can be allocated on that node.
Once the primary shard is allocated, it moves to state initializing
, and the
split process begins. When the split operation completes, the shard will
become active
. At that point, Elasticsearch will try to allocate any
replicas and may decide to relocate the primary shard to another node.
Wait for active shards
editBecause the split operation creates a new index to split the shards to, the wait for active shards setting on index creation applies to the split index action as well.
Path parameters
edit-
<index>
- (Required, string) Name of the source index to split.
-
<target-index>
-
(Required, string) Name of the target index to create.
Index names must meet the following criteria:
- Lowercase only
-
Cannot include
\
,/
,*
,?
,"
,<
,>
,|
, ` ` (space character),,
,#
-
Indices prior to 7.0 could contain a colon (
:
), but that’s been deprecated and won’t be supported in 7.0+ -
Cannot start with
-
,_
,+
-
Cannot be
.
or..
- Cannot be longer than 255 bytes (note it is bytes, so multi-byte characters will count towards the 255 limit faster)
Query parameters
edit-
wait_for_active_shards
-
(Optional, string) The number of shard copies that must be active before proceeding with the operation. Set to
all
or any positive integer up to the total number of shards in the index (number_of_replicas+1
). Default: 1, the primary shard.See Active shards.
-
timeout
-
(Optional, time units) Specifies the period of time to wait for
a response. If no response is received before the timeout expires, the request
fails and returns an error. Defaults to
30s
. -
master_timeout
-
(Optional, time units) Specifies the period of time to wait for
a connection to the master node. If no response is received before the timeout
expires, the request fails and returns an error. Defaults to
30s
.
Request body
edit-
aliases
- (Optional, alias object) Index aliases which include the target index. See Update index alias.
-
settings
- (Optional, index setting object) Configuration options for the target index. See Index Settings.
On this page
ElasticON events are back!
Learn about the Elastic Search AI Platform from the experts at our live events.
Register now