- Elasticsearch Guide: other versions:
- What is Elasticsearch?
- What’s new in 7.10
- Getting started with Elasticsearch
- Set up Elasticsearch
- Installing Elasticsearch
- Configuring Elasticsearch
- Setting JVM options
- Secure settings
- Auditing settings
- Circuit breaker settings
- Cluster-level shard allocation and routing settings
- Cross-cluster replication settings
- Discovery and cluster formation settings
- Field data cache settings
- HTTP
- Index lifecycle management settings
- Index management settings
- Index recovery settings
- Indexing buffer settings
- License settings
- Local gateway settings
- Logging
- Machine learning settings
- Monitoring settings
- Node
- Network settings
- Node query cache settings
- Search settings
- Security settings
- Shard request cache settings
- Snapshot lifecycle management settings
- Transforms settings
- Transport
- Thread pools
- Watcher settings
- Important Elasticsearch configuration
- Important System Configuration
- Bootstrap Checks
- Heap size check
- File descriptor check
- Memory lock check
- Maximum number of threads check
- Max file size check
- Maximum size virtual memory check
- Maximum map count check
- Client JVM check
- Use serial collector check
- System call filter check
- OnError and OnOutOfMemoryError checks
- Early-access check
- G1GC check
- All permission check
- Discovery configuration check
- Bootstrap Checks for X-Pack
- Starting Elasticsearch
- Stopping Elasticsearch
- Discovery and cluster formation
- Add and remove nodes in your cluster
- Full-cluster restart and rolling restart
- Remote clusters
- Set up X-Pack
- Configuring X-Pack Java Clients
- Plugins
- Upgrade Elasticsearch
- Index modules
- Mapping
- Text analysis
- Overview
- Concepts
- Configure text analysis
- Built-in analyzer reference
- Tokenizer reference
- Token filter reference
- Apostrophe
- ASCII folding
- CJK bigram
- CJK width
- Classic
- Common grams
- Conditional
- Decimal digit
- Delimited payload
- Dictionary decompounder
- Edge n-gram
- Elision
- Fingerprint
- Flatten graph
- Hunspell
- Hyphenation decompounder
- Keep types
- Keep words
- Keyword marker
- Keyword repeat
- KStem
- Length
- Limit token count
- Lowercase
- MinHash
- Multiplexer
- N-gram
- Normalization
- Pattern capture
- Pattern replace
- Phonetic
- Porter stem
- Predicate script
- Remove duplicates
- Reverse
- Shingle
- Snowball
- Stemmer
- Stemmer override
- Stop
- Synonym
- Synonym graph
- Trim
- Truncate
- Unique
- Uppercase
- Word delimiter
- Word delimiter graph
- Character filters reference
- Normalizers
- Index templates
- Data streams
- Ingest node
- Search your data
- Query DSL
- Aggregations
- Bucket aggregations
- Adjacency matrix
- Auto-interval date histogram
- Children
- Composite
- Date histogram
- Date range
- Diversified sampler
- Filter
- Filters
- Geo-distance
- Geohash grid
- Geotile grid
- Global
- Histogram
- IP range
- Missing
- Nested
- Parent
- Range
- Rare terms
- Reverse nested
- Sampler
- Significant terms
- Significant text
- Terms
- Variable width histogram
- Subtleties of bucketing range fields
- Metrics aggregations
- Pipeline aggregations
- Bucket aggregations
- EQL
- SQL access
- Overview
- Getting Started with SQL
- Conventions and Terminology
- Security
- SQL REST API
- SQL Translate API
- SQL CLI
- SQL JDBC
- SQL ODBC
- SQL Client Applications
- SQL Language
- Functions and Operators
- Comparison Operators
- Logical Operators
- Math Operators
- Cast Operators
- LIKE and RLIKE Operators
- Aggregate Functions
- Grouping Functions
- Date/Time and Interval Functions and Operators
- Full-Text Search Functions
- Mathematical Functions
- String Functions
- Type Conversion Functions
- Geo Functions
- Conditional Functions And Expressions
- System Functions
- Reserved keywords
- SQL Limitations
- Scripting
- Data management
- ILM: Manage the index lifecycle
- Overview
- Concepts
- Automate rollover
- Manage Filebeat time-based indices
- Index lifecycle actions
- Configure a lifecycle policy
- Migrate index allocation filters to node roles
- Resolve lifecycle policy execution errors
- Start and stop index lifecycle management
- Manage existing indices
- Skip rollover
- Restore a managed data stream or index
- Monitor a cluster
- Frozen indices
- Roll up or transform your data
- Set up a cluster for high availability
- Snapshot and restore
- Secure a cluster
- Overview
- Configuring security
- User authentication
- Built-in users
- Internal users
- Token-based authentication services
- Realms
- Realm chains
- Active Directory user authentication
- File-based user authentication
- LDAP user authentication
- Native user authentication
- OpenID Connect authentication
- PKI user authentication
- SAML authentication
- Kerberos authentication
- Integrating with other authentication systems
- Enabling anonymous access
- Controlling the user cache
- Configuring SAML single-sign-on on the Elastic Stack
- Configuring single sign-on to the Elastic Stack using OpenID Connect
- User authorization
- Built-in roles
- Defining roles
- Granting access to Stack Management features
- Security privileges
- Document level security
- Field level security
- Granting privileges for data streams and index aliases
- Mapping users and groups to roles
- Setting up field and document level security
- Submitting requests on behalf of other users
- Configuring authorization delegation
- Customizing roles and authorization
- Enabling audit logging
- Encrypting communications
- Restricting connections with IP filtering
- Cross cluster search, clients, and integrations
- Tutorial: Getting started with security
- Tutorial: Encrypting communications
- Troubleshooting
- Some settings are not returned via the nodes settings API
- Authorization exceptions
- Users command fails due to extra arguments
- Users are frequently locked out of Active Directory
- Certificate verification fails for curl on Mac
- SSLHandshakeException causes connections to fail
- Common SSL/TLS exceptions
- Common Kerberos exceptions
- Common SAML issues
- Internal Server Error in Kibana
- Setup-passwords command fails due to connection failure
- Failures due to relocation of the configuration files
- Limitations
- Watch for cluster and index events
- Command line tools
- How To
- Glossary of terms
- REST APIs
- API conventions
- Compact and aligned text (CAT) APIs
- cat aliases
- cat allocation
- cat anomaly detectors
- cat count
- cat data frame analytics
- cat datafeeds
- cat fielddata
- cat health
- cat indices
- cat master
- cat nodeattrs
- cat nodes
- cat pending tasks
- cat plugins
- cat recovery
- cat repositories
- cat segments
- cat shards
- cat snapshots
- cat task management
- cat templates
- cat thread pool
- cat trained model
- cat transforms
- Cluster APIs
- Cluster allocation explain
- Cluster get settings
- Cluster health
- Cluster reroute
- Cluster state
- Cluster stats
- Cluster update settings
- Nodes feature usage
- Nodes hot threads
- Nodes info
- Nodes reload secure settings
- Nodes stats
- Pending cluster tasks
- Remote cluster info
- Task management
- Voting configuration exclusions
- Cross-cluster replication APIs
- Data stream APIs
- Document APIs
- Enrich APIs
- Graph explore API
- Index APIs
- Add index alias
- Analyze
- Clear cache
- Clone index
- Close index
- Create index
- Delete index
- Delete index alias
- Delete component template
- Delete index template
- Delete index template (legacy)
- Flush
- Force merge
- Freeze index
- Get component template
- Get field mapping
- Get index
- Get index alias
- Get index settings
- Get index template
- Get index template (legacy)
- Get mapping
- Index alias exists
- Index exists
- Index recovery
- Index segments
- Index shard stores
- Index stats
- Index template exists (legacy)
- Open index
- Put index template
- Put index template (legacy)
- Put component template
- Put mapping
- Refresh
- Rollover index
- Shrink index
- Simulate index
- Simulate template
- Split index
- Synced flush
- Type exists
- Unfreeze index
- Update index alias
- Update index settings
- Resolve index
- List dangling indices
- Import dangling index
- Delete dangling index
- Index lifecycle management APIs
- Ingest APIs
- Info API
- Licensing APIs
- Machine learning anomaly detection APIs
- Add events to calendar
- Add jobs to calendar
- Close jobs
- Create jobs
- Create calendars
- Create datafeeds
- Create filters
- Delete calendars
- Delete datafeeds
- Delete events from calendar
- Delete filters
- Delete forecasts
- Delete jobs
- Delete jobs from calendar
- Delete model snapshots
- Delete expired data
- Estimate model memory
- Find file structure
- Flush jobs
- Forecast jobs
- Get buckets
- Get calendars
- Get categories
- Get datafeeds
- Get datafeed statistics
- Get influencers
- Get jobs
- Get job statistics
- Get machine learning info
- Get model snapshots
- Get overall buckets
- Get scheduled events
- Get filters
- Get records
- Open jobs
- Post data to jobs
- Preview datafeeds
- Revert model snapshots
- Set upgrade mode
- Start datafeeds
- Stop datafeeds
- Update datafeeds
- Update filters
- Update jobs
- Update model snapshots
- Machine learning data frame analytics APIs
- Create data frame analytics jobs
- Create trained models
- Update data frame analytics jobs
- Delete data frame analytics jobs
- Delete trained models
- Evaluate data frame analytics
- Explain data frame analytics
- Get data frame analytics jobs
- Get data frame analytics jobs stats
- Get trained models
- Get trained models stats
- Start data frame analytics jobs
- Stop data frame analytics jobs
- Migration APIs
- Reload search analyzers API
- Repositories metering APIs
- Rollup APIs
- Search APIs
- Searchable snapshots APIs
- Security APIs
- Authenticate
- Change passwords
- Clear cache
- Clear roles cache
- Clear privileges cache
- Clear API key cache
- Create API keys
- Create or update application privileges
- Create or update role mappings
- Create or update roles
- Create or update users
- Delegate PKI authentication
- Delete application privileges
- Delete role mappings
- Delete roles
- Delete users
- Disable users
- Enable users
- Get API key information
- Get application privileges
- Get builtin privileges
- Get role mappings
- Get roles
- Get token
- Get users
- Grant API keys
- Has privileges
- Invalidate API key
- Invalidate token
- OpenID Connect prepare authentication
- OpenID Connect authenticate
- OpenID Connect logout
- SAML prepare authentication
- SAML authenticate
- SAML logout
- SAML invalidate
- SSL certificate
- Snapshot and restore APIs
- Snapshot lifecycle management APIs
- Transform APIs
- Usage API
- Watcher APIs
- Definitions
- Migration guide
- Release notes
- Elasticsearch version 7.10.2
- Elasticsearch version 7.10.1
- Elasticsearch version 7.10.0
- Elasticsearch version 7.9.3
- Elasticsearch version 7.9.2
- Elasticsearch version 7.9.1
- Elasticsearch version 7.9.0
- Elasticsearch version 7.8.1
- Elasticsearch version 7.8.0
- Elasticsearch version 7.7.1
- Elasticsearch version 7.7.0
- Elasticsearch version 7.6.2
- Elasticsearch version 7.6.1
- Elasticsearch version 7.6.0
- Elasticsearch version 7.5.2
- Elasticsearch version 7.5.1
- Elasticsearch version 7.5.0
- Elasticsearch version 7.4.2
- Elasticsearch version 7.4.1
- Elasticsearch version 7.4.0
- Elasticsearch version 7.3.2
- Elasticsearch version 7.3.1
- Elasticsearch version 7.3.0
- Elasticsearch version 7.2.1
- Elasticsearch version 7.2.0
- Elasticsearch version 7.1.1
- Elasticsearch version 7.1.0
- Elasticsearch version 7.0.0
- Elasticsearch version 7.0.0-rc2
- Elasticsearch version 7.0.0-rc1
- Elasticsearch version 7.0.0-beta1
- Elasticsearch version 7.0.0-alpha2
- Elasticsearch version 7.0.0-alpha1
- Dependencies and versions
Transform limitations
editTransform limitations
editThe following limitations and known problems apply to the 7.10.2 release of the Elastic transform feature:
Transforms UI will not work during a rolling upgrade from 7.2
editIf your cluster contains mixed version nodes, for example during a rolling upgrade from 7.2 to a newer version, and transforms have been created in 7.2, the transforms UI (earler data frame UI) will not work. Please wait until all nodes have been upgraded to the newer version before using the transforms UI.
Transforms reassignment suspended during a rolling upgrade from 7.2 and 7.3
editIf your cluster contains mixed version nodes, for example during a rolling upgrade from 7.2 or 7.3 to a newer version, transforms whose nodes are stopped will not be reassigned until the upgrade is complete. After the upgrade is done, transforms resume automatically; no action is required.
Data frame data type limitation
editData frames do not (yet) support fields containing arrays – in the UI or the API. If you try to create one, the UI will fail to show the source index table.
Up to 1,000 transforms are supported
editA single cluster will support up to 1,000 transforms. When using the
GET transforms API a total count
of transforms
is returned. Use the size
and from
parameters to enumerate through the full
list.
Aggregation responses may be incompatible with destination index mappings
editWhen a transform is first started, it will deduce the mappings
required for the destination index. This process is based on the field types of
the source index and the aggregations used. If the fields are derived from
scripted_metrics
or bucket_scripts
,
dynamic mappings will be used. In some instances the
deduced mappings may be incompatible with the actual data. For example, numeric
overflows might occur or dynamically mapped fields might contain both numbers
and strings. Please check Elasticsearch logs if you think this may have occurred.
You can view the deduced mappings by using the
Preview transform API.
See the generated_dest_index
object in the API response.
If it’s required, you may define custom mappings prior to starting the transform by creating a custom destination index using the Create index API. As deduced mappings cannot be overwritten by an index template, use the Create index API to define custom mappings. The index templates only apply to fields derived from scripts that use dynamic mappings.
Batch transforms may not account for changed documents
editA batch transform uses a composite aggregation which allows efficient pagination through all buckets. Composite aggregations do not yet support a search context, therefore if the source data is changed (deleted, updated, added) while the batch data frame is in progress, then the results may not include these changes.
Continuous transform consistency does not account for deleted or updated documents
editWhile the process for transforms allows the continual recalculation of the transform as new data is being ingested, it does also have some limitations.
Changed entities will only be identified if their time field has also been updated and falls within the range of the action to check for changes. This has been designed in principle for, and is suited to, the use case where new data is given a timestamp for the time of ingest.
If the indices that fall within the scope of the source index pattern are removed, for example when deleting historical time-based indices, then the composite aggregation performed in consecutive checkpoint processing will search over different source data, and entities that only existed in the deleted index will not be removed from the data frame destination index.
Depending on your use case, you may wish to recreate the transform entirely after deletions. Alternatively, if your use case is tolerant to historical archiving, you may wish to include a max ingest timestamp in your aggregation. This will allow you to exclude results that have not been recently updated when viewing the destination index.
Deleting a transform does not delete the destination index or Kibana index pattern
editWhen deleting a transform using DELETE _transform/index
neither the destination index nor the Kibana index pattern, should one have been
created, are deleted. These objects must be deleted separately.
Handling dynamic adjustment of aggregation page size
editDuring the development of transforms, control was favoured over performance. In the design considerations, it is preferred for the transform to take longer to complete quietly in the background rather than to finish quickly and take precedence in resource consumption.
Composite aggregations are well suited for high cardinality data enabling pagination through results. If a circuit breaker memory exception occurs when performing the composite aggregated search then we try again reducing the number of buckets requested. This circuit breaker is calculated based upon all activity within the cluster, not just activity from transforms, so it therefore may only be a temporary resource availability issue.
For a batch transform, the number of buckets requested is only ever adjusted downwards. The lowering of value may result in a longer duration for the transform checkpoint to complete. For continuous transforms, the number of buckets requested is reset back to its default at the start of every checkpoint and it is possible for circuit breaker exceptions to occur repeatedly in the Elasticsearch logs.
The transform retrieves data in batches which means it calculates several
buckets at once. Per default this is 500 buckets per search/index operation. The
default can be changed using max_page_search_size
and the minimum value is 10.
If failures still occur once the number of buckets requested has been reduced to
its minimum, then the transform will be set to a failed state.
Handling dynamic adjustments for many terms
editFor each checkpoint, entities are identified that have changed since the last time the check was performed. This list of changed entities is supplied as a terms query to the transform composite aggregation, one page at a time. Then updates are applied to the destination index for each page of entities.
The page size
is defined by max_page_search_size
which is also used to
define the number of buckets returned by the composite aggregation search. The
default value is 500, the minimum is 10.
The index setting index.max_terms_count
defines
the maximum number of terms that can be used in a terms query. The default value
is 65536. If max_page_search_size
exceeds index.max_terms_count
the
transform will fail.
Using smaller values for max_page_search_size
may result in a longer duration
for the transform checkpoint to complete.
Continuous transform scheduling limitations
editA continuous transform periodically checks for changes to source data. The functionality
of the scheduler is currently limited to a basic periodic timer which can be
within the frequency
range from 1s to 1h. The default is 1m. This is designed
to run little and often. When choosing a frequency
for this timer consider
your ingest rate along with the impact that the transform
search/index operations has other users in your cluster. Also note that retries
occur at frequency
interval.
Handling of failed transforms
editFailed transforms remain as a persistent task and should be handled appropriately, either by deleting it or by resolving the root cause of the failure and re-starting.
When using the API to delete a failed transform, first stop it using
_stop?force=true
, then delete it.
Continuous transforms may give incorrect results if documents are not yet available to search
editAfter a document is indexed, there is a very small delay until it is available to search.
A continuous transform periodically checks for changed entities between the time since
it last checked and now
minus sync.time.delay
. This time window moves
without overlapping. If the timestamp of a recently indexed document falls
within this time window but this document is not yet available to search then
this entity will not be updated.
If using a sync.time.field
that represents the data ingest time and using a
zero second or very small sync.time.delay
, then it is more likely that this
issue will occur.
Support for date nanoseconds data type
editIf your data uses the date nanosecond data type, aggregations are nonetheless on millisecond resolution. This limitation also affects the aggregations in your transforms.
Data streams as destination indices are not supported
editTransforms update data in the destination index which requires writing into the destination. Data streams are designed to be append-only, which means you cannot send update or delete requests directly to a data stream. For this reason, data streams are not supported as destination indices for transforms.
ILM as destination index may cause duplicated documents
editILM is not recommended to use as a transform destination index. Transforms update documents in the current destination, and cannot delete documents in the indices previously used by ILM. This may lead to duplicated documents when you use transforms combined with ILM in case of a rollover.
If you use ILM to have time-based indices, please consider using the
Date index name instead. The processor works without duplicated
documents if your transform contains a group_by
based on date_histogram
.
Using scripts in transforms
editTransforms support scripting in every case when aggregations support them. However, there are certain factors you might want to consider when using scripts in transforms:
- Transforms cannot deduce index mappings for output fields when the fields are created by a script. In this case, you might want to create the mappings of the destination index yourself prior to creating the transform.
- Scripted fields may increase the runtime of the transform.
-
Transforms cannot optimize queries when you use scripts for all the
groupings defined in
group_by
, you will receive a warning message when you use scripts this way.
On this page
- Transforms UI will not work during a rolling upgrade from 7.2
- Transforms reassignment suspended during a rolling upgrade from 7.2 and 7.3
- Data frame data type limitation
- Up to 1,000 transforms are supported
- Aggregation responses may be incompatible with destination index mappings
- Batch transforms may not account for changed documents
- Continuous transform consistency does not account for deleted or updated documents
- Deleting a transform does not delete the destination index or Kibana index pattern
- Handling dynamic adjustment of aggregation page size
- Handling dynamic adjustments for many terms
- Continuous transform scheduling limitations
- Handling of failed transforms
- Continuous transforms may give incorrect results if documents are not yet available to search
- Support for date nanoseconds data type
- Data streams as destination indices are not supported
- ILM as destination index may cause duplicated documents
- Using scripts in transforms