- Elasticsearch Guide: other versions:
- What is Elasticsearch?
- What’s new in 7.10
- Getting started with Elasticsearch
- Set up Elasticsearch
- Installing Elasticsearch
- Configuring Elasticsearch
- Setting JVM options
- Secure settings
- Auditing settings
- Circuit breaker settings
- Cluster-level shard allocation and routing settings
- Cross-cluster replication settings
- Discovery and cluster formation settings
- Field data cache settings
- HTTP
- Index lifecycle management settings
- Index management settings
- Index recovery settings
- Indexing buffer settings
- License settings
- Local gateway settings
- Logging
- Machine learning settings
- Monitoring settings
- Node
- Network settings
- Node query cache settings
- Search settings
- Security settings
- Shard request cache settings
- Snapshot lifecycle management settings
- Transforms settings
- Transport
- Thread pools
- Watcher settings
- Important Elasticsearch configuration
- Important System Configuration
- Bootstrap Checks
- Heap size check
- File descriptor check
- Memory lock check
- Maximum number of threads check
- Max file size check
- Maximum size virtual memory check
- Maximum map count check
- Client JVM check
- Use serial collector check
- System call filter check
- OnError and OnOutOfMemoryError checks
- Early-access check
- G1GC check
- All permission check
- Discovery configuration check
- Bootstrap Checks for X-Pack
- Starting Elasticsearch
- Stopping Elasticsearch
- Discovery and cluster formation
- Add and remove nodes in your cluster
- Full-cluster restart and rolling restart
- Remote clusters
- Set up X-Pack
- Configuring X-Pack Java Clients
- Plugins
- Upgrade Elasticsearch
- Index modules
- Mapping
- Text analysis
- Overview
- Concepts
- Configure text analysis
- Built-in analyzer reference
- Tokenizer reference
- Token filter reference
- Apostrophe
- ASCII folding
- CJK bigram
- CJK width
- Classic
- Common grams
- Conditional
- Decimal digit
- Delimited payload
- Dictionary decompounder
- Edge n-gram
- Elision
- Fingerprint
- Flatten graph
- Hunspell
- Hyphenation decompounder
- Keep types
- Keep words
- Keyword marker
- Keyword repeat
- KStem
- Length
- Limit token count
- Lowercase
- MinHash
- Multiplexer
- N-gram
- Normalization
- Pattern capture
- Pattern replace
- Phonetic
- Porter stem
- Predicate script
- Remove duplicates
- Reverse
- Shingle
- Snowball
- Stemmer
- Stemmer override
- Stop
- Synonym
- Synonym graph
- Trim
- Truncate
- Unique
- Uppercase
- Word delimiter
- Word delimiter graph
- Character filters reference
- Normalizers
- Index templates
- Data streams
- Ingest node
- Search your data
- Query DSL
- Aggregations
- Bucket aggregations
- Adjacency matrix
- Auto-interval date histogram
- Children
- Composite
- Date histogram
- Date range
- Diversified sampler
- Filter
- Filters
- Geo-distance
- Geohash grid
- Geotile grid
- Global
- Histogram
- IP range
- Missing
- Nested
- Parent
- Range
- Rare terms
- Reverse nested
- Sampler
- Significant terms
- Significant text
- Terms
- Variable width histogram
- Subtleties of bucketing range fields
- Metrics aggregations
- Pipeline aggregations
- Bucket aggregations
- EQL
- SQL access
- Overview
- Getting Started with SQL
- Conventions and Terminology
- Security
- SQL REST API
- SQL Translate API
- SQL CLI
- SQL JDBC
- SQL ODBC
- SQL Client Applications
- SQL Language
- Functions and Operators
- Comparison Operators
- Logical Operators
- Math Operators
- Cast Operators
- LIKE and RLIKE Operators
- Aggregate Functions
- Grouping Functions
- Date/Time and Interval Functions and Operators
- Full-Text Search Functions
- Mathematical Functions
- String Functions
- Type Conversion Functions
- Geo Functions
- Conditional Functions And Expressions
- System Functions
- Reserved keywords
- SQL Limitations
- Scripting
- Data management
- ILM: Manage the index lifecycle
- Overview
- Concepts
- Automate rollover
- Manage Filebeat time-based indices
- Index lifecycle actions
- Configure a lifecycle policy
- Migrate index allocation filters to node roles
- Resolve lifecycle policy execution errors
- Start and stop index lifecycle management
- Manage existing indices
- Skip rollover
- Restore a managed data stream or index
- Monitor a cluster
- Frozen indices
- Roll up or transform your data
- Set up a cluster for high availability
- Snapshot and restore
- Secure a cluster
- Overview
- Configuring security
- User authentication
- Built-in users
- Internal users
- Token-based authentication services
- Realms
- Realm chains
- Active Directory user authentication
- File-based user authentication
- LDAP user authentication
- Native user authentication
- OpenID Connect authentication
- PKI user authentication
- SAML authentication
- Kerberos authentication
- Integrating with other authentication systems
- Enabling anonymous access
- Controlling the user cache
- Configuring SAML single-sign-on on the Elastic Stack
- Configuring single sign-on to the Elastic Stack using OpenID Connect
- User authorization
- Built-in roles
- Defining roles
- Granting access to Stack Management features
- Security privileges
- Document level security
- Field level security
- Granting privileges for data streams and index aliases
- Mapping users and groups to roles
- Setting up field and document level security
- Submitting requests on behalf of other users
- Configuring authorization delegation
- Customizing roles and authorization
- Enabling audit logging
- Encrypting communications
- Restricting connections with IP filtering
- Cross cluster search, clients, and integrations
- Tutorial: Getting started with security
- Tutorial: Encrypting communications
- Troubleshooting
- Some settings are not returned via the nodes settings API
- Authorization exceptions
- Users command fails due to extra arguments
- Users are frequently locked out of Active Directory
- Certificate verification fails for curl on Mac
- SSLHandshakeException causes connections to fail
- Common SSL/TLS exceptions
- Common Kerberos exceptions
- Common SAML issues
- Internal Server Error in Kibana
- Setup-passwords command fails due to connection failure
- Failures due to relocation of the configuration files
- Limitations
- Watch for cluster and index events
- Command line tools
- How To
- Glossary of terms
- REST APIs
- API conventions
- Compact and aligned text (CAT) APIs
- cat aliases
- cat allocation
- cat anomaly detectors
- cat count
- cat data frame analytics
- cat datafeeds
- cat fielddata
- cat health
- cat indices
- cat master
- cat nodeattrs
- cat nodes
- cat pending tasks
- cat plugins
- cat recovery
- cat repositories
- cat segments
- cat shards
- cat snapshots
- cat task management
- cat templates
- cat thread pool
- cat trained model
- cat transforms
- Cluster APIs
- Cluster allocation explain
- Cluster get settings
- Cluster health
- Cluster reroute
- Cluster state
- Cluster stats
- Cluster update settings
- Nodes feature usage
- Nodes hot threads
- Nodes info
- Nodes reload secure settings
- Nodes stats
- Pending cluster tasks
- Remote cluster info
- Task management
- Voting configuration exclusions
- Cross-cluster replication APIs
- Data stream APIs
- Document APIs
- Enrich APIs
- Graph explore API
- Index APIs
- Add index alias
- Analyze
- Clear cache
- Clone index
- Close index
- Create index
- Delete index
- Delete index alias
- Delete component template
- Delete index template
- Delete index template (legacy)
- Flush
- Force merge
- Freeze index
- Get component template
- Get field mapping
- Get index
- Get index alias
- Get index settings
- Get index template
- Get index template (legacy)
- Get mapping
- Index alias exists
- Index exists
- Index recovery
- Index segments
- Index shard stores
- Index stats
- Index template exists (legacy)
- Open index
- Put index template
- Put index template (legacy)
- Put component template
- Put mapping
- Refresh
- Rollover index
- Shrink index
- Simulate index
- Simulate template
- Split index
- Synced flush
- Type exists
- Unfreeze index
- Update index alias
- Update index settings
- Resolve index
- List dangling indices
- Import dangling index
- Delete dangling index
- Index lifecycle management APIs
- Ingest APIs
- Info API
- Licensing APIs
- Machine learning anomaly detection APIs
- Add events to calendar
- Add jobs to calendar
- Close jobs
- Create jobs
- Create calendars
- Create datafeeds
- Create filters
- Delete calendars
- Delete datafeeds
- Delete events from calendar
- Delete filters
- Delete forecasts
- Delete jobs
- Delete jobs from calendar
- Delete model snapshots
- Delete expired data
- Estimate model memory
- Find file structure
- Flush jobs
- Forecast jobs
- Get buckets
- Get calendars
- Get categories
- Get datafeeds
- Get datafeed statistics
- Get influencers
- Get jobs
- Get job statistics
- Get machine learning info
- Get model snapshots
- Get overall buckets
- Get scheduled events
- Get filters
- Get records
- Open jobs
- Post data to jobs
- Preview datafeeds
- Revert model snapshots
- Set upgrade mode
- Start datafeeds
- Stop datafeeds
- Update datafeeds
- Update filters
- Update jobs
- Update model snapshots
- Machine learning data frame analytics APIs
- Create data frame analytics jobs
- Create trained models
- Update data frame analytics jobs
- Delete data frame analytics jobs
- Delete trained models
- Evaluate data frame analytics
- Explain data frame analytics
- Get data frame analytics jobs
- Get data frame analytics jobs stats
- Get trained models
- Get trained models stats
- Start data frame analytics jobs
- Stop data frame analytics jobs
- Migration APIs
- Reload search analyzers API
- Repositories metering APIs
- Rollup APIs
- Search APIs
- Searchable snapshots APIs
- Security APIs
- Authenticate
- Change passwords
- Clear cache
- Clear roles cache
- Clear privileges cache
- Clear API key cache
- Create API keys
- Create or update application privileges
- Create or update role mappings
- Create or update roles
- Create or update users
- Delegate PKI authentication
- Delete application privileges
- Delete role mappings
- Delete roles
- Delete users
- Disable users
- Enable users
- Get API key information
- Get application privileges
- Get builtin privileges
- Get role mappings
- Get roles
- Get token
- Get users
- Grant API keys
- Has privileges
- Invalidate API key
- Invalidate token
- OpenID Connect prepare authentication
- OpenID Connect authenticate
- OpenID Connect logout
- SAML prepare authentication
- SAML authenticate
- SAML logout
- SAML invalidate
- SSL certificate
- Snapshot and restore APIs
- Snapshot lifecycle management APIs
- Transform APIs
- Usage API
- Watcher APIs
- Definitions
- Migration guide
- Release notes
- Elasticsearch version 7.10.2
- Elasticsearch version 7.10.1
- Elasticsearch version 7.10.0
- Elasticsearch version 7.9.3
- Elasticsearch version 7.9.2
- Elasticsearch version 7.9.1
- Elasticsearch version 7.9.0
- Elasticsearch version 7.8.1
- Elasticsearch version 7.8.0
- Elasticsearch version 7.7.1
- Elasticsearch version 7.7.0
- Elasticsearch version 7.6.2
- Elasticsearch version 7.6.1
- Elasticsearch version 7.6.0
- Elasticsearch version 7.5.2
- Elasticsearch version 7.5.1
- Elasticsearch version 7.5.0
- Elasticsearch version 7.4.2
- Elasticsearch version 7.4.1
- Elasticsearch version 7.4.0
- Elasticsearch version 7.3.2
- Elasticsearch version 7.3.1
- Elasticsearch version 7.3.0
- Elasticsearch version 7.2.1
- Elasticsearch version 7.2.0
- Elasticsearch version 7.1.1
- Elasticsearch version 7.1.0
- Elasticsearch version 7.0.0
- Elasticsearch version 7.0.0-rc2
- Elasticsearch version 7.0.0-rc1
- Elasticsearch version 7.0.0-beta1
- Elasticsearch version 7.0.0-alpha2
- Elasticsearch version 7.0.0-alpha1
- Dependencies and versions
Word delimiter graph token filter
editWord delimiter graph token filter
editSplits tokens at non-alphanumeric characters. The word_delimiter_graph
filter
also performs optional token normalization based on a set of rules. By default,
the filter uses the following rules:
-
Split tokens at non-alphanumeric characters.
The filter uses these characters as delimiters.
For example:
Super-Duper
→Super
,Duper
-
Remove leading or trailing delimiters from each token.
For example:
XL---42+'Autocoder'
→XL
,42
,Autocoder
-
Split tokens at letter case transitions.
For example:
PowerShot
→Power
,Shot
-
Split tokens at letter-number transitions.
For example:
XL500
→XL
,500
-
Remove the English possessive (
's
) from the end of each token. For example:Neil's
→Neil
The word_delimiter_graph
filter uses Lucene’s
WordDelimiterGraphFilter.
The word_delimiter_graph
filter was designed to remove punctuation from
complex identifiers, such as product IDs or part numbers. For these use cases,
we recommend using the word_delimiter_graph
filter with the
keyword
tokenizer.
Avoid using the word_delimiter_graph
filter to split hyphenated words, such as
wi-fi
. Because users often search for these words both with and without
hyphens, we recommend using the
synonym_graph
filter instead.
Example
editThe following analyze API request uses the
word_delimiter_graph
filter to split Neil's-Super-Duper-XL500--42+AutoCoder
into normalized tokens using the filter’s default rules:
GET /_analyze { "tokenizer": "keyword", "filter": [ "word_delimiter_graph" ], "text": "Neil's-Super-Duper-XL500--42+AutoCoder" }
The filter produces the following tokens:
[ Neil, Super, Duper, XL, 500, 42, Auto, Coder ]
Add to an analyzer
editThe following create index API request uses the
word_delimiter_graph
filter to configure a new
custom analyzer.
PUT /my-index-000001 { "settings": { "analysis": { "analyzer": { "my_analyzer": { "tokenizer": "keyword", "filter": [ "word_delimiter_graph" ] } } } } }
Avoid using the word_delimiter_graph
filter with tokenizers that remove
punctuation, such as the standard
tokenizer.
This could prevent the word_delimiter_graph
filter from splitting tokens
correctly. It can also interfere with the filter’s configurable parameters, such
as catenate_all
or
preserve_original
. We
recommend using the keyword
or
whitespace
tokenizer instead.
Configurable parameters
edit-
adjust_offsets
-
(Optional, Boolean) If
true
, the filter adjusts the offsets of split or catenated tokens to better reflect their actual position in the token stream. Defaults totrue
.Set
adjust_offsets
tofalse
if your analyzer uses filters, such as thetrim
filter, that change the length of tokens without changing their offsets. Otherwise, theword_delimiter_graph
filter could produce tokens with illegal offsets.
-
catenate_all
-
(Optional, Boolean) If
true
, the filter produces catenated tokens for chains of alphanumeric characters separated by non-alphabetic delimiters. For example:super-duper-xl-500
→ [superduperxl500
,super
,duper
,xl
,500
]. Defaults tofalse
.Setting this parameter to
true
produces multi-position tokens, which are not supported by indexing.If this parameter is
true
, avoid using this filter in an index analyzer or use theflatten_graph
filter after this filter to make the token stream suitable for indexing.When used for search analysis, catenated tokens can cause problems for the
match_phrase
query and other queries that rely on token position for matching. Avoid setting this parameter totrue
if you plan to use these queries.
-
catenate_numbers
-
(Optional, Boolean) If
true
, the filter produces catenated tokens for chains of numeric characters separated by non-alphabetic delimiters. For example:01-02-03
→ [010203
,01
,02
,03
]. Defaults tofalse
.Setting this parameter to
true
produces multi-position tokens, which are not supported by indexing.If this parameter is
true
, avoid using this filter in an index analyzer or use theflatten_graph
filter after this filter to make the token stream suitable for indexing.When used for search analysis, catenated tokens can cause problems for the
match_phrase
query and other queries that rely on token position for matching. Avoid setting this parameter totrue
if you plan to use these queries.
-
catenate_words
-
(Optional, Boolean) If
true
, the filter produces catenated tokens for chains of alphabetical characters separated by non-alphabetic delimiters. For example:super-duper-xl
→ [superduperxl
,super
,duper
,xl
]. Defaults tofalse
.Setting this parameter to
true
produces multi-position tokens, which are not supported by indexing.If this parameter is
true
, avoid using this filter in an index analyzer or use theflatten_graph
filter after this filter to make the token stream suitable for indexing.When used for search analysis, catenated tokens can cause problems for the
match_phrase
query and other queries that rely on token position for matching. Avoid setting this parameter totrue
if you plan to use these queries. -
generate_number_parts
-
(Optional, Boolean)
If
true
, the filter includes tokens consisting of only numeric characters in the output. Iffalse
, the filter excludes these tokens from the output. Defaults totrue
. -
generate_word_parts
-
(Optional, Boolean)
If
true
, the filter includes tokens consisting of only alphabetical characters in the output. Iffalse
, the filter excludes these tokens from the output. Defaults totrue
. -
ignore_keywords
-
(Optional, Boolean)
If
true
, the filter skips tokens with akeyword
attribute oftrue
. Defaults tofalse
.
-
preserve_original
-
(Optional, Boolean) If
true
, the filter includes the original version of any split tokens in the output. This original version includes non-alphanumeric delimiters. For example:super-duper-xl-500
→ [super-duper-xl-500
,super
,duper
,xl
,500
]. Defaults tofalse
.Setting this parameter to
true
produces multi-position tokens, which are not supported by indexing.If this parameter is
true
, avoid using this filter in an index analyzer or use theflatten_graph
filter after this filter to make the token stream suitable for indexing. -
protected_words
- (Optional, array of strings) Array of tokens the filter won’t split.
-
protected_words_path
-
(Optional, string) Path to a file that contains a list of tokens the filter won’t split.
This path must be absolute or relative to the
config
location, and the file must be UTF-8 encoded. Each token in the file must be separated by a line break. -
split_on_case_change
-
(Optional, Boolean)
If
true
, the filter splits tokens at letter case transitions. For example:camelCase
→ [camel
,Case
]. Defaults totrue
. -
split_on_numerics
-
(Optional, Boolean)
If
true
, the filter splits tokens at letter-number transitions. For example:j2se
→ [j
,2
,se
]. Defaults totrue
. -
stem_english_possessive
-
(Optional, Boolean)
If
true
, the filter removes the English possessive ('s
) from the end of each token. For example:O'Neil's
→ [O
,Neil
]. Defaults totrue
. -
type_table
-
(Optional, array of strings) Array of custom type mappings for characters. This allows you to map non-alphanumeric characters as numeric or alphanumeric to avoid splitting on those characters.
For example, the following array maps the plus (
+
) and hyphen (-
) characters as alphanumeric, which means they won’t be treated as delimiters:[ "+ => ALPHA", "- => ALPHA" ]
Supported types include:
-
ALPHA
(Alphabetical) -
ALPHANUM
(Alphanumeric) -
DIGIT
(Numeric) -
LOWER
(Lowercase alphabetical) -
SUBWORD_DELIM
(Non-alphanumeric delimiter) -
UPPER
(Uppercase alphabetical)
-
-
type_table_path
-
(Optional, string) Path to a file that contains custom type mappings for characters. This allows you to map non-alphanumeric characters as numeric or alphanumeric to avoid splitting on those characters.
For example, the contents of this file may contain the following:
# Map the $, %, '.', and ',' characters to DIGIT # This might be useful for financial data. $ => DIGIT % => DIGIT . => DIGIT \\u002C => DIGIT # in some cases you might not want to split on ZWJ # this also tests the case where we need a bigger byte[] # see https://en.wikipedia.org/wiki/Zero-width_joiner \\u200D => ALPHANUM
Supported types include:
-
ALPHA
(Alphabetical) -
ALPHANUM
(Alphanumeric) -
DIGIT
(Numeric) -
LOWER
(Lowercase alphabetical) -
SUBWORD_DELIM
(Non-alphanumeric delimiter) -
UPPER
(Uppercase alphabetical)
This file path must be absolute or relative to the
config
location, and the file must be UTF-8 encoded. Each mapping in the file must be separated by a line break. -
Customize
editTo customize the word_delimiter_graph
filter, duplicate it to create the basis
for a new custom token filter. You can modify the filter using its configurable
parameters.
For example, the following request creates a word_delimiter_graph
filter that uses the following rules:
-
Split tokens at non-alphanumeric characters, except the hyphen (
-
) character. - Remove leading or trailing delimiters from each token.
- Do not split tokens at letter case transitions.
- Do not split tokens at letter-number transitions.
-
Remove the English possessive (
's
) from the end of each token.
PUT /my-index-000001 { "settings": { "analysis": { "analyzer": { "my_analyzer": { "tokenizer": "keyword", "filter": [ "my_custom_word_delimiter_graph_filter" ] } }, "filter": { "my_custom_word_delimiter_graph_filter": { "type": "word_delimiter_graph", "type_table": [ "- => ALPHA" ], "split_on_case_change": false, "split_on_numerics": false, "stem_english_possessive": true } } } } }
Differences between word_delimiter_graph
and word_delimiter
editBoth the word_delimiter_graph
and
word_delimiter
filters produce tokens
that span multiple positions when any of the following parameters are true
:
However, only the word_delimiter_graph
filter assigns multi-position tokens a
positionLength
attribute, which indicates the number of positions a token
spans. This ensures the word_delimiter_graph
filter always produces valid
token graphs.
The word_delimiter
filter does not assign multi-position tokens a
positionLength
attribute. This means it produces invalid graphs for streams
including these tokens.
While indexing does not support token graphs containing multi-position tokens,
queries, such as the match_phrase
query, can
use these graphs to generate multiple sub-queries from a single query string.
To see how token graphs produced by the word_delimiter
and
word_delimiter_graph
filters differ, check out the following example.
Example
Both the word_delimiter
and word_delimiter_graph
produce the following token
graph for PowerShot2000
when the following parameters are false
:
This graph does not contain multi-position tokens. All tokens span only one position.
word_delimiter_graph
graph with a multi-position token
The word_delimiter_graph
filter produces the following token graph for
PowerShot2000
when catenate_words
is true
.
This graph correctly indicates the catenated PowerShot
token spans two
positions.
word_delimiter
graph with a multi-position token
When catenate_words
is true
, the word_delimiter
filter produces
the following token graph for PowerShot2000
.
Note that the catenated PowerShot
token should span two positions but only
spans one in the token graph, making it invalid.
On this page