N-gram tokenizer

The ngram tokenizer first breaks text down into words whenever it encounters one of a list of specified characters, then it emits N-grams of each word of the specified length.

N-grams are like a sliding window that moves across the word - a continuous sequence of characters of the specified length. They are useful for querying languages that don’t use spaces or that have long compound words, like German.

Example output

edit

With the default settings, the ngram tokenizer treats the initial text as a single token and produces N-grams with minimum length 1 and maximum length 2:

resp = client.indices.analyze(
    tokenizer="ngram",
    text="Quick Fox",
)
print(resp)

response = client.indices.analyze(
  body: {
    tokenizer: 'ngram',
    text: 'Quick Fox'
  }
)
puts response

const response = await client.indices.analyze({
  tokenizer: "ngram",
  text: "Quick Fox",
});
console.log(response);

POST _analyze
{
  "tokenizer": "ngram",
  "text": "Quick Fox"
}

The above sentence would produce the following terms:

[ Q, Qu, u, ui, i, ic, c, ck, k, "k ", " ", " F", F, Fo, o, ox, x ]

Configuration

edit

The ngram tokenizer accepts the following parameters:

`min_gram`	Minimum length of characters in a gram. Defaults to `1`.
`max_gram`	Maximum length of characters in a gram. Defaults to `2`.
`token_chars`	Character classes that should be included in a token. Elasticsearch will split on characters that don’t belong to the classes specified. Defaults to `[]` (keep all characters). Character classes may be any of the following: `letter` — for example `a`, `b`, `ï` or `京` `digit` — for example `3` or `7` `whitespace` — for example `" "` or `"\n"` `punctuation` — for example `!` or `"` `symbol` — for example `$` or `√` `custom` — custom characters which need to be set using the `custom_token_chars` setting.
`custom_token_chars`	Custom characters that should be treated as part of a token. For example, setting this to `+-_` will make the tokenizer treat the plus, minus and underscore sign as part of a token.

It usually makes sense to set min_gram and max_gram to the same value. The smaller the length, the more documents will match but the lower the quality of the matches. The longer the length, the more specific the matches. A tri-gram (length 3) is a good place to start.

The index level setting index.max_ngram_diff controls the maximum allowed difference between max_gram and min_gram.

Example configuration

edit

In this example, we configure the ngram tokenizer to treat letters and digits as tokens, and to produce tri-grams (grams of length 3):

resp = client.indices.create(
    index="my-index-000001",
    settings={
        "analysis": {
            "analyzer": {
                "my_analyzer": {
                    "tokenizer": "my_tokenizer"
                }
            },
            "tokenizer": {
                "my_tokenizer": {
                    "type": "ngram",
                    "min_gram": 3,
                    "max_gram": 3,
                    "token_chars": [
                        "letter",
                        "digit"
                    ]
                }
            }
        }
    },
)
print(resp)

resp1 = client.indices.analyze(
    index="my-index-000001",
    analyzer="my_analyzer",
    text="2 Quick Foxes.",
)
print(resp1)

response = client.indices.create(
  index: 'my-index-000001',
  body: {
    settings: {
      analysis: {
        analyzer: {
          my_analyzer: {
            tokenizer: 'my_tokenizer'
          }
        },
        tokenizer: {
          my_tokenizer: {
            type: 'ngram',
            min_gram: 3,
            max_gram: 3,
            token_chars: [
              'letter',
              'digit'
            ]
          }
        }
      }
    }
  }
)
puts response

response = client.indices.analyze(
  index: 'my-index-000001',
  body: {
    analyzer: 'my_analyzer',
    text: '2 Quick Foxes.'
  }
)
puts response

const response = await client.indices.create({
  index: "my-index-000001",
  settings: {
    analysis: {
      analyzer: {
        my_analyzer: {
          tokenizer: "my_tokenizer",
        },
      },
      tokenizer: {
        my_tokenizer: {
          type: "ngram",
          min_gram: 3,
          max_gram: 3,
          token_chars: ["letter", "digit"],
        },
      },
    },
  },
});
console.log(response);

const response1 = await client.indices.analyze({
  index: "my-index-000001",
  analyzer: "my_analyzer",
  text: "2 Quick Foxes.",
});
console.log(response1);

PUT my-index-000001
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer": {
          "tokenizer": "my_tokenizer"
        }
      },
      "tokenizer": {
        "my_tokenizer": {
          "type": "ngram",
          "min_gram": 3,
          "max_gram": 3,
          "token_chars": [
            "letter",
            "digit"
          ]
        }
      }
    }
  }
}

POST my-index-000001/_analyze
{
  "analyzer": "my_analyzer",
  "text": "2 Quick Foxes."
}

The above example produces the following terms:

[ Qui, uic, ick, Fox, oxe, xes ]

« Lowercase tokenizer Path hierarchy tokenizer »