Use a Japanese language NLP model in Elasticsearch to enable semantic searches

Quickly finding necessary documents from among the large volume of internal documents and product information generated every day is an extremely important task in both work and daily life. However, if there is a high volume of documents to search through, it can be a time-consuming process even for computers to re-read all of the documents in real time and find the target file. That is what led to the appearance of Elasticsearch® and other search engine software. When a search engine is used, search index data is first created so that key search terms included in documents can be used to quickly find those documents.

However, even if the user has a general idea of what type of information they are searching for, they may not be able to recall a suitable keyword or they may search for another expression that has the same meaning. Elasticsearch enables synonyms and similar terms to be defined to handle such situations, but in some cases it can be difficult to simply use a correspondence table to convert a search query into a more suitable one.

To address this need, Elasticsearch 8.0 released the vector search feature, which searches by the semantic content of a phrase. Alongside that, we also have a blog series on how to use Elasticsearch to perform vector searches and other NLP tasks. However, up through the 8.8 release, it was not able to correctly analyze text in languages other than English.

With the 8.9 release, Elastic added functionality for properly analyzing Japanese in text analysis processing. This functionality enables Elasticsearch to perform semantic searches like vector search on Japanese text, as well as natural language processing tasks such as sentiment analysis in Japanese. In this article, we will provide specific step-by-step instructions on how to use these features.

Fugashi, ipadic, and unidic_lite will be required to use a Japanese model.

After these libraries have been installed, Eland can be installed as well. Eland 8.9.0 or later will be required to use a Japanese model, so be sure to note the version number.

Once installation is complete, use the command below to confirm that Eland can be used.

The primary method of enabling vector search is the same as the method used in English in this article. We will briefly cover the same procedure here to review.

As explained above, the appropriate machine learning model must be imported into Elasticsearch in order to enable NLP processing in Elasticsearch. It is possible to implement a machine learning model yourself using PyTorch, but this also requires sufficient expertise with machine learning and natural language processing as well as the computing power that machine learning requires. However, there is now an online repository, Hugging Face, that is heavily used by researchers and developers in the machine learning and natural language processing spaces, and many models are published to this repository. In this example, we will use a model published to Hugging Face to implement semantic search functionality.

To start, let’s pick a model on Hugging Face that will embed (vectorize) Japanese sentences into numerical vectors. In this article, we will use the model linked below.

• cl-tohoku/bert-base-japanese-v2

Let’s cover some points to note when selecting a Japanese model in 8.9.

First, only the BERT model algorithm is supported. Check the tags on Hugging Face and other information to confirm that the desired NLP model is a BERT trained model.

Additionally, for BERT and other NLP tasks, text that is input is then “pre-tokenized,” which means the text is divided into units at the word level. In this case, a Japanese language morphological analysis engine is used to pre-tokenize our Japanese text. Elasticsearch 8.9 supports morphological analysis using MeCab. On Hugging Face’s models page, open the “Files and versions” tab and view the contents of the tokenizer_config.json file. Confirm that the value for word_tokenizer_type is mecab.

Unfortunately, if the model you want to use has a value other than mecab for word_tokenizer_type, that model is not currently supported by Elasticsearch. We welcome feedback regarding any specific word tokenizer types (word_tokenizer_type) for which support is required.

Once you have decided on a model to import, the steps required to import are the same as for an English model. First, use eland_import_hub_model to import the model into Elasticsearch. Refer to this page for how to use eland_import_hub_model.

Once the model has been imported successfully, it will be displayed under Machine Learning > Model Management > Trained Models in Kibana. Open the model “Config” tab to see that “bert_ja” is used for tokenization and the model is set up correctly to handle Japanese.

Once the model upload is complete, let’s test it out. Click a button in the Actions column to open the menu.

Select “Test model”, then enter any Japanese phrase under “Input text” and click the “Test” button.

You will then be able to see that the model vectorized the Japanese text you input into a numerical string, as shown here. It seems to be working correctly.

When the inference processor is used, the model specified in model_id is applied to the text saved in the target field (in this case, title) and the output thereof is stored in target_field. Additionally, each model expects a different field (in this case, text_field) for the input value for the process. For that reason, field_map is used to specify the correspondence between the actual input field for the target of the process and the field name expected by the ML model.

Once the pipeline is set up, it can be used to create indices. Since fields will be needed in those indices to store vectors, we will define the appropriate mapping. In the example below, the text_embedding.predicted_value field is set up to hold 768-dimensional dense_vector data. Note that the number of dimensions varies by model. Check the model’s page on Hugging Face (hidden_size value in the model’s config.json) or elsewhere and set an appropriate number.

If there is already an index that includes the Japanese text data targeted for searching, then the reindex API can be used. In this example, the original text data is in the japanese-text index. A document containing the vectorization of that text is registered to the japanese-text-embeddings index.

Alternatively, a document can be registered directly for testing purposes by specifying a pipeline created as shown below and storing it in the index.

Once the vectorized document has been registered, it is finally possible to execute a search. A kNN (k-nearest neighbor) search is one available method that makes use of vectors. We will now execute a kNN vector search using the query_vector_builder option in the standard _search API. When query_vector_builder is used, the model specified in model_id can be used to convert the text in model_text into a query containing a vector in which that text is embedded.

When this search query is executed, the following type of response is received.

Search successful! The search also contains fields in which the Japanese has been embedded. In most actual use cases, it is not necessary for this text to be included in the response. In such cases, use the _source parameter or some other method to exclude that information from the response (or take other such action).

For fine-tuning search rankings, the Reciprocal rank fusion (RRF) feature, which adeptly blends the results of vector searches and standard keyword searches, has been released. Be sure to take a look at this as well.

This concludes our discussion of how to enable semantic search using vector searching. Although setting this functionality up may require more work than a standard search and you may encounter some unique machine learning vocabulary, you will be able to execute searches in almost the same way as normal once the setup phase is complete, so please give it a try.

Once the model has been imported successfully, it will be displayed under Machine Learning > Model Management > Trained Models in Kibana.

In this case as well, click “Test model” in the Actions menu.

As before, this will display a dialog box for testing. Enter the text to be classified here, and the text will be classified into POSITIVE, NEUTRAL, or NEGATIVE. As a test, let’s enter: “I’m happy that I’m now able to execute NLP tasks with Elasticsearch.” As shown below, this generated a result of 99.2% positive.

Below is the same process executed via API.

The response is as follows:

Since this process can also be executed with the inference processor, it is possible to attach these analysis results before the Japanese text is indexed. For example, applying this process to the text of comments for specific products could help convert user ratings for those products into numerical values.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use. 

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.