Add BERT Tokenizer as OpenSearch built-in analyzer #3719

zhichao-aws · 2025-04-09T07:39:52Z

Description

This PR add bert-base-uncased tokenizer and bert-base-multilingual-uncased tokenizer as OpenSearch built-in analyzer/tokenizer. Users can use them via analyze API without doing any special settings:

POST /_analyze
{
    "tokenizer" : "mbert-uncased",
    "text" : "hello world 你好"
}

We also have a follow up PR in neural-search to make neural-sparse search work with the analyzer.

Related Issues

Resolves #3708

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: zhichao-aws <[email protected]>

zhichao-aws · 2025-04-09T08:31:00Z

We can merge this PR after 3.0.0-beta1 get released

Signed-off-by: zhichao-aws <[email protected]>

mingshl · 2025-04-14T20:37:54Z

would you consider option 3 from the issue? #3708 (comment)

Signed-off-by: zhichao-aws <[email protected]>

zhichao-aws · 2025-04-15T08:23:21Z

would you consider option 3 from the issue? #3708 (comment)

I've changed to option 3.

mingshl · 2025-04-15T15:21:57Z

ml-algorithms/src/main/java/org/opensearch/ml/engine/analysis/HFModelAnalyzer.java

+ * Custom Lucene Analyzer that uses the HFModelTokenizer for text analysis.
+ * Provides a way to process text using Hugging Face models within OpenSearch.
+ */
+public class HFModelAnalyzer extends Analyzer {


even though this is an internal class, but I am thinking why named it as HFModelAnalyer, it doesn't have to be from huggingface analyzer right? is it can be any custom model?

It's a wrapper class of DJL HuggingFaceTokenizer. So we need to refect it in the class name

mingshl · 2025-04-15T15:25:00Z

ml-algorithms/src/main/java/org/opensearch/ml/engine/analysis/HFModelTokenizer.java

+ * Supports token weighting and handles overflow scenarios.
+ */
+@Log4j2
+public class HFModelTokenizer extends Tokenizer {


similarly, is the the tokenizer can be any customMLTokenizer ?

mingshl · 2025-04-15T15:28:15Z

ml-algorithms/src/main/java/org/opensearch/ml/engine/analysis/HFModelTokenizerFactory.java

+            try {
+                Files.createDirectories(DJLUtils.getMlEngine().getAnalysisRootPath());
+                Path filePath = DJLUtils.getMlEngine().getAnalysisRootPath().resolve(name + ZIP_PREFIX);
+                downloadTokenizerFile(url, filePath, hashCode);


question here, the file path is coded here, so it will download the default tokenizer for bert-uncased, can user change to use other tokenizer instead of this default tokenizer? it looks like it's not allowed at the moment.

Currently we only support to load the tokenizers we've released in artifacts.opensearch.org website. Users can not use other tokenizers from custom urls now.

It's an open discussion to support custom files in the future, feel free to add comments in #3708

mingshl · 2025-04-15T15:30:41Z

ml-algorithms/src/main/java/org/opensearch/ml/engine/analysis/HFModelTokenizerFactory.java

+                downloadTokenizerFile(url, filePath, hashCode);
+                ZipUtils.unzip(filePath.toFile(), DJLUtils.getMlEngine().getAnalysisRootPath().resolve(name));
+            } catch (IOException | PrivilegedActionException e) {
+                log.error("Failed to download analyzer zip file for {}. {}", name, e);


when catched the exception, this class doesn't have the artifact for analyzer, what will happen? does it still worth moving forward when there is no artifact? should it throw a different exception and ask users for retries?

I think if we don't throw exception here it will throw null pointer exception later when we try to use it but it's not clear. So I think it's better to throw the exception here with clear error message. For retry I think we may want to have some auto retry mechanism in downloadTokenizerFile.

If not throw exception, we'll get an error for the analyzer request:

{ "error": { "root_cause": [ { "type": "null_pointer_exception", "reason": "Cannot invoke \"ai.djl.huggingface.tokenizers.HuggingFaceTokenizer.encode(String, boolean, boolean)\" because the return value of \"java.util.function.Supplier.get()\" is null" } ], "type": "null_pointer_exception", "reason": "Cannot invoke \"ai.djl.huggingface.tokenizers.HuggingFaceTokenizer.encode(String, boolean, boolean)\" because the return value of \"java.util.function.Supplier.get()\" is null" }, "status": 500 }

If throw exception, the caller doesn't catch the exception and the OpenSearch node will fail.

Break the OpenSearch node is not acceptable. So we can only log errors instead of throw exceptions during initialization.

I've added retry logics, it will be triggered when the analyzer is invoked, and it tries to download file when the tokenizer is not initialized

mingshl · 2025-04-15T15:32:03Z

ml-algorithms/src/main/java/org/opensearch/ml/engine/analysis/HFModelTokenizerFactory.java

+                this.token_weights = DJLUtils
+                    .fetchTokenWeights(DJLUtils.getMlEngine().getAnalysisRootPath().resolve(name).resolve(TOKEN_WEIGHTS_FILE_NAME));
+            } catch (Exception e) {
+                log.error("Failed to initialize analyzer: {}. {}", name, e);


same here, when it couldn't initialize the analyzer, we should throw meaningful exception, maybe
throw new RuntimeException("Failed to initialize bert-uncased tokenizer. " + e);

bzhangam · 2025-04-15T16:26:26Z

ml-algorithms/src/main/java/org/opensearch/ml/engine/analysis/HFModelTokenizerFactory.java

+                String hash = calculateFileHash(filePath.toFile());
+                if (!hash.equals(hashCode)) {
+                    log.error("Analyzer zip file content hash can't match original hash value.");
+                    throw (new IllegalArgumentException("Analyzer " + name + " zip file content changed."));


Use RuntimeException since the url is hardcoded and it's not something that user can fix?

bzhangam · 2025-04-15T16:52:43Z

ml-algorithms/src/main/java/org/opensearch/ml/engine/analysis/DJLUtils.java

+
+                return action.call();
+            } finally {
+                Thread.currentThread().setContextClassLoader(contextClassLoader);


Do we also need to reset the system properties?

The logics are consistent with other place using djl. We can keep them even finished.

codecov · 2025-04-15T17:30:41Z

Codecov Report

Attention: Patch coverage is 87.39496% with 15 lines in your changes missing coverage. Please review.

Project coverage is 79.14%. Comparing base (2ef82b6) to head (b7b698f).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...ch/ml/engine/analysis/HFModelTokenizerFactory.java	75.75%	7 Missing and 1 partial ⚠️
...pensearch/ml/engine/analysis/HFModelTokenizer.java	89.13%	2 Missing and 3 partials ⚠️
...gine/algorithms/tokenize/SparseTokenizerModel.java	0.00%	1 Missing ⚠️
...va/org/opensearch/ml/engine/analysis/DJLUtils.java	93.75%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3719      +/-   ##
============================================
+ Coverage     79.11%   79.14%   +0.02%     
- Complexity     7116     7147      +31     
============================================
  Files           623      628       +5     
  Lines         31590    31706     +116     
  Branches       3574     3583       +9     
============================================
+ Hits          24993    25093     +100     
- Misses         5057     5068      +11     
- Partials       1540     1545       +5

Flag	Coverage Δ
ml-commons	`79.14% <87.39%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: zhichao-aws <[email protected]>

bert analyzer

955a6ed

Signed-off-by: zhichao-aws <[email protected]>

zhichao-aws requested review from b4sjoo, dhrubo-os, mingshl, jngz-es, model-collapse, rbhavna, ylwu-amzn, zane-neo, Zhangxunmt, austintlee, HenryL27 and xinyual as code owners April 9, 2025 07:39

zhichao-aws requested a deployment to ml-commons-cicd-env-require-approval April 9, 2025 07:41 — with GitHub Actions Waiting

zhichao-aws added 2 commits April 9, 2025 16:46

add license header

e598e29

Signed-off-by: zhichao-aws <[email protected]>

add rest test case

db98ff4

Signed-off-by: zhichao-aws <[email protected]>

zhichao-aws requested a deployment to ml-commons-cicd-env-require-approval April 9, 2025 09:24 — with GitHub Actions Waiting

load from zip

f546c4f

Signed-off-by: zhichao-aws <[email protected]>

zhichao-aws had a problem deploying to ml-commons-cicd-env-require-approval April 14, 2025 07:39 — with GitHub Actions Error

zhichao-aws had a problem deploying to ml-commons-cicd-env-require-approval April 14, 2025 07:39 — with GitHub Actions Failure

Merge branch 'main' into bert_analyzer

6ea0eae

zhichao-aws requested a deployment to ml-commons-cicd-env-require-approval April 15, 2025 02:28 — with GitHub Actions Waiting

change to option 3

b7b698f

Signed-off-by: zhichao-aws <[email protected]>

zhichao-aws temporarily deployed to ml-commons-cicd-env-require-approval April 15, 2025 08:24 — with GitHub Actions Inactive

mingshl reviewed Apr 15, 2025

View reviewed changes

bzhangam reviewed Apr 15, 2025

View reviewed changes

zhichao-aws had a problem deploying to ml-commons-cicd-env-require-approval April 15, 2025 17:31 — with GitHub Actions Failure

zhichao-aws had a problem deploying to ml-commons-cicd-env-require-approval April 15, 2025 17:31 — with GitHub Actions Error

add retry when invoke

10d2b93

Signed-off-by: zhichao-aws <[email protected]>

zhichao-aws requested a deployment to ml-commons-cicd-env-require-approval April 16, 2025 08:22 — with GitHub Actions Waiting

address comments

a626e03

Signed-off-by: zhichao-aws <[email protected]>

zhichao-aws requested a deployment to ml-commons-cicd-env-require-approval April 16, 2025 08:27 — with GitHub Actions Waiting

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add BERT Tokenizer as OpenSearch built-in analyzer #3719

Add BERT Tokenizer as OpenSearch built-in analyzer #3719

zhichao-aws commented Apr 9, 2025 •

edited

Loading

zhichao-aws commented Apr 9, 2025

mingshl commented Apr 14, 2025

zhichao-aws commented Apr 15, 2025

mingshl Apr 15, 2025

zhichao-aws Apr 16, 2025

mingshl Apr 15, 2025

mingshl Apr 15, 2025

zhichao-aws Apr 16, 2025

mingshl Apr 15, 2025

bzhangam Apr 15, 2025

zhichao-aws Apr 16, 2025

zhichao-aws Apr 16, 2025

mingshl Apr 15, 2025

bzhangam Apr 15, 2025

zhichao-aws Apr 16, 2025

bzhangam Apr 15, 2025

zhichao-aws Apr 16, 2025

codecov bot commented Apr 15, 2025 •

edited

Loading

Add BERT Tokenizer as OpenSearch built-in analyzer #3719

Are you sure you want to change the base?

Add BERT Tokenizer as OpenSearch built-in analyzer #3719

Conversation

zhichao-aws commented Apr 9, 2025 • edited Loading

Description

Related Issues

Check List

zhichao-aws commented Apr 9, 2025

mingshl commented Apr 14, 2025

zhichao-aws commented Apr 15, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Apr 15, 2025 • edited Loading

Codecov Report

zhichao-aws commented Apr 9, 2025 •

edited

Loading

codecov bot commented Apr 15, 2025 •

edited

Loading