[DOC] Tokenizers - Character group #8350
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: [email protected] <[email protected]>
leanneeliatra
requested review from
AMoo-Miki,
Naarcha-AWS,
dlvenable,
epugh,
kolchfa-aws,
natebower,
stephen-crawford and
vagimeli
as code owners
|
Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged. Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer. When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review. |
Signed-off-by: [email protected] <[email protected]>
|
@udabhas Will you review this PR for technical accuracy, or have a peer review it? Thank you. |
Signed-off-by: [email protected] <[email protected]>
Signed-off-by: [email protected] <[email protected]>
Signed-off-by: [email protected] <[email protected]>
vagimeli
reviewed
Oct 15, 2024
Signed-off-by: Melissa Vagi <[email protected]>
vagimeli
reviewed
Oct 16, 2024
vagimeli
reviewed
Oct 16, 2024
Signed-off-by: Melissa Vagi <[email protected]>
vagimeli
reviewed
Oct 16, 2024
Signed-off-by: Melissa Vagi <[email protected]>
vagimeli
reviewed
Oct 16, 2024
vagimeli
reviewed
Oct 16, 2024
vagimeli
reviewed
Oct 16, 2024
vagimeli
reviewed
Oct 16, 2024
vagimeli
reviewed
Oct 16, 2024
Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: leanneeliatra <[email protected]>
udabhas
reviewed
Dec 9, 2024
| The following response shows that the specified characters have been removed: | ||
|
|
||
| ``` | ||
| Fast cars drive fast |
There was a problem hiding this comment.
This returns fast!
all tokens -
Fast cars drive fast!
udabhas
reviewed
Dec 9, 2024
|
|
||
| The character group tokenizer accepts the following parameters: | ||
|
|
||
| 1. `tokenize_on_chars`: Specifies a set of characters on which the text should be tokenized. The tokenizer creates a new token upon encountering any character from the specified set, for example, single characters `(e.g., -, @)` and character classes such as `whitespace`, `letter`, `digit`, `punctuation`, and `symbol`. |
There was a problem hiding this comment.
also would be good to add that this tokenizer accepts escape characters
Signed-off-by: Fanit Kolchina <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
natebower
approved these changes
Jan 2, 2025
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
Jan 2, 2025
* finished tokenizers example Signed-off-by: [email protected] <[email protected]> * updating nav order Signed-off-by: [email protected] <[email protected]> * layout cleanup Signed-off-by: [email protected] <[email protected]> * grammar fix Signed-off-by: [email protected] <[email protected]> * doc: small update for page numbers Signed-off-by: [email protected] <[email protected]> * layout fix: correct scentence case for all examples Signed-off-by: [email protected] <[email protected]> * small update: adding copy tag for json segment Signed-off-by: [email protected] <[email protected]> * Update _analyzers/tokenizers/character-group-tokenizer.md Signed-off-by: Melissa Vagi <[email protected]> * Update _analyzers/tokenizers/character-group-tokenizer.md Signed-off-by: Melissa Vagi <[email protected]> * Update _analyzers/tokenizers/character-group-tokenizer.md Signed-off-by: Melissa Vagi <[email protected]> * Apply suggestions from code review Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: leanneeliatra <[email protected]> * Doc review Signed-off-by: Fanit Kolchina <[email protected]> * Reorder index Signed-off-by: Fanit Kolchina <[email protected]> * Add escape characters Signed-off-by: Fanit Kolchina <[email protected]> --------- Signed-off-by: [email protected] <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: leanneeliatra <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Co-authored-by: Melissa Vagi <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> (cherry picked from commit b52ec2f) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
epugh
pushed a commit
to o19s/documentation-website
that referenced
this pull request
Jul 2, 2025
* finished tokenizers example Signed-off-by: [email protected] <[email protected]> * updating nav order Signed-off-by: [email protected] <[email protected]> * layout cleanup Signed-off-by: [email protected] <[email protected]> * grammar fix Signed-off-by: [email protected] <[email protected]> * doc: small update for page numbers Signed-off-by: [email protected] <[email protected]> * layout fix: correct scentence case for all examples Signed-off-by: [email protected] <[email protected]> * small update: adding copy tag for json segment Signed-off-by: [email protected] <[email protected]> * Update _analyzers/tokenizers/character-group-tokenizer.md Signed-off-by: Melissa Vagi <[email protected]> * Update _analyzers/tokenizers/character-group-tokenizer.md Signed-off-by: Melissa Vagi <[email protected]> * Update _analyzers/tokenizers/character-group-tokenizer.md Signed-off-by: Melissa Vagi <[email protected]> * Apply suggestions from code review Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: leanneeliatra <[email protected]> * Doc review Signed-off-by: Fanit Kolchina <[email protected]> * Reorder index Signed-off-by: Fanit Kolchina <[email protected]> * Add escape characters Signed-off-by: Fanit Kolchina <[email protected]> --------- Signed-off-by: [email protected] <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: leanneeliatra <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Co-authored-by: Melissa Vagi <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Eric Pugh <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Addition of the Tokenizer - Character group documentation.
Added in the Analyzers section
Issues Resolved
Part of #1483 addressed in this PR.
Version
all
Frontend features
n/a
Checklist
For more information on following Developer Certificate of Origin and signing off your commits, please check here.