Add Hindi (hi-IN) support for TTS

[quapham]

Important

The Update branch button must only be pressed in very rare occassions.
An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.

What does this PR do ?

Add Hindi (hi-IN) TTS support with char tokenizer

Collection: [Note which collection this PR will affect]
TTS, common

Changelog

• nemo/collections/common/tokenizers/text_to_speech/ipa_lexicon.py
  Added Hindi grapheme character set (hi-IN) with Devanagari script support

• nemo/collections/common/tokenizers/text_to_speech/tts_tokenizers.py
  "HindiCharsTokenizer" class for Hindi grapheme tokenization
  Handles Unicode decomposition correctly (e.g., ड़ = ड + ़ nukta)

Usage

• You can potentially add a usage example below

# Code tests for Hindi character tokenizer
from nemo.collections.common.tokenizers.text_to_speech.tts_tokenizers import HindiCharsTokenizer
tokenizer = HindiCharsTokenizer(
    punct=True,
    apostrophe=True,
    pad_with_space=False
)
text="अंगड़ाई"
ids = tokenizer.encode(text)
decoded = tokenizer.decode(ids)

print(f"IDs:   {ids}")
print(f"Decoded: {decoded}")

#IDS: [74, 138, 90, 100, 141, 124, 77]
#Decoded:अंगड़ाई

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

• New Feature
• Bugfix
• Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

• Related to # (issue)

[XuesongYang]

could you pls add unit tests to verify if they worked as expected? examples are,

1. https://github.com/NVIDIA-NeMo/NeMo/blob/main/tests/collections/tts/g2p/test_modules.py
2. https://github.com/NVIDIA-NeMo/NeMo/blob/main/tests/collections/common/tokenizers/text_to_speech/test_tts_tokenizers.py

[Copilot]

Pull request overview

This PR adds Hindi (hi-IN) language support for TTS functionality by introducing character-based tokenization and IPA (International Phonetic Alphabet) grapheme-to-phoneme (G2P) capabilities for the Devanagari script.

Key Changes:

• Adds Hindi locale ("hi-IN") to supported TTS locales
• Introduces HindiCharsTokenizer class for character-level tokenization of Devanagari script
• Extends Unicode range support to include Devanagari characters (\u0900-\u097F) across the TTS pipeline

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 5 comments.

File	Description
nemo/collections/common/tokenizers/text_to_speech/ipa_lexicon.py	Adds Hindi to supported locales; defines Devanagari grapheme character set and Hindi IPA phoneme set
nemo/collections/common/tokenizers/text_to_speech/tokenizer_utils.py	Extends regex patterns to include Devanagari Unicode range for text processing
nemo/collections/common/tokenizers/text_to_speech/tts_tokenizers.py	Implements new HindiCharsTokenizer class with custom encoding logic for Devanagari combining marks
nemo/collections/tts/g2p/models/i18n_ipa.py	Extends character and punctuation regex patterns to support Devanagari script; adds experimental decorator to IpaG2p class

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[XuesongYang]

"hi-IN" is defined in IPA_CHARACTER_SETS, so I guess you may have implemented phoneme tokenizer as well. Could you pls also add below items if any?

1. Hindi's IPA dictionary;
2. if no phoneme tokenizer is defined, pls ensure IpaG2p and IPATokenizer correctly support Hindi.
3. add unit test following test_ipa_tokenizer_es_es.

[quapham]

Yes, it will have a dictionary. Currently, it works in our tests, so I will add this as well.

[XuesongYang]

nice. pls add those changes.

[XuesongYang]

Let’s merge this grapheme based tokenizer. Would you pls open a new PR adding phoneme based ones?

[subhankar-ghosh]

This PR is just for Char tokenizer correct? and NOT phoneme tokenizer. @quapham You have the phoneme dict but it is not being used anywhere in this PR. Is that the correct understanding?

[quapham]

Thank @XuesongYang . I will create a new PR for the IPA G2P phoneme.
@subhankar-ghosh, you are correct, this PR currently only supports the Hindi chartokenizer.

[chtruong814]

The test failure is likely temporary network issue. It's a 404 from huggingface on a test that has worked before. Will go ahead and merge this.