Train/finetue iic/punc_ct-transformer_cn-en-common-vocab471067-large for a different language

[mark95]

Is it possible the train only the punctuation model for a different language, i found out how to finetune only for the ASR models. If it's possible what trainset structure do i need and how to modify the training/finetuning to work for the punctuation model?

[LauraGPT]

Yes, the CT-Transformer can be trained as a punctuation-only model; it does not need audio. It is a token-classification task: for every input text token, provide exactly one label describing the punctuation after that token.

There is an important version caveat, though: current FunASR main does not ship a complete punctuation training recipe. The historical punc_train.py / PunctuationTask path was removed during the 1.0 refactor, and the current registered datasets do not produce the punc and punc_lengths fields required by CTTransformer.forward. Therefore, changing the model name in an ASR fine-tuning command is not sufficient, and I would not recommend copying an old v0.x command into a current environment.

Data contract

The historical trainer used two Kaldi-style, two-column files with identical utterance IDs. For a character tokenizer:

# text
utt1 hello
utt2 hi

# punc
utt1 ____,
utt2 _?

The aligned labels mean:

h  e  l  l  o
_  _  _  _  ,

_ means "no punctuation". The number of punctuation labels must equal the number of tokens after tokenization. With the historical char tokenizer, do not write _ _ _ _ ,: those spaces are also tokenized and corrupt the alignment. I verified the loader contract and a real forward/backward/optimizer step on the last pre-removal implementation; the required field names were exactly text and punc (historical source).

The historical punctuation vocabulary file was:

<unk>:1.0
_:1.0
,:1.0
.:1.0
?:1.0

The value after : is the class weight. You can replace the punctuation symbols with the classes needed by your language.

Adapting the 471,067-token checkpoint

iic/punc_ct-transformer_cn-en-common-vocab471067-large is tied to its original tokenizer and 471,067-row input embedding.

• If your target-language tokenizer only emits tokens already present in that vocabulary, you can keep the embedding and fine-tune the model.
• If the target script has poor vocabulary coverage and maps heavily to <unk>, create a new token list. That changes vocab_size, so the old embedding cannot be loaded unchanged; initialize the new embedding and reuse only compatible encoder weights.
• If you change the punctuation classes, the decoder output dimension changes, so reinitialize the decoder as well.

Before training, assert this for every sample:

assert len(text_ids) == len(punc_ids)

For current FunASR, the practical route is a small custom dataset/collator that returns text, text_lengths, punc, and punc_lengths, then calls CTTransformer directly. I also verified one current-main issue: the YAML-style punc_weight list reaches .to() during training and raises AttributeError; passing it as a torch.Tensor makes an aligned mini-batch train correctly. So the model architecture is trainable, but the stock current training path is not yet an end-to-end supported recipe.

If you post the target language, tokenizer choice (character/word/subword), and desired punctuation set, we can provide an exact token-alignment example and determine how much of the Chinese/English checkpoint can be reused.