Skip to content

Add VietOCR recognizer for high-accuracy Vietnamese OCR (lang='vi') - #5327

Open
nhducminh wants to merge 2 commits into
opendatalab:masterfrom
nhducminh:add-vietocr-vietnamese-ocr
Open

Add VietOCR recognizer for high-accuracy Vietnamese OCR (lang='vi')#5327
nhducminh wants to merge 2 commits into
opendatalab:masterfrom
nhducminh:add-vietocr-vietnamese-ocr

Conversation

@nhducminh

Copy link
Copy Markdown

Summary

Adds Vietnamese as a first-class OCR language (lang='vi'). PaddleOCR's shared recognizer does not cover Vietnamese tone marks, so pipeline OCR on Vietnamese documents frequently drops/misreads diacritics — e.g. "QUYẾT ĐỊNH" → "QUYT ĐNH". This routes the recognition step to VietOCR (vgg_transformer) while detection stays PaddleOCR, and ships a custom batched decoder that is much faster than VietOCR's stock predict_batch().

Before / after (real Vietnamese PDF page)

Recognized text
PaddleOCR (before) Bng 2. Lưng phân hu co, hu co vi sinh cn cung cp theo kt qu phân tích đt
VietOCR (this PR) Bảng 2. Lượng phân hữu cơ, hữu cơ vi sinh cần cung cấp theo kết quả phân tích đất

Changes

  • ocr_language.py, models_config.yml — register "vi" as a public OCR language. It resolves through validate/normalize unchanged and maps its PaddleOCR det/rec/dict to the ch models (detection is used; the rec model still loads as a valid fallback but is bypassed when VietOCR is active).
  • pytorch_paddle.py — for lang='vi', load VietOCR and use it for the rec step in ocr()/__call__(); confidence for drop_score filtering comes from VietOCR's own softmax probabilities. Falls back to the PaddleOCR recognizer if VietOCR is unavailable or errors (graceful degradation, no crash).
  • vietocr_fast_batch.py (new) — batched VietOCR decoder with a KV-cache (self-attention cache + one-time cross-attention projection over the fixed encoder memory) and per-sequence early-exit. Replaces vietocr's predict_batch()/translate(), which recompute self-attention over the full growing prefix every step and use an unbounded torch.cat() that OOMs on large crop lists. Architecture-guarded: the hand-rolled decode runs only on a post-norm nn.TransformerDecoderLayer stack, otherwise it falls back to an architecture-agnostic decoder — a differently-configured model can't produce silently-wrong output.
  • pyproject.toml — optional vietocr extra. Install with mineru[pipeline,vietocr].
  • docs/en/reference/vietocr_benchmark.md — rationale, correctness, numbers.

Correctness & performance

  • Verified byte-identical to per-image Predictor.predict() — 0 text mismatches across thousands of real crops, for both the KV-cache path and the fallback path.
  • Isolated microbenchmark (2000 real Vietnamese line crops, RTX 5080): stock predict_batch() 155 ms/crop → this decoder 9 ms/crop (~17×), 0/2000 mismatches. ~3× on the largest in-pipeline OCR batches.

Notes

  • Default OCR language is unchanged (ch); vi is opt-in via -l vi.
  • vietocr + PyMuPDF are pulled only by the optional vietocr extra, not by the core install.
  • Backward compatible: no behavior change for any existing language.

@dosubot dosubot Bot added size:XL This PR changes 500-999 lines, ignoring generated files. enhancement New feature or request labels Jul 25, 2026
@github-actions

github-actions Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@nhducminh

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

@nhducminh

Copy link
Copy Markdown
Author

recheck

@nhducminh
nhducminh force-pushed the add-vietocr-vietnamese-ocr branch from 5850ad0 to 5c367dc Compare July 25, 2026 02:47
github-actions Bot added a commit that referenced this pull request Jul 25, 2026
PaddleOCR's shared "latin"/"ch" recognizer does not cover Vietnamese tone
marks, so pipeline OCR on Vietnamese documents frequently drops/misreads
diacritics ("QUYẾT ĐỊNH" -> "QUYT ĐNH"). This registers "vi" as a first-class
OCR language whose recognition is handled by VietOCR (vgg_transformer) while
detection stays PaddleOCR.

- ocr_language.py, models_config.yml: register "vi" as a public OCR language.
  "vi" resolves through validate/normalize unchanged and maps its PaddleOCR
  det/rec/dict to the "ch" models (detection is used; the rec model loads as a
  valid fallback but is bypassed when VietOCR is available).
- pytorch_paddle.py: when lang='vi', load VietOCR and use it for the rec step
  in ocr()/__call__(); confidence for drop_score filtering comes from VietOCR's
  own softmax probs. Falls back to the PaddleOCR recognizer if VietOCR is
  unavailable or errors (graceful degradation, no crash).
- vietocr_fast_batch.py: batched decoder for VietOCR with a KV-cache
  (self-attn cache + one-time cross-attn projection) and per-sequence
  early-exit, replacing vietocr's predict_batch()/translate() which recompute
  self-attention over the full prefix every step and use an unbounded
  torch.cat() that OOMs on large crop lists. Architecture-guarded: the
  hand-rolled decode runs only on a post-norm nn.TransformerDecoderLayer stack,
  else falls back to an architecture-agnostic decoder. ~26x faster than stock
  predict_batch() in isolation, verified byte-identical to per-image predict().
- pyproject.toml: optional "vietocr" extra (install `mineru[pipeline,vietocr]`).

Tested end-to-end on a real Vietnamese PDF page: VietOCR path produces correct
diacritics; forcing VietOCR off exercises the PaddleOCR fallback without error.
Document the KV-cache/early-exit decoder rationale, the architecture guard, the
correctness verification (0 mismatch vs stock predict_batch on 2000 crops), and
the measured ~17x isolated speedup / ~3x in-pipeline speedup.
@nhducminh
nhducminh force-pushed the add-vietocr-vietnamese-ocr branch from 5c367dc to 6eeff10 Compare July 25, 2026 02:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request size:XL This PR changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant