fix(translation): catch untranslated adverbs the residue detector could not enumerate - #179
Merged
Conversation
…erate detectEnglishResidue's mostly-CJK branch only flagged words from a curated function-word list, so ordinary content words shipped untranslated inside otherwise-Chinese paragraphs — "曼联的历史最佳射手 arguably 从未找到自己的 最佳位置", "遭受了 relentless 的高空轰炸" (reported on The Mixer, 28 such segments across chapters 7–38, denser in the late chapters). Enumerating adverbs and adjectives is hopeless, so the branch now also flags word *shapes* that cannot be pinyin or romaji: -ly, the Latinate noun/adjective suffixes (-tion/-ment/-ness/-able/-ive/-ous/-less/-ical…), and -ing/-ed/-al above a length floor that keeps "ping"/"ming"/"xing" out. Curated list additions cover the connectives with no inflection to key off (somehow, otherwise, meanwhile…). Tuned the same way as the previous pass — a scan over all 180k mostly-CJK production segments. The list alone flagged 8 segments (0.004%); list plus shapes flags 335 (0.19%), and sampling the new flags shows they are real leftovers, with the known false-positive shapes (quoted terms, bracketed glosses, pinyin runs, Title-Case names) still excluded. A false positive costs one retry that is only kept when it strictly reduces residue. Applied to all four mirrors (translate-worker.ts, translate-core.ts, eval-translation.mjs, scan-residue-corpus.mjs) with regression tests from the reported sentences in both suites. Adds scripts/backfill-residue.ts (yarn backfill-residue): re-runs the current detector over a book's stored translations and re-translates the flagged rows with the production prompt + glossary, updating by row id and writing a JSONL before/after log for rollback. translateText is now exported so the backfill can't drift from the pipeline's prompt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KoxwKGP9yhzhf8NQmDUcNk
…etries Running the backfill for real turned up three sharp edges: `--file` writes upload a temp file per row and answered "fetch failed" mid-run, a whole book's candidate rows in one response triggered the same error, and a one-book-at-a-time flag makes a library sweep a manual chore. Statements now go through --command as an argv element (execFileSync, no shell, so translated text needs no escaping), reads are paged 300 rows at a time, every wrangler call retries three times with backoff, and --all walks every book while a GLOB prefilter keeps segments without latin text out of the transfer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KoxwKGP9yhzhf8NQmDUcNk
… rows Three ways the sweep could have damaged good text. Nodes over the pipeline's 3000-char split threshold would come back from a single LLM call truncated — and a truncated tail reads as residue-free, so the guard that only accepts cleaner output would have waved it through; those rows are now skipped. A replacement under 60% of the original's length is treated as a dropped tail and rejected. And --high-cjk-only restricts a run to the branch the word-shape fix actually changed: without it a library sweep also picks up credits pages, TOC headings and name lists, whose English is deliberate and predates this work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KoxwKGP9yhzhf8NQmDUcNk
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
ovid | fdc59e9 | Commit Preview URL Branch Preview URL |
Aug 17 2026, 02:59 AM |
Nodes past the pipeline's 3000-char split threshold were skipped outright. They now go through translateLargeNode — the same paragraph-chunked path the pipeline uses for them — so a long node gets repaired instead of left behind, without the truncation risk that made a single call unsafe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KoxwKGP9yhzhf8NQmDUcNk
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
问题
用户在《混战》(The Mixer) 后几章读到大量未翻译的副词：
detectEnglishResidue的 mostly-CJK 分支只认一份人工维护的虚词列表（simply/exactly/further…），所以这类普通实词副词、形容词从未触发重译，直接发到读者眼前。《混战》全书 28 段命中，分布在第 7–38 章（后半更密，但不只在最后几章）。改动
副词/形容词是列不完的，所以这一版改为同时匹配词形——拼音和罗马字都不可能出现的英语屈折后缀：
-ly（arguably、inevitably、momentarily）-tion/-sion/-ment/-ness/-able/-ible/-ive/-ous/-ful/-less/-ical/-istic（possession、cohesive、relentless）-ing/-ed/-al，带长度下限，把ping/ming/xing这类拼音音节挡在外面外加把没有屈折可依的连接词补进原列表（somehow、otherwise、meanwhile…）。
调参方式沿用上一轮 (a67c715)：对生产库全部 18 万条 mostly-CJK 译文做离线扫描。原列表只命中 8 条 (0.004%)，加词形后 335 条 (0.19%)，抽样检查新增命中基本都是真残留；引号内英文、括号注、拼音串、Title-Case 人名等既有的 false-positive 形态仍被排除。误报的代价只是一次重译，而且只有当残留严格变少时才会采用。
四份镜像同步：
translate-worker.ts、translate-core.ts、eval-translation.mjs、scan-residue-corpus.mjs；两个测试套件都加了取自真实句子的回归用例。新增
yarn backfill-residue用当前检测器扫已有书的译文，对命中段落用生产同款 prompt + glossary 重译并按 row id
UPDATE（不删不重插，xpath 映射不动），逐行写 before/after JSONL 回滚日志。写库前的三道闸：translateLargeNode分块路径，而不是一次调用--all扫全库，--high-cjk-only只处理这次修复覆盖的分支（不限分支会连版权页、目录、人名列表一起命中，那些英文本来就该留着）。为此导出了translateText/translateLargeNode，避免脚本里再抄一份 prompt。线上数据回填（已完成）
18 段故意跳过——英文是作者有意留的：《我们赖以生存的隐喻》的语言学讨论（"n't 否定的是 leave 而不是 think"）、《詹姆斯·乔伊斯如何成名》的拉康法语双关（franchement / poubellication / père-version）、《从爱丁堡到印度与缅甸》的古登堡勘误注记、《比德威尔游记》的方言引语。回滚日志留存在本地。
测试
yarn test310 passed；services/translator173 passed。🤖 Generated with Claude Code
https://claude.ai/code/session_01KoxwKGP9yhzhf8NQmDUcNk