Skip to content

V2_TEST_REPORT.md

Latest commit

 

History

History
127 lines (92 loc) · 3.79 KB

V2_TEST_REPORT.md

File metadata and controls

127 lines (92 loc) · 3.79 KB
Raw

LLM Post-processing V2 Test Report

Date: 2026-01-02 Test Video: Chicago Fire S12E01 (41.8 min, 2505s) Test Setup:

  • V1: batch_size=10, plain text parsing
  • V2: batch_size=20, JSON parsing, Agent Loop, Similarity Validation

Summary

Metric V1 V2 Change
Parsing Success Rate 87% (~13% fallback) 100% ✅ +13%
Herrmann Consistency 75% (3/4) 100% (4/4) ✅ +25%
Kylie Estevez Fix 100% 100% ✅ Maintained
Segments Modified Unknown 61/582 (10.5%) ⚠️ High
Processing Time ~3 min ~3 min ✅ Same
RTFx (ASR only) 240.87x 238.57x ✅ Same

Key Improvements ✅

1. Parsing Reliability

  • V1: ~13% parsing failures (batch 30: 4/30 batches failed)
  • V2: 0% parsing failures ✅
  • Cause: JSON structured output + json-repair library + Agent Loop retry

2. Name Consistency

Herrmann:

  • V1: Herman (line 255), Herrmann (3 other places) → 75% consistency
  • V2: Herrmann everywhere → 100% consistency ✅

Galo (character name):

  • V1: Gala (wrong)
  • V2: Galo ✅

3. Spelling Corrections

  • V1: permagonate → V2: permanganate ✅

Concerns ⚠️

1. Over-Modification Rate

  • 10.5% of segments (61/582) were modified
  • Many changes introduce new errors instead of fixes

2. New Semantic Errors

Line V1 (Correct) V2 (Wrong) Issue
14 "So be it, Brett" "So be Brett" ❌ Lost meaning
21 "Arsenal was his side" "Arsenal was his sidekick" ❌ Changed meaning
44 "That can wait" "That can happen" ❌ Wrong word
51 "this fits great" "this size down" ❌ Nonsense
89 "thirty seconds" "40 seconds" ❌ Wrong number
94 "breach" "bridge" ❌ Wrong word

3. Analysis

Why Over-Modification?

  1. Similarity threshold (30%/60%) may be too permissive
  2. GPT-4o-mini may "hallucinate" corrections
  3. Not enough emphasis on "minimal changes" in prompt

Recommendations

Option A: Keep V2 (Recommended)

Pros:

  • 100% parsing success (critical improvement)
  • 100% name consistency (key goal achieved)
  • Most changes are minor/cosmetic

Cons:

  • 10.5% modification rate higher than ideal
  • Some new errors introduced

Action: Document limitations, ship as V2

Option B: Tune Similarity Validation

Changes:

  1. Increase similarity threshold: 30%/60% → 50%/80%
  2. Add stricter validation for number changes
  3. Emphasize "DO NOT change unless absolutely certain" in prompt

Risk: May reduce beneficial corrections too

Option C: Hybrid Approach

Changes:

  1. Keep V2 architecture (Agent Loop + JSON)
  2. Add conservative mode: llm_postprocess.conservative=true
  3. In conservative mode: stricter validation (70%/85% similarity)

Conclusion

V2 is production-ready ✅ with caveats:

  1. Major Win: Parsing reliability (87% → 100%)
  2. Key Goal Achieved: Name consistency (75% → 100%)
  3. Trade-off: Some over-modification (10.5% change rate)
  4. User Expectation: Document that LLM may introduce new errors occasionally

User's Philosophy Alignment:

  • ✅ "不需要完全正確，只要一致性好" - Consistency improved significantly
  • ✅ "承認 AI 辨識有侷限性" - Documented in limitations
  • ⚠️ "主要是能盡量一致性就好" - Achieved, but at cost of some over-modification

Verdict: Ship V2, consider Option C (conservative mode) for future enhancement.

Test Artifacts

  • Baseline (no LLM): baseline_no_llm.srt
  • V1 (batch_size=10): with_gpt4o_mini.srt
  • V2 (batch_size=20, Agent Loop): v2_with_gpt4o_mini.srt

Cost:

  • V1: ~$0.06/episode (GPT-4o-mini)
  • V2: ~$0.06/episode (same, slightly more tokens but same batch count)