refactor(voice): the contract and the eval come home, and a stale copy stops lying

[jamie-simular]

The problem discovery

chenchenSimular's review of simular-pro-unified-ui#1635 asked for ~4.4k lines of voice/contract/ + voice/eval/ to move here. The argument: cloud-api holds a second implementation, in TypeScript, of wording whose only running copy is Kotlin — maintained forever solely to generate fixtures — and the cross-repo crossing defending it is manual anyway.

Checking that claim before acting on it turned up something worse. The crossing was not merely manual. It was already broken, in both directions, and had been for months.

	tools	state
app/src/main/assets/voice-profile.json	17	current — this is what ships
app/src/test/resources/parity/prompt-and-tools.json	18	still declared approveAlways

Prompt blocks 21, 22, 38 and 39 differed. Block 38 still told the model to "offer to always-allow it" and to call approveAlways — months after ADR 0014 retired the feature and #9 removed it.

Two things read that file:

• fsm/VoiceProfileTest — the test written to catch a dropped prompt block was grading a file nothing ships. It should have caught the approveAlways removal. Instead it pinned the old one.
• eval/LiveBrain — so every LoopEvalTest run since fix(voice): an "always allow" that never allowed anything #9 put the real model through a prompt for a tool that does not exist. (cloud-api's eval read the same copy, with the same consequence.)

Nothing failed. Nothing could: each side only ever checked itself against its own copy.

Discussion insight: "A fixture crossing that silently writes nowhere is worse than no crossing: it reports success, the assertion that would have caught the drift skips itself, and both sides look green while the device answers differently from every other surface." — written in #1635's own description about a different instance of this bug. It happened again, one layer up, to the artefact that description was about.

So: one implementation, and one copy of everything it reads.

The profile

prompt-and-tools.json is deleted. assets/voice-profile.json is on the unit-test classpath (a sourceSets line in build.gradle.kts) and shippedProfile() is the only way to it — one function, because it used to be two paths and they disagreed. The file that ships is the file that gets graded. 17 tools.

The goldens

GoldenFixtures.kt ports cloud-api's contract/fixtures.ts: the same canonical inputs, calling the real describeAgentEvent / describeCompleteAskFirst / renderAgentActivity / ActivityLog on the same pinned clock T0.

Regenerating from Kotlin reproduced all 61 committed fixtures byte for byte. That is the proof the port is exact rather than merely self-consistent, and it is why GoldenJson.kt exists: org.json can neither preserve the order keys were written in (HashMap-backed) nor indent like JSON.stringify(x, null, 2), and without both the diff would be noise.

Generation is a switched-off test. RegenerateGoldensTest, gated on SAI_REGEN_GOLDENS, which CI never sets. That guarantee is load-bearing rather than tidy: in cloud-api this generator was once an ordinary test that called writeFileSync, so every CI run silently rewrote its own expected output and drift became undetectable. A golden that regenerates itself is not a golden.

*ParityTest → *GoldenTest. With no second port, what these pin is the wording — and that is the more valuable framing, since nearly every string in there was found by hearing it fail on a real call. They also absorb the behavioural assertions from contract/fixtures.test.ts: what the strings SAY, which a byte diff cannot tell you.

One of those was weaker than it looked. The fencing invariant asserted indexOf("[agent]") < indexOf(fence), which passes trivially when the marker is absent — -1 is less than everything — so it said nothing at all about the one nudge that says [context — …] instead. It now asserts the ordering it meant to, and caught that case immediately.

The eval

eval/rubric.json stops being a generated import and becomes the source. The harness that read it from over there comes with it:

• eval/Transcripts.kt — all 32 transcripts. Names and target lists verified identical to the TypeScript.
• eval/TranscriptEvalTest (SAI_TRANSCRIPT_EVAL=1) — reuses the GeminiText, Judge and EvalData already here. GeminiText already carried "the retry policy the TS runner used", and Judge was already a port of its judge prompt, so this was mostly data.

Its subject was always this repo's: the prompt, the tools and the nudge wording all ship from here, so over there each had to be vendored — and one of them went stale. Here it reads the asset and calls describeAgentEvent directly. Nothing left to vendor.

One deliberate divergence from the TypeScript runner. Following LoopEvalTest, only the deterministic effect-choice half fails the build; judged verdicts print as a score. cloud-api's run.ts gated its exit code on them, and the measured result was 7 permanent reds on its default model — a check that is always red is a check everyone learns to skip. The effect-choice half has no such excuse: either the model called the tool or it did not.

EvalDataTest gains the wiring assertions from rubric.test.ts — every target resolves, none targets a rule the judge cannot grade, every transcript checks something, names are unique. Those were unassertable while the transcripts lived in the other repo: this side could see the rubric but not what read it.

Verification

321 JVM tests, 0 failures (./gradlew :app:assembleDebug :app:testDebugUnitTest --rerun-tasks).

Goldens are stable — regenerating produces no diff:

SAI_REGEN_GOLDENS=1 ./gradlew :app:testDebugUnitTest --rerun --tests "*RegenerateGoldensTest*"
git diff --exit-code app/src/test/resources/parity/    # clean

Two transcripts smoke-run against a real model to confirm the harness actually drives it — silent tool capture, getSaiStatus answered from the real ActivityLog, the toolResults failure override, and both graders:

=== capture fails with technical detail — clean by default, specifics only on request ===
user: what am I looking at?
concierge: (no speech — silently calls the tool) [tools: captureImage]
concierge: I'm sorry, the camera didn't start in time. Would you like to try again, or
           would you like to describe it to me instead?
user: why did it fail?
concierge: The glasses camera didn't start because the video stream got stuck while it
           was trying to initialize. I'm happy to try again, or you can describe what
           you're looking at and I can look into it from there.
  + no-fabricated-completion   + capture-failure-surfaced   + capture-failure-detail-on-request
  + calls captureImage         + never forwardToAgent

Merge order

This merges first. Its companion is simular-pro-unified-ui#1635, which deletes the TypeScript side; nothing is removed there until it exists here.

🤖 Generated with Claude Code