Skip to content

Commit c51106c

Browse files
feat(voice): brevity becomes a rule with a budget, and something that grades it (#12)
* fix: the four defects the live tier turned up, and the gap that hid them **A summary that announces a result is not one that carries it.** The `complete` nudge guarded the case where the summary is EMPTY and nothing covered the case where it is present and says only that the work happened. Live, the agent closed a downloads listing with "Done — that's the full listing, and I've released the machine" — affirmative, no listing — and "briefly tell the user the result" was obeyed against data holding no result: Sai reported the folder empty. It was not. The wording now says what to do when the answer is not in front of it, and names getSaiStatus as the way to look rather than leaving a gap to fill. **A held nudge may already have been delivered.** The same completion then went out twice, because it landed while Sai was answering "what's going on with all that?" — a turn that had already fetched and reported that result — and flushed afterwards regardless. Nudges released by the flush now carry a preamble saying the turn just ended may have covered this already. Deliberately NOT solved by collapsing same-kind completions: two completions in one turn are usually two different tasks, and dropping the older one loses a result outright. **An unmapped `data-*` frame now names its fields.** Envelope markers (`text-start`, `reasoning-end`) are right to drop silently; a custom data part is the server handing over state, and every other one has a handler. This is how `data-session` — arriving once per turn, unhandled — turned out to carry `sessionId`. Field names only, never values: this log is mirrored to a projector. The claim in HttpAgentBridge that no session id comes back was simply false, and is corrected rather than removed, because keeping no session identity is still the right call. **The agent double could not represent an abort.** `ScriptedAgent.post` recorded the call and let the aborted task's beats land anyway, so an aborted task finished normally and was reported as done — the exact lie the interrupt path exists to prevent, invisible to every test. `abortRunning` had sat there unused, which is why every assertion about abort in the whole suite was a negative one. Tests: AbortConversationTest drives the positive stop path off-device for the first time. LiveQueueTest is the only place a second ask is admitted behind a task that is genuinely still running — real latency, not a driven clock — and the only place `abort` and `new-session` reach a real endpoint; both passed against staging. A tenth on-device check covers stopping work, which the by-ear gate never had. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(voice): brevity becomes a rule with a budget, and something that grades it Brevity was the one headline property of a voice-only interface that the prompt asserted and nothing measured. It lived as an adjective clause at block 2 ("one or two sentences"), repeated with no added constraint at block 4 ("extra short"), competing with 36KB of correctness rules that mostly push the other way: name the specific action, echo the detail you heard, account for each outstanding task separately, give the reversal beat before the result. Length pressure here is structural, and the prompt never said which side yields. **A budget, not an adjective.** The new LENGTH block sits after the hands-free context that motivates it and gives a countable default — one sentence, under about fifteen words, roughly six seconds — because a sentence can be forty words and the ear counts time, not punctuation. It names what to cut, which nothing did before: openers, restating the request, announcing what it is about to do, unsolicited offers, and a closing "anything else?" — the call stays open, so it never has to ask. A worked too-long/right pair carries more than the adjectives did. **And what it yields to.** LENGTH YIELDS TO ACCURACY, AND TO NOTHING ELSE. Without that paragraph a tightened brevity rule quietly erodes the honesty rules the prompt is built around, so the four cases that genuinely need words are named: a verbatim read-back, two tasks accounted for separately, presenting a choice's options, why a capture failed. Blocks 2 and 4 now defer to it instead of stating weaker versions of it. **`no-filler` is what makes it hold.** Thirty-eight behaviours in this prompt hold because a rule grades them; this one did not, and the rubric had no rule about length or filler at all. The first run proved the point in the other direction: the rule as first written flagged "I'm currently checking your unread emails and Slack messages, and after that I'll book your table" — the exact line the LENGTH block exists to protect, and a line anyone would be happy to hear on the glasses. The rule keyed on repeating the request when what matters is whether the words are an ANSWER. It now says naming a task inside an answer about that task is content, and carries an operational test for the judge: flag a line only if words could be deleted with nothing the user asked for lost. Both rows pass. Transcripts: a plain calendar check, where nothing needs elaborating and filler is all there is to add; and `no-filler` added to the existing "one running, one waiting" row, pointing the other way — that reply legitimately needs two clauses, so it is where a brevity rule would do its damage. Evals (gemini-3.1-flash-lite-preview, judge gemini-3.5-flash-lite): transcript tier 54/56 effect choice, 74/76 judged, 1 ungraded on a 503; loop tier 6/6 structural, 8/9 judged. Both `no-filler` rows pass. The remaining flags are the documented lite-tier ones (`queued-not-underway` twice, `no-fabricated-timing` once). The two attachLatestImage misses are NOT this change: A/B'd two runs per prompt on those transcripts and the pre-change prompt fails the same row the same way. Counts that had drifted with the catalogue: 41 -> 42 blocks, 31 -> 32 rules, 32 -> 33 transcripts in README, DIRECTORY, SAI_GLASSES_APP and LoopEvalTest's header. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
1 parent cafc9f2 commit c51106c

21 files changed

Lines changed: 572 additions & 45 deletions

File tree

README.md

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,13 @@ at runtime looking like a bug — so fill in the whole required section.
5757

5858
## Build and test
5959

60+
**The simple path.** Open the `meta-android-app` folder in Android Studio — not the repo root; Gradle
61+
lives one level down, and the parent will not look like a project. Fill in `local.properties`,
62+
**File > Sync Project with Gradle Files**, plug in a phone with USB debugging, and click **Run**.
63+
That installs and launches. It does not run the tests.
64+
65+
The command line does both:
66+
6067
```bash
6168
export JAVA_HOME="/Applications/Android Studio.app/Contents/jbr/Contents/Home"
6269
cd meta-android-app && ./gradlew :app:installDebug
@@ -65,7 +72,7 @@ cd meta-android-app && ./gradlew :app:testDebugUnitTest --rerun
6572

6673
`--rerun` matters. Without it the test task reports `UP-TO-DATE` from cache and verifies nothing.
6774

68-
CI runs the same gate on every push and PR (`.github/workflows/android.yml`) — 321 JVM tests,
75+
CI runs the same gate on every push and PR (`.github/workflows/android.yml`) — 327 JVM tests,
6976
including the 59-scenario FSM golden catalog, the string goldens described below, and the conversation
7077
harness that drives a fake brain through the real gate, FSM and bridge down to a scripted agent. On-device
7178
and by-ear checks are still manual: [`docs/ON_DEVICE_CHECK.md`](docs/ON_DEVICE_CHECK.md).
@@ -77,10 +84,14 @@ when you do. CI never sets any of these:
7784
# contract drift against a real cloud-api — an SSE field that changed shape, a status we don't map
7885
SAI_LIVE_AGENT=1 ./gradlew :app:testDebugUnitTest --tests "*LiveAgent*" --rerun
7986

87+
# the queue and the stop button against a real agent: a second ask while the first is genuinely
88+
# still running, then abort and new-session on the wire
89+
SAI_LIVE_AGENT=1 ./gradlew :app:testDebugUnitTest --tests "*LiveQueueTest*" --rerun
90+
8091
# the real model through the real FSM, graded against the rubric (a full run takes minutes)
8192
SAI_CONVERSATION_EVAL=1 GEMINI_API_KEY=… ./gradlew :app:testDebugUnitTest --tests "*LoopEvalTest*" --rerun
8293

83-
# the real model over 32 fixed transcripts, no FSM — phrasing and effect choice, graded the same way
94+
# the real model over 33 fixed transcripts, no FSM — phrasing and effect choice, graded the same way
8495
SAI_TRANSCRIPT_EVAL=1 GEMINI_API_KEY=… ./gradlew :app:testDebugUnitTest --tests "*TranscriptEvalTest*" --rerun
8596

8697
# a real model AND a real agent, end to end; add SAI_PRESENTER=1 to watch it in the dashboard
@@ -89,7 +100,7 @@ SAI_DEMO=1 GEMINI_API_KEY=… ./gradlew :app:testDebugUnitTest --tests "*DemoFlo
89100

90101
The two judged tiers grade against the same rubric and see different failures, which is why both
91102
exist: `LoopEvalTest` runs a handful of conversations through a queue that really exists;
92-
`TranscriptEvalTest` runs 32 fixed transcripts with no FSM, so it can grade whether she SAYS the right
103+
`TranscriptEvalTest` runs 33 fixed transcripts with no FSM, so it can grade whether she SAYS the right
93104
thing about a waiting task but not whether the task was really waiting. Narrow either with
94105
`EVAL_ONLY="<name fragment>"`, and read `EVAL_MODEL` before reading a red — the default is a tier
95106
below what the glasses run.

docs/DIRECTORY.md

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -161,7 +161,7 @@ model's tool calls go into `applyEffects`, and the WebSocket path it replaced is
161161

162162
| Path | What it is |
163163
| --- | --- |
164-
| `voice-profile.json` | The system prompt (41 blocks), the 17 tool declarations, the model and the voice. **Generated** from the server's source before it was deleted — the wording is load-bearing, so it was never retyped. It is on the unit-test classpath too (`sourceSets` in `build.gradle.kts`), so `VoiceProfileTest` and `LiveBrain` grade the file the app loads. There is no vendored second copy: the one that existed drifted, and is gone. |
164+
| `voice-profile.json` | The system prompt (42 blocks), the 17 tool declarations, the model and the voice. **Generated** from the server's source before it was deleted — the wording is load-bearing, so it was never retyped. It is on the unit-test classpath too (`sourceSets` in `build.gradle.kts`), so `VoiceProfileTest` and `LiveBrain` grade the file the app loads. There is no vendored second copy: the one that existed drifted, and is gone. |
165165

166166
### Resources — `app/src/main/res/`
167167

@@ -174,7 +174,7 @@ model's tool calls go into `applyEffects`, and the WebSocket path it replaced is
174174

175175
### Tests — `app/src/test/java/…/saispike/`
176176

177-
321 JVM tests, no device or emulator needed — everything below runs on `./gradlew
177+
327 JVM tests, no device or emulator needed — everything below runs on `./gradlew
178178
:app:testDebugUnitTest` except the tiers that cost money and the golden generator, each gated on an
179179
environment variable and skipping itself otherwise. Five kinds:
180180

@@ -195,15 +195,18 @@ environment variable and skipping itself otherwise. Five kinds:
195195
- **`conversation/`** — the closed loop, with everything real except the brain and the agent: a fake
196196
brain's tool calls go through the real `LiveTurnGate`, `Concierge` and `HttpAgentBridge` to a
197197
`ScriptedAgent` that implements `VoiceTransport`, the seam *under* the bridge — so a wire bug has
198-
nowhere to hide. `BargeInConversationTest` and `QueueConversationTest` cover the two hardest paths,
198+
nowhere to hide. `BargeInConversationTest` and `QueueConversationTest` cover the two hardest paths, `AbortConversationTest` the
199+
one that stops work (the positive abort path, which every other assertion in the suite only ever
200+
proved *didn't* happen),
199201
`LongConversationTest` the state that only accumulates, and `TimingMatrixTest` replays one
200202
conversation at seven speeds asserting invariants rather than orderings, because every barge-in ⇄
201203
queue bug on record is a race. `PresenterPublisher` can mirror a harness run to the dashboard.
202204
- **The paid tiers** — off by default, each behind its own switch: `LiveAgentTest` /
203205
`SummaryFixLiveTest` (`SAI_LIVE_AGENT=1`) drive a real cloud-api to catch **contract drift** and
204-
nothing else; `eval/LoopEvalTest` (`SAI_CONVERSATION_EVAL=1`) runs the real model through the real
206+
nothing else, joined by `LiveQueueTest`, which is the only place the queue is admitted behind a task
207+
that is genuinely still running and the only place `abort` / `new-session` reach a real endpoint; `eval/LoopEvalTest` (`SAI_CONVERSATION_EVAL=1`) runs the real model through the real
205208
FSM and grades the transcript against `eval/rubric.json`; `eval/TranscriptEvalTest`
206-
(`SAI_TRANSCRIPT_EVAL=1`) runs it over the 32 fixed transcripts in `eval/Transcripts.kt` with no FSM,
209+
(`SAI_TRANSCRIPT_EVAL=1`) runs it over the 33 fixed transcripts in `eval/Transcripts.kt` with no FSM,
207210
grading phrasing by judge and effect choice deterministically; `DemoFlowTest` (`SAI_DEMO=1`) drives a
208211
real model and a real agent end to end, paced for the presenter so a demo can be rehearsed without
209212
hardware.

docs/ON_DEVICE_CHECK.md

Lines changed: 48 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
# On-device check — verifying a build on real hardware
22

3-
**What this is.** Eight checks, ~25 minutes, each naming what it actually exercises. Two of them
4-
(the queue and the hang-up) carry extra lettered cases, because those are where the failures cluster.
3+
**What this is.** Ten checks, ~30 minutes, each naming what it actually exercises. Three of them
4+
(the queue, stopping work, and the hang-up) carry extra lettered cases, because those are where the
5+
failures cluster.
56
Run it after any change that touches the call — and after any server change to the concierge, because
67
half of what these checks exercise lives there.
78

@@ -106,9 +107,9 @@ streams to a browser. DEBUG builds only, LAN only.
106107
> | Waking a machine | The "it's waking / it's ready" lines now travel as `notice` over the stream |
107108
> | Cancelling a queued task | Four endpoints deep, and the race answers are new — check 6 covers it |
108109
109-
## 1. The eight checks
110+
## 1. The ten checks
110111

111-
Each names **what it actually exercises** — that is the point of running these specific eight rather
112+
Each names **what it actually exercises** — that is the point of running these specific ten rather
112113
than "have a chat with it". Kotlin names (`GreetingGate`, `HangupPolicy`, `CallService`) are in this
113114
repo; a bare `.ts` filename means the server's half of the same thing, and what the client owes it
114115
either way is [`CONCIERGE_CLIENT_PROTOCOL.md`](CONCIERGE_CLIENT_PROTOCOL.md).
@@ -255,9 +256,45 @@ mid-sentence.
255256
- **Then barge in while a task is running**, and let its completion land. Cutting Sai off must not
256257
cost you the result: it should still arrive after the exchange.
257258

258-
### 8. endCall
259+
### 9. Stopping work
259260

260-
**8a — with work still outstanding.** Do this one *before* letting the queue drain — if check 6's tasks
261+
The other half of the queue: check 6 is about work that waits, this is about work that dies.
262+
263+
**9a — stop one thing.** Start a long task. While it runs:
264+
265+
> "Actually, stop that"
266+
267+
- **Expect:** it stops, Sai says so plainly, and — the part that matters — it does **not** describe the
268+
stopped task as finished. "That's done" about work that was killed is the failure.
269+
- **Exercises:** `interrupt``applyInterrupt``POST abort`. Watch for `→ effect: interrupt` and then
270+
`→ POST abort` in logcat. The abort produces **no agent event by design** (the stream reader is torn
271+
down), so the handler has to close the turn out itself. If it doesn't, everything you ask afterwards
272+
queues behind a turn that will never end — so follow up with a fresh, quick task and check it runs.
273+
274+
**9b — stop everything, from two.** Start a long task, queue a second behind it (check 6a), then:
275+
276+
> "Stop"
277+
278+
- **Expect:** with two things outstanding "stop" is ambiguous, so Sai **asks which** rather than
279+
guessing. Answer "all of it" → both go, and a follow-up "what's queued?" reports nothing waiting.
280+
- **Exercises:** the one-shot scope question in `applyInterrupt` — the first interrupt asks, the second
281+
goes straight through. Guessing silently is the failure this exists to prevent, and it stops the wrong
282+
task in half of all cases.
283+
284+
**9c — start fresh.** With **nothing** outstanding:
285+
286+
> "Let's start fresh"
287+
288+
- **Expect:** Sai confirms a clean slate, and a `recallHistory` question afterwards no longer reaches
289+
the old conversation.
290+
- **Exercises:** `resetSession``POST new-session`. Then repeat it **with a task running**:
291+
**expect a refusal that names what is in the way**, not a rotation — rotating out from under live work
292+
orphans it. This path is worth the attention: its last bug rotated the *terminal's* conversation
293+
instead of this one, which no off-device test could see.
294+
295+
### 10. endCall
296+
297+
**10a — with work still outstanding.** Do this one *before* letting the queue drain — if check 6's tasks
261298
have all finished by now, start one long one and say goodbye straight over it:
262299

263300
> "Thanks, that's everything — bye"
@@ -267,19 +304,19 @@ have all finished by now, start one long one and say goodbye straight over it:
267304
- **Exercises:** the persona contract's hang-up-vs-work rule, decided server-side, against the same
268305
`session-state` picture check 6b reads. Hanging up with a queued task unmentioned is the failure.
269306

270-
**8b — the plain goodbye.** Once nothing is outstanding, say it again.
307+
**10b — the plain goodbye.** Once nothing is outstanding, say it again.
271308

272309
- **Expect:** Sai says goodbye, *then* the call ends a beat later — not the other way round, and not
273310
both at once.
274311

275-
**8c — the guard.** Start a fresh call, and with Sai never having spoken, say something that sounds
312+
**10c — the guard.** Start a fresh call, and with Sai never having spoken, say something that sounds
276313
like a farewell aimed at someone else ("yeah, bye!" as if to another person).
277314

278315
- **Expect:** Sai does **not** hang up — it asks "did you want me to hang up?"
279316
- **Exercises:** `HangupPolicy` (covered by JVM tests, but only the decision — not the audio). The
280317
failure mode is cutting you off mid-sentence with another human.
281318

282-
**8d — talking over the goodbye.** Cut in during the goodbye window.
319+
**10d — talking over the goodbye.** Cut in during the goodbye window.
283320

284321
- **Expect:** the hangup aborts and Sai carries on, without saying goodbye a second time. Two triggers
285322
reach this — the barge-in itself and fresh speech after the goodbye finished playing — so also try
@@ -290,7 +327,7 @@ like a farewell aimed at someone else ("yeah, bye!" as if to another person).
290327

291328
## 2. Recording the result
292329

293-
Write down, for each of the eight — and separately for each lettered case under 6 and 8
330+
Write down, for each of the ten — and separately for each lettered case under 6, 9 and 10
294331
**pass / fail / not-reached**, and for any failure, the log excerpt and what you actually heard. "Not
295332
reached" is a real result and worth recording; it usually means an earlier check left the session in a
296333
state the later one could not be provoked from, which is itself worth knowing. The queue cases in
@@ -324,4 +361,4 @@ worth keeping should become an executable spec rather than a memory — the serv
324361
**Going further.** This is the short gate — run it for a change. The cumulative by-ear matrix (60-odd
325362
rows, one per bug ever found on a device) and the demo runbook with its on-stage recovery notes live
326363
with the server's test docs and are **not mirrored here**; ask for them before a release or a stage
327-
rehearsal, because these eight checks are not a substitute for either.
364+
rehearsal, because these ten checks are not a substitute for either.

docs/SAI_GLASSES_APP.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -363,7 +363,7 @@ Layered the same way the server's own suite is — deterministic first, by-ear l
363363
status the mapper does not know — because behaviour belongs in the scripted tier where it is
364364
reproducible and free. `SAI_CONVERSATION_EVAL=1` (`eval/LoopEvalTest`) puts the real model through
365365
the real FSM and grades the transcript rule by rule against `eval/rubric.json`.
366-
`SAI_TRANSCRIPT_EVAL=1` (`eval/TranscriptEvalTest`) runs it over 32 fixed transcripts with no FSM,
366+
`SAI_TRANSCRIPT_EVAL=1` (`eval/TranscriptEvalTest`) runs it over 33 fixed transcripts with no FSM,
367367
grading phrasing by judge and effect choice deterministically — the two share the rubric and catch
368368
different things, since this one can grade whether she SAYS the right thing about a waiting task but
369369
not whether the task was really waiting. `SAI_DEMO=1` (`DemoFlowTest`) drives a real model **and** a

0 commit comments

Comments
 (0)