Skip to content

Commit a100952

Browse files
jamie-simularclaude
andcommitted
feat(voice): stop means stop — the abort now reaches the reader and the camera
The device found three failures in one call, all of them the same shape: "stop" was reported as done while the work carried on. **The abort never stopped anything local.** `applyInterrupt` has always claimed the turn's reader was torn down; nothing did it, so an aborted task was read to its natural end and Sai reported the result of work she had just been told to abandon. And a turn about what the user is looking at spends most of its life on this phone waiting for the camera, so a "wait, stop" still ended with a shutter firing and a photo arriving for a task that no longer existed. `abort()` now stops reading, cancels the device's own work, and only then asks the server. A generation counter covers the gap `ensureActive` cannot see: an event already parsed and parked on the FSM's mutex — the mutex the abort itself holds. **There was no way to say "drop this one and carry on."** `interrupt` took the queue with it, `cancelQueued` cannot touch a running task, and `relayToAgent` only narrows one from the inside — so the model reached for `resetSession`, which wipes the conversation and stops no work at all. `interrupt` gains a scope; an absent or unknown one still means everything, because a bare "stop" means stop. **And `resetSession` was one mis-heard word from a wipe.** It now confirms once. The flag is cleared by any batch that is not another reset, because a held reset happens with nothing running, so no turn starts to clear it and a declined "no, just drop that" left the yes standing. Two more from the same log. Ask-first fired after a 40s capture and silenced a result the user was sitting there waiting for — quiet is now measured to the moment work STARTED, keeping the first stamp after they speak, so the camera spinning up ends their silence rather than the forward tens of seconds later. Its nudge also says plainly that it is a delay and not a discard, after Sai offered to retry a task whose answer she was holding; `ActivityLog` keeps that outcome so `getSaiStatus` can answer with it. The camera's retries were self-inflicted: the frame counter used `take(N).collect`, which cancels the subscription on the Nth frame, so every `capturePhoto` was issued against a stream nobody was reading. The collector now stays subscribed until the still is in hand. Calls no longer mint their own session. A conversation outlives a call — five quiet minutes, folded glasses or a misheard goodbye all end one — and each mint added a page to the user's sidebar. What made the original hazard unescapable was a second bug, since fixed: "start fresh" rotated the terminal's session instead of this client's. One review fix on top: the abort POST is best-effort in code and not only in its comment. It threw, which returned before the turn was closed out, leaving the FSM in `working` with its reader already gone — no event could arrive to end it, and admission held every later task behind a turn that could not finish. Docs: VOICE_FSM gains the two one-shot confirmations and the conversation-per-call trade; three stale counts corrected (63 scenarios, fourteen effects). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 08e14c8 commit a100952

38 files changed

Lines changed: 1659 additions & 338 deletions

docs/CONCIERGE_CLIENT_PROTOCOL.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -208,6 +208,15 @@ from; moving retires the one behind you. Two things follow, and a client owes th
208208
app's half, shared with the hang-up, and it exists because the FSM is rebuilt on a switch and would
209209
otherwise drop a queued task it had promised out loud.
210210

211+
**Do not rotate on your own initiative.** `POST /new-session` is the user's command, not the client's
212+
housekeeping: a send already resolves the channel's current session, so there is nothing a client has
213+
to do to stay in one conversation. sai-fi used to mint a session per call and the rotation was
214+
visible — every call added a page to the user's sidebar in the desktop app. The unit matters more than
215+
it sounds. A *call* ends on five minutes of quiet, on the wearer folding the glasses, or on the model
216+
mishearing a goodbye, so a client that rotates per call rotates several times inside one thing the
217+
user would call a conversation. Rotate when asked, and leave it alone otherwise. (Session growth is
218+
the real cost of not rotating; if it needs bounding, bound it on the session's age, not on a call.)
219+
211220
## 8. Keeping a port honest
212221

213222
If you implement this in another language, mirror the guards that already exist rather than inventing

docs/ON_DEVICE_CHECK.md

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -301,9 +301,14 @@ The other half of the queue: check 6 is about work that waits, this is about wor
301301
- **Expect:** it stops, Sai says so plainly, and — the part that matters — it does **not** describe the
302302
stopped task as finished. "That's done" about work that was killed is the failure.
303303
- **Exercises:** `interrupt``applyInterrupt``POST abort`. Watch for `→ effect: interrupt` and then
304-
`→ POST abort` in logcat. The abort produces **no agent event by design** (the stream reader is torn
305-
down), so the handler has to close the turn out itself. If it doesn't, everything you ask afterwards
306-
queues behind a turn that will never end — so follow up with a fresh, quick task and check it runs.
304+
`[voice] stopped following the turn — it was aborted`. No agent event follows, *because* of that line:
305+
the abort stops this device reading the turn, so the handler has to close the turn out itself. If it
306+
doesn't, everything you ask afterwards queues behind a turn that will never end — so follow up with a
307+
fresh, quick task and check it runs.
308+
- **Also watch for what must NOT appear:** any `` progress line, a `✓ done:`, or a `→ nudge: complete`
309+
after the stop. On 2026-08-20 all three did — nothing tore the reader down, the cancelled task was
310+
read to its natural end, and Sai reported its result to a user who had just stopped it. If you see
311+
`[voice] dropped … from an abandoned turn` instead, that is the guard working.
307312

308313
**9b — stop everything, from two.** Start a long task, queue a second behind it (check 6a), then:
309314

@@ -326,6 +331,16 @@ The other half of the queue: check 6 is about work that waits, this is about wor
326331
orphans it. This path is worth the attention: its last bug rotated the *terminal's* conversation
327332
instead of this one, which no off-device test could see.
328333

334+
**9d — two calls, one page.** With the desktop app's sidebar visible: make a call, ask for anything
335+
that reaches the agent, hang up. Start a second call and ask for something else.
336+
337+
- **Expect:** **one** conversation page total, with both calls' work in it — not a new page per call.
338+
Then say "start fresh" and expect a second page to appear, since that one is asked for.
339+
- **Exercises:** the absence of a per-call mint (`VOICE_FSM.md` §7b). Nothing off-device can see this
340+
`VoiceSession` has no unit test and the harness fakes reimplement the transport — and the symptom
341+
is only visible in the sidebar, which is why it went unnoticed. Note that a **machine switch** still
342+
starts a new page and that is expected: `machineId` is in the server's session key.
343+
329344
### 10. endCall
330345

331346
**10a — with work still outstanding.** Do this one *before* letting the queue drain — if check 6's tasks

docs/SAI_GLASSES_APP.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,7 @@ while nothing is running, for instance.
103103
| `ActivityLog` / `ConciergeProtocol` | The canonical nudge and activity-log wording (`describeAgentEvent` with prompt-injection fencing). Originally ported verbatim from the server's `core/activity-log.ts` / `core/nudges.ts`; those are gone and these are the only copy, pinned by the string goldens (§8.1). Feed `getSaiStatus` + the UI activity view. |
104104
| `GlassesGestureSession` | DAT `DeviceSession` (no display/camera capability) reacting to the only temple gestures DAT surfaces: tap = mute/unmute Sai, tap-and-hold/doff/fold = end (all just `DeviceSessionState`; no gesture is remappable — see §6 "Glasses gestures"). |
105105
| `SaiFiApp` / `MainActivity` | App init (`Wearables.initialize` once); `MainActivity` is now just the DAT-registration deep-link callback host (`saiwearables`) — the CameraAccess sample UI was pruned in productization. |
106-
| `fsm/` (13 files) | **The conversation state machine**, ported from the server: modes and transitions, the bounded effect grammar and its parse boundary, the admission rule that holds a mid-turn task instead of folding it in, the local held-task queue, the cost guard, and every line the FSM speaks. Everything but `Concierge.kt` is pure, which is what makes the 59-scenario golden catalog runnable as JVM tests; `Concierge.kt` serialises all four input kinds through one `Mutex`, because two forwards interleaving at a suspension point books the restaurant twice. Design: [`VOICE_FSM.md`](VOICE_FSM.md). **This drives every call**`CallService.buildConcierge` builds one `VoiceSession` per call and feeds the model's tool calls straight into it. |
106+
| `fsm/` (13 files) | **The conversation state machine**, ported from the server: modes and transitions, the bounded effect grammar and its parse boundary, the admission rule that holds a mid-turn task instead of folding it in, the local held-task queue, the cost guard, and every line the FSM speaks. Everything but `Concierge.kt` is pure, which is what makes the 63-scenario golden catalog runnable as JVM tests; `Concierge.kt` serialises all four input kinds through one `Mutex`, because two forwards interleaving at a suspension point books the restaurant twice. Design: [`VOICE_FSM.md`](VOICE_FSM.md). **This drives every call**`CallService.buildConcierge` builds one `VoiceSession` per call and feeds the model's tool calls straight into it. |
107107

108108
## 3. The client contract (frozen; the Kotlin app ports the browser reference client)
109109

@@ -176,6 +176,11 @@ programmatic channel, as with a rebound Telegram link), so coming back to a mach
176176
`recallHistory` will not reach what you did there before. Within a call that costs nothing: the Live
177177
model holds the whole call in its own context.
178178

179+
That retirement is the server's, and the client no longer adds a rotation of its own on top of it.
180+
Ending a call and starting another stays in one conversation — one page in the desktop sidebar — and
181+
the only thing that rotates the session is the user saying "start fresh". See
182+
[`VOICE_FSM.md`](VOICE_FSM.md) §7b for what that trades away.
183+
179184
**Client-local voice tools** (declared server-side, handled on-device): `getSaiStatus` (status
180185
pull), `recallHistory` (recent machine history via `GET /v1/agents/context` — recall questions
181186
answered without waking the agent), `switchMachine` (rebuilds the concierge against another owned VM; the Live audio session keeps

docs/VOICE_FSM.md

Lines changed: 66 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ Everything in `State.kt`, `Effects.kt`, `Speech.kt` and `AgentIngest.kt` is pure
6464
input go in, a new state comes out. No coroutines, no clock, no I/O. `Concierge.kt` is the only part
6565
that suspends, and the ports (`AgentBridge`, `VoiceChannel`) are the only way it reaches the world.
6666

67-
That split is what makes 59 golden scenarios runnable as plain JVM tests. It is the same shape
67+
That split is what makes 63 golden scenarios runnable as plain JVM tests. It is the same shape
6868
`GlassesLink` uses, for the same reason.
6969

7070
**Do not make the pure parts call the clock or the network.** The moment they do, the catalog stops
@@ -104,7 +104,7 @@ to different questions.
104104

105105
## 4. Effects: the conversation is open, the capabilities are not
106106

107-
The model can say anything. It can *do* only the fifteen things in `Effects.kt`. `parseEffect` is the
107+
The model can say anything. It can *do* only the fourteen things in `Effects.kt`. `parseEffect` is the
108108
boundary, and an effect it does not recognise — or whose payload is the wrong shape — is **dropped**,
109109
not guessed at. A newer model inventing a capability does not get to exercise it.
110110

@@ -115,6 +115,9 @@ The parse rules are deliberately asymmetric, and the asymmetry is the contract:
115115
- `askAndWait` and `setState` **reject outright** on a bad enum — an invented mode must never become
116116
a state.
117117
- An unrecognised `urgency` **degrades to normal** rather than dropping the task.
118+
- An unrecognised `interrupt` **`scope` widens to `everything`**, for the same reason pointed the
119+
other way: a bare "stop" means stop, and a scope a build does not know must never quietly
120+
narrow a cancellation into leaving work running the user believes they stopped.
118121

119122
`askAndWait` does **not** speak. It is a pure state signal: the Live model has already voiced the
120123
question, and speaking it again doubles it up and interrupts the model mid-sentence.
@@ -152,6 +155,33 @@ hears "on it" waits for a result nothing is producing. So the spoken line names
152155

153156
> "Got it — I'll start that as soon as I'm done with: …"
154157
158+
## 6b. Two questions the FSM asks before it acts, each exactly once
159+
160+
`interrupt` and `resetSession` both guard themselves with a one-shot flag, and both flags are cleared
161+
by `startTurn` — new work is a new context, and a stale yes is a yes to something else.
162+
163+
**The interrupt's scope question** fires when more than one thing is outstanding, counting running
164+
PLUS queued: since admission holds a second request rather than folding it in, "one running, one
165+
queued" is the same question with the same stakes. A second `interrupt` while the flag is set reads as
166+
"all of it" and goes straight through. A scoped `running` interrupt skips the question entirely,
167+
because the user has already answered it — the waiting list is explicitly being kept.
168+
169+
**The reset confirmation** exists because a rotation cannot be undone, and because "forget it", "never
170+
mind" and "drop that" are almost always about the last thing said while sharing their vocabulary with
171+
"forget everything we talked about". On a device call a bare "forget it" — about a question Sai had
172+
just asked — arrived as `resetSession` and cleared the conversation. The prompt says not to do that;
173+
the prompt is not a guarantee, and there is nothing behind it. So the first call asks and the second
174+
rotates.
175+
176+
That flag needs one thing `startTurn` cannot give it: a held reset happens with **nothing running**,
177+
so no turn starts to clear it. A user who answers "no, just drop that" would leave the yes standing
178+
for the next stray "forget it", minutes and subjects later. So `applyEffects` also clears it after any
179+
batch that is not another `resetSession` — the user having moved on. Asking again is the safe
180+
direction; the failure being avoided is a wipe nobody asked for.
181+
182+
Neither question is a `say`. Both are `instruct`, so the model asks in its own words and nothing is
183+
voiced that describes an action as already taken — see §5.
184+
155185
## 7. The queue is local, and that is a trade
156186

157187
Held tasks live in this FSM and **nowhere else**. Nothing is written server-side when a task is
@@ -170,6 +200,39 @@ the world that starts a held task.** So every path that can leave `mode` at `IDL
170200
It runs after each agent event, which covers every way a turn ends today; miss one and a task the
171201
user was told was coming simply never runs.
172202

203+
## 7b. The conversation outlives the call, and that is a trade
204+
205+
The client does **not** rotate the agent session on its own initiative. A send resolves the server's
206+
`{uid}_{machineId}_{channel}` pointer to whatever session is already current, and the only thing that
207+
moves that pointer is the user asking — `resetSession`, on "start fresh". Two consecutive calls are
208+
one conversation, and one page.
209+
210+
**Why it used to rotate per call.** `VoiceSession` minted a fresh `api` session on its first forward,
211+
and the hazard it was guarding is real: everything in that transcript is read back as the agent's own
212+
prior turns on every later call, so one bad turn is not a bad turn — it is a permanent change of
213+
behaviour. That is not hypothetical. A stubbed reply written during local testing was still being
214+
imitated by the real agent days later, on a machine doing real work, and fixing the code that wrote
215+
it did not help, because a code fix does not reach the data.
216+
217+
**Why that no longer justifies it.** What made that episode unescapable was a *second* bug: the one
218+
command for getting out — "start fresh" — was rotating the terminal's `cli` session instead of this
219+
client's `api` one, so the user could not rotate away from the poisoned transcript even by asking.
220+
That is fixed and pinned by a test. Meanwhile the cost of auto-rotation was being paid constantly,
221+
because **a call ends far more easily than a conversation does**: five quiet minutes trips the idle
222+
guard, folding or removing the glasses ends it, and the model can decide it heard a goodbye. A chat,
223+
glasses off for a minute, two more questions, then a machine switch produced four pages in the user's
224+
sidebar for what they experienced as one conversation.
225+
226+
**What this costs, plainly.** The `api` transcript now grows without bound, and a bad turn persists
227+
across calls until the user says "start fresh". That is a step backwards on blast radius, accepted
228+
knowingly. If a long-lived session turns out to degrade the agent measurably, the next move is
229+
**age-bounded rotation** — not a return to per-call, which trades a rare problem for a constant one.
230+
231+
One part of this is not reachable from here: a **machine switch** still starts a new page, because
232+
`machineId` is part of the server's session key. That needs a cloud-api change — dropping `machineId`
233+
from the `api` channel's key, or a `sessionId` parameter on `POST /message` so a client could pin the
234+
session the `data-session` frame already names.
235+
173236
## 8. The races that used to be here
174237

175238
`Races.kt` is gone. It guarded three cases that arose because held work lived in two uncoordinated
@@ -241,7 +304,7 @@ changed. Concretely: two forwards both observe an empty `inFlight` before either
241304
take the immediate path, and the user's restaurant is booked twice.
242305

243306
**Testability.** `Dispatchers.Main` has no implementation in a plain JUnit run, and nothing in this
244-
suite installs one. An FSM that named it could not be unit-tested at all — and the 59 golden
307+
suite installs one. An FSM that named it could not be unit-tested at all — and the 63 golden
245308
scenarios are the only thing that proves this port matches the server's behaviour.
246309

247310
The server does the same job with a promise-tail chain. One `Mutex` is the same guarantee.

0 commit comments

Comments
 (0)