You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feat(voice): brevity becomes a rule with a budget, and something that grades it (#12)
* fix: the four defects the live tier turned up, and the gap that hid them
**A summary that announces a result is not one that carries it.** The
`complete` nudge guarded the case where the summary is EMPTY and nothing
covered the case where it is present and says only that the work happened.
Live, the agent closed a downloads listing with "Done — that's the full
listing, and I've released the machine" — affirmative, no listing — and
"briefly tell the user the result" was obeyed against data holding no
result: Sai reported the folder empty. It was not. The wording now says
what to do when the answer is not in front of it, and names getSaiStatus as
the way to look rather than leaving a gap to fill.
**A held nudge may already have been delivered.** The same completion then
went out twice, because it landed while Sai was answering "what's going on
with all that?" — a turn that had already fetched and reported that result —
and flushed afterwards regardless. Nudges released by the flush now carry a
preamble saying the turn just ended may have covered this already.
Deliberately NOT solved by collapsing same-kind completions: two
completions in one turn are usually two different tasks, and dropping the
older one loses a result outright.
**An unmapped `data-*` frame now names its fields.** Envelope markers
(`text-start`, `reasoning-end`) are right to drop silently; a custom data
part is the server handing over state, and every other one has a handler.
This is how `data-session` — arriving once per turn, unhandled — turned out
to carry `sessionId`. Field names only, never values: this log is mirrored
to a projector. The claim in HttpAgentBridge that no session id comes back
was simply false, and is corrected rather than removed, because keeping no
session identity is still the right call.
**The agent double could not represent an abort.** `ScriptedAgent.post`
recorded the call and let the aborted task's beats land anyway, so an
aborted task finished normally and was reported as done — the exact lie the
interrupt path exists to prevent, invisible to every test. `abortRunning`
had sat there unused, which is why every assertion about abort in the whole
suite was a negative one.
Tests: AbortConversationTest drives the positive stop path off-device for
the first time. LiveQueueTest is the only place a second ask is admitted
behind a task that is genuinely still running — real latency, not a driven
clock — and the only place `abort` and `new-session` reach a real endpoint;
both passed against staging. A tenth on-device check covers stopping work,
which the by-ear gate never had.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(voice): brevity becomes a rule with a budget, and something that grades it
Brevity was the one headline property of a voice-only interface that the
prompt asserted and nothing measured. It lived as an adjective clause at
block 2 ("one or two sentences"), repeated with no added constraint at
block 4 ("extra short"), competing with 36KB of correctness rules that
mostly push the other way: name the specific action, echo the detail you
heard, account for each outstanding task separately, give the reversal
beat before the result. Length pressure here is structural, and the
prompt never said which side yields.
**A budget, not an adjective.** The new LENGTH block sits after the
hands-free context that motivates it and gives a countable default — one
sentence, under about fifteen words, roughly six seconds — because a
sentence can be forty words and the ear counts time, not punctuation. It
names what to cut, which nothing did before: openers, restating the
request, announcing what it is about to do, unsolicited offers, and a
closing "anything else?" — the call stays open, so it never has to ask.
A worked too-long/right pair carries more than the adjectives did.
**And what it yields to.** LENGTH YIELDS TO ACCURACY, AND TO NOTHING
ELSE. Without that paragraph a tightened brevity rule quietly erodes the
honesty rules the prompt is built around, so the four cases that
genuinely need words are named: a verbatim read-back, two tasks
accounted for separately, presenting a choice's options, why a capture
failed. Blocks 2 and 4 now defer to it instead of stating weaker
versions of it.
**`no-filler` is what makes it hold.** Thirty-eight behaviours in this
prompt hold because a rule grades them; this one did not, and the rubric
had no rule about length or filler at all. The first run proved the
point in the other direction: the rule as first written flagged "I'm
currently checking your unread emails and Slack messages, and after that
I'll book your table" — the exact line the LENGTH block exists to
protect, and a line anyone would be happy to hear on the glasses. The
rule keyed on repeating the request when what matters is whether the
words are an ANSWER. It now says naming a task inside an answer about
that task is content, and carries an operational test for the judge:
flag a line only if words could be deleted with nothing the user asked
for lost. Both rows pass.
Transcripts: a plain calendar check, where nothing needs elaborating and
filler is all there is to add; and `no-filler` added to the existing
"one running, one waiting" row, pointing the other way — that reply
legitimately needs two clauses, so it is where a brevity rule would do
its damage.
Evals (gemini-3.1-flash-lite-preview, judge gemini-3.5-flash-lite):
transcript tier 54/56 effect choice, 74/76 judged, 1 ungraded on a 503;
loop tier 6/6 structural, 8/9 judged. Both `no-filler` rows pass. The
remaining flags are the documented lite-tier ones (`queued-not-underway`
twice, `no-fabricated-timing` once). The two attachLatestImage misses
are NOT this change: A/B'd two runs per prompt on those transcripts and
the pre-change prompt fails the same row the same way.
Counts that had drifted with the catalogue: 41 -> 42 blocks, 31 -> 32
rules, 32 -> 33 transcripts in README, DIRECTORY, SAI_GLASSES_APP and
LoopEvalTest's header.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: docs/DIRECTORY.md
+8-5Lines changed: 8 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -161,7 +161,7 @@ model's tool calls go into `applyEffects`, and the WebSocket path it replaced is
161
161
162
162
| Path | What it is |
163
163
| --- | --- |
164
-
|`voice-profile.json`| The system prompt (41 blocks), the 17 tool declarations, the model and the voice. **Generated** from the server's source before it was deleted — the wording is load-bearing, so it was never retyped. It is on the unit-test classpath too (`sourceSets` in `build.gradle.kts`), so `VoiceProfileTest` and `LiveBrain` grade the file the app loads. There is no vendored second copy: the one that existed drifted, and is gone. |
164
+
|`voice-profile.json`| The system prompt (42 blocks), the 17 tool declarations, the model and the voice. **Generated** from the server's source before it was deleted — the wording is load-bearing, so it was never retyped. It is on the unit-test classpath too (`sourceSets` in `build.gradle.kts`), so `VoiceProfileTest` and `LiveBrain` grade the file the app loads. There is no vendored second copy: the one that existed drifted, and is gone. |
165
165
166
166
### Resources — `app/src/main/res/`
167
167
@@ -174,7 +174,7 @@ model's tool calls go into `applyEffects`, and the WebSocket path it replaced is
174
174
175
175
### Tests — `app/src/test/java/…/saispike/`
176
176
177
-
321 JVM tests, no device or emulator needed — everything below runs on `./gradlew
177
+
327 JVM tests, no device or emulator needed — everything below runs on `./gradlew
178
178
:app:testDebugUnitTest` except the tiers that cost money and the golden generator, each gated on an
179
179
environment variable and skipping itself otherwise. Five kinds:
180
180
@@ -195,15 +195,18 @@ environment variable and skipping itself otherwise. Five kinds:
195
195
-**`conversation/`** — the closed loop, with everything real except the brain and the agent: a fake
196
196
brain's tool calls go through the real `LiveTurnGate`, `Concierge` and `HttpAgentBridge` to a
197
197
`ScriptedAgent` that implements `VoiceTransport`, the seam *under* the bridge — so a wire bug has
198
-
nowhere to hide. `BargeInConversationTest` and `QueueConversationTest` cover the two hardest paths,
198
+
nowhere to hide. `BargeInConversationTest` and `QueueConversationTest` cover the two hardest paths, `AbortConversationTest` the
199
+
one that stops work (the positive abort path, which every other assertion in the suite only ever
200
+
proved *didn't* happen),
199
201
`LongConversationTest` the state that only accumulates, and `TimingMatrixTest` replays one
200
202
conversation at seven speeds asserting invariants rather than orderings, because every barge-in ⇄
201
203
queue bug on record is a race. `PresenterPublisher` can mirror a harness run to the dashboard.
202
204
-**The paid tiers** — off by default, each behind its own switch: `LiveAgentTest` /
203
205
`SummaryFixLiveTest` (`SAI_LIVE_AGENT=1`) drive a real cloud-api to catch **contract drift** and
204
-
nothing else; `eval/LoopEvalTest` (`SAI_CONVERSATION_EVAL=1`) runs the real model through the real
206
+
nothing else, joined by `LiveQueueTest`, which is the only place the queue is admitted behind a task
207
+
that is genuinely still running and the only place `abort` / `new-session` reach a real endpoint; `eval/LoopEvalTest` (`SAI_CONVERSATION_EVAL=1`) runs the real model through the real
205
208
FSM and grades the transcript against `eval/rubric.json`; `eval/TranscriptEvalTest`
206
-
(`SAI_TRANSCRIPT_EVAL=1`) runs it over the 32 fixed transcripts in `eval/Transcripts.kt` with no FSM,
209
+
(`SAI_TRANSCRIPT_EVAL=1`) runs it over the 33 fixed transcripts in `eval/Transcripts.kt` with no FSM,
207
210
grading phrasing by judge and effect choice deterministically; `DemoFlowTest` (`SAI_DEMO=1`) drives a
208
211
real model and a real agent end to end, paced for the presenter so a demo can be rehearsed without
0 commit comments