Skip to content

Commit 3dd9557

Browse files
jamie-simularclaude
andcommitted
feat(voice): ask before leaving work behind, and stop cutting off the sign-off
Three things the device found, all in the same corner of teardown. **A machine switch discarded outstanding work, silently.** `applyMachineSwitch` builds a fresh VoiceSession, so the FSM goes with the old one: the queue, the in-flight turn and any pending approval. A task Sai had promised out loud disappeared with nothing said — the hazard VOICE_FSM §7 records for a dropped call, reachable by an ordinary voice command. `close()` only discards the stream, so the work kept running on the machine being left and its result reached nobody. So the switch now asks first, and so does the hang-up, through one shared LeavingWorkPolicy — they lose the same work in the same way, and the ask should not read differently depending on which one you said. It names what is outstanding, says it keeps running where it started, and points at the Sai app for the result. It never stops anything: aborting work nobody asked to cancel would be its own failure. One-shot, like applyInterrupt's scope question — someone who says "switch to my laptop" twice means it. Running and queued are named separately, the same rule interruptScopeQuestion follows, because reading them as one list describes a queued task as underway. From the picker there is no question to ask, so it switches and then says what was left behind. **The idle sign-off was cut mid-sentence.** Teardown waited a flat 1.8s; "It's been quiet for a bit, so I'll hang up to save battery. Tap to start again." runs about four. It now waits for the line to start, for the play queue to drain, and for a short tail — capped at 12s, because a teardown that hangs on a sign-off that never came leaves the mic open and has not actually ended anything. Both endCall-with-reason paths get it; the goodbye window is left alone on purpose, since its 1.8s doubles as the barge-in cancel window that check 10d is written against. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 961b8f1 commit 3dd9557

5 files changed

Lines changed: 353 additions & 2 deletions

File tree

docs/ON_DEVICE_CHECK.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -273,6 +273,14 @@ Mid-call, with more than one machine on the account:
273273
your ear is `RESELECT_NUDGE`'s failure in a new place. Switching from the **phone picker** instead
274274
must be silent — a button press carries no tool response, so the correction goes in as a nudge, and
275275
Sai narrating what you just pressed is the failure.
276+
- **Switch with a task still running** and Sai must **ask first** — naming the task, saying it keeps
277+
going on the machine you are leaving and that this call will not hear the result, and offering to
278+
stop it. Answer either way and the switch then goes through; the question is one-shot, like
279+
`interrupt`'s scope question. This is `LeavingWorkPolicy`, shared with the hang-up in check 10a, and
280+
it exists because `applyMachineSwitch` builds a fresh `VoiceSession`: before it, a switch silently
281+
discarded the queue, the running turn and any pending approval — work Sai had promised out loud. From
282+
the **picker** there is no question to ask (a tap is not a conversation), so it switches and then says
283+
what was left behind.
276284
- **Switch onto a hibernated machine** and it should wake, announced: "Switched to ⟨label⟩", then
277285
`MACHINE_WAKING` verbatim, then `MACHINE_AWAKE` on its own a minute or so later — all before you ask
278286
for anything. Both switch paths go through `applyMachineSwitch`, so the picker behaves the same.

meta-android-app/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/saispike/AudioIo.kt

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -281,6 +281,21 @@ class AudioIo(
281281
}
282282

283283
/** Barge-in: drop everything queued for playback so the model goes quiet immediately. */
284+
/**
285+
* Is there model audio still waiting to be written to the track?
286+
*
287+
* For the teardown paths, which have to let a spoken sign-off finish before cutting the audio. They
288+
* used to wait a flat 1.8 s, which is less than the lines they were waiting for: the idle-timeout
289+
* reason ("It's been quiet for a bit, so I'll hang up to save battery. Tap to start again.") ran
290+
* past it and was cut mid-sentence on device.
291+
*
292+
* Only the QUEUE — what the track itself still holds is a fraction of a second and is covered by the
293+
* caller's tail wait. Exact accounting there would mean polling `getPlaybackHeadPosition` against a
294+
* moving write cursor, which is a lot of arithmetic to shave 200 ms off a hang-up.
295+
*/
296+
val playbackPending: Boolean
297+
get() = playQueue.isNotEmpty()
298+
284299
fun flushPlayback() {
285300
playQueue.clear() // drop everything not yet written, not just what the track already holds
286301
track?.let {

meta-android-app/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/saispike/CallService.kt

Lines changed: 105 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -117,6 +117,15 @@ class CallService : Service() {
117117

118118
/** The in-flight "is it awake yet" watcher, if any. A machine switch supersedes the previous one. */
119119
private var wakeWatch: Job? = null
120+
121+
/**
122+
* The user has been asked once about work they are leaving behind — see [LeavingWorkPolicy].
123+
*
124+
* Shared by the hang-up and the machine switch on purpose: they are the same question about the same
125+
* work, and asking it twice in a row ("shall I stop that first?" … "shall I stop that first?") is
126+
* the trap the one-shot exists to avoid. Reset per call, like [hangupGuardUsed].
127+
*/
128+
@Volatile private var leavingWorkAsked = false
120129
// Nudges withheld while muted (they'd be spoken into the void and lost). Replayed on unmute.
121130
private val heldNudges = HeldNudgeQueue(MAX_HELD_NUDGES)
122131
// Keepalive throttle: the last time we told the server a human is still present (see maybeKeepalive).
@@ -234,6 +243,7 @@ class CallService : Service() {
234243
lastUserText = ""
235244
lastSaiText = ""
236245
hangupGuardUsed = false
246+
leavingWorkAsked = false
237247
lastStepFailureNudgeAt = 0L
238248
callActive = true
239249
audioPaused = false
@@ -750,6 +760,17 @@ class CallService : Service() {
750760
"last sai: \"${excerpt(lastSaiText)}\"" +
751761
(if (spokeThisTurn) " | Sai spoke in this turn" else " | Sai said nothing in this turn"),
752762
)
763+
// Work first, farewell second. HangupPolicy answers "did they mean me?"; this answers "do they
764+
// know what they are leaving?" — and an unanswered task outlives the call either way, so the
765+
// question has to come before the goodbye rather than after it.
766+
val leaving = leavingWork(Leaving.CALL)
767+
if (leaving is LeavingWorkAction.Ask) {
768+
leavingWorkAsked = true
769+
log("⏻ endCall held — work outstanding; asking before leaving it behind")
770+
live?.injectNudge("endcall-work-outstanding", leaving.nudge)
771+
status("live — talk (a hang-up was held back)")
772+
return
773+
}
753774
when (val action =
754775
HangupPolicy.decide(
755776
spokeThisTurn = spokeThisTurn,
@@ -847,7 +868,7 @@ class CallService : Service() {
847868
}
848869
notifyReason(reason)
849870
scope.launch {
850-
if (canSpeak) delay(1_800) // let the spoken reason land before cutting audio
871+
if (canSpeak) awaitSignOff()
851872
stopAll()
852873
status(reason) // stopAll resets status to "Idle" — keep the reason visible on the ended call
853874
}
@@ -1001,7 +1022,7 @@ class CallService : Service() {
10011022
)
10021023
}
10031024
scope.launch {
1004-
if (canSpeak) delay(1_800) // let the spoken sign-off land before cutting audio
1025+
if (canSpeak) awaitSignOff() // the line runs past any fixed delay worth picking
10051026
stopAll()
10061027
status(line) // keep the reason visible after the call ends
10071028
}
@@ -1016,6 +1037,16 @@ class CallService : Service() {
10161037
private fun switchMachine(name: String): String {
10171038
val decision = MachineSwitcher.resolve(name, machines, currentMachineId)
10181039
if (decision is MachineSwitch.SwitchTo) {
1040+
// Asked BEFORE the switch, because the switch is what destroys the answer: applyMachineSwitch
1041+
// builds a fresh VoiceSession, so the queue, the in-flight turn and any pending approval go with
1042+
// the old one. Returned as the TOOL REPLY rather than spoken, so the model asks in its own words
1043+
// and the switch simply has not happened yet.
1044+
val leaving = leavingWork(Leaving.MACHINE)
1045+
if (leaving is LeavingWorkAction.Ask) {
1046+
leavingWorkAsked = true
1047+
log("↺ switchMachine held — work outstanding; asking before leaving it behind")
1048+
return leaving.nudge
1049+
}
10191050
applyMachineSwitch(decision.machine, notifyModel = false) // the reply carries the context update
10201051
}
10211052
return when (decision) {
@@ -1034,7 +1065,16 @@ class CallService : Service() {
10341065
return
10351066
}
10361067
if (match.machineId == currentMachineId) return
1068+
// A button press is not a question, so this does not hold — the user did the thing deliberately and
1069+
// a modal-by-voice over a tap would be worse than the loss. But it must not be SILENT: the same
1070+
// work is being left behind, so say what it was.
1071+
val leftBehind = leavingWork(Leaving.MACHINE)
10371072
applyMachineSwitch(match, notifyModel = true)
1073+
if (leftBehind is LeavingWorkAction.Ask) {
1074+
leavingWorkAsked = true
1075+
log("↺ picker switch left work behind — telling the user rather than asking")
1076+
live?.injectNudge("left-work-behind", leftBehind.nudge)
1077+
}
10381078
}
10391079

10401080
/**
@@ -1152,6 +1192,61 @@ class CallService : Service() {
11521192
}
11531193
}
11541194

1195+
/**
1196+
* Is there outstanding work, and has the user been told? See [LeavingWorkPolicy].
1197+
*
1198+
* Reads the FSM rather than tracking a parallel copy: the queue and the in-flight turn already live
1199+
* there, and a second count is a second thing to get wrong.
1200+
*/
1201+
private fun leavingWork(leaving: Leaving): LeavingWorkAction {
1202+
val state = concierge?.state() ?: return LeavingWorkAction.Proceed
1203+
return LeavingWorkPolicy.decide(
1204+
state = state,
1205+
leaving = leaving,
1206+
alreadyAsked = leavingWorkAsked,
1207+
muted = saiMuted,
1208+
)
1209+
}
1210+
1211+
/**
1212+
* Wait for a spoken sign-off to actually finish before tearing the audio down.
1213+
*
1214+
* Replaces a flat 1.8 s, which was shorter than the lines it was waiting for — the idle-timeout
1215+
* reason runs about four seconds and was cut mid-sentence on device. Waits for the line to START
1216+
* (the model has to generate it first), then for the play queue to drain, then a short tail for what
1217+
* the AudioTrack still holds.
1218+
*
1219+
* Every wait is bounded. A teardown that hangs because a sign-off never came is worse than one that
1220+
* clips a word: the mic stays open and the guard that fired has not actually ended anything.
1221+
*/
1222+
private suspend fun awaitSignOff() {
1223+
val io = audioIo ?: return
1224+
// 1. Wait for it to begin. If nothing is ever queued the model produced no audio at all — muted,
1225+
// dropped socket, a refusal — and there is nothing to wait out.
1226+
val startBy = SystemClock.elapsedRealtime() + SIGNOFF_START_MS
1227+
while (!io.playbackPending && SystemClock.elapsedRealtime() < startBy) delay(50)
1228+
if (!io.playbackPending) {
1229+
log("sign-off: nothing was spoken — ending now")
1230+
return
1231+
}
1232+
// 2. Wait for it to finish, capped. `playbackPending` goes false between chunks as well as at the
1233+
// end, so require it to stay quiet for a beat rather than trusting one observation.
1234+
val endBy = SystemClock.elapsedRealtime() + SIGNOFF_MAX_MS
1235+
var quietSince = 0L
1236+
while (SystemClock.elapsedRealtime() < endBy) {
1237+
delay(50)
1238+
if (io.playbackPending) {
1239+
quietSince = 0L
1240+
continue
1241+
}
1242+
if (quietSince == 0L) quietSince = SystemClock.elapsedRealtime()
1243+
if (SystemClock.elapsedRealtime() - quietSince >= SIGNOFF_QUIET_MS) break
1244+
}
1245+
if (SystemClock.elapsedRealtime() >= endBy) log("sign-off: capped at ${SIGNOFF_MAX_MS / 1000}s")
1246+
// 3. The track's own buffer, which `playbackPending` cannot see.
1247+
delay(SIGNOFF_TAIL_MS)
1248+
}
1249+
11551250
/**
11561251
* Act on a [WakePolicy] decision: speak the line verbatim, or log why not.
11571252
*
@@ -1547,6 +1642,14 @@ class CallService : Service() {
15471642
private const val WAKE_WATCH_MS = 3 * 60_000L
15481643
/** Poll interval while waiting for a wake. Cheap (`GET /machines`), and nothing pushes this. */
15491644
private const val WAKE_POLL_MS = 10_000L
1645+
/** How long to wait for a sign-off to start before concluding none is coming. */
1646+
private const val SIGNOFF_START_MS = 2_500L
1647+
/** Hard cap on waiting for a sign-off to finish. The mic is still open until it expires. */
1648+
private const val SIGNOFF_MAX_MS = 12_000L
1649+
/** How long the play queue must stay empty to count as finished rather than between chunks. */
1650+
private const val SIGNOFF_QUIET_MS = 350L
1651+
/** For what the AudioTrack still holds after the queue drains. */
1652+
private const val SIGNOFF_TAIL_MS = 400L
15501653
private const val NOTIF_ID = 42
15511654
private const val REASON_NOTIF_ID = 43
15521655
/** Default "ask before delivering an update" threshold; overridable via StartParams (app setting). */
Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
/*
2+
* sai-fi — voice concierge (leaving work behind).
3+
*/
4+
5+
// Two ways to walk away from work in progress — hanging up, and changing machines — and one rule for
6+
// both: ask first, and say where the results will be instead.
7+
//
8+
// A pure decision, so it can be tested without a device. The reason it is shared rather than written
9+
// twice: the two paths lose work in exactly the same way, and the one that had no ask at all lost it
10+
// silently. `applyMachineSwitch` builds a fresh VoiceSession, so the FSM — queue, in-flight turn,
11+
// pending approval — is replaced wholesale. Work the user was PROMISED OUT LOUD disappears with
12+
// nothing said, which is the hazard docs/VOICE_FSM.md §7 records for a dropped call, reachable by an
13+
// ordinary voice command.
14+
//
15+
// Note what this does NOT do: it never stops anything. A task left running keeps running on the
16+
// machine it started on, and the honest thing is to say so and point at the app, not to abort work
17+
// nobody asked to cancel.
18+
19+
package com.meta.wearable.dat.externalsampleapps.cameraaccess.saispike
20+
21+
import com.meta.wearable.dat.externalsampleapps.cameraaccess.saispike.fsm.ConciergeState
22+
import com.meta.wearable.dat.externalsampleapps.cameraaccess.saispike.fsm.hasOutstandingWork
23+
import com.meta.wearable.dat.externalsampleapps.cameraaccess.saispike.fsm.readBackList
24+
25+
/** Where the user is going, which is all that differs between the two questions. */
26+
enum class Leaving {
27+
/** The call is ending. Results have nowhere to arrive. */
28+
CALL,
29+
/** The machine is changing. The old machine keeps working; this call stops hearing about it. */
30+
MACHINE,
31+
}
32+
33+
/** What [LeavingWorkPolicy.decide] concluded. */
34+
sealed interface LeavingWorkAction {
35+
/** Do not go yet. [nudge] is model-facing: it carries the facts and asks for a decision. */
36+
data class Ask(val nudge: String) : LeavingWorkAction
37+
38+
/** Nothing outstanding, or the user has already been asked. Go. */
39+
data object Proceed : LeavingWorkAction
40+
}
41+
42+
object LeavingWorkPolicy {
43+
44+
/**
45+
* Ask before leaving work behind — once.
46+
*
47+
* One-shot for the same reason `applyInterrupt`'s scope question is: a user who says "hang up" or
48+
* "switch to my laptop" twice means it, and a question that cannot be got past is a trap. The caller
49+
* owns the flag so the two paths do not consume each other's ask.
50+
*
51+
* @param muted nothing asked can be heard, so there is no question to put — the caller decides what
52+
* to do with that, because the right answer differs: a hang-up should proceed silently, and a
53+
* switch should proceed and be reported on unmute.
54+
*/
55+
fun decide(
56+
state: ConciergeState,
57+
leaving: Leaving,
58+
alreadyAsked: Boolean,
59+
muted: Boolean,
60+
): LeavingWorkAction {
61+
if (alreadyAsked || muted) return LeavingWorkAction.Proceed
62+
if (!state.hasOutstandingWork()) return LeavingWorkAction.Proceed
63+
return LeavingWorkAction.Ask(nudge(state, leaving))
64+
}
65+
66+
/**
67+
* The facts, and what to do with them. Model-facing, like `HangupPolicy.UNCONFIRMED_NUDGE` — the
68+
* content varies with what is outstanding, so the phrasing is the model's and only the facts are
69+
* ours.
70+
*
71+
* Running and queued are named SEPARATELY, the same rule `interruptScopeQuestion` follows: one is
72+
* work in progress the user may not want to lose, the other has not happened at all, and reading
73+
* them as one list describes a queued task as underway.
74+
*/
75+
private fun nudge(state: ConciergeState, leaving: Leaving): String {
76+
val clauses = mutableListOf<String>()
77+
if (state.inFlight.isNotEmpty()) {
78+
clauses += "still working on ${readBackList(state.inFlight)}"
79+
}
80+
if (state.queue.isNotEmpty()) {
81+
clauses += "${readBackList(state.queue.map { it.text })} hasn't started yet"
82+
}
83+
if (state.pendingApprovalId != null) {
84+
clauses += "there's a request waiting on their answer"
85+
}
86+
87+
val going =
88+
when (leaving) {
89+
Leaving.CALL -> "hanging up"
90+
Leaving.MACHINE -> "moving to another machine"
91+
}
92+
// The part that makes the answer actionable, and the part that was missing: a task left running is
93+
// not lost, it is just somewhere else. Without this the choice reads as "abandon it or stay", and
94+
// the user picks staying for work they could simply have read later.
95+
val whereItGoes =
96+
when (leaving) {
97+
Leaving.CALL ->
98+
"Anything left running keeps going on their machine — they just won't hear the result " +
99+
"here, and can pick it up in the Sai app."
100+
Leaving.MACHINE ->
101+
"Anything left running keeps going on the machine they're leaving — this call won't " +
102+
"hear the result once you move, and they can pick it up in the Sai app."
103+
}
104+
105+
return "[system] NOTHING has happened yet — you have not $going, and you must not until they " +
106+
"answer. They are ${clauses.joinToString(", and ")}. $whereItGoes " +
107+
"Tell them what's outstanding in ONE short line, and ask whether to stop it first or leave " +
108+
"it running. Then do what they say: to stop it, call interrupt; to leave it, ask again and " +
109+
"it will go through. Do not describe any of it as finished."
110+
}
111+
}

0 commit comments

Comments
 (0)