Problem or use case
omnigent's premise is that one person's attention is the scarce resource across
many agents. Today that attention is chained to a browser tab.
The measurable failure is stall time: an agent hits an approval gate and
sits dead until someone comes back to the screen. Not seconds, but a coffee, a
standup, a walk to the whiteboard. With five agents that is an annoyance. With
twenty it is the throughput ceiling of the product.
Three moments recur, and none of them are solved by typing faster:
- The stalled gate. An agent asks "run this migration?" while you are away.
The lane is dead for twenty minutes.
- Eyes already busy. You are reading session A's diff when session B blocks.
Ears are the only free channel.
- Reaching a session you are not in. Answering B means navigating away from
A and losing your place.
Evidence the job exists: users already build the missing piece themselves. Around
agent tools there is a standing ecosystem of duct tape: ntfy and Pushover
bridges, Slack relays, macOS say hooks, "agent finished" notification scripts.
Nobody writes duct tape for a problem they do not have.
omnigent is unusually close to this already. WS /v1/dictation/stream ships
a pluggable streaming STT engine (register_engine("sherpa"|"remote"|"fake"),
advertised as dictation_available on GET /v1/info), a 16 kHz AudioWorklet
capture path in web/src/lib/dictation.ts, and a mic button wired to all of it.
What is missing is speech out and any notion of a spoken command.
Worth noting: voice mode was feature #1 in the original scope of #1600 and
was dropped; #2092 later shipped the STT substrate that a voice mode would sit
on. This issue is a proposal for the half that was never built.
Proposed solution
A hands-free voice mode in the web UI, structured so that the risky parts are
optional and separable.
The analogy that shaped it
Not a smart speaker and not a screen reader. An air-traffic controller's
headset over a radar you never stop watching. Three independent reviewers
reached that analogy separately. It demands: callsigns, closed phraseology,
readback before anything dangerous, silence when nothing is happening,
push-to-talk, and the visual UI staying primary for every non-binary decision.
What it does
1. Address a session and act. A callsign binds the target, the rest of the
sentence is the instruction. It lands as an ordinary turn in that session:
“bravo, rebase onto main and run the tests”
The screen never has to leave the session you were reading.
2. Ask the fleet a question. "Who's idle?" is a question, not a dashboard
scan. Twenty sessions answer in one sentence.
3. Be told when something blocks (opt-in). On state change only: a gate
opened, a run failed, a session went idle. Never progress narration; that is
how a voice feature becomes noise you switch off on day two.
4. Answer the gates that are safe to answer by voice, with a closed grammar
and a check token.
Where the line sits
Voice may request. A second modality commits.
Unconfirmed speech recognition is not an authorization primitive. Production
deploys, merges, force-pushes, deletes, secret access, and spending are
not answerable by voice at all. Voice surfaces them, defers them, or denies
them (denial is always safe), and the accept path stays on screen.
Below that line, the reply carries a spoken check token (allow bravo, never
yes) so a podcast, a meeting, or the system's own speaker cannot produce a
valid approval. That is the FAA readback rule and it is the cheapest safety
property available here.
Demo
An interactive mockup built against omnigent's own design tokens
(--brand-accent, --ease-otto, --radius-otto-md) and a recorded walkthrough:
The moment that matters most is voice refusing to approve a production
deploy, staging it for a click instead:
Full 77s walkthrough ·
mockup ·
scenario
Architecture
Four decisions, each chosen to keep the review surface small:
Speech out mirrors the STT registry, not a new stack.
register_speech_engine(...) beside the existing register_engine(...), with
speech_available beside dictation_available. omnigent already depends on
sherpa-onnx for STT and
sherpa-onnx also does TTS (Piper,
Kokoro, VITS) — so speech output adds no new dependency, and the browser's
speechSynthesis covers the zero-install path.
Intent resolution is a pure function. (transcript, fleet snapshot) → Intent | Ambiguous | Refused. No audio, no network, no browser in its signature, so
the whole grammar — including every safety refusal — is unit-testable.
Voice gets no private write path. Every action is an endpoint the click path
already uses (POST /v1/sessions/{id}/messages, the elicitation resolve
endpoint). Nothing is approvable by voice that is not approvable by click, and
voice inherits the same auth, policy and audit trail. There is no "voice API".
Risk classification is server-side and reads the action: target
environment, ref, command class, rather than the card's styling or an
agent-authored schema. A browser deciding what counts as destructive is a boundary in the wrong
place.
Callsigns solve the ASR problem. auth-bug-2 and omni-dev are exactly the
proper nouns speech recognition mangles, and a voice UI gets about one strike.
So callsigns come from a fixed ~20-word phonetic list, auto-assigned and shown
beside the real name (auth-bug-2 · Bravo), fed to sherpa-onnx as hotwords.
Titles stay labels; they are never a routing key.
Suggested phasing
Each phase is independently useful, and the later ones are deletable if
telemetry says nobody uses them:
- Speech-out registry + read-aloud. Small, self-contained, immediately
useful for accessibility on its own.
- Announce on state change (opt-in). This is the kernel — it has none of
ASR's failure modes.
- Closed-grammar reply to gates, flag-gated. Delete in one file if a month
of telemetry shows nobody answers by voice.
- User-initiated addressing and fleet queries.
Alternatives considered
A push notification with an Approve button. This is the strongest
alternative and it should be said plainly: perfect recognition accuracy, works
with the lid closed, works in a café, same audit trail. It is a complement
rather than a cheaper substitute, since it needs a service worker, VAPID keys
and a sender, which is more plumbing than speechSynthesis, not less. If someone
wants to build it, it is the better first slice for the away-from-desk case
specifically.
Realtime speech-to-speech (OpenAI Realtime, Gemini Live). Rejected as the
default: for supervised agent work the explicit transcript is the feature, not
the overhead. It is a reviewable intermediate you can gate on, log and replay. S2S
makes it implicit, has lower tool-call accuracy, is cloud-only and
single-vendor, and costs roughly 10× a chained pipeline for a latency prize of
~200 ms. It fits fine as a pluggable engine later.
Doing nothing. Genuinely defensible, and the counter-argument deserves
stating: voice in developer tools has failed on retention nearly everywhere
except accessibility tooling. Reading beats listening for the actual work
(700 wpm vs 150). If agents interrupt often enough for voice to become habitual,
that may indicate a permissions-design problem instead. The phasing above is
built so that a maintainer can take phase 1–2 and stop, capturing most of the
value and none of the ASR risk.
Notes
I have a design document, a working mockup, and the recorded walkthrough above,
and I am happy to build any part of this — or none of it, if the direction is
wrong. Filing as an issue first rather than a PR precisely because the scope
question ("is the announcer enough?") is the maintainers' call, not mine.
The design was cross-checked with three flagship models at maximum reasoning
effort and adjudicated by a synthesis panel; their dissent is recorded rather
than smoothed over, and it changed the proposal materially. An earlier draft
let voice approve a production deploy behind a spoken confirmation, and that was
cut as safety theatre.
Problem or use case
omnigent's premise is that one person's attention is the scarce resource across
many agents. Today that attention is chained to a browser tab.
The measurable failure is stall time: an agent hits an approval gate and
sits dead until someone comes back to the screen. Not seconds, but a coffee, a
standup, a walk to the whiteboard. With five agents that is an annoyance. With
twenty it is the throughput ceiling of the product.
Three moments recur, and none of them are solved by typing faster:
The lane is dead for twenty minutes.
Ears are the only free channel.
A and losing your place.
Evidence the job exists: users already build the missing piece themselves. Around
agent tools there is a standing ecosystem of duct tape: ntfy and Pushover
bridges, Slack relays, macOS
sayhooks, "agent finished" notification scripts.Nobody writes duct tape for a problem they do not have.
omnigent is unusually close to this already.
WS /v1/dictation/streamshipsa pluggable streaming STT engine (
register_engine("sherpa"|"remote"|"fake"),advertised as
dictation_availableonGET /v1/info), a 16 kHzAudioWorkletcapture path in
web/src/lib/dictation.ts, and a mic button wired to all of it.What is missing is speech out and any notion of a spoken command.
Worth noting: voice mode was feature #1 in the original scope of #1600 and
was dropped; #2092 later shipped the STT substrate that a voice mode would sit
on. This issue is a proposal for the half that was never built.
Proposed solution
A hands-free voice mode in the web UI, structured so that the risky parts are
optional and separable.
The analogy that shaped it
Not a smart speaker and not a screen reader. An air-traffic controller's
headset over a radar you never stop watching. Three independent reviewers
reached that analogy separately. It demands: callsigns, closed phraseology,
readback before anything dangerous, silence when nothing is happening,
push-to-talk, and the visual UI staying primary for every non-binary decision.
What it does
1. Address a session and act. A callsign binds the target, the rest of the
sentence is the instruction. It lands as an ordinary turn in that session:
The screen never has to leave the session you were reading.
2. Ask the fleet a question. "Who's idle?" is a question, not a dashboard
scan. Twenty sessions answer in one sentence.
3. Be told when something blocks (opt-in). On state change only: a gate
opened, a run failed, a session went idle. Never progress narration; that is
how a voice feature becomes noise you switch off on day two.
4. Answer the gates that are safe to answer by voice, with a closed grammar
and a check token.
Where the line sits
Voice may request. A second modality commits.
Unconfirmed speech recognition is not an authorization primitive. Production
deploys, merges, force-pushes, deletes, secret access, and spending are
not answerable by voice at all. Voice surfaces them, defers them, or denies
them (denial is always safe), and the accept path stays on screen.
Below that line, the reply carries a spoken check token (
allow bravo, neveryes) so a podcast, a meeting, or the system's own speaker cannot produce avalid approval. That is the FAA readback rule and it is the cheapest safety
property available here.
Demo
An interactive mockup built against omnigent's own design tokens
(
--brand-accent,--ease-otto,--radius-otto-md) and a recorded walkthrough:The moment that matters most is voice refusing to approve a production
deploy, staging it for a click instead:
Full 77s walkthrough ·
mockup ·
scenario
Architecture
Four decisions, each chosen to keep the review surface small:
Speech out mirrors the STT registry, not a new stack.
register_speech_engine(...)beside the existingregister_engine(...), withspeech_availablebesidedictation_available. omnigent already depends onsherpa-onnx for STT and
sherpa-onnx also does TTS (Piper,
Kokoro, VITS) — so speech output adds no new dependency, and the browser's
speechSynthesiscovers the zero-install path.Intent resolution is a pure function.
(transcript, fleet snapshot) → Intent | Ambiguous | Refused. No audio, no network, no browser in its signature, sothe whole grammar — including every safety refusal — is unit-testable.
Voice gets no private write path. Every action is an endpoint the click path
already uses (
POST /v1/sessions/{id}/messages, the elicitation resolveendpoint). Nothing is approvable by voice that is not approvable by click, and
voice inherits the same auth, policy and audit trail. There is no "voice API".
Risk classification is server-side and reads the action: target
environment, ref, command class, rather than the card's styling or an
agent-authored schema. A browser deciding what counts as destructive is a boundary in the wrong
place.
Callsigns solve the ASR problem.
auth-bug-2andomni-devare exactly theproper nouns speech recognition mangles, and a voice UI gets about one strike.
So callsigns come from a fixed ~20-word phonetic list, auto-assigned and shown
beside the real name (
auth-bug-2 · Bravo), fed to sherpa-onnx as hotwords.Titles stay labels; they are never a routing key.
Suggested phasing
Each phase is independently useful, and the later ones are deletable if
telemetry says nobody uses them:
useful for accessibility on its own.
ASR's failure modes.
of telemetry shows nobody answers by voice.
Alternatives considered
A push notification with an Approve button. This is the strongest
alternative and it should be said plainly: perfect recognition accuracy, works
with the lid closed, works in a café, same audit trail. It is a complement
rather than a cheaper substitute, since it needs a service worker, VAPID keys
and a sender, which is more plumbing than
speechSynthesis, not less. If someonewants to build it, it is the better first slice for the away-from-desk case
specifically.
Realtime speech-to-speech (OpenAI Realtime, Gemini Live). Rejected as the
default: for supervised agent work the explicit transcript is the feature, not
the overhead. It is a reviewable intermediate you can gate on, log and replay. S2S
makes it implicit, has lower tool-call accuracy, is cloud-only and
single-vendor, and costs roughly 10× a chained pipeline for a latency prize of
~200 ms. It fits fine as a pluggable engine later.
Doing nothing. Genuinely defensible, and the counter-argument deserves
stating: voice in developer tools has failed on retention nearly everywhere
except accessibility tooling. Reading beats listening for the actual work
(700 wpm vs 150). If agents interrupt often enough for voice to become habitual,
that may indicate a permissions-design problem instead. The phasing above is
built so that a maintainer can take phase 1–2 and stop, capturing most of the
value and none of the ASR risk.
Notes
I have a design document, a working mockup, and the recorded walkthrough above,
and I am happy to build any part of this — or none of it, if the direction is
wrong. Filing as an issue first rather than a PR precisely because the scope
question ("is the announcer enough?") is the maintainers' call, not mine.
The design was cross-checked with three flagship models at maximum reasoning
effort and adjudicated by a synthesis panel; their dissent is recorded rather
than smoothed over, and it changed the proposal materially. An earlier draft
let voice approve a production deploy behind a spoken confirmation, and that was
cut as safety theatre.