[DRAFT/prototype] Native audio reading experience (#1580)

[openlibrary-bot]

Draft — prototype for discussion, not for merge. Opened by openlibrary-bot on behalf of @mekarpeles' exploration of #1580.

Refs #1580.

What this is

A working local prototype of the audio-first reading mode: load a URL, get cover + table of contents + playback controls (no page-image view), press play, and a real archive.org book is read aloud client-side from its OCR text.

Demo: /BookReaderDemo/demo-audioreader.html?ocaid=theworksofplato01platiala

Verified against theworksofplato01platiala — public, English, 524 leaves, 18 resolvable TOC entries from Open Library.

What works

• Minimal view. Cover, TOC (jumping to chapters), transport, playback rate. The paragraph being read is displayed with per-segment styling so the progressive chunking is visible, not just audible.
• Reads real OCR aloud via speechSynthesis, sourced from BookReaderGetTextWrapper.php — the same endpoint the existing read-aloud plugin uses.
• Progressive chunk rendering exactly as the issue specifies: first 3 words as a "landmark", then the rest of that sentence, then the rest of the paragraph. The tail is split per sentence rather than emitted as one blob, so synthesis units stay small and a seek abandons little work.
• ≤5-paragraph lookahead buffer, hydrated strictly in priority order by a single-worker queue, so background loading cannot saturate the CPU.
• Loading spinner shown exactly when the segment being waited on is not yet synthesized.
• Seek throttling. next/prev stop audio immediately, but a burst of presses coalesces into one buffer rebuild (verified: 6 clicks → 1 rebuild), and synthesis that is no longer the most urgent work is aborted rather than left to delay what the patron is waiting for.
• No page images are ever requested — asserted in a browser test, not just intended.

Shape, and the questions it raises

src/audioreader/ is a standalone bundle (250KB) with its own webpack entry. It never boots BookReader.js, so no plugin registry, no page model, no jQuery, no bergamot. It reuses the read-aloud plugin's PageChunk parser and WebTTSSound (keeping its hard-won Chrome/Firefox/Safari speech workarounds) via a jqueryShim, so the reuse costs nothing.

That is deliberate but not a recommendation: whether audio mode should be a lean standalone bundle or a BookReader mode that disables plugins is exactly the productionization decision this prototype does not settle. Today plugins are wired at the webpack entry level, so "don't load translate" means a different bundle, not a runtime flag.

PocketTTS — synthesis works, browser integration pending

The reference runtime is Python, so this is a port: text conditioner → flow_lm_main per frame → flow_lm_flow integration → mimi decoder, threading all 18 flow and 56 mimi state tensors explicitly. PocketTtsSynthesizer is runtime-agnostic, so the same code runs under onnxruntime-web in a Worker and under Node.

Verified against the real int8 weights (node scripts/verify-pocket-tts.mjs <modelDir>): 5.28s of audio for an 18-word sentence of real Plato OCR — 3.4 words/sec — at RTFx 1.7x on a single wasm thread. Since I cannot listen to it, I compared its acoustics to the human reference clip: spectral centroid 1560Hz vs 1620Hz, 98.9% vs 99.4% of energy below 4kHz, envelope dynamics 27× vs 21×, RMS 0.030 vs 0.029. That profile is speech, not noise. Intelligibility by ear is unverified.

Two pieces of unexpected work:

• The tokenizer. PocketTTS ships a raw sentencepiece ModelProto and there is no JS sentencepiece. SentencePieceUnigram parses the protobuf and implements Viterbi with byte fallback, validated token-for-token against sentencepiece 0.2.2 (68 tests: ASCII, accents, Japanese, emoji, tabs, decimals, abbreviations). Worth knowing if anyone else touches this: the 256 byte pieces carry score 0.0, higher than every real piece's log probability, so they must be kept out of the lattice or the tokenizer spells everything out byte by byte.
• Resampling. The only ungated voice source is a 16kHz clip and the encoder wants 24kHz, so there is a windowed-sinc resampler — naive interpolation colours exactly the timbre being cloned.

⚠️ The 8 named voices are not usable from a browser. They live in the gated kyutai/pocket-tts (fetching alba.safetensors returns 401). The voice is therefore cloned from ungated reference audio through the mimi encoder. Worth deciding deliberately, since "run PocketTTS client-side" implies either cloning from a clip we own or an arrangement with kyutai.

It runs in the browser. Synthesis is in a Web Worker (per-frame ONNX on the main thread would stall the transport controls), the ~146MB bundle is cached in the Cache API so it downloads once rather than per page load, and the UI shows the download rather than sitting silent. build-ort copies onnxruntime's wasm into BookReader/ort/, following the build-bergamot precedent.

The hybrid preview — ?engine=hybrid

The arrangement the issue suggests, and the answer to PocketTTS's one real weakness: at ~RTFx 1.7 on one wasm thread, a paragraph the buffer has not reached costs seconds of silence.

The rule falls out of when each decision is made. At synthesis time, PocketTTS gets a short grace period — once the lookahead is warm it wins easily, because synthesis started paragraphs ago, so no preview is produced at all. If the grace expires, hand back a WebSpeech preview and leave PocketTTS running. At play time, check again: if PocketTTS landed in between, play that instead. Reading straight through is high quality; a cold seek speaks immediately and recovers quality from the next segment. Late PocketTTS audio is still cached for a seek back.

Verified in a browser: {quality: 1, preview: 1, upgraded: 1} — first segment previewed, then PocketTTS took over with its samples reaching the device.

Tests

• 233 jest tests in tests/jest/audioreader/ covering segmentation, paragraph walking, queue ordering/abort/eviction, the player state machine, the tokenizer against real sentencepiece, the resampler, and the generation loop (with stand-in ONNX sessions, so the suite needs no model weights).
• 19 Playwright tests in tests/playwright/ against the live archive.org endpoints, with screenshots. Playwright is new to this repo (the committed e2e suite is testcafe) and is additive.

One environment caveat, stated plainly: Chromium under Playwright on this machine has no audio device, so its AudioContext clock does not advance (0.005s per 2 real seconds) and speechSynthesis never fires end. Audio starting is verifiable — utterance start events with a real voice, and for the PCM path a captured sample count and non-zero peak amplitude — but a sound finishing is not observable in-browser here. Continuous advancement driven by genuine sound completion is covered by the jest suite instead. This also motivated a real fix: PcmAudioOutput now has a duration-based watchdog so a context that stops rendering (backgrounded mobile tab, device removed) cannot strand the reader on one segment, and it counts those completions separately so they are never mistaken for playback.

Not done / out of scope

Everything #1580 asks for is now built: the minimal view, real OCR read aloud, progressive chunk rendering, the ≤5-paragraph buffer, the spinner, seek throttling, PocketTTS client-side in WASM, and the hybrid preview. Deliberately out of scope: the accessibility word-window mode (the issue calls it future work), mobile packaging, cross-browser testing, production polish. SyntheticPcmEngine renders measurable tones, not speech — it exists to verify the pipeline and to exercise the PCM path. Model weights are gitignored and fetched at runtime.

[openlibrary-bot]

Progress update — PocketTTS synthesis now works, and the branch has moved to this repo.

Branch location. audioreader is now pushed directly to internetarchive/bookreader (it was previously only on an openlibrary-bot fork, which is what this PR's head still points at). All five commits are now authored by openlibrary-bot. A replacement draft PR from the in-repo branch is pending a permission gate on my side; if a maintainer prefers, this PR can simply be retargeted by closing it in favour of one opened from internetarchive/bookreader:audioreader — same commits.

PocketTTS is producing speech. The reference runtime is Python, so the flow-matching loop is now ported to JS (src/audioreader/pocket/) and verified against the real int8 weights, not mocks:

bundle english_2026-04: 24000Hz, 12.5 frames/s, latent 32, 18 flow states, 56 mimi states
session signatures: flowLmMain 20/20 | flowLmFlow 4/1 | mimiDecoder 57/57 | mimiEncoder 1/1
voice embeddings: [1, 112, 1024] (cloned from reference audio, resampled 16k -> 24k)
RESULT: 126720 samples, 5.28s audio in 3.0s (RTFx 1.73x), peak 0.2550, rms 0.0303

The session signatures corroborate the port independently: 20 = 2 real + 18 state tensors, 57 = 1 + 56, exactly the bundle.json manifests.

I cannot listen to the output, so I compared its acoustics against the human reference clip: spectral centroid 1560Hz vs 1620Hz, 98.9% vs 99.4% of energy below 4kHz, envelope dynamics 27x vs 21x, RMS 0.030 vs 0.029, and 18 words in 5.28s (3.4 words/sec). That is a speech profile rather than noise. Intelligibility by ear remains unverified.

Two pieces of incidental work that may be of wider interest:

• There is no JS sentencepiece, and PocketTTS ships a raw ModelProto, so SentencePieceUnigram parses the protobuf and implements Viterbi with byte fallback — validated token-for-token against sentencepiece 0.2.2 across ASCII, accents, Japanese, emoji, tabs and decimals. Trap for anyone who touches it: the 256 byte pieces carry score 0.0, higher than every real piece's log probability, so they must be excluded from the lattice or the tokenizer spells everything out byte by byte.
• A windowed-sinc resampler, because the only ungated voice source is a 16kHz clip and the encoder wants 24kHz.

⚠️ Voice licensing question for maintainers. The 8 named PocketTTS voices live in kyutai/pocket-tts, which is gated — fetching alba.safetensors returns 401. They are therefore unusable from a browser, and the weights are not ours to rehost. This prototype clones a voice from ungated reference audio via the mimi encoder instead. If audio mode ships, that is a deliberate decision to make: clone from a clip IA owns, or reach an arrangement with kyutai.

Tests: 216 jest + 14 Playwright, lint clean. The generation loop is covered with stand-in ONNX sessions so the suite needs none of the ~146MB of weights; real-weight verification is scripts/verify-pocket-tts.mjs.

Still to do: Web Worker wrapper, Cache API model storage, ?engine=pocket wiring, and an end-to-end browser test proving non-silent PocketTTS audio in a real browser. Still a draft; not for review.

[openlibrary-bot]

Milestone: PocketTTS now runs client-side in the browser and produces real speech. This closes the loop on the part of #1580 that was genuinely uncertain.

Real audio from the real model, in a real browser

A Playwright test drives the demo with ?engine=pocket, waits for the worker to load the bundle and clone a voice, presses play, and then asserts on the synthesized samples themselves rather than on playback side effects:

PocketTTS synthesized: {
  "key": "12:0#0",
  "text": "THE APOLOGY OF",
  "length": 28800,
  "sampleRate": 24000,
  "peak": 0.1726,
  "rms": 0.0218
}

28,800 samples at 24kHz is 1.2s of audio for the landmark segment of the first paragraph, drawn from this book's own OCR. The assertions are deliberately the ones silence would fail: non-zero peak, non-zero RMS, and RMS well below peak (a constant tone or DC offset would not satisfy that). A second test then confirms those samples reach the audio device with a non-zero peak and a running AudioContext.

The in-browser voice-clone result matches the Node verification exactly — 24kHz, 112 voice frames — which is a useful independent check that the worker path and the direct path agree.

How it is put together

• pocket-tts-worker.js — synthesis runs in a Web Worker. Per-frame ONNX inference on the main thread would stall rendering and make the transport controls unresponsive, which is the opposite of the experience this issue is about.
• modelStore.js — the ~146MB int8 bundle goes into the Cache API keyed by bundle name, so it is downloaded once rather than per page load. Fetches are sequential on purpose: eight parallel downloads of that size compete with the book's own text requests and make progress reporting meaningless. Progress is reported per byte, since "3 of 5 files" tells a patron nothing when one file is 76MB.
• PocketTtsEngine.js — implements the same small engine interface as the WebSpeech path, so the buffering, spinner and seek-cancellation logic already in place needed no changes at all. Cancellation propagates into the worker, because a stale frame loop would otherwise hold the single synthesis slot against work the patron is actually waiting for.
• onnxruntime wasm assets are copied into BookReader/ort/ by a new build-ort npm script, following the same pattern build-bergamot already uses for the translator worker.
• The UI now shows the model download with a progress bar. ~146MB of silence with no explanation reads as a broken page.

Test status

216 jest + 17 Playwright, lint clean. New coverage in this milestone:

• PocketTTS produces non-silent PCM in a real browser (above).
• PocketTTS samples reach the audio device.
• The lookahead holds at exactly 5 paragraphs sampled repeatedly during steady-state playback — not just at startup — and the test proves the cursor really advanced while it was measuring, so it cannot pass by sitting still.

New screenshots: 12-pocket-loading.png (model download), 13-pocket-playing.png.

The PocketTTS browser tests skip rather than fail when the weights are not present locally, so a clean checkout still runs the rest of the suite. They point at a gitignored local copy instead of HuggingFace to keep the run fast; the fetch commands are documented in the spec file.

Honest limits

• I have not listened to the output. The evidence is acoustic: sample counts, peak/RMS structure, and a spectral profile that matches the human reference clip (centroid 1560Hz vs 1620Hz, 98.9% vs 99.4% of energy below 4kHz, envelope dynamics 27x vs 21x). That distinguishes speech from noise; it does not prove the words are the right words.
• Synthesis is slower than real time (~RTFx 1.7x on one wasm thread; threads need cross-origin isolation the demo page does not have). The 5-paragraph lookahead absorbs this once reading is under way, but the first segment after a cold seek still costs a wait. The hybrid WebSpeech-preview path the issue suggests is the answer and is not built yet.
• ⚠️ Voice licensing remains an open decision. The 8 named PocketTTS voices are in the gated kyutai/pocket-tts (401 without a token), so this clones a voice from ungated reference audio via the mimi encoder. If audio mode ships, IA should either use a clip it owns or reach an arrangement with kyutai.

Still a draft — not for review, not for merge.

[openlibrary-bot]

Milestone: the hybrid preview path is built, which was the last substantive piece the issue describes. ?engine=hybrid.

What it does

The issue floats this directly — "a hybrid of PocketTTS and WebSpeechSynthesis (much faster) to provide an instant fallback preview while the higher quality voices are loading" — and it is the answer to PocketTTS's one real weakness in a browser: at ~RTFx 1.7 on a single wasm thread, a paragraph the buffer has not reached yet costs seconds of silence.

The rule falls out of when each decision gets made:

1. At synthesis time, give PocketTTS a short grace period (400ms by default). Once the lookahead buffer is warm it wins easily, because synthesis for that segment started paragraphs ago — so ordinary reading is high quality and no preview is ever produced.
2. If the grace period expires, hand back a WebSpeech preview and leave PocketTTS running.
3. At play time, check again. If PocketTTS landed in between, play that instead.

So reading straight through sounds like PocketTTS; a cold seek speaks immediately and recovers quality from the next segment. Nothing is thrown away — late PocketTTS audio stays in the queue's cache for a seek back.

Verified in a real browser: {quality: 1, preview: 1, upgraded: 1} — the first segment previewed because nothing was synthesized yet, then PocketTTS took over, and its samples reached the device with a non-zero peak.

Two bugs this shook out

Both were mine, both in the hybrid, both would have silently degraded the feature rather than crashing:

• The upgrade never fired. "Has this promise already resolved?" cannot be answered synchronously, and my check raced the quality promise against an immediate one — which the immediate one always wins by a microtask. Every segment would have stayed on the preview voice forever. Fixed by recording the result onto the holder when it settles, rather than re-racing at play time.
• A failed segment was played by the wrong engine. When PocketTTS failed outright I returned a bare fast-engine sound, which was indistinguishable from a quality sound, so playback routed it to PocketTTS. Now it is wrapped too, and play() always knows which engine owns the audio.

An honest note on how the takeover is tested

The preview half tests cleanly with WebSpeech: the utterance starts immediately, with a real voice, speaking the paragraph's landmark segment.

The takeover half cannot be observed with WebSpeech as the preview voice, for the environment reason documented earlier in this PR: a WebSpeech sound never fires end under Playwright here, so playback parks on the first preview and PocketTTS never gets a turn. That is a limitation of the test browser, not of the feature.

Rather than assert something weaker, the preview engine is now selectable (&fast=…), and the takeover test uses the measurable PCM engine as the preview voice so reading actually advances. The hybrid is engine-agnostic by design, so this exercises the real handover logic with real PocketTTS audio on the quality side. Both halves are covered; neither pretends to cover the other.

Tests

233 jest + 19 Playwright, lint clean. 17 new unit tests pin the hybrid's decisions: quality-wins-inside-grace (and that no preview is even synthesized then), fallback outside it, upgrade-before-play, upgrade counted once across replays, permanent fallback on failure, pause/resume reaching whichever engine is live, stop silencing both mid-handover, and rate changes applying to both so a handover does not change speed.

New screenshots: 14-hybrid-preview.png, 15-hybrid-quality.png.

Where this leaves the prototype

Everything issue #1580 asks for is now built and demonstrable: the minimal audio-first view, real OCR read aloud, progressive chunk rendering, the ≤5-paragraph buffer, the spinner, seek throttling, PocketTTS client-side in WASM, and the hybrid preview. Out of scope and untouched: the accessibility word-window mode (the issue calls it future work), mobile packaging, and cross-browser support.

⚠️ Still one decision for maintainers, unchanged: the named PocketTTS voices are gated (401), so this clones a voice from ungated reference audio. If audio mode ships, IA should either use a clip it owns or reach an arrangement with kyutai.

Still a draft — not for review, not for merge.