feat: span UI audio playback for STT/TTS spans (LAT-720) - #3804
Closed
geclos wants to merge 4 commits into
Closed
Conversation
…LAT-720) Document current capabilities and gaps across UI rendering, OTLP parsers, rollup gates, and framework telemetry. Playable audio is feasible when customers attach uri/blob audio parts, but frameworks today emit text only. Co-authored-by: Gerard <gerard@latitude.so>
Shift product direction from conversation tab to span detail: add VoiceSpanSection with native audio controls, parseVoice OTLP parser for flat text/audio attrs, and transcribe/speech span icons. Co-authored-by: Gerard <gerard@latitude.so>
cursor
Bot
changed the title
Co-authored-by: Gerard <gerard@latitude.so>
Add OTLP transform integration test for STT/TTS audio attrs, ClickHouse roundtrip test for voice message JSON, and VoiceSpanSection component tests that verify audio controls and transcript rendering. Co-authored-by: Gerard <gerard@latitude.so>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds voice span support end-to-end: OTLP ingestion maps STT/TTS audio attributes to GenAI message parts, ClickHouse stores/retrieves them, and the span detail UI renders playable audio controls on the Spans tab (not the Conversation tab).
QA performed
Automated QA added and run locally:
voice.test.ts(6 tests)voice-transform.test.ts— STT input audio URI + output text, TTS input text + output audio URIspan-repository.test.ts—findBySpanIdroundtrip for voice messages with audio JSONvoice-span-section.test.tsx—isVoiceSpan, audio<audio>src, lazy transcript text, no-audio text-only@domain/spans,@app/web@domain/spans checkNot QA'd in this environment: full browser E2E (Docker unavailable in cloud VM). Manual verification steps for local dev:
gen_ai.input.audio.uri/gen_ai.output.audio.uriattrs<audio controls>players, Input/Output sectionsChanges
parseVoiceparser — maps flat voice attrs (gen_ai.*, LiveKit,voice.*) to GenAI messages with audio partsVoiceSpanSection— span detail UI for transcribe/speech operations and spans with audio message partsspan-icon.tsx— MicIcon (transcribe), Volume2Icon (speech)specs/voice-audio-spans.md— instrumentation guide for customers emitting audio URIs/blobsNotes
Linear Issue: LAT-720