Skip to content

Deepgram plugin stops sending audio before Deepgram can endpoint, so speech_final never arrives #2242

Description

@CampbellMBXJ

Describe the bug

AudioEnergyFilter gates the frames the Deepgram plugin sends to the websocket. Once the 1 second cooldown elapses with no audio above its RMS threshold, the plugin stops sending audio entirely (KeepAlive still goes out, so the socket looks healthy).

The problem is that Deepgram's endpointing can only fire by observing silence in the audio stream. If we stop sending before it has, the utterance is never endpointed. No final transcript arrives, and the stream stays stuck until the caller speaks again, at which point the pending final is flushed.

That's bad on its own, but it also stalls the agent. runEOUDetection bails when there's no accumulated transcript:

// agents/src/voice/audio_recognition.ts
if (this.stt && !this.audioTranscript && this.turnDetectionMode !== 'manual') {
  this.logger.debug('skipping EOU detection');

The Python plugin doesn't have this problem, it sends every frame unconditionally:

for frame in frames:
    self._audio_duration_collector.push(frame.duration)
    await ws.send_bytes(frame.data.tobytes())

In agents-js only the deepgram and sarvam plugins use AudioEnergyFilter, and the
cooldown isn't configurable from STTOptions.

Relevant log output

Instrumented the plugin to log when the energy filter starts and stops forwarding frames.

A healthy turn, Deepgram finalises before the audio is cut:

20.25  interim transcript   "make an appointment"
20.87  speech_final received
21.11  energy filter stops forwarding audio

A stalled turn from the same call, audio cut 0.20s after the last interim:

31.38  interim transcript   "It just says"
31.58  energy filter stops forwarding audio
       ... nothing from Deepgram for the entire gap ...
42.35  energy filter resumes forwarding (10.8s with no audio sent)
42.54  interim transcript   "It just says,"

Across that call Deepgram needed 0.5-1.2s after the last interim to send speech_final. Every
turn where the filter cut before that stalled; every turn where it cut after was fine.

Describe your environment

OS: Amazon Linux in ECS
Node: 24.14.0
@livekit/agents: 1.4.6
@livekit/agents-plugin-deepgram: 1.4.6
@livekit/agents-plugin-silero: 1.4.6

Have validated bug still exists in 1.6.2 as well.

Minimal reproducible example

The easiest way to trigger it is bump up the endpointing to a large number,
To trigger it you want audio quiet enough to fall under the filter's rms > 0.004 threshold near the end of an utterance, which is easy with phone audio and someone trailing off, and hard with a close mic.

Additional information

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions