gemini live function calling options : Addition of SILENT, WHEN_IDLE, and INTERRUPT modes + other improvements

[remisharrock]

pipecat version

No response

Python version

No response

Operating System

No response

Question

I've received this email from Google today and was wondering if pipecat is ready for the Addition of SILENT, WHEN_IDLE, and INTERRUPT modes ?

You’re receiving this communication because you’re a user of Gemini 2.5 Live API Native Audio (General Availability) model ID: gemini-live-2.5-flash-native-audio.

We’re implementing a scheduled product update on April 28, 2026, to add new product features and improvements.

We’ve provided additional information below to help you understand the details of this update.

What you need to know

New features:

Function call modes: Addition of SILENT, WHEN_IDLE, and INTERRUPT modes for more granular control over how and when the AI triggers functions.
Vertex AI Studio integration: The ability to define and test function calls directly in Vertex AI Studio.
Improved transcription quality: New support for a language-specific flag to enhance transcription accuracy.
Product improvements:

Function calling improvements:
“No Response” issues: Minimized instances where the model remains silent during long-running tool calls.
“Trigger Repeat”issues: Minimized the issue where the model sends the same function call multiple times before receiving a response.
Note: We still recommend client-side filtering for duplicate function calls as a best practice.
Transcription quality: Resolved the transcription spacing issues for a higher quality output.

What I've tried

No response

Context

No response

[remisharrock]

official documentation is

https://ai.google.dev/gemini-api/docs/live-api/tools#async-function-calling

especially that part:

NON-BLOCKING ensures the function runs asynchronously while you can continue interacting with the model.

Then you need to tell the model how to behave when it receives the FunctionResponse using the scheduling parameter. It can either:

Interrupt what it's doing and tell you about the response it got right away (scheduling="INTERRUPT"),
Wait until it's finished with what it's currently doing (scheduling="WHEN_IDLE"),
Or do nothing and use that knowledge later on in the discussion (scheduling="SILENT")

# for a non-blocking function definition, apply scheduling in the function response:
  function_response = types.FunctionResponse(
      id=fc.id,
      name=fc.name,
      response={
          "result": "ok",
          "scheduling": "INTERRUPT" # Can also be WHEN_IDLE or SILENT
      }
  )

[elliottventures]

Adding a data point from a reproduction today (2026-04-21) in case it's useful for prioritization.

Env: Pipecat 1.0.0 (pypi), google-genai 1.73.1, Python 3.14.4, macOS arm64, AI Studio API key, Daily transport direct mode.

Config: Swapped the model from gemini-3.1-flash-live-preview to gemini-2.5-flash-native-audio-latest on an otherwise-working Pipecat pipeline. No other changes. Function handlers are sync, <10ms each, no behavior/scheduling (Pipecat doesn't surface those fields).

Result: First tool call (an advance_step equivalent) fired cleanly end-to-end in ~2s. After that, the model went silent for 68 seconds — no further user transcription frames, no VAD events, no interruptions, no tool calls — until we disconnected. This matches the "No Response" bug described in the email this issue forwards ("Minimized instances where the model remains silent during long-running tool calls") — in our case the tool was already <10ms, so it appears the silence trigger can happen without an actually-long tool call as long as the sync protocol is in use.

Reverted to gemini-3.1-flash-live-preview for stability. Deferring any native-audio migration until either the 2026-04-28 server-side fix lands and/or Pipecat surfaces the behavior: NON_BLOCKING + scheduling: INTERRUPT|WHEN_IDLE|SILENT wire fields.

Happy to run additional tests or provide more log detail if helpful.

[elliottventures]

Following up on my 2026-04-21 comment. I chased the "No Response" symptom further and found two Pipecat-side bugs that reproduce it on gemini-3.1-flash-live-preview independently of any async/NON_BLOCKING scenario:

1. fix: send realtime input after tool_result so Gemini 3.x runs inference #4366 — GeminiLiveLLMService._tool_result missing the Gemini 3.x realtime-input kick. The in-file comment at _create_initial_response and _create_single_response already documents that "Gemini 3.x wants turn_complete=True, but also won't run inference without a realtime input." The tool-result path was the remaining send-site missing this workaround. 3-line fix mirroring the existing two sites.

2. fix: re-trigger deferred context frame push on UserStoppedSpeakingFrame #4367 — LLMAssistantAggregator silently drops context push on a user-speaking race. The bot-speaking branch sets _push_context_on_bot_stopped_speaking and re-triggers on BotStoppedSpeakingFrame. The user-side had no equivalent — a short trigger utterance whose transcription-driven turn-start races the function-call result would leave the tool response permanently un-delivered. Fix mirrors the bot-side pattern.

Both bugs are entirely separate from the async function calling support this issue tracks — but they produce the same observable symptom (model goes silent after a tool call), which made them easy to conflate with "No Response." Sharing here so future searchers land on the specific root causes that reproduce on 3.1 Flash Live as of April 2026. Full repro + test coverage in each PR description.