Skip to content

Sessions: grow realtime sessions into a multi-provider framework (OpenAI-only today) #666

Description

@AbirAbbas

Umbrella tracking issue from #654.

#654 ships a deliberately provider-agnostic session DX — provider is an explicit, required knob and the transport matrix is multi-provider — but OpenAI WebRTC is the only fully-wired realtime path today. realtime-offer hard-gates to provider=openai+transport=webrtc and posts to api.openai.com/v1/realtime/calls; the declared openrouter: audio_turns pair has no handler.

This issue tracks growing sessions into genuinely multi-provider realtime ingress, so AgentField is the provider boundary for more than just OpenAI.

Candidate providers / transports

  • OpenRouter (voice-capable models) — tracked in Sessions: implement real OpenRouter provider support + voice-capable models (audio_turns is currently a no-op) #665.
  • Azure OpenAI Realtime — near drop-in; Azure documents the same SDP-proxy pattern (/openai/v1/realtime/calls, ephemeral via /openai/v1/realtime/client_secrets). Mostly an endpoint + auth swap in createOpenAIRealtimeCall.
  • Google Gemini Live — different protocol (WebSocket bidi streaming, not WebRTC/SDP); needs a dedicated control-plane adapter.
  • Deepgram Voice Agent / others — evaluate transport + tool-calling fit.

Design principles to hold across providers

Acceptance criteria (behavior)

  • At least one non-OpenAI provider produces a working realtime session end-to-end through the control plane, key server-side.
  • Adding a provider is a documented extension point (handler + matrix entry + session-config mapping), not a fork of the OpenAI path.

Related: #662 (sideband tools), #663 (VAD/turn-detection config), #664 (websocket transport + live loop), #665 (OpenRouter).

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:aiAI/LLM integrationarea:control-planeControl plane server functionalityenhancementNew feature or requestepicTracking issue for a coordinated initiativehelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions