Skip to content

use inference gateway in the readme #3665

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

use inference gateway in the readme #3665

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,9 +99,9 @@ async def entrypoint(ctx: JobContext):
session = AgentSession(
vad=silero.VAD.load(),
# any combination of STT, LLM, TTS, or realtime API can be used
stt=deepgram.STT(model="nova-3"),
llm=openai.LLM(model="gpt-4o-mini"),
tts=elevenlabs.TTS(),
stt="assemblyai/universal-streaming:en",
llm="openai/gpt-4.1-mini",
tts="cartesia/sonic-2:9626c31c-bec5-4cca-baa8-f8ba9e84c8bc",
)

await session.start(agent=agent, room=ctx.room)
Expand Down Expand Up @@ -177,9 +177,9 @@ async def entrypoint(ctx: JobContext):
userdata = StoryData()
session = AgentSession[StoryData](
vad=silero.VAD.load(),
stt=deepgram.STT(model="nova-3"),
llm=openai.LLM(model="gpt-4o-mini"),
tts=openai.TTS(voice="echo"),
stt="deepgram/nova-3",
llm="openai/gpt-4o",
Copy link
Contributor

@bcherry bcherry Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we're recommending openai/gpt-4.1.mini as the new default

Copy link
Member Author

@theomonnom theomonnom Oct 17, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This model hallucinates a lot, can't get the drive-thru tests to pass

Copy link
Contributor

@bcherry bcherry Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh we should probably not be using it as default then - will revive the convo in slack

Copy link

@tmshapland tmshapland Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bcherry @theomonnom I can't find the slack thread where you're discussing what the default should be. Looks like you landed on 4o. Should we have the examples and the quickstart also use 4o by default?

When we were talking about this for the gateway before launching, I remember DZ saying we should use 4.1 bc OAI will eventually deprecate 4o. But I was also just talking to a user who said, "4o 2024-05-13 is better at tool calling, and oai models are not robotic, and the intelligence doesn’t drop with large context, all for the best trade off on latency. 4.1 and gemini 2.5 flash comes close, but the may version of 4o is the best. They even charge more for it." Lots of users have different preferences, but it was interesting to hear this guy back up what he likes about 4o, and specify the may version specifically.

image

Copy link
Contributor

@bcherry bcherry Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i posted in slack but got no replies on it https://live-kit.slack.com/archives/C05P6DRC6MV/p1760744284787699

I don't have an opinion, happy to do whatever the team thinks is the right answer but want to be consistent across the board and also get this PR landed!

tts="elevenlabs/eleven_turbo_v2_5:Xb7hH8MSUJpSbSDYk0k2",
Copy link
Contributor

@bcherry bcherry Oct 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
tts="elevenlabs/eleven_turbo_v2_5:Xb7hH8MSUJpSbSDYk0k2",
tts="cartesia/sonic-2:9626c31c-bec5-4cca-baa8-f8ba9e84c8bc",

userdata=userdata,
)

Expand Down