Skip to content

Ensure support for all o models, including o3 and o4-mini #398

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
deepdive101 opened this issue Apr 17, 2025 · 2 comments
Open

Ensure support for all o models, including o3 and o4-mini #398

deepdive101 opened this issue Apr 17, 2025 · 2 comments

Comments

@deepdive101
Copy link

At this point, it is just wrong for OpenAI to release models without updating tiktoken at the same time.
@indrasvat indrasvat mentioned this issue Apr 18, 2025
Closed
@DavidAmat
Copy link

when it is expected to add o4- in MODEL_PREFIX_TO_ENCODING in tiktoken/model.py ? they are basically using the same tokenizer as o3, so no reason for not updating tiktoken right away

@kaimatzu
Copy link

Yeah, it's all o200k_base at this point. But they should specify this explicitly on model release lol. It's confusing for developers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@DavidAmat @kaimatzu @deepdive101