[Usability]: Please pin the exact version of tokenizers and other fast-moving libraries it depends on

[prrao87]

What happened?

This morning, an update to the Hugging Face tokenizers library caused a breaking change to all DSPy users, because DSPy pulls the latest tokenizers version. I was able to bypass this issue by downgrading the version of tokenizers myself, but this took a lot of time to dig through the issues and find that downgrading the tokenizers library would solve the problem.

dependencies = [
    "dspy>=3.0.0b3",
    "tokenizers==0.21.2",
]

This is imo a little too lax, and introduced unintended breaking points because tokenizers is a fast-moving project and can introduce breaking changes like this, this totally breaks DSPy installs too, causing unnecessary headaches for users. Because Tokenizers is not a critical dependency for DSPy (not every user needs it and is even aware it's being installed under the hood), and because DSPy itself moves at a rapid pace, it's worth pinning the non-critical deps like this so that such annoying breaking changes upstream don't block newcomers to DSPy who are still learning the ropes.

Steps to reproduce

uv add dspy

This is broken and fails to work (as of 5 pm UTC on 28 July 2025).

DSPy version

3.0.0b3

[prrao87]

Apologies, I accidentally opened this as a bug report instead of a feature request. Please change the label when possible.

[okhat]

Thanks @prrao87 ! Is tokenizers even a dependency of DSPy? I think something is missing here. Why would tokenizers have anything to do with DSPy ..

[prrao87]

Yes, it gets generated as part of uv.lock because the litellm dependency (which is the default) depends on tokenizers. TBH all of this makes DSPy seem really heavy. I understand that LiteLLM is the default LM provider, but the larger the community gets, expecting them to get a LiteLLM key or even use it by default seems a little too restrictive. Personally, I get everything I want out of DSPy and OpenRouter for now, the LiteLLM dependency is not something I need now.

This issue was resolved upstream by Hugging Face yesterday, but it led to several hours of downtime where users like me who were creating new DSPy environments that don't need/require LiteLLM or tokenizers needlessly went down this rabbit hole. It can break CI pipelines too that are trying to run tests.

I think this issue has a larger scope than this, and could be renamed/repurposed as such.

[prrao87]

Upon further reading, I can see why LiteLLM is chosen as a necessary dependency for the project - it allows arbitrary LM providers to be plugged into DSPy at any time. So this makes sense. However, I'm not 100% sure how the tokenizers dependency snuck in there. Clearly there's something in the dependency resolution of DSPy that causes libraries like OpenAI's tiktoken and Hugging Face's tokenizers to be pulled by default, even if the user just does uv add dspy.