[Feature] Allow multiple modules with different LMs to optimize together (SIMBA compatibility) #8588
Description
What feature would you like to see?
I'd like DSPy to support SIMBA (or similar) joint optimization across complex pipelines where different modules use different LMs. This is a real-world use case, as many advanced systems are modular and require bespoke models for different components. Currently, DSPy does not allow this—optimization fails if more than one LM instance is detected.
Currently, attempting this results in the error: "Multiple LMs are being used in the module. There's no unique LM to return." This makes it impossible to optimize modular pipelines with multiple LMs, even though this is a common real-world need.
Would you like to contribute?
- Yes, I'd like to help implement this.
- No, I just want to request it.
Additional Context
If I understand correctly, SIMBA optimization should not require all modules to use the same LM. Real-world and research workflows increasingly involve pipelines where each module is best served with different model providers or capabilities. Removing this limitation would unlock much greater flexibility and relevance for DSPy.