[Paltform][LiteLLM] Add Bridge #872
Conversation
welcoMattic
requested review from
Nyholm,
OskarStark and
chr-hertel
as code owners
welcoMattic
force-pushed
the
platform/litellm
branch
from
cdb9a18 to
0f4f0e0
Compare
|
Hey, thanks for that, that's actually an interesting one - the feature of litellm is to be a generic bridge to whatever based on the original OpenAI contract - right? So i think we should work on the design shift here more - in case you're up for that. For example the contract got centralized already - and as long as the bridge uses that original openai compliant contract, no additional normalizers need to be registered with the platform. i think the same should apply for some kind generic endpoint handling (model client + result converter) so i think it would be great that we could use generic services (like litellm, openrouter, replicate) that conform to that first openai contract from payload and endpoint handling point of view without even installing a bridge. WDYT? (if that's off-scope for your current context, we can go ahead with this, merge it for now and potentially remove it later in favor of a generalized/centralized solution) |
OskarStark
changed the title
| - model_name: mistral-small-latest | ||
| litellm_params: | ||
| model: mistral/mistral-small-latest | ||
| api_key: "os.environ/MISTRAL_API_KEY" |
There was a problem hiding this comment.
Can't we append the os.environ/ part internally if its not provided?
There was a problem hiding this comment.
It refers to the .env file that must contains MISTRAL_API_KEY
| volumes: | ||
| - ./litellm/config.yaml:/app/config.yaml | ||
| env_file: | ||
| - .env |
There was a problem hiding this comment.
.env.local can be added here to read your own keys if needed.
4 participants
This PR adds support for LiteLLM as Bridge
For now, it is a very simple support (only chat), see it as ready-to-evolve bridge.
I've added a litellm service in Docker Compose file of examples to easily test it without having to install LiteLLM locally on your host.