Litellm Prompt Management with Langfuse Integration

[ZACSAD]

Prompt Usage and Prompt Fallback

Configuration in Langfuse :

Created 2 prompts in Langfuse with 2 models as follows:

• prompt A as "movie-critic" with model gpt-4.1
• prompt B as "joke-critic" with model gpt-4o

Configuration in Litellm :

proxy_config:
  model_list:
    - model_name: my-langfuse-model
      litellm_params:
        model: langfuse/gpt-4.1
        prompt_id: "movie-critic"
    - model_name: my-langfuse-fallback
      litellm_params:
        model: langfuse/gpt-4o
        prompt_id: "joke-critic"

  router_settings:
    enable_tag_filtering: true
    fallbacks: [{"my-langfuse-model": ["my-langfuse-fallback"]}]


  litellm_settings:
    success_callback: ["langfuse"]

Usecase for prompts:

• Calling models in litellm should use prompts

response = client.chat.completions.create(
    model="my-langfuse-model",
    messages = [
        {
            "role": "user",
            "content": "write a poem"
        }
    ]
    ,
    extra_body={
        "prompt_variables": { 
            "key": "movie-critic"
        }
    }
)

Observation
The observation here is sometimes it is using the prompt A and sometimes it is not using the prompts as per traces in langfuse. Also, the prompt_variables is not being used it seems and anything can be given in the key.

Expectation
The model should always use prompt in this case while calling "my-langfuse-model" it should always use prompt A.

Usecase for prompts fallback:

• Calling a model and if that model fails it should fallback to another model with a prompt
  Now, in this scenario the wrong key is configured in litellm for the model "my-langfuse-model", so that it fallbacks to "my-langfuse-fallback"

response = client.chat.completions.create(
    model="my-langfuse-model",
    messages = [
        {
            "role": "user",
            "content": "write a poem"
        }
    ]
    ,
    extra_body={
        "prompt_variables": { 
            "key": "movie-critic"
        }
    }
)

Observation
The observation here is it is falling to "my-langfuse-fallback" and gpt-4o is being used for response, however in case of prompts sometimes it is using prompt A, sometimes prompt B and sometimes no prompt is being used as per langfuse traces.

Expectation
The model should always use a prompt in this case it can either be prompt A or prompt B , however the behaviour is random in using both prompts and no prompt as well.

Ask
Are there any missing configurations as per the expectations or behaviour or .
There is no defined behaviours in case of prompts as of now.

Please help here .... thanks