fix(api): model prices are fetched from the internet at import, so cost depends on when a worker booted

[mmabrouk]

What happens

Every cost number we report depends on whatever was on litellm's main branch at the moment the worker process booted. Two workers started at different times can price the same call differently, and nothing tells us when that happens.

Why

When litellm is imported it fetches its price map over the network from https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json, falling back to the copy inside the installed package only if the fetch fails. This is documented in litellm/litellm_core_utils/get_model_cost_map.py and is switched off by setting the environment variable LITELLM_LOCAL_MODEL_COST_MAP to True.

We do not set it. Confirmed inside a running API container:

LITELLM_LOCAL_MODEL_COST_MAP = None
model_cost entries: 2985

The packaged copy has a different number of entries, so the fetch is really happening.

Why it matters

Three separate problems.

Prices are not reproducible. The cost we computed for a trace last week cannot be recomputed today, because the input changed underneath us.

Two workers can disagree. A worker restarted after a litellm price change prices differently from one that has been up for a week, in the same deployment, with no signal.

It is a network call on the import path. A self-hosted deployment behind a firewall, or one running while GitHub is having a bad day, takes a startup penalty for something that should be a local file read.

Suggested fix

Set LITELLM_LOCAL_MODEL_COST_MAP=True in the API environment, add it to api/oss/src/utils/env.py so it is configuration rather than a loose variable, and treat the litellm version bump as the moment prices change. That makes pricing a property of a release rather than a property of the day.

[linear-code]

AGE-4069

[Aman-goel-04]

I've reproduced this locally and confirmed the fix.

That confirms the diagnosis exactly: with the env var unset, litellm fetches the live map from GitHub (2985 entries) instead of using the version-pinned local backup (2923 entries).

I'll go with what you suggested: set LITELLM_LOCAL_MODEL_COST_MAP=True, move it into env.py as a real config field instead of a loose env var, log the model count at startup so a mismatch is actually visible, and add a test pinning the count at 2923 so this can't silently regress. That makes pricing tied to the litellm version bump, not to boot time.

One thing I want to check before opening the PR, whether anything imports litellm before env.py does in the startup chain, since the var needs to be set before that first import happens.

Could this be assigned to me? I'll open the PR once that's confirmed.