What happens
Traced token cost is overstated for any model that supports prompt caching. On a workload that reuses a long prefix, the overstatement is large. On one measured Gemini call with a 25,978-token prompt of which 24,540 were served from cache, the correct cost is $0.005955 and we would report $0.039084. That is 6.6 times too high.
Why
Two things, and both have to be fixed.
We never record the cached token count. Nothing under api/oss/src/core/tracing/ or the Python SDK's tracing writes cache_read_input_tokens or anything like it. Only prompt and completion are recorded.
The one place that prices tokens does not pass it. api/oss/src/core/tracing/utils/trees.py:636 calls litellm's cost_per_token with model, prompt_tokens and completion_tokens only. That function accepts cache_read_input_tokens and cache_creation_input_tokens, and litellm's price map carries a separate, much lower rate for cached input. For Gemini 3.5 Flash the cached rate is exactly one tenth of the normal input rate.
So every cached token is billed at the full input rate in what we report to the user.
Where
api/oss/src/core/tracing/utils/trees.py:636, the only cost calculation in the API.- Wherever token counts are written into
ag.metrics.tokens.incremental, which needs a third field.
Impact
This runs in the open source edition, so it affects every self-hosted user as well as cloud. It gets worse as prompt caching becomes more common, and it is worst exactly on agent workloads, which replay a long prefix on every call.
Suggested fix
Record the cached input count alongside prompt and completion, then pass it through to cost_per_token. Both providers we care about report it in the usage object already: OpenAI and Google both put it at prompt_tokens_details.cached_tokens.
What happens
Traced token cost is overstated for any model that supports prompt caching. On a workload that reuses a long prefix, the overstatement is large. On one measured Gemini call with a 25,978-token prompt of which 24,540 were served from cache, the correct cost is $0.005955 and we would report $0.039084. That is 6.6 times too high.
Why
Two things, and both have to be fixed.
We never record the cached token count. Nothing under
api/oss/src/core/tracing/or the Python SDK's tracing writescache_read_input_tokensor anything like it. Onlypromptandcompletionare recorded.The one place that prices tokens does not pass it.
api/oss/src/core/tracing/utils/trees.py:636calls litellm'scost_per_tokenwithmodel,prompt_tokensandcompletion_tokensonly. That function acceptscache_read_input_tokensandcache_creation_input_tokens, and litellm's price map carries a separate, much lower rate for cached input. For Gemini 3.5 Flash the cached rate is exactly one tenth of the normal input rate.So every cached token is billed at the full input rate in what we report to the user.
Where
api/oss/src/core/tracing/utils/trees.py:636, the only cost calculation in the API.ag.metrics.tokens.incremental, which needs a third field.Impact
This runs in the open source edition, so it affects every self-hosted user as well as cloud. It gets worse as prompt caching becomes more common, and it is worst exactly on agent workloads, which replay a long prefix on every call.
Suggested fix
Record the cached input count alongside prompt and completion, then pass it through to
cost_per_token. Both providers we care about report it in the usage object already: OpenAI and Google both put it atprompt_tokens_details.cached_tokens.