[Bug]: Job "update_spend" raised an exception

[heiko-braun]

What happened?

Testing 1.74.12-nightly (litellm[proxy]==1.74.12)fails with an exception

Job "update_spend" raised an exception

It looks like this completely broke spend tracking?

Relevant log output

File "/home/litellm/venv/lib/python3.12/site-packages/prisma/engine/utils.py", line 175, in handle_response_errors
    raise exc(error)
prisma.errors.FieldNotFoundError: Could not find field at `createManyLiteLLM_SpendLogs.data.mcp_namespaced_tool_name`

What LiteLLM version are you on ?

v1.74.12-nightly

Observations

Apparently the DB migrations did run correctly:

• LiteLLM_DailyUserSpend prisma def contain mcp_namespaced_tool_name
• Database Table LiteLLM_DailyUserSpend contains the column as well

[heiko-braun]

Related to #12894 ?

[heiko-braun]

Another, related error:

File "/home/litellm/venv/lib/python3.12/site-packages/litellm/proxy/management_endpoints/common_daily_activity.py", line 367, in get_daily_activity
    grouped_data[date_str]["breakdown"] = update_breakdown_metrics(
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/litellm/venv/lib/python3.12/site-packages/litellm/proxy/management_endpoints/common_daily_activity.py", line 121, in update_breakdown_metrics
    if record.mcp_namespaced_tool_name:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/litellm/venv/lib/python3.12/site-packages/pydantic/main.py", line 991, in __getattr__
    raise AttributeError(f'{type(self).__name__!r} object has no attribute {item!r}')
AttributeError: 'LiteLLM_DailyUserSpend' object has no attribute 'mcp_namespaced_tool_name'

[ishaan-jaff]

#12894 fixes this @heiko-braun

[heiko-braun]

@ishaan-jaff Looks like there is still a catch with 1.74.15.post1

Not sure if it's on my end...

File "/home/litellm/venv/lib/python3.12/site-packages/prisma/engine/utils.py", line 175, in handle_response_errors
    raise exc(error)
prisma.errors.FieldNotFoundError: Could not find field at `upsertOneLiteLLM_DailyTagSpend.where.tag_date_api_key_model_custom_llm_provider_mcp_namespaced_tool_name`

[jugaldb]

@krrishdholakia @ishaan-jaff this is still happening on v1.75.0, see logs:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/apscheduler/executors/base.py", line 181, in run_coroutine_job
    retval = await job.func(*job.args, **job.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/litellm/proxy/utils.py", line 3424, in update_spend
    await ProxyUpdateSpend.update_spend_logs(
  File "/usr/local/lib/python3.12/site-packages/litellm/proxy/utils.py", line 3378, in update_spend_logs
    _raise_failed_update_spend_exception(
  File "/usr/local/lib/python3.12/site-packages/litellm/proxy/utils.py", line 3457, in _raise_failed_update_spend_exception
    raise e
  File "/usr/local/lib/python3.12/site-packages/litellm/proxy/utils.py", line 3354, in update_spend_logs
    await prisma_client.db.litellm_spendlogs.create_many(
  File "/usr/local/lib/python3.12/site-packages/prisma/actions.py", line 12564, in create_many
    resp = await self._client._execute(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prisma/_base_client.py", line 543, in _execute
    return await self._engine.query(builder.build(), tx_id=self._tx_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prisma/engine/_query.py", line 402, in query
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prisma/engine/_http.py", line 233, in request
    return self._process_response_data(data=data, response=response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prisma/engine/_http.py", line 87, in _process_response_data
    return utils.handle_response_errors(response, errors_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prisma/engine/utils.py", line 181, in handle_response_errors
    raise prisma_errors.DataError(data[0])
prisma.errors.DataError: The column `mcp_namespaced_tool_name` does not exist in the current database.

Reported over slack

[heiko-braun]

Yep, same here

[ishaan-jaff]

@jugaldb do you have a way to repro the issue ?

[heiko-braun]

Not sure, but how about this:

• Start with 1.74.12, create users/keys
• upgrade to 1.75.0

Maybe its missing steps during the migration?

[ishaan-jaff]

Tried that too and it works

[heiko-braun]

I found it. The error was on my side.

We build our own images and had an old schema.prisma in the repo.

Sorry for keeping you busy with this.