[BUG] ChronosBoltPipeline fails with transformers>=4.41 due to DynamicCache changes

[PouyaGhahramanian]

Bug report

Bug report checklist

• I provided code that demonstrates a minimal reproducible example.
• I confirmed bug exists on the latest mainline of Chronos via source install.

Describe the bug
When using ChronosBoltPipeline with the latest transformers library (>=4.41), the following error occurs due to changes in Hugging Face's caching mechanism:

AttributeError: 'DynamicCache' object has no attribute 'is_updated'

Expected behavior
ChronosBoltPipeline.predict() should work correctly with the latest versions of Hugging Face transformers without requiring manual patches or downgrades.

To reproduce

from chronos import ChronosBoltPipeline

pipeline = ChronosBoltPipeline.from_pretrained("amazon/chronos-bolt-base")
context = [...]  # your input data
prediction_length = 10
forecast = pipeline.predict(context, prediction_length=prediction_length)

This results in:

AttributeError: 'DynamicCache' object has no attribute 'is_updated'

Environment description
Operating system: Ubuntu 20.04 LTS
Python version: 3.11
CUDA version: 12.1
PyTorch version: 2.5.1
HuggingFace transformers version: 4.43.2
HuggingFace accelerate version: 1.9.0

---

Cause
Starting from transformers>=4.41, Hugging Face introduced a new caching mechanism (DynamicCache) for models like T5, which no longer has the is_updated attribute. ChronosBolt still assumes the old cache implementation.

Temporary Workarounds

1. Downgrade transformers to 4.40.2:

   pip install transformers==4.40.2

2. Patch modeling_t5.py:
   Replace:

   past_key_value.is_updated[self.layer_idx] = True

   with:

   if hasattr(past_key_value, "is_updated"):
       past_key_value.is_updated[self.layer_idx] = True

Suggested Fix

• Pin transformers<=4.40.2 in chronos-forecasting dependencies until a proper fix is implemented, or
• Update ChronosBolt's T5 wrapper to support DynamicCache by adding compatibility checks as shown above.

Additional Note
I am willing to work on a fix for this issue and can submit a PR to make ChronosBoltPipeline compatible with the new DynamicCache system in transformers>=4.41.

[abdulfatir]

@PouyaGhahramanian I am unable to reproduce the issue.

# Save as github.py

import torch
import transformers
from chronos import ChronosBoltPipeline

pipeline = ChronosBoltPipeline.from_pretrained("amazon/chronos-bolt-base")
context = torch.rand(512)
prediction_length = 10
forecast = pipeline.predict(context, prediction_length=prediction_length)
print(f"{forecast.shape=}")
print(f"{torch.__version__=}, {transformers.__version__=}")

Running the script with:

uv run --no-project --with chronos-forecasting github.py

gives me

forecast.shape=torch.Size([1, 9, 10])
torch.__version__='2.6.0+cu124', transformers.__version__='4.49.0'

[PouyaGhahramanian]

Hi, thanks for the quick response!

I followed your instructions exactly, using a clean conda environment with:

• transformers==4.49.0
• torch==2.6.0
• chronos-forecasting installed via:

  pip install git+https://github.com/amazon-science/chronos-forecasting.git

When I run the same script, I get the following error:

UnboundLocalError: cannot access local variable 'is_updated' where it is not associated with a value

This occurs in:
transformers/models/t5/modeling_t5.py, at:

if is_cross_attention and past_key_value is not None and is_updated:

In transformers >= 4.41, the T5 architecture uses a new DynamicCache system that does not define is_updated anymore, which causes this crash. Based on this, I believe your local version of Chronos (possibly built with uv or from an internal repository) may already include a compatibility fix for the new caching mechanism, which hasn't yet been published in the public repo.

To address this, I’d be happy to submit a pull request that safely adds a compatibility check like:

is_updated = getattr(past_key_value, "is_updated", False)

to make ChronosBolt work with both old and new cache APIs.

Let me know if you'd prefer that approach!

[abdulfatir]

@PouyaGhahramanian I tried again in a new conda env but I am unable to reproduce the issue. Here are the commands I ran:

conda create -n chronos-test python=3.11
conda activate chronos-test                                      
pip install transformers==4.49.0 torch==2.6.0
pip install git+https://github.com/amazon-science/chronos-forecasting.git
python github.py

Here github.py is the snippet I posted above.

Output:

forecast.shape=torch.Size([1, 9, 10])
torch.__version__='2.6.0+cu124', transformers.__version__='4.49.0'

Note that I do get this warning but the code does not break.

Passing a tuple of past_key_values is deprecated and will be removed in Transformers v4.48.0. You should pass an instance of EncoderDecoderCache instead, e.g. past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values).

[PouyaGhahramanian]

Hi @abdulfatir , thanks for checking this so quickly and for the repro commands! 🙏

I double-checked my environment and realized I’d mis-reported my transformers version. I was actually on 4.54.1 (not 4.49.0).
Starting with 4.53 the HF team removed the backward-compat shim that still exposed DynamicCache.is_updated, so Chronos’ call to

past_key_value.is_updated[layer_idx] = True

now raises

AttributeError: 'DynamicCache' object has no attribute 'is_updated'

Pinning to transformers < 4.53 (e.g. pip install "transformers==4.52.4") restores the old behavior and the pipeline works fine again.

---

Possible fixes

1. Quick patch in chronos/modeling_t5.py

   if hasattr(past_key_value, "is_updated"):
       past_key_value.is_updated[self.layer_idx] = True

2. Version guard in setup.cfg / pyproject.toml

   transformers>=4.41,<4.53
   

3. Longer-term: migrate to the new EncoderDecoderCache API.

Happy to open a PR with (1) if that helps. Sorry for the confusion earlier, and thanks again for your help!

—Pouya

[abdulfatir]

@PouyaGhahramanian Thank you for your patience. I was able to reproduce the issue on transformers v4.54. I have opened a PR (#313) to fix this. Note that the issue was more subtle than you had described. Please check the PR description for details.

[PouyaGhahramanian]

@abdulfatir Thanks for the fix! The PR explanation clarifies the root cause—much appreciated.