Add utility for Reload Transformers imports cache for development workflow #35508

[sambhavnoobcoder]

Problem Statement

When developing or modifying Transformers models, cached imports can prevent code changes from being picked up without manually restarting Python. This makes the development workflow more cumbersome, especially when making iterative changes to model implementations.

Fixes #35508

Root Cause Analysis

The issue stems from Python's module caching behavior in sys.modules. Additionally, Transformers uses a custom _LazyModule system for imports that maintains its own object cache. Both these caches need to be cleared to fully reload modified code.

Implementation

Added a clear_import_cache() utility function to transformers.utils.import_utils that:

1. Identifies all Transformers modules in sys.modules
2. Clears internal caches of _LazyModule instances
3. Removes modules from sys.modules
4. Forces reload of the main Transformers module

The implementation is minimally invasive and works with Transformers' existing module system.

Testing

Added a test case that verifies:

1. Initial module imports work correctly
2. Cache clearing removes modules from sys.modules
3. Modules can be reimported after clearing
4. The lazy loading system continues to function properly

The test ensures the cache clearing is thorough while maintaining backward compatibility.

Screenshots

cc : @ArthurZucker @SunMarc @Rocketknight1 and anyone interested in development workflow improvements.

[ArthurZucker]

Sorry for my late review! Thanks for the PR 🤗 just missing a bit of doc, maybe in How to hack a model? As I guess this is the usecase? Also mentionning when would people use this (based of the issue!)

[sambhavnoobcoder]

Hey @ArthurZucker , i have added the necessary documentation in 388b2ac . Do tell me if anything else is required , i'll make those fixes as well .

[ArthurZucker]

Should be good to go, thanks

[Rocketknight1]

Hey! This test is failing in the CI

[sambhavnoobcoder]

anything specific i need to pass them @Rocketknight1 ? i'll do it right away .

[sambhavnoobcoder]

Hi @Rocketknight1,

I've done a thorough analysis of the failing tests in relation to the changes introduced in this PR. Let me break down each failure type and explain why I believe they're unrelated to these changes:

1. pipelines_torch -> NumPy ValueError (6 failures):

Calling nonzero on 0d arrays is not allowed. Use np.atleast_1d(scalar).nonzero() instead.

These failures occur in question-answering pipeline tests across ERNIE, ALBERT, and RemBERT models. They're specifically about NumPy array dimensionality operations. The changes from the PR only touch the module import system and don't interact with any NumPy operations or array handling.

2. tests_non_model -> Unpickling Error (1 failure):

_pickle.UnpicklingError -> Weights only load failed

This is related to model weight deserialization. The clear_import_cache() function doesn't interact with model weights or serialization mechanisms - it only manages module imports in sys.modules.

3. tests_torch -> Module Import AssertionError (1 failure):

assert 'transformers.models.auto.modeling_auto' in {'PIL': <module...}

While this is import-related, the cache clearing function:

• Only runs when explicitly called
• Only affects modules starting with "transformers."
• Preserves the module hierarchy when reloading
• Is isolated in its test file
  The error seems to be about PIL module verification, which the PR changes don't touch.

4. Generic AssertionError (2 failures):

AssertionError -> False is not true

These are basic assertion failures in test logic, unrelated to module importing or caching mechanisms.

The changes in this PR are focused solely on providing a development utility to clear the transformers module cache when needed. The function:

• Only affects sys.modules entries starting with "transformers."
• Only clears caches when explicitly called
• Maintains proper module hierarchy during reloading

That said if you see any way that the changes from the PR could be causing these failures, I'm more than happy to investigate further and make any necessary corrections. Please let me know if you'd like me to explore any specific aspects of the implementation.

[dvrogozh]

@sambhavnoobcoder : the test seems to somehow invalidate the state of the test engine. Some (not all) of the next tests start to fail if executed after this one though otherwise they pass. See:

• Some of test/utils tests fail being invalidated by tests/utils/test_import_utils.py::test_clear_import_cache #36334

[sambhavnoobcoder]

thank you for pointing it out . #36345 PR solves this issue . Hoping that gets merged quickly and solves this issue .