Add support for transformers>=5.0#416
Conversation
|
Thanks a lot @kashif! Do you think something also needs changing in Chronos and Chronos-2? Could you also update the bound after the stable version of v5 is available? We can test and get this merged after |
kashif
changed the title
transformers==5.0transformers>=5.0
Remove comments about weight initialization in transformers.
Cyrilvallez
left a comment
Cyrilvallez
left a comment
There was a problem hiding this comment.
Yep, LGTM from the transformers side!
abdulfatir
left a comment
abdulfatir
left a comment
There was a problem hiding this comment.
Thanks for the PR, Kashif! It looks like the most critical change is the T5Stack does not expected embed_tokens anymore. The initialization stuff can still be kept the same as before (pure pytorch). What do you think?
| factor = self.config.initializer_factor | ||
| if isinstance(module, Chronos2LayerNorm): | ||
| module.weight.data.fill_(factor * 1.0) | ||
| init.constant_(module.weight, factor * 1.0) |
There was a problem hiding this comment.
Is there a reason why we need to move the init logic from pure pytorch to transformers? I would prefer keeping this part as is.
There was a problem hiding this comment.
You can keep pytorch, but it should use the init module! I.e. not do it in-place, as we check a flag on the weight itself before re-initializing or not.
So keeping it as module.weight.data.fill_ will skip the check, and always re-init randomly, even if it was just loaded
|
I am looking forward to transformers>=5.0 as this would unlock using https://github.com/huggingface/huggingface_hub v1.x |
4 participants
Issue #, if available:
Description of changes:
Added some fixes to chronos-bolt for Transformers V5rc
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.