fix(huggingface): detect max_length from nested text_config (Gemma3 multimodal)#3916
Open
OrionArchitekton wants to merge 1 commit into
Conversation
HFLM.max_length scanned only the top level of model.config for n_positions / max_position_embeddings / n_ctx. Multimodal configs such as Gemma3 nest max_position_embeddings under text_config, so the value was missed and the tokenizer's infinite model_max_length fell through to _DEFAULT_MAX_LENGTH (2048), silently truncating a long-context model. Add a guarded text_config fallback after the top-level scan and before the tokenizer default; text-only configs are unaffected. This keeps the existing precedence of config over tokenizer.model_max_length, now applied to nested configs, so it also covers other multimodal families that expose text_config. HFMultimodalLM inherits this property, so the hf processor path from the issue is fixed too. Fixes EleutherAI#3460 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
HFLM.max_lengthauto-detects a model's context length by scanningself.model.configforn_positions,max_position_embeddings, orn_ctx, but it only checked the top level of the config. Multimodal configs such asGemma3Confignestmax_position_embeddingsundertext_config, so the value is missed; the processor's tokenizer reports an infinitemodel_max_length, which then falls through to_DEFAULT_MAX_LENGTH(2048). The long-context model is silently truncated to 2048 tokens.Fixes #3460.
Change
After the existing top-level scan, and before the tokenizer default, fall back to a nested
text_config, guarded so text-only configs are untouched.HFMultimodalLMinherits this property with no override, so the "hf processor" path named in the issue is fixed by the same change.This keeps the harness's existing precedence of config over
tokenizer.model_max_length(the top-level scan already returns before the tokenizer) and extends it to nested configs. It therefore applies to any config exposingtext_config(Gemma3 is the reported case, but the same holds for other multimodal families), which is intentional and consistent with how top-level configs are already treated.Verification
New
tests/models/test_huggingface_max_length.py(no model weights downloaded; exercises the realHFLM.max_lengthgetter):text_configresolves to the real value, not 2048 (the reported bug);model_max_length, pinning the intended precedence for multimodal families;transformers.Gemma3Configresolves to itstext_config.max_position_embeddings(131072), pinning the fix against future library changes.ruff checkandruff formatare clean on the changed files; the change introduces no new lint. Verified against transformers 5.13.0.🤖 Generated with Claude Code