Skip to content

fix(huggingface): detect max_length from nested text_config (Gemma3 multimodal)#3916

Open
OrionArchitekton wants to merge 1 commit into
EleutherAI:mainfrom
OrionArchitekton:codex/fix-gemma3-nested-text-config-maxlen-20260706
Open

fix(huggingface): detect max_length from nested text_config (Gemma3 multimodal)#3916
OrionArchitekton wants to merge 1 commit into
EleutherAI:mainfrom
OrionArchitekton:codex/fix-gemma3-nested-text-config-maxlen-20260706

Conversation

@OrionArchitekton

Copy link
Copy Markdown

Summary

HFLM.max_length auto-detects a model's context length by scanning self.model.config for n_positions, max_position_embeddings, or n_ctx, but it only checked the top level of the config. Multimodal configs such as Gemma3Config nest max_position_embeddings under text_config, so the value is missed; the processor's tokenizer reports an infinite model_max_length, which then falls through to _DEFAULT_MAX_LENGTH (2048). The long-context model is silently truncated to 2048 tokens.

Fixes #3460.

Change

After the existing top-level scan, and before the tokenizer default, fall back to a nested text_config, guarded so text-only configs are untouched. HFMultimodalLM inherits this property with no override, so the "hf processor" path named in the issue is fixed by the same change.

This keeps the harness's existing precedence of config over tokenizer.model_max_length (the top-level scan already returns before the tokenizer) and extends it to nested configs. It therefore applies to any config exposing text_config (Gemma3 is the reported case, but the same holds for other multimodal families), which is intentional and consistent with how top-level configs are already treated.

Verification

New tests/models/test_huggingface_max_length.py (no model weights downloaded; exercises the real HFLM.max_length getter):

  • nested text_config resolves to the real value, not 2048 (the reported bug);
  • a top-level config still takes precedence (no regression);
  • a nested config value takes precedence over a smaller tokenizer model_max_length, pinning the intended precedence for multimodal families;
  • a real transformers.Gemma3Config resolves to its text_config.max_position_embeddings (131072), pinning the fix against future library changes.

ruff check and ruff format are clean on the changed files; the change introduces no new lint. Verified against transformers 5.13.0.

🤖 Generated with Claude Code

HFLM.max_length scanned only the top level of model.config for
n_positions / max_position_embeddings / n_ctx. Multimodal configs such
as Gemma3 nest max_position_embeddings under text_config, so the value
was missed and the tokenizer's infinite model_max_length fell through to
_DEFAULT_MAX_LENGTH (2048), silently truncating a long-context model.

Add a guarded text_config fallback after the top-level scan and before
the tokenizer default; text-only configs are unaffected. This keeps the
existing precedence of config over tokenizer.model_max_length, now
applied to nested configs, so it also covers other multimodal families
that expose text_config. HFMultimodalLM inherits this property, so the
hf processor path from the issue is fixed too.

Fixes EleutherAI#3460

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@CLAassistant

CLAassistant commented Jul 6, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

max_model_length for gemma3-12b is set to 2048 (_DEFAULT_MAX_LENGTH) with the hf processor

2 participants