-
Notifications
You must be signed in to change notification settings - Fork 3.3k
Support Gemma3 with Clip fused attention #24187
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
qk_nodes = self.model.match_parent_path( | ||
matmul_qkv, ["Cast", "Cast", "Softmax", "Add", "Mul", "MatMul"], [0, 0, 0, 0, 0, 0] | ||
) | ||
# If attention mask is not used, we can still match the qk path. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Suggest to change the condition so that layout is more friendly:
Before:
if qk_nodes is None:
...
else:
add_mask = qk_nodes[1]
To
if qk_nodes is not None:
add_mask = qk_nodes[1]
else:
...
Another possible change is to use match_parent_paths.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done in #24280
The errors seem to be some docker image authorization issues. |
Closing this to "not using fork" for triggering CI (authorization to docker images) |
Description
Essentially, the vision model is traced differently (this time it's without mask.), and the input indices of op.Add and op.MatMul can be different. Also, fp16 and fp32 need different tracing patterns (op.Cast).
Motivation and Context
To optimize Gemma3 multi-modal model, the changes are needed. https://huggingface.co/google/gemma-3-4b-it