feat(transformers): add VaultGemma (v4.57.1)

[alien-0119]

What does this PR do?

Adds # (feature)
Add VaultGemma model and fast ut.

Usage Example:

from transformers import AutoTokenizer
from mindone.transformers import VaultGemmaForCausalLM
import mindspore as ms

model_id = "google/vaultgemma-1b"
model = VaultGemmaForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)

prompt = "Tell me an unknown interesting biology fact about the brain."
inputs = tokenizer(prompt, return_tensors="np")
inputs = {k: ms.tensor(v) for k, v in inputs.items()}

# Generate
outputs = model.generate(**inputs, max_new_tokens=32)
print(tokenizer.decode(outputs[0]))
# I'm not sure if this is the right place to ask this, but I'm curious about the

Performance:
Experiments were tested on Ascend Atlas 800T A2 machines with mindspore 2.7.0 pynative mode.

model	precision	weight load(s)	s/step
google/vaultgemma-1b	fp32	48.753	0.090
google/vaultgemma-1b	fp16	46.820	0.089
google/vaultgemma-1b	bf16	35.317	0.102

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you make sure to update the documentation with your changes? E.g. record bug fixes or new features in What's New. Here are the
  documentation guidelines
• Did you build and run the code without any errors?
• Did you report the running environment (NPU type/MS version) and performance in the doc? (better record it for data loading, model inference, or training tasks)
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@xxx

[gemini-code-assist]

Summary of Changes

Hello @alien-0119, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly expands the mindone/transformers library by integrating the VaultGemma model. This addition provides users with a new powerful causal language model, enhancing the library's capabilities for various natural language processing tasks within the MindSpore ecosystem. The changes involve adding the model's core architecture, updating auto-loading mechanisms, and ensuring robust testing.

Highlights

• New Model Integration: The VaultGemma model (v4.57.1) has been successfully integrated into the mindone/transformers library, expanding the range of available transformer models.
• Core Model Implementation: Full MindSpore implementations for key VaultGemma components, including VaultGemmaRMSNorm, VaultGemmaMLP, VaultGemmaAttention, VaultGemmaDecoderLayer, VaultGemmaRotaryEmbedding, VaultGemmaPreTrainedModel, VaultGemmaModel, and VaultGemmaForCausalLM, have been added.
• Auto-Configuration and Auto-Modeling Support: The library's auto-configuration and auto-modeling utilities have been updated to seamlessly recognize and load VaultGemma models and their configurations.
• Comprehensive Testing: New unit tests have been introduced to validate the correctness and precision of the VaultGemma model's MindSpore implementation against its PyTorch counterpart.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces the VaultGemma model, which appears to be a port from the Hugging Face Transformers library. The implementation is comprehensive, including the model architecture and corresponding tests. My review focuses on ensuring code quality and correctness. I've identified a couple of areas for improvement: one is an incorrect type hint in the attention mechanism, and the other concerns the use of a wildcard import, which could be made more explicit for better code clarity. Overall, the changes are well-structured and the inclusion of tests is a great practice.

[gemini-code-assist]

The return type hint of the construct method is incorrect. It's specified as tuple[Tensor, Optional[Tensor], Optional[tuple[Tensor]]] but the function returns a tuple of two elements: attn_output and attn_weights. The third element, presumably for past_key_values, is not returned. Since the cache is updated in-place, this is fine, but the type hint should be corrected to match the actual return signature for code clarity and to aid static analysis.

Suggested change

	) -> tuple[Tensor, Optional[Tensor], Optional[tuple[Tensor]]]:
	) -> tuple[Tensor, Optional[Tensor]]:

[gemini-code-assist]

Using a wildcard import (*) is generally discouraged by PEP 8 as it can lead to namespace pollution and makes it difficult to track where names are coming from. While I see this pattern is used elsewhere in the repository, it's a good practice to explicitly import the public API. This improves code readability and helps static analysis tools.

Suggested change

	from .modeling_vaultgemma import *
	from .modeling_vaultgemma import (
	VaultGemmaForCausalLM,
	VaultGemmaModel,
	VaultGemmaPreTrainedModel,
	)
	__all__ = ["VaultGemmaForCausalLM", "VaultGemmaModel", "VaultGemmaPreTrainedModel"]