Add Nemotron GGUF Loading Support #34725
Conversation
| @@ -129,6 +129,9 @@ def load_gguf_checkpoint(gguf_checkpoint_path, return_tensors=False): | |||
| ) | |||
| model_size = m.group().strip("-") # only keeps `7b` | |||
|
|
|||
| if "nemotron" in architecture: | |||
There was a problem hiding this comment.
Why you explicitly assign architecture to updated one if it is the same?
There was a problem hiding this comment.
I'm sorry if my answer seems obvious, but isn't it for addressing cases where the "architecture" does not only contain "nemotron"? I took reference on what you did for the qwen2moe, so I think It's better to also do it for nemotron. But I tested it without these lines and it passes through. What do you think? And thank you for reviewing! As this is my first time contributing, please let me know if anything seems odds or is there any better implementation. Thank you!
There was a problem hiding this comment.
No, for qwen2moe, I explicitly assigned another architecture name, because gguf file contains qwen2moe, but later, execution chain expects to get qwen2_moe for config, model processing and so on. You provided the same name "nemotron". So, there is no reason to explicitly assign updated architecture to the same name and even to mention nemotron, because gguf processing takes it from config by default.
What does this PR do?
Add Nemotron GGUF loading support
Before submitting
Pull Request section?
to it if that's the case. Link: Community contribution: Adding GGUF support for more architectures #33260
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@SunMarc @LysandreJik @ArthurZucker