Skip to content

Commit 0bd7a29

Browse files
localai-botmudler
andauthored
feat(gallery): add Gemma 4 llama.cpp MTP variants; fix gemmable-4-12b-mtp (#10876)
Google shipped the Gemma 4 MTP drafter heads and llama.cpp merged native support in ggml-org/llama.cpp#23398. LocalAI's pinned llama.cpp already carries it, and the config plumbing (draft_model + core/config/mtp.go) was built for exactly this path, but no official Gemma 4 gallery entry wired it up. Add llama.cpp draft-mtp speculative-decoding variants for the dense sizes, sourced from the unsloth QAT GGUF repos (target UD-Q4_K_XL + mtp-*.gguf drafter + BF16 mmproj): - gemma-4-e2b-it-qat-mtp - gemma-4-e4b-it-qat-mtp - gemma-4-12b-it-qat-mtp - gemma-4-31b-it-qat-mtp These replace the previously commented-out attempts, which were disabled because the Janvitos/boxwrench drafter GGUFs declared the architecture as `gemma4_assistant` (underscore) and failed to load on stock llama.cpp. The unsloth drafters use the upstream `gemma4-assistant` (hyphen) spelling that mtp.go's isDraftOnlyAssistantArch expects, so they load without any backend patch. The 26B-A4B MoE is intentionally omitted (the upstream PR reports no meaningful MTP speedup for it). Also fix gemmable-4-12b-mtp: it loaded the draft-only `-mtp` GGUF as the main model with no draft_model set, which cannot run standalone. It now loads the target as the model, wires the drafter via draft_model, enables spec_type:draft-mtp, and downloads both files. All sha256 pins were taken from the HuggingFace API lfs.oid (reliable content hash even for Xet-backed repos). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
1 parent 45b8047 commit 0bd7a29

1 file changed

Lines changed: 229 additions & 168 deletions

File tree

0 commit comments

Comments
 (0)