Commit 0bd7a29
feat(gallery): add Gemma 4 llama.cpp MTP variants; fix gemmable-4-12b-mtp (#10876)
Google shipped the Gemma 4 MTP drafter heads and llama.cpp merged native
support in ggml-org/llama.cpp#23398. LocalAI's pinned llama.cpp already
carries it, and the config plumbing (draft_model + core/config/mtp.go)
was built for exactly this path, but no official Gemma 4 gallery entry
wired it up.
Add llama.cpp draft-mtp speculative-decoding variants for the dense
sizes, sourced from the unsloth QAT GGUF repos (target UD-Q4_K_XL +
mtp-*.gguf drafter + BF16 mmproj):
- gemma-4-e2b-it-qat-mtp
- gemma-4-e4b-it-qat-mtp
- gemma-4-12b-it-qat-mtp
- gemma-4-31b-it-qat-mtp
These replace the previously commented-out attempts, which were disabled
because the Janvitos/boxwrench drafter GGUFs declared the architecture as
`gemma4_assistant` (underscore) and failed to load on stock llama.cpp.
The unsloth drafters use the upstream `gemma4-assistant` (hyphen) spelling
that mtp.go's isDraftOnlyAssistantArch expects, so they load without any
backend patch. The 26B-A4B MoE is intentionally omitted (the upstream PR
reports no meaningful MTP speedup for it).
Also fix gemmable-4-12b-mtp: it loaded the draft-only `-mtp` GGUF as the
main model with no draft_model set, which cannot run standalone. It now
loads the target as the model, wires the drafter via draft_model, enables
spec_type:draft-mtp, and downloads both files.
All sha256 pins were taken from the HuggingFace API lfs.oid (reliable
content hash even for Xet-backed repos).
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>1 parent 45b8047 commit 0bd7a29
1 file changed
Lines changed: 229 additions & 168 deletions
0 commit comments