llama.cpp has been growing its own answer to the question we ship gguf-parser for: llama-fit-params (and the common_init_result: fitting params to device memory path inside llama-server itself) computes what fits on the available GPUs from the same allocator the engine actually uses. That overlaps heavily with what we run gguf-parser for. This issue is context for a focused spike into whether the engine can shed its second helper binary; nothing here is committed work.
The short version
We vendor gguf-parser as the measuring tape: it reads a GGUF and predicts VRAM per model per device, and the placement planner turns those predictions into a layout. The prediction is a reimplementation of llama.cpp's memory math in Go, which means it drifts every time llama.cpp changes its allocator, and we inherit that drift as estimation bugs. A fit oracle that lives inside llama.cpp cannot drift from llama.cpp. The catch is that today's llama-fit-params answers a different question ("what settings would I pick for this hardware") rather than the one placement needs ("how many bytes will this exact configuration use per device"), so this is an upstream-maturity watch item, not a swap we can do now.
Why we ship gguf-parser today
Before anything starts, the planner decides which GPU each role lands on, how a big model splits across cards, how many parallel slots the context divides into, and whether co-tenants share a device. All of that needs per-device byte predictions ahead of launch, for models that may not even be downloaded yet (remote parsing during lilbee model pull). gguf-parser provides exactly that: structured per-device estimates, projector-aware, offload-aware, with remote HTTP range reads.
What the drift costs us in practice
The placement-estimation campaign that closed with the tight-placement and projector-correction work documented five separate upstream estimation defects in gguf-parser's vision path, each found by diffing its numbers against measured llama-server residency: an unknown-projector-type fallback that inflated one family by ~10 GiB, flash-attention ignored in projector math, mixed-modality projectors detected as the wrong type, an audio position-table capacity bug, and a missing image-size default that understates one family by ~0.3 GiB. lilbee now fences all of these (estimates advise, never refuse; the projector charge is floored by file size; the only hard refusal left is weights larger than total fleet VRAM), but every fence is code we maintain because the measuring tape reads wrong. An in-engine oracle would have had none of these by construction.
Upstream status
The fit machinery lives behind the design discussion ggml-org/llama.cpp#18049, which is still open. The projector half of the gap has already been fixed upstream: issue 19980 and issue 18181 (both closed) were resolved by merged PR 21489, which added mtmd_get_memory_usage and made llama-server's in-process --fit account for vision and audio encoder memory. That fix lives in the server path, not in the standalone llama-fit-params tool this issue proposes as an estimator backend: the tool still rejects --mmproj, and no machine-readable per-device output exists anywhere in the fit path. Neither remaining gap has an open issue yet; both would need to be filed as feature requests referencing the discussion above before this migration can start.
What llama-fit-params offers today
- It runs llama.cpp's real memory planning against the real device topology, so its notion of "fits" is the engine's own, including flash attention, KV quantization, and current allocator behavior.
- It picks
--n-gpu-layers, --tensor-split, and context settings that fit, which is a stronger form of the advice our estimates currently feed into plan_placement.
Where it falls short of the job (as of the research spike)
- It rejects
--mmproj, so the vision path that motivated this whole campaign is exactly the path it cannot size yet.
- It prints the CLI decisions it made, not per-device byte footprints, so the planner has nothing structured to reason with for co-tenancy and swap groups.
- There is no unified "total footprint for this argv" query, and no remote mode: it wants the weights on disk, while our pull-time gating sizes models before download via HTTP range reads.
The shape of a migration, when upstream is ready
Step 1 (measure) is the only step that changes: _estimate_role would call a fit oracle bundled with the engine wheel instead of shelling out to gguf-parser, and the correction layer plus its upstream-defect fences would be deleted rather than ported. Placement policy, swap groups, slot math, and the physics bound all stay: they consume byte predictions and do not care where the bytes come from. Remote pull-time sizing needs its own answer (either upstream range-read support or keeping a thin header parser for that one call site).
What would settle it
A small contained experiment once upstream moves: for the same model corpus the campaign used, ask the fit oracle for per-device footprints, launch, and diff against measured residency. The bar the current stack sets is 947 corpus pairs with zero refusals and zero floor violations, and 10 vision families loading on a 12 GB card with the whole fleet peaking under budget. If the in-engine oracle matches or beats that with mmproj support and structured output, gguf-parser and its correction layer go away.
Watch items
--mmproj support in the standalone tool (the in-server fit already has it via PR 21489)- machine-readable per-device output
- sizing without a full local copy of the weights
The last two have no open upstream issue as of this writing.
llama.cpp has been growing its own answer to the question we ship gguf-parser for:
llama-fit-params(and thecommon_init_result: fitting params to device memorypath inside llama-server itself) computes what fits on the available GPUs from the same allocator the engine actually uses. That overlaps heavily with what we run gguf-parser for. This issue is context for a focused spike into whether the engine can shed its second helper binary; nothing here is committed work.The short version
We vendor gguf-parser as the measuring tape: it reads a GGUF and predicts VRAM per model per device, and the placement planner turns those predictions into a layout. The prediction is a reimplementation of llama.cpp's memory math in Go, which means it drifts every time llama.cpp changes its allocator, and we inherit that drift as estimation bugs. A fit oracle that lives inside llama.cpp cannot drift from llama.cpp. The catch is that today's llama-fit-params answers a different question ("what settings would I pick for this hardware") rather than the one placement needs ("how many bytes will this exact configuration use per device"), so this is an upstream-maturity watch item, not a swap we can do now.
Why we ship gguf-parser today
Before anything starts, the planner decides which GPU each role lands on, how a big model splits across cards, how many parallel slots the context divides into, and whether co-tenants share a device. All of that needs per-device byte predictions ahead of launch, for models that may not even be downloaded yet (remote parsing during
lilbee model pull). gguf-parser provides exactly that: structured per-device estimates, projector-aware, offload-aware, with remote HTTP range reads.What the drift costs us in practice
The placement-estimation campaign that closed with the tight-placement and projector-correction work documented five separate upstream estimation defects in gguf-parser's vision path, each found by diffing its numbers against measured llama-server residency: an unknown-projector-type fallback that inflated one family by ~10 GiB, flash-attention ignored in projector math, mixed-modality projectors detected as the wrong type, an audio position-table capacity bug, and a missing image-size default that understates one family by ~0.3 GiB. lilbee now fences all of these (estimates advise, never refuse; the projector charge is floored by file size; the only hard refusal left is weights larger than total fleet VRAM), but every fence is code we maintain because the measuring tape reads wrong. An in-engine oracle would have had none of these by construction.
Upstream status
The fit machinery lives behind the design discussion ggml-org/llama.cpp#18049, which is still open. The projector half of the gap has already been fixed upstream: issue 19980 and issue 18181 (both closed) were resolved by merged PR 21489, which added
mtmd_get_memory_usageand made llama-server's in-process--fitaccount for vision and audio encoder memory. That fix lives in the server path, not in the standalonellama-fit-paramstool this issue proposes as an estimator backend: the tool still rejects--mmproj, and no machine-readable per-device output exists anywhere in the fit path. Neither remaining gap has an open issue yet; both would need to be filed as feature requests referencing the discussion above before this migration can start.What llama-fit-params offers today
--n-gpu-layers,--tensor-split, and context settings that fit, which is a stronger form of the advice our estimates currently feed intoplan_placement.Where it falls short of the job (as of the research spike)
--mmproj, so the vision path that motivated this whole campaign is exactly the path it cannot size yet.The shape of a migration, when upstream is ready
Step 1 (measure) is the only step that changes:
_estimate_rolewould call a fit oracle bundled with the engine wheel instead of shelling out to gguf-parser, and the correction layer plus its upstream-defect fences would be deleted rather than ported. Placement policy, swap groups, slot math, and the physics bound all stay: they consume byte predictions and do not care where the bytes come from. Remote pull-time sizing needs its own answer (either upstream range-read support or keeping a thin header parser for that one call site).What would settle it
A small contained experiment once upstream moves: for the same model corpus the campaign used, ask the fit oracle for per-device footprints, launch, and diff against measured residency. The bar the current stack sets is 947 corpus pairs with zero refusals and zero floor violations, and 10 vision families loading on a 12 GB card with the whole fleet peaking under budget. If the in-engine oracle matches or beats that with mmproj support and structured output, gguf-parser and its correction layer go away.
Watch items
--mmprojsupport in the standalone tool (the in-server fit already has it via PR 21489)The last two have no open upstream issue as of this writing.