Skip to content

gguf-compute-in-quant-gemm.md

Latest commit

 

History

History
550 lines (480 loc) · 40.1 KB

gguf-compute-in-quant-gemm.md

File metadata and controls

550 lines (480 loc) · 40.1 KB
Raw

Leaf spec: GGUF compute-in-quant GEMM — ggml tensor-traits port into vt::cpu

Row: QUANT-GGUF-CIQ-GEMM (quantization-matrix.md, leaf of the QUANT-GGUF-COMPUTE block) · status: ACTIVE — G1-G4 landed (2026-07-22): the portable tier-0 quant GEMM is complete, gated at the OP level, and now routed end to end — vt::MatmulBT dispatches a block-dtype weight to kMatmulBTQuant, keep-quant is the production DEFAULT wherever that op is registered, and the six routed encodings compute in quant. Measured on the binding host: decode 3.45×, prefill 4.16×, peak RSS 1.16× less, output tokens byte-identical. G5-G8 open, and the measurement RE-RANKS them (see the G4 result below: the remaining CPU gap is now the ELEMENTWISE bf16/f16 GEMM, not the quant one) · upstream pin: llama.cpp local fork 237ad9b96 (b9892) · parent evidence: B4 decision row (parity-ledger.md L290) — the "vendor IF proven faster" criterion is MET; this leaf is the structural fix: keep GGUF blocks resident and quantize the ACTIVATION to int8 to meet them (llama.cpp's mul_mat), instead of expanding weights to bf16 at load. Research estimate on top of the threadpool: ~3.3× decode, ~10× prefill, ~3.3× weight RAM (expansion map wave 3).

Scope

  • In scope: quantized-block weight storage in vt:: (new block dtypes + traits); activation quantization f32→Q8_0/Q8_K (from_float); the per-type vec_dot GEMM core dispatched exactly like ggml's type_traits_cpu table; portable generic C++ kernels FIRST for the types our GGUF families need — Q8_0, Q4_K, Q5_K, Q6_K, Q3_K (APEX 35B Compact/Balanced + the Q8_K_XL bench file) plus already-materialized Q4_0; then per-arch SIMD tiers (x86 AVX2, Arm NEON/dotprod/i8mm for GB10) and the repack/interleave tier (q4_Kx8-style) as separate work rows. CPU backend only.
  • Out of scope: loader/persistence of the quant weights (leaf QUANT-GGUF-KEEPQ-LOADER); the threadpool itself (leaf QUANT-GGUF-CPU-THREADPOOL); i-quants/TQ/MXFP4/NVFP4 GGUF types (their own encoding rows: QUANT-GGUF-IQ2_S, QUANT-GGUF-IQ4_XS, …); CUDA GGUF compute (future backend row); llamafile/KleidiAI/AMX accelerators (see Risks/decisions).
  • Dispatch behavior (mirrors ggml exactly): for weight type T, vec_dot_type(T) selects the activation quant (Q8_0 for Q4_0/Q8_0; Q8_K for Q3_K/Q4_K/Q5_K/Q6_K per type_traits_cpu); src1 (activations) is quantized once into scratch, then each output element is one vec_dot(K, w_row, act_row). Repack tier, when present for (type, arch, N%8), replaces this with interleaved gemv/gemm — same selection precedence as ggml_repack_get_optimal_repack_type.

Upstream chain

Pinned local fork /home/mudler/_git/llama.cpp @ 237ad9b96.

Component Upstream anchor Content
Trait table ggml/src/ggml-cpu/ggml-cpu.c:211-406 type_traits_cpu[]: from_float, vec_dot, vec_dot_type, nrows per type; Q8_0→(q8_0,q8_0,nrows 2 on i8mm) :262-271, Q3_K :295-300, Q4_K :301-310, Q5_K :311-316, Q6_K :317-326, Q4_0 :230-239
GEMM driver ggml-cpu.c:1245-1443 src1→vec_dot_type quant into wdata (:1313-1349, block-split across threads), chunk grid + atomic stealing (threadpool leaf), num_rows_per_vec_dot 2-row mmla guard :1426-1433
Chunk worker ggml-cpu.c:1155-1243 16×16 tile, vec_dot per output, dst column memcpy
Block layouts ggml/src/ggml-common.h:242-245 (q8_0), :305-310 (q3_K), :317-327 (q4_K), :334-345 (q5_K), :352-357 (q6_K), :361-365 (q8_K: f32 d, i8 qs[256], i16 bsums[16]) byte-exact structs; our gguf_dequant.cpp already mirrors the weight-side layouts
Activation quant reference ggml/src/ggml-quants.c:238 (quantize_row_q8_0_ref), :2696 (quantize_row_q8_K_ref incl. bsums) portable reference; CPU wrappers ggml-cpu/quants.c:45,117
Generic vec_dot kernels (portable C++ tier) ggml-cpu/quants.c:174 (q4_0·q8_0), :400 (q8_0·q8_0), :566 (q3_K·q8_K), :645 (q4_K·q8_K), :720 (q5_K·q8_K), :800 (q6_K·q8_K) integer block dots + bsums tricks; this is the byte-for-byte port source for tier 0
x86 SIMD tier ggml-cpu/arch/x86/quants.c:1170 (q8_0·q8_0), :1900 (q4_K·q8_K), :2288 (q6_K·q8_K), … AVX2/AVX512 variants, generic fallback tail
Arm SIMD tier ggml-cpu/arch/arm/quants.c:1076 (q8_0·q8_0), :2260 (q4_K·q8_K), :2890 (q6_K·q8_K), … NEON/dotprod/SVE/i8mm (nrows==2 mmla paths) — GB10's tier
SIMD abstraction ggml-cpu/simd-mappings.h, arch-fallback.h per-ISA macro mapping; our tiers keep the same generic-fallback naming discipline
Tensor-traits hook ggml-cpu/traits.h:20-36, traits.cpp:9-38 tensor_traits::{work_size,compute_forward} + extra_buffer_type probe — the seam repack plugs into
Repack tier ggml-cpu/repack.cpp:4153-4250 (traits class), :4253 (forward_mul_mat), :4528-4726 (ggml_repack_get_optimal_repack_type: q4_K_8x8 on AVX2/i8mm, q6_K_8x8/8x4, q8_0_4x4/4x8, q5_K, q2_K …), :4727 (extra hook), :4745-4830 (buffer type); activation ggml_quantize_mat_t<4/8, Q8_0> :318-357; generic gemv/gemm: q4_K :887,:1822,:1905, q5_K impl :551,:647, q6_K impl :357,:447, q8_0 :1274-1368; arch SIMD: arch/x86/repack.cpp, arch/arm/repack.cpp prefill engine: 4/8 interleaved weight rows × 4/8 activation rows per kernel call

Runtime-trace: CPU dispatch is static given (type, arch, N alignment); the B4 oracle run (20-thread llama.cpp, repack active on AVX2) is the traced denominator. When gating on GB10 Arm, confirm which tier llama.cpp selects there (i8mm repack expected) before comparing.

Our baseline

Port map

Upstream Local Notes / deviations
ggml.h type ids + ggml-common.h block structs extend vt::DType with kQ4_0,kQ8_0,kQ3_K,kQ4_K,kQ5_K,kQ6_K,kQ8_K + block traits (block_elems, block_bytes) in src/vt/dtype.cpp; cross-checked against GgmlTraits (gguf_reader.cpp:191) design decision recorded below; kQ8_K is activation-only (never a weight/storage dtype in files)
type_traits_cpu[] (ggml-cpu.c:213-471) new src/vt/cpu/cpu_quant_traits.{h,cpp}: {from_float, vec_dot, vec_dot_type, nrows} per block dtype same table shape so upstream diffs port mechanically
quantize_row_q8_0_ref / quantize_row_q8_K_ref (ggml-quants.c:238,2696) src/vt/cpu/cpu_quant_act.cpp byte-for-byte incl. Q8_K bsums; f32 input (our activations are bf16 → widen first, matching ggml's f32 src1 contract)
generic ggml_vec_dot_* (ggml-cpu/quants.c:174,400,566,645,720,800) src/vt/cpu/cpu_quant_dot.cpp (tier 0, pure portable C++) keep *_generic names in comments + upstream line cites per kernel
ggml_compute_forward_mul_mat src1-quant + chunk loop (ggml-cpu.c:1245-1443,1155-1243) new op OpId::kMatmulBTQuant in include/vt/ops.h + CPU kernel in cpu_ops.cpp; vt::MatmulBT call sites route on b.dtype being a block dtype (no signature change) scratch for quantized activations: per-call 64B-aligned buffer sized rows × row_size(vec_dot_type, K) mirroring ggml_graph_plan wdata (ggml-cpu.c:2752-2980); reuses ParallelFor from the threadpool leaf
arch/x86/quants.c, arch/arm/quants.c src/vt/cpu/arch/{x86,arm}/cpu_quant_dot_{avx2,neon}.cpp (tier 1) compile-time ISA gates + runtime CPU-feature probe mirroring ggml_cpu_has_*; generic fallback always built (CI runs it)
repack.cpp traits + gemv/gemm + quantize_mat_t src/vt/cpu/cpu_quant_repack.cpp (tier 2) weight repack at LOAD (from the keep-quant loader's resident blocks) selected per (dtype, arch, N%8) exactly as :4528-4726; brings the 4×8/8×8 GEMM used for prefill
traits.h tensor_traits/extra_buffer_type classes NOT ported as a polymorphic hook our dispatch is the static op registry; ggml needs the hook for pluggable backends we don't have — recorded deviation, revisit if a second accelerator (AMX/KleidiAI-class) is ported

Tests to port

Upstream test Local tier / file Notes
llama.cpp tests/test-quantize-fns.cpp (thresholds :17-28: total quant error ≤0.002 4-bit+, ≤0.0040 3-bit, reference error ≤0.0001, dot-product error ≤0.02, ≤0.04 low-bit) T-unit tests/vt/test_ops_quant_dot.cpp (new) per-type: quantize→dequant RMSE bounds AND vec_dot vs f64 reference dot on synthetic rows; same thresholds, cite each
llama.cpp tests/test-backend-ops.cpp MUL_MAT cases (:4260-4450, NMSE ≤ max_nmse_err() :1133-1151 ≈5e-4) T-unit, same file kMatmulBTQuant vs the dequant-to-f32 composite (DequantGgufRowToF32 + MatmulBTKernel) at model shapes (K=hidden, N=proj dims, M∈{1,4,32,512}) per type
llama.cpp tests/test-quantize-perf.cpp tooling only (bench harness), not a ctest gate microbench per vec_dot tier; evidence for the ledger, never a pass/fail gate
our existing goldens T-parity tests/parity/test_op_parity.cpp:34 add matmul_bt_quant runner + goldens; goldens+runner land in the SAME commit or the op goes in PendingRunnerOps() (workflow.md anti-stale-golden gate)
same-file engine gates T-e2e tests/parity/test_qwen36_gguf_engine.cpp:143 + the 2B Q8_K_XL file greedy 16/16 vs same-file llama.cpp oracle with compute-in-quant ACTIVE; new goldens (see Risks — tokens may legitimately differ from the bf16-expansion path; the oracle is llama.cpp, not our old path)

Gates

  1. Op correctness: ported test-quantize-fns bounds green for all six types (generic tier on CI; SIMD tiers on their boxes); MUL_MAT NMSE ≤ 5e-4 vs the dequant-f32 composite at all listed shapes.
  2. Oracle discipline: VT_CPU_REF=1 env forces the dequant-to-bf16 + bf16-GEMM path (loader leaf provides it): existing goldens stay byte-identical there — the parity oracle is never lost. Compute-in-quant is NOT bit-identical to it by design; its correctness bar is the llama.cpp same-file oracle.
  3. E2E token gate: greedy 16/16 token-exact vs same-file llama.cpp CPU (237ad9b96 build) on (a) Qwen3.5-2B-UD-Q8_K_XL, (b) APEX 35B Compact, (c) APEX 35B Balanced — the existing gate harness recipe.
  4. Perf (B4-anchored, same box/file/recipe as parity-ledger.md L290, threadpool active, 20 threads, ≥2 reproducing runs):
    • tier 0 (generic vec_dot) milestone: decode ≥ 0.25× llama.cpp tg32 (≥3.2 t/s), prefill ≥ 0.1× pp128;
    • tier 1 (arch SIMD) milestone: decode ≥ 0.7× tg32;
    • tier 2 (repack) closes the row: match or beat llama.cpp on decode t/s, prefill t/s AND peak RSS on the same file/box (mirror-as-floor; BACKEND-GATE-CPU-LLAMACPP consumes this evidence; GB10 Arm rerun recorded there too).
  5. Memory: peak RSS with compute-in-quant ≤ 1.15× llama.cpp same-run (2.80 GiB on the 2B) — the bf16 expansion (7.43 GiB) must be gone (loader leaf) and no hidden f32 staging may reintroduce it.

Dependencies

  • QUANT-GGUF-CPU-THREADPOOL (ParallelFor + chunk loop; perf gates meaningless without it). Op-level tier-0 kernels + unit tests may land before it (single-thread), but the row cannot pass gate 4 without it.
  • QUANT-GGUF-KEEPQ-LOADER for gates 3-5 (resident quant weights + the VT_CPU_REF switch). Unit gates 1-2 need only synthetic blocks.
  • Hardware: 20-core x86 box (AVX2 tier + gates); GB10 (Arm NEON/i8mm tier); CI runs generic tier. No GPU lock; CPU runs need an otherwise-idle box per benchmark-protocol.
  • Oracle: llama.cpp fork build at 237ad9b96 (same as B4; keep the local NVFP4/Q1_0 fork patch noted — irrelevant to these six types).
  • Models: 2B Q8_K_XL + APEX 35B Compact/Balanced GGUFs (already on the boxes; checkpoint-gated tests skip when absent, as today).

Work breakdown

W Row (claim-sized) Content Depends Status
G1 block dtypes + traits vt::DType block entries, traits table, kMatmulBTQuant op skeleton + generic-composite fallback, trait cross-check vs GgmlTraits - DONE 2026-07-22
G2 activation quant quantize_row_q8_0/q8_K ports + scratch sizing; unit tests G1 DONE 2026-07-22
G3 tier-0 vec_dot kernels the six generic vec_dot ports + GEMM wiring + test-quantize-fns/backend-ops ports + parity goldens G1, G2 DONE 2026-07-22
G4 e2e enablement route MatmulBT call sites on block dtype; engine gates (3) + VT_CPU_REF A/B; first B4-recipe measurement (tier-0 milestone) G3, loader L2-L3, threadpool W2 DONE 2026-07-22
G5 x86 AVX2 tier arch/x86 vec_dot ports + feature probe; tier-1 milestone measurement G3 open
G6 Arm NEON/i8mm tier arch/arm ports incl. nrows==2 mmla; GB10 measurement G3 open
G7 repack tier repack-at-load + gemv/gemm + quantize_mat_t; selection parity vs :4528; closes gates 4-5 G4 + loader L2 open
G8 ledger + matrix closure full A/B series, ledger row, flip C/E/P cells per encoding row (Q8_0/Q4_K/Q5_K/Q6_K/Q3_K/Q4_0), roadmap update G4-G7 open

G5/G6 are parallel; G7 may start once G4's layout is fixed.

Risks/decisions

  • vt::DType extension vs side-struct (internal design, decided here): block dtypes IN the enum mirror ggml's "type carries layout" model, keep Tensor self-describing, and let dispatch key on b.dtype — the alternative (parallel QuantTensor) would fork every op signature. Blocked dtypes are storage-only: any elementwise op on them is a VT_CHECK failure.
  • Goldens shift: enabling compute-in-quant changes CPU e2e numerics (not bit-equal to bf16 expansion). Decision: goldens for quant-GGUF engine gates are regenerated against the llama.cpp oracle in the SAME change that flips the default, with the VT_CPU_REF path keeping the old goldens alive. No silent tolerance widening.
  • Repack memory transient: repack-at-load temporarily holds source+dest blocks per tensor; do it tensor-by-tensor (llama.cpp repacks in-buffer per tensor) to keep gate 5.
  • Accelerator hooks not ported (llamafile sgemm ggml-cpu.c:1286-1310, AMX, KleidiAI): each is an extra tier behind the same dispatch seam; llama.cpp's B4 arm did not need them to hit the floor numbers (stock CPU build). Revisit only if gate 4 tier-2 stalls below 1.0× — record then.
  • nrows==2 boundary guards (ggml-cpu.c:1426-1433) must be ported with the mmla kernels or i8mm results corrupt on odd shapes — covered by the non-divisor shape cases in the ported MUL_MAT tests.
  • Upstream behavior is not reopened: quant math, thresholds, and tier selection all mirror llama.cpp; the only product-ish calls are the two recorded above (enum design, golden regeneration protocol).

G1 result (2026-07-22)

Landed. vt::DType gains the seven block entries (kQ4_0 kQ8_0 kQ3_K kQ4_K kQ5_K kQ6_K kQ8_K) with block geometry in dtype.cpp and the storage-only rule enforced — SizeOf on a block dtype THROWS, so any elementwise kernel that reaches one fails loudly. RowSizeBytes mirrors ggml_row_size (rows are whole blocks). The type_traits_cpu mirror is cpu_quant_traits.cpp behind the new public vt/quant.h; vec_dot_type and nrows are populated, vec_dot/from_float stay nullptr until G3/G2. OpId::kMatmulBTQuant + the CPU kernel run the generic dequant-composite fallback for every type.

Deviation from the port map (recorded). The six dequantize_row_* decoders were MOVED, not duplicated: they now live at cpu_quant_dequant.cpp as the traits table's to_float column (upstream keeps to_float in ggml.c's device-neutral type_traits; vt has no such second table, so it rides the CPU one), and the GGUF loader delegates. One implementation serves both the loader oracle and the GEMM fallback. The code is byte-identical to what it replaced, so numerics are unchanged, and test_gguf_dequant plus a new byte-for-byte equivalence case gate that.

Trait cross-check — NO mismatch. test_ops_quant_traits.cpp compares THREE independent statements of the same llama.cpp facts: vt's block geometry table, the GGUF reader's GgmlTraits, and the block-struct arithmetic written out from ggml-common.h. All seven types agree on block_elems, block_bytes and type id; the id map round-trips; Q8_K (id 15) is correctly absent from the reader's table because it is activation-only. 8 cases / 5,694 assertions green.

G2 + G3 result (2026-07-22)

Landed together. The portable tier-0 compute-in-quant GEMM is complete.

G2 — activation quant. cpu_quant_act.cpp ports quantize_row_q8_0_ref (ggml-quants.c:238) and quantize_row_q8_K_ref (ggml-quants.c:2696, bsums included) plus nearest_int (:563); the CPU-tier wrappers those sit behind are ggml-cpu/quants.c:45,117, which on the generic tier are bare calls to the _ref forms. Scratch sizing (QuantActRowBytes / QuantActScratchBytes) mirrors ggml_row_size + the mul_mat wdata computation in ggml_graph_plan (ggml-cpu.c:1313-1349, 2752-2980). Only the two ACTIVATION encodings get a from_float: nothing in this project quantizes an activation into a k-quant, so upstream's k-quant encoders stay unported.

G3 — the six generic vec_dot. cpu_quant_dot.cpp ports ggml-cpu/quants.c :174 (q4_0·q8_0), :400 (q8_0·q8_0), :566 (q3_K·q8_K), :645 (q4_K·q8_K), :720 (q5_K·q8_K), :800 (q6_K·q8_K), with upstream's auto-vectorization-shaped structure (aux8 decode split, 8-wide aux16/aux32 staging, deferred sums[8]) preserved verbatim — that structure also FIXES the reduction order, which is what makes the GEMM bit-reproducible. Each kernel asserts nrc == 1; nrows == 2 stays unreachable until G6 brings the mmla kernels AND the ggml-cpu.c:1426-1433 boundary guards. cpu_quant_blocks.h is a static_asserted mirror of the ggml-common.h block structs. cpu_quant_gemm.cpp now takes the quantized path (quantize src1 ONCE into scratch, then one integer vec_dot per output) for the six types; the generic dequant-composite remains for any block dtype without a vec_dot (today only Q8_K, activation-only).

Correctness — gated against an INDEPENDENT reference, not a second copy. test_ops_quant_dot.cpp, 16 cases / 78,052 assertions green. The primary gate dequantizes BOTH operands through BlockToFloat (the loader-side dequantize_row_* decoders — a separate port that walks the layout differently from each vec_dot's inline decode) and dots them in f64, with the tolerance set relative to the dot's L1 magnitude so sign cancellation cannot hide an error (|got - ref| <= 1e-5 * L1, actual agreement ~1e-6). Coverage: all six types × nblocks ∈ {1, 2, 3, 5, 7, 16} — single-block rows, ODD multiples that catch an unroll-by-2 assumption, and the Q6_K/Q3_K super-block structure. Ragged K (not a whole number of blocks) is gated to THROW at every layer (from_float, vec_dot, QuantActRowBytes, kMatmulBTQuant) rather than round down. The G2→G3 handoff is covered by feeding real from_float output into every vec_dot case. Upstream's own thresholds are ported unwidened: test-quantize-fns.cpp:17-28 (total 0.002 / reference 0.0001 / dot-product 0.02) and test-backend-ops.cpp:4277 MUL_MAT NMSE ≤ 5e-4 at M ∈ {1,4,32,512} × N ∈ {1,7,16}. Determinism: vec_dot and the full GEMM are asserted bit-identical run-to-run and across thread counts 1/2/4 (byte memcmp, not Approx), and the GEMM is asserted bit-equal to a hand-driven per-element vec_dot so the wiring provably adds no reordering or stride slip.

Mutation-tested (14 mutants, 13 caught). To prove the gate has teeth rather than merely passing: q6_K −32 bias→−31, q4_K mins[j/2]→[j/4], q3_K hmask polarity flip, q5_K high-bit 16→8, q4_0 nibble bias −8→−7, q8_0 dropped activation delta, q4_K kmask1 corruption, q3_K scale bias −32→−31, q8_K bsums short/doubled, q8_K iscale −127→−128, q8_0 amax/127→/128, q8_0 stored-delta fed back into the quants — all caught. One mutant (roundf→truncation in Q8_0) initially PASSED every statistical bound: upstream's 8-bit RMSE/NMSE thresholds genuinely cannot distinguish rounding modes. That gap is closed by an added byte-exact encoder gate comparing from_float output against a reference written from upstream's prose (round-half-away-from-zero for Q8_0, round-half-to-even for nearest_int) rather than by calling the same library function; it also catches the nearest_int→truncation and stored-delta mutants. The single uncaught mutant is Q8_K's MIN(127, v), which is provably unreachable (|iscale| = 127/amax and |x| ≤ amax ⇒ |iscale·x| ≤ 127); it is kept for upstream fidelity and the test records why no case can distinguish it.

DGX CONFIRMATION RUN (dgx.casa, ~/work/vllm.cpp-ciq-g3, transferred by git archive — never rsync — production flags -DVLLM_CPP_CUTLASS_DIR=$HOME/cutlass-4.5.0 -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.0/bin/nvcc -DVLLM_CPP_TRITON=ON -DCMAKE_CUDA_ARCHITECTURES=121a, GPU verified idle by nvidia-smi with no compute apps, every stage under flock $HOME/gpu.lock): clean CUDA -Werror build to 100%, 0 warnings. Regression set ALL UNCHANGED, each gate run STANDALONE — 27B 235/235, 35B 315/315, Qwen3-Coder 6/6 (138 assertions; strict 5/6, max gap 0 nats), Qwen3-dense 16/16 on BOTH 0.6B and 4B (664 assertions; strict 10/16 and 11/16, max gap 0.25 nats), OPT 6/6 (36 assertions), DeepSeek-V2-Lite 8/8 (223 assertions; strict 5/8, 92/128 tokens strictly exact, 0 forward-divergent). test_qwen36_gguf_engine run STANDALONE on the real APEX files: 28/28 assertions, 2/2 cases, 16/16 tokens each on Compact and Balanced. The three new/updated CPU units also pass on dgx's aarch64 with identical counts (test_ops_quant_dot 16 cases / 78,052 assertions, test_ops_quant_traits 8 / 5,615, test_gguf_dequant 11 / 202) — the portable tier is architecture-independent, as intended. Golden corpus md5 (475 files under tests/parity/goldens) identical before and after the series: 2965ef5772b556d3f3f86fedf4221b2f. One transient to record honestly: a first pass that ran four engine gates back-to-back inside a single shell aborted DeepSeek-V2 partway (95 of 223 assertions, 1 failure); re-run STANDALONE it passes 223/223 / 8/8, which is the known co-scheduled-memory effect on this box, not a regression.

Nothing existing moved. No model call site routes to kMatmulBTQuant (grep over src/vllm/ finds no reference — G4 owns that), every GGUF weight still expands to bf16 at load, and the per-encoding C cells therefore stay -. test_gguf_dequant 11 cases / 202 assertions and the BlockToFloat == loader-dequant byte-for-byte case remain green, so the decode numerics are untouched. test_ops_quant_traits is 8 cases / 5,615 assertions (was 5,694): its composite-fallback case now covers Q8_K alone, because the six weight types legitimately no longer take the composite path — the drop is that retarget, not lost coverage, and the six types' GEMM behaviour moved to the new file's independent-reference and NMSE cases.

G4 grounding — what the measurement says it is worth (2026-07-22)

CLAIM-BENCH-CPU-LLAMA-REMEASURE-1 re-measured the floor and profiled the engine. Full write-up: CPU floor re-measurement. Three results change how this row should be planned and judged.

1. G4 is not an increment — it is most of the gap. An op-dispatch profile that accounted for 100.0 % of wall time found kMatmul at 95.37 % (748 calls, 10.915 ms/call); attention, the GDN recurrence, norms, conv, sampling and all host glue are under 5 % combined. The CPU plan is a GEMM plan.

2. The tier-0 kernels this row already landed are 14–44× faster than the GEMM production runs. Op-level harness at this model's shapes (aarch64, 20 threads, best of 3):

shape (M×N×K) production kMatmul [K,N] bf16 kMatmulBT [N,K] bf16 tier-0 kMatmulBTQuant Q8_0 ratio
decode qkv 1×3072×2048 11.79 GFLOP/s 17.91 287.54 24.4×
decode down 1×2048×6144 6.11 18.48 266.50 43.6×
decode lm_head 1×248320×2048 14.66 24.68 211.43 14.4×
prefill qkv 128×3072×2048 18.42 24.62 416.79 22.6×
prefill gate_up 128×12288×2048 14.86 24.52 393.48 26.5×

x86 (indicative, contended box): production 6.1–12.6, Q8_0 tier-0 130–200, so 13–20×.

3. Gate 4's tier-0 milestone is set far too low. Projecting the table above onto this model (arithmetic on measured op throughput, NOT an end-to-end run): per-token decode GEMM 24 × (0.044+0.032+0.226+0.094) + 4.811 = 14.3 ms ⇒ ~70 t/s cache-warm, bounded in practice by DRAM at llama.cpp's own 25.80 t/s; 128-token prefill GEMM 24 × (3.864+2.710+16.373+8.295) = 750 ms ⇒ ~171 t/s against llama.cpp pp128 174.63; peak RSS ≈ file + activations ≈ 2.9–3.0 GiB against llama.cpp's 2.798. The stated tier-0 milestone (decode ≥ 0.25× tg32, prefill ≥ 0.1× pp128) should be cleared by a wide margin. Measure G4 against PARITY, not against the milestone — and re-scope G7 afterwards, because tier-0 at M=128 is already within ~2 % of pp128 on GEMM alone, which is far less headroom than the repack tier was budgeted for.

Orientation is a G4 PREREQUISITE, newly identified. The GGUF loader today dequantises to bf16 and transposes into [K,N] (qwen3_5_gguf_weights.cpp:192 — "dequant to bf16 and transpose to Matmul-B [K, N]"), so the hot path runs kMatmul, whose inner loop strides by N down K. Measured, that orientation alone costs 1.3–3.0×. GGUF's native layout is already [N,K] — exactly what MatmulBTQuant requires — so G4's first step is to stop doing extra load-time work to reach the slower kernel.

Inertness re-confirmed at 1cb5f64 by three independent checks: no MatmulBTQuant reference under src/vllm/; vt::MatmulBT (ops.cpp:163) hard-requires IsFloat(b.dtype) with no block branch; and VT_GGUF_KEEP_QUANT=1 fails at load on both boxes with vt: matmul_bt: float inputs and f32/bf16 output required.

G4 result (2026-07-22) — routed, default-ON, measured

Landed. The machinery G1-G3 and L1-L3 built is now on the executed path.

What routes. ONE dispatch point does the whole job: vt::MatmulBT sends a block-dtype b to MatmulBTQuant and falls through to its unchanged validation + kMatmulBT for everything else. That is sufficient because every model matmul helper already routes an nk == true weight to MatmulBT (qwen3_5.cpp:1067,1078 device-in/out, :743,760 host), and the keep-quant loader already produces exactly that: a block-typed OwnedTensor in the file's [N,K] order. No forward, no signature and no call site changed. Safetensors paths are unreachable from the new branch by construction — no safetensors loader produces a block dtype.

The default flip. GgufLoadPolicy::FromEnv now defaults keep_quant to GgufQuantComputeAvailable() — vt::OpRegistered(kMatmulBTQuant, CurrentPlatform().device_type()) — rather than to false. The condition is the honest one ("does a block weight have a consumer on the device this process will run on"), so a CUDA build keeps expanding to bf16 (CUDA GGUF compute-in-quant remains a future backend row) and the day that kernel is registered elsewhere the default follows with no edit. VT_GGUF_KEEP_QUANT=0/off/false is the opt-out; VT_CPU_REF=1 still wins over everything.

The transpose is gone — for BOTH residencies. Kept blocks never had one (L2). The measurement's lever 2 was folded in as GgufLoadPolicy::expand_nk: a matmul weight that must EXPAND (an f16/f32 file tensor, or an unported encoding) is now materialised in the file's own [N,K] order with nk = true instead of being transposed into Matmul-B [K,N]. It rides the same availability condition and is forced off by VT_CPU_REF. This is numerically free on the CPU tier by construction — MatmulOneChunk<kBT> differs from <false> ONLY in the weight offset, keeping the same sequential f32 K reduction — and a unit asserts the two loads are element-for-element transposes of each other. It is deliberately NOT applied on a device whose GEMM picks its algorithm from the operand layout (cuBLASLt), where orientation is not numerically free.

This mattered more than expected. The bench file is MIXED, which is the normal case: Qwen3.5-2B-UD-Q8_K_XL.gguf is 103 q8_0 + 56 f16 + 176 f32 tensors, and the f16 side is not small tensors — it is 1.615 GiB including the 970 MiB tied token_embd/lm_head and whole layers of ffn. Without expand_nk the largest GEMM in the model would still have been paying the slow orientation.

Correctness — NO token movement anywhere

  • E2E gate, quant path DEFAULT-ON, run STANDALONE on a CPU-only dgx build (which is where keep-quant is live): test_qwen36_gguf_engine PASSES on the real APEX files — 2/2 cases, 16/16 greedy tokens each on Compact ({F32,Q3_K,Q4_K,Q6_K}) and Balanced ({F32,Q8_0,Q5_K,Q6_K}) against the same-file llama.cpp oracle. Five of the six routed encodings are exercised end to end by that pair. Peak RSS during it was 18.7 GiB on the 17.3 GiB Compact file — visibly the file, not a bf16 expansion of it.
  • Oracle A/B on the bench model: the output token streams of the BEFORE arm (VT_GGUF_KEEP_QUANT=0), the AFTER arm (production default) and the ORACLE arm (VT_CPU_REF=1) are byte-identical — one md5, d235db12f2cd304007530286a1755c95, across all three. So VT_CPU_REF reproduces the historical result exactly, and compute-in-quant did not move a single token on this file.
  • A CPU-only build fails four SACRED gates — CONTROLLED as pre-existing, not a G4 regression. Running the full dgx ctest against a CPU-only build (the configuration that exercises the quant path) fails test_qwen36_weights, test_qwen3_paged_engine, test_qwen27_paged_engine and test_deepseek_v2_paged_engine, because those gates' goldens were captured on CUDA and a CPU forward is not bit-identical to it. The BASE tree (c603ebc) was built CPU-only on the same box and run against the same four: identical failures, identical assertion counts (51/2, 245/2, 11/2, 95/1). None of the four loads a GGUF file, and all four pass on the CUDA build (235/235, 315/315, 664/664, 223/223). The regression bar for this repo is the CUDA build; a CPU-only run of a CUDA-goldened gate is not one.
  • The spec anticipated that quantized compute "is NOT bit-identical to the dequant path by design". On the measured workloads it did not need the latitude: zero divergence, so nothing had to be diagnosed and no golden was regenerated. Q8_0 · bf16 is the favourable case (the weight values are identical either way; only the accumulation differs, and the integer path is the more accurate one), so this is a result about these files, not a proof that no k-quant file can ever differ.

Benchmark — binding, dgx.casa (GB10, aarch64, 20 cores), idle

Whole series under one flock $HOME/gpu.lock; load average 0.11 at series start, nvidia-smi reporting no compute apps; transferred by git archive; goldens md5-verified 475 files / 2965ef5772b556d3f3f86fedf4221b2f before and after. Model Qwen3.5-2B-UD-Q8_K_XL.gguf, llama.cpp 237ad9b96 (b9892).

llama.cpp (llama-bench -t 20 -ngl 0 -r 5): pp512 212.49±0.74, pp128 175.71±2.54, tg128 25.90±0.69, tg32 25.87±1.36; isolated tg32 25.26±1.34 with peak RSS 2,934,124 KB = 2.798 GiB. (Yesterday's numbers on this box reproduce to within noise, so the denominator is fresh, not carried.)

Ours — SAME BINARY, three arms, VLLM_CPP_CPU_THREADS=20, --input-len 128 --output-len 32 --concurrency 1 --seed 0 --temperature 0, 3 reps each, medians:

arm TTFT (ms) TPOT (ms) peak RSS (KB)
BEFORE — VT_GGUF_KEEP_QUANT=0 (the historical path) 24,859.25 451.27 7,788,196
AFTER — production default 5,970.21 130.72 6,712,368
ORACLE — VT_CPU_REF=1 24,822.20 451.89 7,787,940

Rep spread on the AFTER arm: TTFT 1.41 %, TPOT 2.03 %. The BEFORE arm reproduces the 2026-07-22 binding baseline (24,594 ms / 450.7 ms / 7,788,220 KB) to 1.1 % / 0.1 % / 24 KB, which is what makes this a valid same-binary A/B.

axis BEFORE AFTER ours: A/B vs llama.cpp BEFORE vs llama.cpp AFTER
Prefill (128 / TTFT) 5.149 t/s 21.44 t/s 4.16× 34.1× behind 8.20× behind
Decode (1000 / TPOT) 2.216 t/s 7.650 t/s 3.45× 11.7× behind 3.38× behind (3.30× vs isolated tg32)
Peak RSS 7.428 GiB 6.401 GiB 1.16× less 2.66× worse 2.29× worse

Did the 9-17× projection hold? NO — and the reason is measurable

It did not. The projection was 9-17× end to end with RSS falling to ~2.9 GiB; the measurement is 3.45× decode / 4.16× prefill and RSS 6.40 GiB. The projection was arithmetic over the quant GEMM's op-level throughput applied to the WHOLE model, and that step is where it broke: it assumed every GEMM takes the quant path. On this file most of the weight mass cannot.

bytes in file params routed to kMatmulBTQuant?
q8_0 (103 tensors) 1.062 GiB 1.073 B yes
f16 (56 tensors, incl. the 970 MiB tied token_embd/lm_head) 1.615 GiB 0.867 B no — f16 is not a block encoding
f32 (176 tensors) 0.006 GiB — no (norms/vectors, never eligible)

So 60 % of the weight bytes and 45 % of the parameters — including the single biggest GEMM in the model — kept running the elementwise bf16 kernel at 17-25 GFLOP/s while the routed 40 % moved to 211-417 GFLOP/s. The measured 3.4-4.2× is what that mixture predicts; the projection's 9-17× is what a fully-q8_0 file would have given. Nothing about the quant kernels under-delivered.

The same mixture explains the RSS floor: 2.68 GiB of touched file pages (blocks are COPIED, not mmapped in place — the recorded L2 follow-up) + 1.615 GiB of f16→bf16 expansion + a SECOND 970 MiB copy of the tied head + 1.06 GiB of kept blocks ≈ 6.35 GiB, against 6.401 GiB measured.

Consequences for the plan (supersedes the ranking in the floor re-measurement)

  1. The elementwise bf16/f16 GEMM is now the #1 CPU lever, ahead of G5-G7. It owns 60 % of this file's weight mass, EVERY safetensors CPU path, and every mixed GGUF (which is what published quants actually are). Its two known defects are stated in the floor re-measurement lever 3: no SIMD vec_dot_bf16/f16, and MatmulOneChunk's per-element LoadF32 dtype switch INSIDE the K loop. Closing it is what would let this file approach llama.cpp, which computes those same f16 rows with ggml_vec_dot_f16.
  2. G5/G6 (SIMD quant tiers) now improve only the 40 % that is already fast, so their end-to-end share on a mixed file is bounded by Amdahl at well under what the spec assumed. G7 (repack) should stay parked: prefill is 8.2× behind and the quant GEMM is not what is holding it.
  3. RSS needs two loader changes, not a kernel: mmap-in-place residency instead of copying (already recorded as an L2 follow-up, llama.cpp supports both), and sharing the tied lm_head with the embedding table instead of materialising the vocab matrix twice. Together they are ~1.9 GiB of the 3.6 GiB that separates us from llama.cpp; the rest is the f16 expansion, which the lever-1 work removes as a side effect if f16 weights stay f16.
  4. Gate 4's tier-0 milestone is MET. Its bar is decode ≥ 0.25× tg32 and prefill ≥ 0.1× pp128: measured 0.296× (7.650 / 25.87) and 0.122× (21.44 / 175.71), against 0.086× and 0.029× before G4. Gate 5 (peak RSS ≤ 1.15× llama.cpp) is NOT met — 2.29× — and stays open with L4 plus the two loader changes in item 3.

Reproduction

# dgx.casa, aarch64, idle, ONE flock for the whole series
R=$HOME/work/bench-cpu-llama
M=$R/models/Qwen3.5-2B-UD-Q8_K_XL.gguf
LB=$R/llamacpp/build/bin
B=$HOME/work/vllm.cpp-ciq-g4/build-cpu/examples/vllm-bench

cmake -S vllmcpp -B build-cpu -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF
cmake --build build-cpu -j 20 --target vllm-bench

flock $HOME/gpu.lock sh -c '
  uptime; nvidia-smi --query-compute-apps=pid,used_memory --format=csv
  LD_LIBRARY_PATH='"$LB"' '"$LB"'/llama-bench -m '"$M"' -p 512,128 -n 128,32 -t 20 -r 5 -ngl 0
  LD_LIBRARY_PATH='"$LB"' /usr/bin/time -v '"$LB"'/llama-bench -m '"$M"' -p 0 -n 32 -t 20 -r 3 -ngl 0
  for arm in "VT_GGUF_KEEP_QUANT=0" "" "VT_CPU_REF=1"; do
    for rep in 1 2 3; do
      env $arm VLLM_CPP_CPU_THREADS=20 /usr/bin/time -v '"$B"' --model '"$M"' \
        --num-prompts 1 --input-len 128 --output-len 32 --concurrency 1 \
        --seed 0 --temperature 0 --output-token-ids /tmp/ids.$rep.txt
    done
  done
'
# The three arms MUST agree on /tmp/ids.*.txt (md5 d235db12f2cd304007530286a1755c95).

Gates at G1: clean CPU -Werror build (0 warnings); full CPU ctest 151/151; gguf ctest 4/4. DGX CONFIRMATION RUN (dgx.casa, ~/work/vllm.cpp-ciq-g1, production flags -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.0/bin/nvcc -DVLLM_CPP_CUTLASS_DIR=/home/mudler/cutlass-4.5.0 -DVLLM_CPP_TRITON=ON, one flock $HOME/gpu.lock for the whole series): clean CUDA -Werror build to 100%, 0 warnings. Regression set ALL UNCHANGED — 27B 235/235, 35B 315/315, Qwen3-Coder 6/6 (strict 5/6, max gap 0 nats), Qwen3-dense 16/16 (strict 11/16, max gap 0.25 nats), OPT 6/6 (36 assertions), DeepSeek-V2-Lite 8/8 (223 assertions). test_qwen36_gguf_engine run STANDALONE (it OOMs co-scheduled) on the real APEX files: 28/28 assertions, 2/2 cases, 16/16 tokens each on Compact and Balanced — the k-quant dequant path is byte-stable across the decoder move. Gates 1-5 of this spec are untouched: they need G2/G3 (op correctness) and the loader leaf (e2e/perf/RSS).