Pure-Rust, Metal-only inference engine for Qwen3.6-27B, Qwen3.6-35B-A3B and Qwen3.8-27B (GGUF), optimized for a single Apple Silicon machine (M5 Max). Manual fork of the laguna/maxuna engine: candle-based, mmap no-copy weight loading, vendored Metal kernels, speculative decoding, and an HTTP server speaking Anthropic Messages and OpenAI Chat Completions.
Status: runs, and matches upstream. The 35B-A3B generates correctly end to end
(greedy, chat template, thinking split, clean stops) at ~59 tok/s decode, and both
checkpoints now pass the parity gate against upstream llama.cpp on identical GGUF
weights (docs/parity.md). No DeltaNet Metal kernels yet. See TODO.md for the
priority ledger and docs/log.md for the narrative.
docs/decisions.md— every deliberate choice and refuted direction, with evidencedocs/log.md— dated engineering narrativedocs/parity.md— verification runbook (vs upstream llama.cpp)CLAUDE.md— agent context: ground truth, architecture cheat sheet, hazardsTODO.md— deferred-work ledger
System Rust toolchain, system Apple CLT SDK (the nix shell deliberately provides no
SDK — see flake.nix). cargo build --release. Ops tests need a Metal device.
Three checkpoints, all Q4_K_M, resolved through the HF cache and downloaded on first use:
| Full name | Repo | --model-size |
Drafter |
|---|---|---|---|
Qwen3.6-27B |
ggml-org/Qwen3.6-27B-GGUF |
27b |
DFlash block drafter, 3.5 GB |
Qwen3.6-35B-A3B |
ggml-org/Qwen3.6-35B-A3B-GGUF |
35b (default) |
DFlash block drafter, 0.8 GB |
Qwen3.8-27B |
ggml-org/Qwen3.8-27B-GGUF |
3.8-27b |
MTP head, 3.2 GB |
Two vocabularies, deliberately. The CLI takes the short aliases above (and the full names); the HTTP APIs take the full names only. Qwen3.8-27B (added 2026-08-14) runs the same graph as Qwen3.6-27B — its config is byte-identical — so it needs no model math of its own. It ships no DFlash sidecar, but it does ship a first-party MTP head, which is a different drafter shape and became a second drafter implementation (2026-08-15); all three checkpoints now speculate.
Every checkpoint ships a drafter and speculates, in one of two kinds. The Qwen 3.6
pair carry DFlash sidecars: block drafters that propose a whole block out of one forward,
so depth is nearly free (--draft-max defaults to 15, a structural ceiling rather than a
fitted number). Qwen3.8-27B carries a first-party MTP head, which is a different shape —
one extra transformer layer that chains a forward per step and feeds itself, so depth
costs linearly and is fitted rather than capped (4).
Drafting is opt-out as of 2026-07-29: a zero-flag run speculates with the
checkpoint's official sidecar, --no-draft decodes plain, and --draft <gguf> swaps in
a custom drafter. Speculation is decided per checkpoint, not per process: a server
drafts for each checkpoint it loads with that checkpoint's own sidecar, and the
dashboard's draft cell reports what the LOADED checkpoint is doing rather than what was
configured. The default costs a sidecar load per run (3.5 GB on the 27B, 0.8 GB on the
35B-A3B, 3.2 GB on the 3.8-27B).
Measured over plain decode on the same machine state (greedy, 128 tokens, warm, medians
of 3 reps, arms interleaved, lowpowermode 0 on AC):
| Checkpoint | code | chat | acceptance | fitted |
|---|---|---|---|---|
Qwen3.6-27B |
+46 to +52% | +46 to +52% | 78-86% | 2026-08-08 |
Qwen3.6-35B-A3B |
+26 to +28% | +15 to +17% | 68-74% | 2026-08-08 |
Qwen3.8-27B |
+44 to +45% | +37 to +38% | 78-80% | 2026-08-15 |
Acceptance trades against draft length, so raising --draft-p-min buys acceptance and
loses tok/s. Ranges span the medians each shipped configuration was measured at; compare
a drafted number only against the plain arm of its OWN sweep, never against another
session's.
Both defaults are per-checkpoint. --draft-p-min is 0.5 on the 27B, 0.3 on the
35B-A3B and 0.7 on the 3.8-27B (Model::draft_p_min_default); --draft-max is 15 on the
two block drafters and 4 on the MTP head (Model::draft_max_default, keyed by drafter
kind). The 27B's target forward is expensive, so it wants short confident drafts; the
35B-A3B's is cheap enough to profit from drafting deeper at lower acceptance. On the
3.8-27B depth is the knob that matters and the floor barely does — across a 3x3 sweep the
floor moved throughput by at most 1.8% at fixed depth where depth moved it 12%. Passing
either flag, or draft.p_min / draft.max in a serve config, overrides it — and since
one server loads whichever checkpoint a request names, an explicit value pins one setting
for every checkpoint it serves; leave them unset to give each its own.
--draft-pause-margin stays a single shared 1.0. bun scripts/retune-draft.ts re-fits
these and prints recommendations; it never edits a default. See docs/decisions.md
"Speculative decoding".
--draft should reproduce --no-draft; bun scripts/spec-equivalence.ts checks that on
all three models in two modes — greedy, and sampled at a fixed seed. Greedy is the
gate. Sampled mode is a diagnostic and currently fails on healthy builds: the batched
verify forward reassociates its f32 sums differently from the single-token forward, so at
temperature a near tie can resolve to a different token, and the shipped 27B diverges on
the chat fixture at every seed tried. Read a sampled divergence only against a control
run of another checkpoint, and note the script's "a first-line fork means the sampler
stream" rule is a heuristic that mis-grades (TODO.md). What actually separates a near tie
from a sampler-stream bug is seed-dependence: a stream bug diverges at every seed.
xwen batch answers N chat items that share a prompt prefix: one JSON request on
stdin, one JSON response on stdout, progress lines on stderr. The shared prefix is
prefilled once and the KV cache snapshotted there; every item restores that snapshot
and prefills only its own tail, so nine questions about the same document cost one
prefill of it rather than nine. The checkpoint comes from the payload, not a flag.
xwen batch < request.json > response.json{
"model": "Qwen3.6-35B-A3B",
"defaults": { "max_tokens": 32 },
"items": [
{
"id": "sentiment",
"messages": [
{ "role": "system", "content": "You classify support email." },
{ "role": "user", "content": "<the email>\n\nOverall sentiment?" }
],
"schema": {
"type": "object",
"properties": {
"label": { "enum": ["positive", "mixed", "negative"], "include_score": true }
},
"required": ["label"],
"additionalProperties": false
}
}
]
}{
"model": "Qwen3.6-35B-A3B",
"items": [
{
"id": "sentiment",
"content": "{\"label\":\"mixed\"}",
"text": "{\"label\":\"mixed\"}",
"json": { "label": { "value": "mixed", "score": 0.5601 } },
"finish_reason": "stop",
"usage": { "prompt_tokens": 349, "cached_prefix_tokens": 317, "completion_tokens": 9 }
}
],
"stats": {
"shared_prefix_tokens": 317,
"snapshot_ms": 148.3,
"items": 1,
"prefill_tokens": 391,
"prefill_ms": 361.0,
"decode_tokens": 9,
"decode_ms": 92.0,
"load_ms": 3470.0,
"total_ms": 1828.0
}
}An item takes messages, an optional JSON schema (constrained decode), thinking
(false / true / a string to inject), an assistant prefill, max_tokens and
sampling; defaults sets any of them batch-wide. Batch sampling is greedy and
thinking is off unless a request says otherwise — both differ from the chat surface on
purpose (docs/decisions.md "Batch"). Per-item failures land as error on that item and
the rest of the batch still runs.
shared_prefix. A request-level string prepended verbatim to every item's first
message content, so a large shared document is spelled once on the wire instead of once
per item. Purely a body-size measure: the prompts (and answers, and scores) are
byte-identical to writing the document into every item, and the one-prefill snapshot
above works the same either way.
include_score. Annotate an enum or boolean property with include_score and the
item leaves the grammar path: xwen writes the JSON skeleton itself and picks each
field's value by scoring every allowed option against the model, reporting
{"value": …, "score": …} instead of a bare value ("all" adds the full scores
table and the escape). escape is the probability the model would rather have
written something NO option opens, measured over the whole vocabulary at the field's
choice point with formatting factored out — whitespace-led spellings of an allowed
value count as that value, pure-whitespace layout tokens count as neither side
(2026-08-11; before that it was raw opener mass and read ≈1 on a document's first
bare-literal field). v1 accepts a flat all-required object of enum/boolean fields and
refuses anything else by name.
XWEN_BATCH_NO_CACHE=1 runs every item from a reset cache — the A/B lever for what the
snapshot saves (35B demo: 1203 ms cached vs 1989 ms cold). The two arms decode the same
answers but not always the same bytes; see docs/decisions.md "Batch".
bun scripts/classify-demo.ts is the worked example: one support email classified along
nine taxonomies as nine batch items, with ground truth embedded, run on both checkpoints
through POST /xwen/v1/batch — against a server already running on the default port (or
--url), else one the script spawns and tears down itself. Running both checkpoints
through one server is a deliberate exercise of the checkpoint swap.
xwen serve runs an HTTP server over the engine. Routes: POST /v1/messages (+
/v1/messages/count_tokens) in the Anthropic dialect, POST /v1/chat/completions in
the OpenAI dialect, POST /xwen/v1/generate and POST /xwen/v1/batch (the native
surface), GET /v1/models, GET /health. xwen serve --init writes a commented
config template; every setting is also a flag.
One server serves every checkpoint (2026-08-11). --model/--model-size picks the
DEFAULT checkpoint; any request may name another one and the engine lazy-loads it,
imaging the live conversation out first (the same path an idle unload takes) — one
model resident at a time, always, and idle_unload applies to whichever is loaded.
On the wire a checkpoint has exactly one name: its full name (Qwen3.6-27B,
Qwen3.6-35B-A3B, Qwen3.8-27B) — 2026-08-14. The CLI's short aliases are a CLI
spelling and are refused by every API. Selection by surface:
GET /v1/modelslists each checkpoint once, the served one first, under exactly the string amodelfield selects it by — every listed id is selectable, which is the point of a listing. A served GGUF that is none of the official checkpoints (a custom--modelpath) leads the list under its file name, which is then its only id.- The compat dialects,
/v1/messages/count_tokensand/xwen/v1/batchall resolvemodelthe same way: absent or empty means the served file, this server's own id means the served file, another checkpoint's full name selects that checkpoint out of its own hub file, and anything else is a 400 in that dialect's error format naming the valid ones. The response echoes the canonical name of what answered, not the client's string. /xwen/v1/generatecarries no model field and always runs on the served file.
An SDK's own model id (gpt-4o, claude-…) is therefore a 400 rather than a silent
answer from whichever checkpoint happened to be the default; point the client at a full
name, or omit the field.
A custom GGUF answers under its own id and no other. If the served file is none of
the official checkpoints, it is served as its architecture's checkpoint (a startup line
says which) but reported and selected by its file name — and a request naming an
official checkpoint gets that checkpoint's real hub file, downloading it if need be,
rather than these weights under an official name. --model-size names the checkpoint a
file that says nothing about itself holds; a flag that contradicts a file that DOES say
is a startup error rather than a server that 500s every request.
Swapping costs a full model load (~3 s warm) plus losing the outgoing checkpoint's warm
KV slots, so interleaving checkpoints request-by-request is legal but slow. The on-disk
prefix cache stays bound to the default checkpoint; the other checkpoint runs without
it. A requested checkpoint (or its drafter) missing from the HF cache is downloaded
inside the request — a one-line notice in the server log; hf-hub's byte-level progress
bar goes to raw stderr, so under --tui it draws over the dashboard (TODO.md).
Request bodies are capped at 100 MB (real cost is judged in tokens by the queue and
context_length, not in bytes). context_length — default: the checkpoint's trained
256k window — is a ceiling, not an allocation: the KV cache starts at 8k positions and
grows on demand as a conversation lengthens, and an idle unload drops whatever it grew
to, so the next load starts small again. The one-shot CLI commands' --max-ctx works
the same way and defaults to 131072.
The default bind is loopback because with no api_key the server accepts every request;
set host = "0.0.0.0" (or --host) together with an api_key to serve the LAN.
Any change to model math re-runs the parity gate. It compares our forward pass against upstream llama.cpp on the identical GGUF, so it needs the oracle built once:
just init # fetch the llama.cpp submodule (pinned)
bash scripts/build-llamacpp.sh
bun scripts/parity-gate.ts # 35B-A3B, all tiers
bun scripts/parity-gate.ts --model-size 27b # 27b densedocs/parity.md is the runbook: tiers, floors, tap mapping, and the pinned oracle
commit.