Add experimental MoE DFlash integration and evidence gates - #214
Closed
PhilipJohnBasile wants to merge 24 commits into
Closed
Add experimental MoE DFlash integration and evidence gates#214PhilipJohnBasile wants to merge 24 commits into
PhilipJohnBasile wants to merge 24 commits into
Conversation
…after market scan Adversarially-verified design for coordinating MTP self-drafting with DFlash block drafting: draft_source seam, soft-q through the existing acceptance lanes, DraftSourcePolicy controller with live cost EWMAs, no tree in v1, exactness invariants (one-forward independence, no sampling under mx.compile, truncation measurability), corrected performance envelopes, phased plan with go/no-go gates, and the drafter scan addendum (satgeze DSpark head-to-head, 35B-A3B Jun-19 retrain, official poolside Laguna-S drafter, public trainers, DDTree blocked on GDN targets). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s, full-cycle cost, Phase-0 upgrades) + baseline runner fixes Codex (0.145.0) adversarial review verdict FLAWED on implementation-readiness framing; architectural convergence with the existing phased plan. Accepted deltas folded into doc section 11: drafter KV-cache transaction requirement, full-cycle controller objective (lazy bonus + rejection repair included), per-position lane dispatch acknowledged as new verifier code, Phase-0 measurement upgrades (verify_ratio to M=16 + dispatch logging, dual AR baselines, tau histograms, enriched u(M), tap-cost prototype), grammar scoped out of the exactness claim, one-hot-first ordering (all three reviews now agree). Harness: load_draft signature compat shim + per-prompt MLX RNG seeding (seed was recorded but never applied). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…R baselines, T_V(M) curves on all three bodies Measured on the quiet M5 Max, 2026-07-30. z-lab 27B DFlash drafter accepts 4.31 tokens/cycle greedy (4/4 gate) and 3.92 at the product sampler on stock 27B-8bit, 24/24 coding prompts clean. True AR: 27B-8bit 18.0 tok/s (93% of bandwidth), 35B-A3B 112.4, Laguna 55.0. Verify-cost curves: MoE wide-block fear disproven (M16 tax 2.96x/3.84x, not ~10x); 27B-8bit shows the stock-qmm plateau at M10+ that the vk M8..16 port targets; measured block-8 beats block-16 e2e on 8-bit, confirming the kernel-regime map. Cost model cross-validates: curve-predicted MTP D3 2.6x vs 2.71x measured. Laguna + official poolside drafter now estimates 2.0-2.2x (~110-120 tok/s) at block 8. DFlash lane proceeds to Phase 2, one-hot first. Includes probe scripts, per-arm JSONs, SUMMARY.md, and the engine-runaway forensic snapshot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…emantics, add MoE varied-token curve + GDN cache finding Codex follow-up review raised the plan to SOUND_WITH_CONCERNS and audited the measurements. Corrections applied verbatim: - 'Gate GREEN' -> 'reference-arm build gate GREEN, release/performance gate pending'; scope banner added (all numbers are reference-impl or stock mlx_lm, not the MTPLX path; 4-bit body unmeasured). - tau defined as committed tokens per cycle (accepted + target replacement); block 16 = 1 anchor + 15 draft rows. - Corrected the 'reference drafts greedily' claim: the MLX backend samples the draft too (model_mlx.py:506,515), so the temp-0.6 arm is sampled-prefix behavior; only the greedy arm maps to our one-hot lane. - '24/24 clean' -> 24/26 validations (both JSON-tool cases failed). - AR baselines reported as run pairs; suite bounds stated (160-192 token caps, single seed, single ~1k context). - Varied-token T_V re-measure: Laguna M8/M16 tax rises 2.30->2.63 / 3.84->4.40 (+14%) vs identical rows, confirming the audit finding. - 35B varied-token curve blocked: stock mlx_lm GatedDeltaNet raises a conv spatial-dims error at small M and trim_prompt_cache does not restore GDN conv state. Documented as a Phase-2 task-1 datapoint: rejection rollback on hybrid-GDN targets cannot be a naive trim. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…8 wins, but reference DFlash lands BELOW MTP D3 on both quant widths Filled Codex's biggest measurement gap (the 4-bit body was never measured). - tau 4.29 (4-bit) vs 4.31 (8-bit): acceptance is quant-independent, so the gate result transfers across bodies. - Block 8 beats block 16 on both widths (+31% 4-bit, +17% 8-bit), independently validating the fixed-block-8 arm as task 3. - Measured AR: 4-bit 31.1 tok/s, 8-bit 18.0. Reference DFlash multipliers are therefore 1.55x/2.09x (B8) vs MTP D3's measured 2.23x/2.71x. Conditions are not perfectly matched, but same machine and stock kernels: the reference implementation does not beat MTP self-speculation. Any DFlash win must come from NAX 16-row kernels + engine machinery + block-8 routing, not from raw drafter acceptance. Strengthens 'implementation green, release gate pending' and keeps the MTP floor as the promotion bar. - Hybrid-GDN verify forwards are not measurable off-engine on any of the three bodies (conv window < kernel width after trim; trim_prompt_cache does not restore GDN conv state). The 8-bit curve completed only by accident of call ordering. Probe now records per-M failures instead of dying. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…surements Stock AR measured at 31.1 (4-bit) / 18.0 (8-bit) tok/s puts the reference DFlash arm at 1.55x/2.09x, below the measured MTP D3 multipliers. The expected end-state rows now hold only if NAX wide-row kernels favour 9-16-row blocks substantially more than MTP's 4 rows -- a bounded decision experiment to run before the build. Laguna (no MTP head, floor 1.0x) is flagged as the strongest remaining case for the lane. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…), plus the real finding and the corrected 4-bit curve The probe built a 480-token prompt and sliced verify rows from [500:516] -- empty. Every verify forward was shape (1,0); S=0 raised the conv error. The signature was there and was misread: only the real-tokenizer run failed, at all 16 M values, while synthetic-id runs completed. Probe fixed (200-rep prompt, tail slice, shape asserts) and the 27B-4bit curve now measures cleanly at M=1..16 on a hybrid-GDN body. The real defect found in the process: trim_prompt_cache is a SILENT no-op on hybrid caches (ArraysCache has no trim -> returns 0 without trimming attention either; measured max logit delta 1.17 after an 8-row verify + trim, no error). Production code never calls it and the reference port guards, so tau=4.29 is uncontaminated -- but rollback must use captured-state restore, never a trim. Corrected 4-bit curve: T_V(1)=32.2ms (matches AR 31.0 tok/s), T_V(4)=38.5, T_V(9)=79.8, T_V(16)=110.8; cliff at M=6 and a plateau from M=10. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… NO-GO, Laguna is the surviving case With the probe fixed, every body measures at M=1..16 with zero failures and T_V(1) tracks 1/AR throughout (the cross-check that these curves measure what they claim). The 35B varied-token curve — previously recorded as an unobtainable gap with a ~3.3-3.4x estimate — comes in at 3.00x at M16, essentially unchanged from identical tokens, so MoE routing diversity costs less than feared. The verdict follows directly from the curves: MTP D3 verifies 4 rows in the nearly-free region while DFlash B8 verifies 9 past the cliff, buying ~12% more committed tokens for ~2x the verify cost. Measured: MTP D3 2.32x/2.84x vs DFlash B8 1.55x/2.09x (4-bit/8-bit). With NAX on both arms, 2.50x vs 1.99x => R = 0.80, below the NO-GO line. Laguna-S remains the surviving case (no MTP head, floor 1.0x, official drafter, ~1.4-1.5x implied). Stale text describing the retracted GDN blocker removed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… same-suite head-to-head Closes both of Codex's INSUFFICIENT DATA objections (cross-suite comparison, untested small blocks). All arms measured on Fable-711-4bit with calibration_coding.jsonl, greedy, seed 0. MTP D5 57.9 tok/s > MTP D3 55.5 > DFlash B5 52.0 > MTP D8 51.8 > DFlash B4 50.8 > DFlash B6 43.5 > DFlash B8 40.1 > MTP D15 29.3. Against Codex's required-tau table (B4 3.02 / B5 3.84 / B6 4.57 / B8 5.06), measured tau is 2.785 / 3.051 / 3.173 / 3.367 -- every block misses its bar. Small blocks cut verify cost but lose acceptance faster; large blocks buy acceptance at more than its worth. No block size beats the model's own MTP heads. Also: drafter acceptance degrades on fine-tuned targets (tau 3.98 stock -> 3.367 Fable-711 at B8). Spin-off: depth 5 beats depth 3 (57.9 vs 55.5) and mtplx tune only searches D1-D3. Laguna estimate corrected upward to ~2.1-2.3x per Codex's arithmetic. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…shable Single-prompt runs showed depth 5 at 57.9 vs depth 3's 55.5 (+4.3%), flagged as needing repeats. With 3 prompts per run and interleaved ordering: depth 3 = [56.29, 56.18] (spread 0.11), depth 5 = [57.60, 55.79] (spread 1.80). The challenger's own spread exceeds the margin it won by; the original comparison paired its lucky sample against the incumbent's ordinary one. No tune-range win. The depth-6 dip IS reproducible (47.5 then 46.53, ~17% below trend at M=7 verify rows) and is now the one open anomaly worth chasing -- whatever costs 17% at M=7 may also shape M=8-9, where any external drafter must operate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… repeat-confirmed The dense NO-GO does not generalize. On Qwen3.6-35B-A3B (4-bit, NAX on, greedy/24 prompts/256 cap): DFlash B8 = 188.22 and 188.30 tok/s (tau 3.809, runs agreeing to 0.04%) vs MTP D3 = 100.4 mean and AR = 105.1. DFlash is 1.79x AR; MTP is 0.96x -- a net LOSS on this body, matching the published mlx-lm result that one MTP layer cannot predict expert routing. Mechanism: cheap verify rows (M9 = 2.10x vs dense 2.48x, expert gather amortizes), a 386M drafter (4x smaller than the 27B one), and a weak incumbent. None of those hold on dense bodies. Also found: z-lab's own MLX backend cannot load z-lab's newer drafters -- load_draft reads flat rope_theta/block_size while the June-retrained MoE drafter nests them under rope_parameters/dflash_config (three KeyErrors). Local clone patched for both formats; worth upstreaming. This likely explains why community MLX benchmarks all use the older, weaker 27B drafter. Revised recommendation: MoE first (drafter exists, Apache-2.0, no port needed), Laguna second, dense Qwen stays NO-GO. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eference default of 16 B4 191.1 / B5 193.8+194.2 / B6 193.7 / B8 188.2+188.3 / B16 147.6 tok/s. Tau rises monotonically with block size (2.95 -> 4.30) while throughput peaks at B5-B6 and falls 24% by B16 -- the same acceptance-vs-verify-cost tradeoff seen on the dense body, but with the peak far above the incumbent instead of below it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ocol mismatch; MoE direction holds, numbers restated Two of my errors, both caught by the audit: (1) the aggregation helper returned the first matching key anywhere in the nested JSON, printing identical MTP and AR values -- an impossible result I reported; the earlier MTP D3 table mixed AR values with wrong fields. (2) DFlash arms ran with thinking enabled while MTP/AR ran --disable-thinking. Corrected with thinking off, aggregating tokens over decode-seconds across 24 prompts: DFlash B8 217.1 tok/s (2.07x compare-AR), B5 207.6 (1.98x), MTP D1 111.0 (1.05x), AR 104.9-105.4, MTP D2 94.3 (0.90x), MTP D3 77.2 (0.74x). Disabling thinking raised DFlash 188->217 at B8 and flipped the block optimum back to B8. MTP on this MoE peaks at D1 and degrades with depth -- the earlier 'net loss' claim was reported off its worst depth; at its best it is a marginal 1.05x win. Remaining caveats recorded: compare-AR vs stock AR baseline, the DFlash arm uses no NAX (stock mlx_lm path), drafter quality confounded with architecture, 160-192 token caps, no token-stream equivalence yet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…GO holds but margin narrows to ~18% Every DFlash arm ran with thinking enabled while every MTP arm ran --disable-thinking, so the dense conclusion rested on the same mismatch just corrected for MoE. Rerun with thinking off: dense DFlash B5 58.6 tok/s (1.89x AR 31.0, up from 1.55x), B8 52.2 (1.68x); optimum shifts B8 -> B5. MTP D3 remains 2.23-2.32x, so the dense NO-GO survives -- but the margin is ~18%, not the ~44% first reported, and the earlier B4-B16 dense sweep ran under the wrong protocol throughout (ordering suspect, conclusion intact). Architecture split with matched protocol both sides: dense DFlash 1.89 vs MTP 2.23-2.32 (MTP wins ~18%); MoE DFlash 2.07 vs MTP 1.05 (DFlash wins ~2x). The split is real; the dense margin is tight enough that a finished 27B drafter could plausibly close it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Author
|
Hi @youssofal — I wanted to share this as an evidence-first experimental The encouraging result is that the staged DFlash path reaches about The branch includes the full research and correction trail, including the I would especially value your guidance on two things:
No expectation to review the 1.48M-line evidence diff as-is. I am happy to |
Owner
|
read the whole protocol. the blocker: the receipts are greedy-only (24 prompts, greedy, thinking disabled) and dflash=ar equality is 9/24 even there. our line for any speculative path is exact probabilistic speculative sampling at the product sampler (temp 0.6, top_p 0.95, top_k 20), the same bar the MTP path clears with probability-ratio acceptance + residual correction. a greedy-only lane can't ship as a product route, flag-gated or not. if the dflash lane grows correct variable-temperature acceptance, we're interested, and the evidence-gate structure you built is the right harness to prove it with. |
Contributor
Author
|
Implemented the requested product-sampler correctness lane in What is now in the branch:
CPU-only validation on the pushed commit:
I deliberately did not run a model/GPU benchmark because the machine's GPU is occupied by an active training job. Accordingly, the PR remains draft and the product/distribution gate remains HOLD: there is no real |
…ark drafter it is Forge correctly refuses DeepSeek-V4-Flash-0731 with backend_pending_mtp:deepseek-v4-mtp, but the entry misdescribes the architecture. V4 ships no MTP head: config's num_nextn_predict_layers=1 maps to zero tensors, and the checkpoint's mtp.0/1/2.* weights are a 3-stage DSpark block drafter (~20B, embed/lm_head shared, taps at layers 40-42, dspark_block_size=5). Any backend needs the DSpark block-draft contract, not the V3 MTP one. arch_id is left as-is (it is a public identifier in diagnostics, artifacts and tests); display_name, backend id, aliases and notes now state the truth, and the notes record the prerequisite: mlx-lm has no deepseek_v4 module at all, and V4 is not a V3 delta (MQA with joint 512-dim K=V, per-head q-norm, attention sinks, inverse-RoPE on the attention output, grouped low-rank O, per-layer compression ratios, frozen hash-routing layers, 4-copy Sinkhorn residual). tests/test_artifacts.py + tests/test_forge_cli.py pass unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…le) scaffold The compiled target-prefix control is only 13/24 byte-identical to generate_ar, which blocks the DFlash release; the divergence predates DFlash. This records a static analysis of why, plus a scaffold for the eventual measurement. Both are marked for what they are rather than what was hoped. The analysis proposed accumulation-order numerics as the leading hypothesis. External review superseded that framing: the supporting evidence came from the wide lane, whose eight-row verify falls through to the stock MoE and so differs on an axis staged-K1 does not have. A correction banner is added in place rather than rewriting, so the reasoning error stays visible. The scaffold does NOT run: it calls prepare_a3b_compiled_target_prefix with the wrong signature (TypeError after loading the model) and compares two stock-MoE prefill forwards with no attention_phase, no mx.compile and no shadow cache -- none of the axes in dispute. Guarded to fail immediately with an explanation, kept so the rewrite has something to diff against. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…y audited before the deciding measurement Its value is the timestamp: it lands before the measurement it governs, so no outcome can be rationalized afterward. It is a proposal, not a finished gate -- its own audit corrections are folded in and the contradicted original rules are replaced in place, not merely appended to. Separates two questions that byte-equality conflates: (A) is the drafter's declared q the distribution it sampled from -- the Leviathan-Chen guarantee and MTPLX's identity claim -- versus (B) is the compiled target path numerically identical to another implementation. Byte-equality vs generate_ar tests B. Receipts print separate A and B verdicts plus a final A-AND-B release verdict. Audit corrections applied, several against the document's own first draft: the L0-L6 ladder is NOT one-axis and a first-nonzero rung names the first EXPOSED difference (a provisional hypothesis, not a mechanism verdict); the proposed thresholds were looser than batch_equivalence's existing 1e-3 while claiming to be anchored to the strictest in-tree gate; rule-of-three needs zero flips; positions within a continuation are correlated; the A-side test as specified was confounded and must use the same compiled verifier as its control. Hand-verified rather than taken on trust: the suite genuinely cannot emit PASS (release_gate_pass hardcoded False at all four sites, deliberately fail-closed), both named A-blockers are absent, and -- load-bearing -- the whole-MoE dispatch docstring claims byte-comparability with generate_ar as its design intent. That demotes the numerics hypothesis: 13/24 reads as a possible defect in already-shipping compiled-target-prefix behaviour, not as evidence the gate measures the wrong thing. Also adds BACKLOG.md and, per the pre-registration's own same-push requirement, points the superseded gate statements in mtp-dflash-coordination.md and dflash_engine_suite.py at it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…gine evidence Enumerates the exact committed-token law for vocab-4 / block-3 by brute force over every draft draw and accept/reject outcome, across adversarial (p,q) families, with a planted defect proving the check can fail. It exercises the Leviathan-Chen ARITHMETIC and nothing else. External audit verdict: NOT EVIDENCE. Two reasons, both recorded in the file: 1. It never calls the production path. Oracle and planted defect are both local functions, so recomputing acceptance and residuals independently buys nothing -- independence of implementation is worthless without contact with the artifact. A1-A4 in the pre-registration all remain unmet. 2. Its conditioning 'fix' was a weakening. The first version asserted an unconditional two-token joint, it failed, and the assertion was conditioned until it passed. That was backwards: in a real decoder a rejection does not end generation, so over the emitted stream the unconditional law holds. The assertion was right and the ORACLE was wrong, terminating at the first rejection instead of continuing into the next cycle. Filed under tests/prototypes/ with a non-test filename so default collection does not count 47 passing cases as coverage for a claim they do not support. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… claimed were folded in Post-push verification caught that commit bdec012 overclaimed: it said the audit corrections were applied in place, but section 4.2 still asserted the false strictest-gate anchor, section 5 still issued definitive mechanism verdicts, section 7 still said 'two verdicts never one', section 9 still said H1 removes the B blocker, and the backlog and divergence doc repeated the one-axis/names-the-mechanism claims plus a link to the oracle's old path. All corrected at the point of claim now: 4.2's threshold table is explicitly marked looser than batch_equivalence's 1e-3 and unsettled; the sample plan is marked superseded (zero flips, clustered positions, B6 non-certifying); 5's table header says every mechanism verdict is provisional; 7 prints separate A and B verdicts plus a final A-AND-B; 9 states H1 does not remove the B blocker. The irony is noted: the commit that landed the 'correct the claim where it is made' discipline itself corrected claims only in an appendix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
|
verdict: not taking this lane into the product tree — closing as research-complete rather than wrong. the honesty of the red gate is exactly the discipline we want here, and it's also the answer: the product bar for a draft lane is committed-sequence identity with the authoritative model, and by your own receipts DFlash sits at 9/24 (wide) with the staged lane inheriting its MTP-K1 control's 13/24 vs AR. an experimental lane that can't clear its own gate stays research, and a 1.48M-line branch isn't reviewable or carryable in-tree regardless of verdict. what's preserved and valuable: the negative result, the adversarial protocol corrections, and the corrected artifact-contract evidence — all linkable from this PR. if a future drafter clears spec==AR at depth on real prompts, open a fresh code-and-tests-only PR (the split you already offered) and that gets a real review. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Status
Draft and experimental. This is not a release candidate: the full correctness
gate remains red, and DFlash stays disabled unless
MTPLX_DFLASH_DRAFT=1is set.What changed
commit-only companion-cache handling
mode, execution switches, and aggregation
depth, forward-call counts, compiled route, and fallback count
forge_provenance.mtp_contractmetadata while preservingcanonical top-level
mtp_contractprecedencethat led to the current decision
Corrected artifact-contract evidence
Target:
Youssofal/Qwen3.6-35B-A3B-MTPLX-Optimized-SpeedDrafter:
z-lab/Qwen3.6-35B-A3B-DFlash@f181eece646affea2c38b2765f1aaa01a9734ccd24 coding prompts, greedy, thinking disabled, 112-token cap:
The staged DFlash lane introduces no additional divergence relative to its MTP
K1 control, but that control still diverges from authoritative AR on 11/24
prompts. The PR intentionally preserves that failed gate rather than redefining
success around MTP parity.
Validation
passed, with one expected skip
pre_norm/post_norm/localcontractReview scope note
This branch contains the complete 17-commit research lineage: 83 files and
roughly 1.48 million added lines, mostly raw benchmark receipts. They are
included so the negative results and protocol corrections remain auditable.
If a smaller review surface is preferred, I can follow with a code-and-tests
only branch while keeping the evidence branch linked.
Remaining blocker
Fix or explain the authoritative AR versus compiled M1/M2 target-prefix stream
divergence before considering enablement or release.