Merge pull request #246 from TobiBu/test/f35-treecode-production-seam

test: measure the treecode production seam, closing F35's correctness half

Author: TobiBu

docs/refactor_audit_2026-08.md

@@ -56,7 +56,7 @@ against `26eac02` rather than read.
 | F20 | 1534 bare `Array`, 0 shaped | **3174 bare, 381 shaped** — stale in both directions |
 | F33 | `fmm_strict_run.py` 55% | **81%**; 70 statements left, inside two 500-line bodies |
 | F34 | `distributed/fmm.py` 19% | **34%** CPU, 83% on two cards |
-| F35 | production reaches `experimental/` | still: `_interaction_cache.py:1402` |
+| F35 | production reaches `experimental/` | **split**: layering settled by G.5 (`:1402` stays); the coverage half closed — 0% (invisible) -> **93%** |
 | F38 | 16 env reads outside `runtime/` | **2 files** |
 | F40 | 126 runtime-typecheck failures | **0** — closed; 126 -> 9 -> 6 -> green, and the cause changed entirely on the way |
 
@@ -951,7 +951,7 @@ Risk = probability the change breaks something. Effort = S (<1h), M (a day), L (
 | F32 | ~~`operators/real_harmonics.py:1120-1298`~~ **CLOSED `ab58c3d`** | dead-code | The Wigner reference family (`_wigner_D_complex`, `_real_wigner_rotation`, `_rotation_to_z_angles`, `_dehnen_real_basis_scale_diag_{multipole,local}`, and **6** public `*_wigner` names) is exported in `__all__` but reachable only from itself. **Corrected: 6, not 8** — the original count was wrong. `m2l_real_wigner` has zero references outside `__all__`. Grepped `jaccpot/`, `tests/`, `bench/`, `examples/`, `docs/` | `_real_wigner_rotation`'s own docstring says *"this exists to check them"* — and it never does. ~260 lines of independent 30-digit reference, unused **and unusable** (F39). The NaN-guard at `:1277-1292` was even applied to this unreachable copy | low | M | no → **section G** |
 | F33 | `runtime/fmm_strict_run.py` | test-gap | **55%** coverage (502 stmts, 227 missed), including most of `strict_run_v2` (428 lines, `:389`) and `_refresh_large_n_same_topology` (515 lines, `:890`). `runtime/fmm_strict_cap_profile.py` 35%; `runtime/fmm_autotune.py` 15%. **Re-measured 2026-08-21 on an A100, full suite: `fmm_strict_run.py` 55% (276/502, 226 missed) — unchanged, so GPU execution does not enter this lane and 2.6 did not unblock it. `fmm_strict_cap_profile.py` 49% (79/160), `fmm_autotune.py` 15% (18/119, unchanged).** **Substantially closed on CPU, 2026-08-27 — and none of it needed the card this row was waiting for.** `fmm_autotune.py` **15% -> 92%**, `fmm_strict_run.py` **55% -> 79%**, `fmm_strict_cap_profile.py` **35% -> 94%** (160 stmts, 5 missed). The last of those is `tests/unit/runtime/test_strict_cap_profile.py`: the module is a JSON catalogue of traversal capacities plus a selection policy, entirely host-side. Two behaviours it pins are worth more than the percentage. **The selection policy fails toward a larger cap, silently** — on an exact context-key miss it keeps `tree_mode` and `leaf` and takes the largest `max_pair_queue` under that prefix, because too small costs a retry and a recompile on the per-step hot path; nothing reports the fallback, so the tests assert on `_strict_profiled_context_key`. **Both halves are fail-open** — the loader wraps its whole body in `except Exception`, so a corrupt profile, an unreadable file and no file at all are indistinguishable, and the writer swallows too. That is the right choice and precisely because it is silent it now has a test saying so. Also pinned: caps ratchet upward only; a zero does not clobber a live cap; capacity-compatibility is asymmetric "fits inside" whose missing-keys-read-as-0 rule makes an unrecognised profile look *compatible*. **`fmm_strict_run.py` 79% -> 81%**, 2026-08-27 (83 -> 70 missed) via `tests/unit/runtime/test_strict_run_fail_fast.py`. Thirteen statements, and they are the entry contract: the four rejection guards and the profile-key accounting, all of which run *before* any device work. **The premise this row was filed under is wrong in the same way F27's was.** `_is_large_n_gpu_production_profile` checks four **config** values — `preset`, `tree_type`, `expansion_basis`, `execution_backend` — and no backend at all, so the strict lane's entry path opens on CPU with two constructor arguments. Third instance in this audit of "needs a GPU" meaning "needs the right profile". Two of the guards are contracts rather than implementation: `refresh_every != 1` raises because velocity-Verlet is endpoint-correct only when the state is refreshed every step, and fused mode **raises rather than silently degrading** when N is outside `JACCPOT_STATIC_STRICT_FUSED_PROFILE_SET` — a silent fallback there is a performance cliff nobody would notice. **What remains is 70 statements, and the percentage is the wrong way to read them**: they are the *interiors* of `strict_run_v2` (522 lines) and `_refresh_large_n_same_topology` (562), plus `_compiled_profile_from_prepared_state` (`:213-262`). Those need a real prepared state and a real stepping run, not another guard test — which is a different kind of work from everything F33 has absorbed so far. | The strict/refresh lane is the per-step hot path for science runs and carries the velocity-Verlet update. Half of it is unexercised | low | L | yes |
 | F34 | `distributed/fmm.py` | test-gap | **19%** coverage (520 stmts, 423 missed) when filed; **34%** on CPU as of 2026-08-27 (565/367), 83% on two cards. All 8 distributed suites skip below 2 devices (`device_count() < 2`) | Documented and expected, but it means the whole `distributed/` layer is unrefactorable in this pass — record it, do not touch it. **Run on two cards 2026-08-21, for the first time ever: `distributed/fmm.py` 19% -> 83% (431/522), and `19 passed, 5 failed, 1 skipped` in 48:55 across 25 tests in 9 files (this row's "24 tests across 10 files" is stale). Four of the five failures are a REAL cross-domain far-field bug, not uncalibrated tolerances — though **not for the reason first recorded here.** That argument ran: the omission baseline `norm(a_cross)/norm(a_full) = 0.008814` is below the measured error (0.018223 dehnen, 0.039704 bh), and an approximation cannot be worse than dropping the term. The principle is right; the baseline was not. It masks same-domain pairs by INPUT order (`i // per`), but `partition_for_devices` Morton-sorts first, so on this IC it measured the *cluster*-masked field. Masked by the real domain assignment it is **0.426192**, 48x larger, and the measured error is 0.043x of it — by that criterion, correctly applied, these failures would have read as a tolerance question. What does establish the bug: the far term is off by 124% of the direct sum over exactly the pairs it approximates, and 2 of 12 accepted far pairs physically overlap their own source region (worst true `(r_src+r_tgt)/d` = 1.193 against the 0.104 the MAC computed). The omission baseline bounds the WHOLE cross term while the far path handles only the accepted fraction, so it is too loose a denominator to clear the far field. The near limit (`theta_cross<=0.01`) is exact at 3e-6, so decomposition, halo import, cross P2P and reassembly are all correct and the error is entirely in the far pairs. Diagnostic: `bench/diagnose_cross_domain_far.py`. So this layer is not merely uncovered — four of its tests were correct and caught a defect the first time they ran.** **ROOT CAUSE AND FIX (2026-08-21):** the coarse tree's MAC extents bounded the frontier's centres of mass, not the particles behind them, so the cross MAC accepted pairs that were not well separated and the M2L was evaluated inside its own source region. In **yggdrax** (`distributed/let.py`), fixed by TobiBu/yggdrax#47 (`CoarseFrontier.radius` + `compute_tree_geometry(particle_radius=)`). jaccpot then dropped `theta_cross`, which existed only to compensate for it: aggL2 **0.018223 -> 0.000006** at the default, with the far path still engaged. Diagnosis and all measurements: `docs/distributed_cross_domain_far_diagnosis.md`; pinned in CPU CI by `tests/integration/test_distributed_cross_domain_far_extents.py`. | low | — | yes |
-| F35 | `runtime/_interaction_cache.py:892` | structure | Production `distributed/fmm.py` (`local_walk="treecode"`) reaches `experimental/treecode_far_near.py`. Already documented in STYLE_GUIDE §8 as *"the weakest-covered production option in the tree"*; my measurement confirms `treecode_far_near.py` at 0% | Not a new finding, but it is the one documented layering violation that is a *correctness* risk rather than a style one. Listed so the work plan does not disturb it **Still open, verified 2026-08-27:** `runtime/_interaction_cache.py:1402` still does `from jaccpot.experimental.treecode_far_near import ...` — production reaching `experimental/`. | — | — | yes → **section G** |
+| F35 | `runtime/_interaction_cache.py:892` | structure | Production `distributed/fmm.py` (`local_walk="treecode"`) reaches `experimental/treecode_far_near.py`. Already documented in STYLE_GUIDE §8 as *"the weakest-covered production option in the tree"*; my measurement confirms `treecode_far_near.py` at 0% | Not a new finding, but it is the one documented layering violation that is a *correctness* risk rather than a style one. Listed so the work plan does not disturb it. **Split and half-closed 2026-08-27.** The two halves had different answers and were being carried as one row. **Layering: already decided.** The import is `_interaction_cache.py:1402`, inside `_build_treecode_artifacts_strict_streamed`, and it stays. G.5 was answered 2026-08-20 — *left as-is, exposure accepted* — and `test_experimental_is_not_on_an_import_path.py` records the scope: the eager import graph is guarded, a lazy function-local import is the accepted exposure. Nothing here reopens that. **Correctness: closed.** The module measured 0% because `pyproject.toml` omitted it, and the stated reason — *"reached only from the distributed treecode lane, whose test skips below 2 devices, so CPU CI never enters it"* — is untrue on both counts. There is a second, **single-device** caller (`_interaction_cache.py:1058`, gated by `JACCPOT_STATIC_STRICT_FUSED_TREECODE_WALK`, an env var and not a device count), and the module runs on CPU in seconds — `tests/experimental/` already held three passing CPU tests for it, deselected by the marker. That is the **third** entry in that omit list whose reason did not survive checking; the comment above it documents the other two, which is a pattern rather than a coincidence. Un-omitted, and `tests/unit/runtime/test_treecode_production_seam.py` now exercises the production entry on every commit: **0% (invisible) -> 93%** (69 stmts, 2 missed). It pins the invariant the no-double-count argument rests on — *every far target is a leaf*, over theta = 0.3/0.5/0.9 with a vacuity guard — because an internal far target would be pushed down by the L2L cascade **and** counted again at the child's own far term, which is a wrong force in the lane whose tests do not run by default. | — | — | yes → **section G** |
 | F36 **✔`23ccc9d`** | 13 modules incl. `jaccpot/__init__.py`, `runtime/_fmm_impl.py`, `runtime/_interaction_cache.py`, `runtime/_nearfield_cache.py`, and 9 `__init__.py` | structure | Missing `from __future__ import annotations` (STYLE_GUIDE §1 says "make it 80"). 41 modules missing `__all__` | Mechanical, and the `_fmm_impl.py`/`_interaction_cache.py` cases are real modules, not just package inits | low | S | no |
 | F37 | `runtime/fmm_overrides.py:253` (`num_particles <= 8192`), `:806` (`n >= 262_144`); `runtime/_large_n_types.py:269, 415, 448` (`65536` three times); `solver.py:75` + `runtime/_fmm_impl.py:1152` (`upward_leaf_batch_size=2048` twice) | magic-number | Thresholds inline rather than in `runtime/fmm_constants.py`, which has 40 named ones | Small, but the repeated `65536` and the duplicated `2048` are the ones that will drift apart **✔ VERIFIED CLOSED 2026-08-27** — every constant this row cites is now named in `runtime/fmm_constants.py` (43 of them): `_JIT_TREE_CPU_SMALL_N_MAX = 8192`, `_CLASS_MAJOR_CPU_PARTICLE_THRESHOLD = 262144`, `_LARGE_CPU_PARTICLE_THRESHOLD = 65536`, `_NEARFIELD_BUCKETED_CPU_EDGE_CHUNK_LARGE = 2048`. The two policies that cross at 262144 are named apart with a comment saying why — which is what this row asked for. | low | S | yes (policy thresholds) |
 | F38 | `jaccpot/_env.py` vs STYLE_GUIDE §8 | structure | The guide says only `runtime/` reads env vars; `_env.py`'s docstring says any layer may. 16 reads outside `runtime/` | Contributors will follow whichever they read first **Re-measured 2026-08-27: 2 files** outside `runtime/` read `os.environ` (`operators/m2l_real_rot_scale.py`, `mutual/farfield.py`), against the 16 reads this row records. | — | S | no → **section G** |

pyproject.toml

@@ -252,9 +252,7 @@ omit = [
   # suite fully covers. Measured:
   #
   #   octree_fmm_uvwx.py    0%    reached only under `pytest -m experimental`
-  #   treecode_far_near.py  0%    reached only from the distributed treecode lane
-  #                               (`local_walk="treecode"`), whose test skips
-  #                               below 2 devices, so CPU CI never enters it
+  #   treecode_far_near.py  0%    the stated reason was UNTRUE -- see below
   #   treecode_walk.py    100%    NOT omitted -- see below
   #
   # `treecode_walk.py` is the pure-JAX reference that
@@ -267,7 +265,22 @@ omit = [
   # is measured until someone decides otherwise -- which is the right default,
   # since the failure this replaces was a glob quietly absorbing covered code.
   "jaccpot/experimental/octree_fmm_uvwx.py",
-  "jaccpot/experimental/treecode_far_near.py",
+  # `treecode_far_near.py` used to be omitted here, on the stated grounds that it
+  # is "reached only from the distributed treecode lane (`local_walk="treecode"`),
+  # whose test skips below 2 devices, so CPU CI never enters it". Untrue on both
+  # counts, and this is the THIRD entry in this list whose reason did not survive
+  # being checked -- the two above are the others.
+  #
+  #   * There is a second caller, and it is single-device:
+  #     `_interaction_cache.py:1058`, gated by the env var
+  #     JACCPOT_STATIC_STRICT_FUSED_TREECODE_WALK, not by device count.
+  #   * The module runs on CPU in seconds. `tests/experimental/` already had three
+  #     passing CPU tests for it, deselected by the `experimental` marker, and
+  #     `tests/unit/runtime/test_treecode_production_seam.py` now exercises the
+  #     production entry on every commit (audit F35).
+  #
+  # Removed, so the one production option that reaches `experimental/` is measured
+  # like any other module rather than being invisible by assertion.
   # `jaccpot/pallas/m2l_real_fused.py` used to be omitted here too, on the stated
   # grounds that it "has no interpret path, so CPU CI structurally cannot reach
   # it". That was untrue, and it is the same failure this list already documents