Merge pull request #6 from nonlinear-xyz/feat/factory-verification-score

Feat/factory verification score

Author: nonlinearnishu

.factory-check.json

@@ -0,0 +1,3 @@
+{
+  "ignorePaths": ["**/__tests__/**", "check/rules/**"]
+}

.github/workflows/factory-conformance.yml

@@ -0,0 +1,62 @@
+# Factory conformance — the kit dogfooding its own checker.
+#
+# This is NOT the shipped template (templates/factory-conformance.yml, which
+# npx's the published package). The kit IS the package, so it runs the checker
+# from source: npm ci → build → node bin/factory-kit-check.js. Same contract —
+# sticky scorecard, gate only on a new critical — proven against itself.
+name: Factory conformance
+
+on:
+  pull_request:
+
+permissions:
+  contents: read
+  pull-requests: write
+
+jobs:
+  conformance:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0 # full history so the base ref resolves for the delta
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+          cache: npm
+
+      - run: npm ci
+      - run: npm run build
+
+      - name: Run factory-kit-check on itself
+        run: |
+          set -o pipefail
+          node bin/factory-kit-check.js . \
+            --base "origin/${{ github.base_ref }}" --md > scorecard.md \
+            && echo "GATE=pass" >> "$GITHUB_ENV" \
+            || echo "GATE=block" >> "$GITHUB_ENV"
+          cat scorecard.md
+
+      - name: Post sticky scorecard
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const fs = require('fs');
+            const body = fs.readFileSync('scorecard.md', 'utf8');
+            const marker = '<!-- factory-kit-check:scorecard -->';
+            const { owner, repo } = context.repo;
+            const issue_number = context.issue.number;
+            const { data: comments } = await github.rest.issues.listComments({ owner, repo, issue_number });
+            const existing = comments.find((c) => c.body && c.body.includes(marker));
+            if (existing) {
+              await github.rest.issues.updateComment({ owner, repo, comment_id: existing.id, body });
+            } else {
+              await github.rest.issues.createComment({ owner, repo, issue_number, body });
+            }
+
+      - name: Enforce delta gate
+        if: env.GATE == 'block'
+        run: |
+          echo "::error::This PR introduces a new critical-severity finding. See the conformance scorecard on the PR."
+          exit 1

CLAUDE.md

@@ -35,6 +35,7 @@ Skills auto-load on the `factory-*` namespace:
 - **factory-ci** — single `ci.yml` merge gate, ephemeral PR DB, coverage floor, `anthropics/claude-code-action@v1` as required check, branch protection
 - **factory-commits** — Conventional Commits + required Linear issue ID; commitlint config, Husky hook, opencommit wiring
 - **factory-pitfalls** — flat cross-skill index of Failure mode blocks + process-level pitfalls without a skill home
+- **factory-verification** — four-tier eval spectrum (CLI rule / test / agent / human gate), evals-graduate-downward promotion pipeline, banded conformance score with severity-aware coverage disclosure, delta-gated GitHub Action that guards the PR boundary not the inner loop
 
 ## Specialist subagents (callable via Agent tool)
 
@@ -49,6 +50,7 @@ Skills auto-load on the `factory-*` namespace:
 - **llm-workflow-engineer** — LangGraph workflows, RAG, structured output, streaming
 - **security-engineer** — threat modeling, AI-code review, sensitive-data handling
 - **code-reviewer** — PR review against factory-pitfalls digest
+- **verification-engineer** — designs the verification strategy for a change (blast radius → required eval tiers → gaps); sister to `code-reviewer` (finds defects) and generalization of `db-migration-engineer`'s verify-stage to all changes
 
 
 ## Slash commands (auto-loaded into `~/.claude/commands/`)

README.md

@@ -60,7 +60,7 @@ factory-kit-check                 # check the current directory
 factory-kit-check ../some-repo    # check another repo
 ```
 
-It is **read-only** — it reads and judges, it never writes to your code. It exits non-zero on any `critical`/`high` finding, so it drops straight into a pre-push hook or CI step.
+It is **read-only** — it reads and judges, it never writes to your code. By default it exits non-zero on any `critical`/`high` finding; with `--base <ref>` it gates on the *delta* instead (see the conformance gate below). Run it on demand locally, or as the PR-boundary Action — not as a blocking inner-loop hook (see `factory-verification.md §Guardrails at the boundary`).
 
 Why deterministic (not an LLM): the rules are cheap, reproducible, and cost nothing per run, so you can run them on every save without thinking about it. Each rule is greppable code with a documented heuristic and a citation — no black box.
 
@@ -77,15 +77,34 @@ Why deterministic (not an LLM): the rules are cheap, reproducible, and cost noth
 
 Detection is regex/line-heuristic in v0 — precision-first, tuned against real repos. Rules are language-tagged (TS today; the seam for a Python `ast` sidecar is in place) and the report footer lists how many known pitfalls are not yet covered, so the tool never implies full coverage.
 
+### Score & coverage
+
+The report leads with a **banded verdict** — `pass` / `warn` / `fail`, severity-gated (one critical ⇒ fail; one high ⇒ warn). The precision of the verdict matches the precision of the instrument: a band, never a false-precision number. Alongside it, **coverage** — how many of the ~40 named pitfalls are machine-checked — with a severity-aware caveat naming any critical-class pitfall that has *no* rule, so a passing grade can't impersonate "nothing critical is wrong." The model lives in `check/score.ts` (pure, tested); the doctrine is `factory-verification.md`.
+
+### Conformance gate (GitHub Action)
+
+```sh
+factory-kit-check . --base origin/main --md    # render the PR scorecard
+npx @nonlinear-labs/factory-kit add-ci          # drop the Action into a repo
+```
+
+The `Factory conformance` Action posts one sticky scorecard comment per PR and fails the check **only on a newly-introduced critical** — it gates the *delta*, not the absolute, so pre-existing debt never blocks a PR and the developer's inner loop is never interrupted. Tighten to new-highs with `"gateOnHigh": true` in `.factory-check.json`. With `--base`, the CLI exit code follows this delta gate; without it, the legacy whole-repo critical/high gate applies, so it still drops into a simple CI step.
+
 ### Configure
 
-Drop a `.factory-check.json` in the repo to disable a rule:
+Drop a `.factory-check.json` in the repo to disable a rule, ignore paths, or tighten the gate:
 
 ```json
-{ "disabledRules": ["update-delete-no-where"] }
+{
+  "disabledRules": ["update-delete-no-where"],
+  "ignorePaths": ["**/__tests__/**"],
+  "gateOnHigh": false
+}
 ```
 
-The report prints how many rules are disabled. If you find yourself disabling more than a handful, the rule design is wrong — open an issue, don't paper over it.
+- `disabledRules` blinds a rule across the whole repo. The report prints how many are disabled; if you disable more than a handful, the rule design is wrong — open an issue, don't paper over it.
+- `ignorePaths` (fast-glob, relative to repo root) excludes paths from the walk — e.g. a rule suite's own test files, fixtures, and rule-definition sources, which contain the very patterns they detect (as test bait or as detection heuristics). Path exclusion scopes *where* rules apply without blinding the rule itself. The kit ships this exact config to skip its own `__tests__/` tree and `check/rules/`.
+- `gateOnHigh` tightens the PR delta gate to also block a newly-introduced high (default: new-critical only).
 
 ### From source
 
@@ -122,6 +141,7 @@ Synthesized cross-build conventions. Auto-loaded by Claude Code from `~/.claude/
 | `factory-ci` | Single `ci.yml` merge gate, ephemeral PR DB, coverage floor, Claude Code reviewer as required check |
 | `factory-commits` | Conventional Commits + required Linear-ID; commitlint config |
 | `factory-pitfalls` | Flat cross-skill index of Failure mode blocks + process-level pitfalls without a skill home |
+| `factory-verification` | Four-tier eval spectrum, evals-graduate-downward pipeline, banded conformance score with coverage disclosure, delta-gated conformance Action |
 
 ### Agents (specialist subagents)
 
@@ -139,7 +159,8 @@ Each is a Claude Code subagent file (YAML frontmatter + markdown body). Callable
 | `data-pipeline-engineer` | CSV ingestion, Python services, simulation envelopes |
 | `llm-workflow-engineer` | LangGraph workflows, RAG, structured output, streaming |
 | `security-engineer` | Threat-model a feature, audit AI-generated code, sensitive-data handling |
-| `code-reviewer` | PR review against `factory-pitfalls.md` checklist |
+| `code-reviewer` | PR review against `factory-pitfalls.md` checklist — finds defects |
+| `verification-engineer` | Designs the verification strategy for a change (blast radius → eval tiers → gaps); sister to `code-reviewer`, generalizes the migration verify-stage |
 
 ### Slash commands
 

agents/verification-engineer.md

@@ -0,0 +1,78 @@
+---
+name: verification-engineer
+description: Use to design the verification strategy for a change — what would prove it correct, given its blast radius, and what is currently unverifiable. Read-only — outputs a verification plan and a gap list, not a review and not code. Sister to `code-reviewer` (which finds defects in a diff) and the generalization of `db-migration-engineer`'s verify-stage discipline to all changes. Carries the four-tier eval spectrum and the score model from `factory-verification.md`. Invoke before merging a nontrivial change, when onboarding a risky area, or when asking "how would we know this is right?"
+tools: Read, Grep, Glob, Bash
+model: sonnet
+---
+
+You are the **verification-engineer** subagent. Your job is to design *how a change would be proven correct* — and to name what currently can't be. You **do not write or edit code, and you do not hunt for defects** (that's `code-reviewer`). Your output is a verification strategy: which tiers of the eval spectrum apply, which are satisfied, and where the gaps are. Read `~/.claude/skills/factory-verification.md` first.
+
+The distinction from `code-reviewer` is load-bearing:
+- `code-reviewer` answers *"what's wrong in this diff?"* — it finds defects.
+- **You** answer *"what would establish this diff is right, and what part of that is missing?"* — you design the proof.
+
+A change can pass a defect review and still be unverified — no test exercises the new branch, no human can explain the tricky part, nothing checks the convention it relies on. That gap is your subject.
+
+## How to think (in order)
+
+1. **What's the scope and the blast radius?** Identify the change (files / diff range / feature). Then classify its blast radius — this sets how much verification is warranted:
+   - **Cosmetic** — copy, styling, docs. Reversible, no behavior change.
+   - **Behavioral** — logic, control flow, a new code path.
+   - **Data-shape** — schema, migration, anything that changes persisted state.
+   - **Destructive / irreversible** — prod data mutation, deletion, anything you can't trivially undo.
+
+   If scope is ambiguous, ask. Don't strategize the whole repo by default.
+
+2. **Map the change onto the four-tier spectrum.** For each tier, state what it *would* take to verify this change, and whether that's present:
+   - **CLI rule** (static inspection): does `factory-kit-check` cover the conventions this change touches? Run it (`--diff`/`--base`) and read the band + coverage. Note any relevant pitfall that sits in `UNCOVERED` — that's a known blind spot for this change.
+   - **Test** (execution): is there a test that exercises the new behavior — not just that it compiles, but that it does the right thing? New branch with no new test = a gap. Check `__tests__/` co-location.
+   - **Agent** (judgment): is there a question here only judgment answers — right abstraction, missing case, security reasoning — that no rule or test will catch?
+   - **Human gate** (comprehension): is there a part of this change a human must be able to explain to own it? Flag the spots where "the checks passed" is not enough.
+
+3. **Match required tiers to blast radius.** Cosmetic needs tiers 1–2 at most. Behavioral needs 1–3. Data-shape and destructive need all four, human gate last — for destructive prod writes, defer to `db-migration-engineer`'s preflight/mutate/verify/rollback runbook; your job there is to confirm that discipline is being followed, not to re-derive it.
+
+4. **Find the gaps — this is the core output.** The hardest and most valuable findings are about verification that *should* exist and doesn't:
+   - New behavior with no test exercising it.
+   - A convention the change depends on that nothing enforces (a silent verification gap — name it; it may be a rule-promotion candidate).
+   - A tricky section no human has signed off on comprehending.
+   - A critical-class pitfall in the change's area that `factory-kit-check` doesn't cover.
+
+5. **Name rule-promotion candidates.** If a gap is decidable by static inspection and likely to recur, say so explicitly: it should graduate into a `check/rules/` rule (see `factory-verification.md §Evals graduate downward`). This feeds the factory's backlog.
+
+6. **Don't gold-plate.** Match verification cost to blast radius. Demanding an E2E test for a copy change is the same failure as shipping a migration with no rollback — verification spent out of proportion to the risk. Say what's *enough*, not the maximum.
+
+## Output format
+
+```
+## Change under verification
+- Scope: <files / diff range; count>
+- Blast radius: <cosmetic | behavioral | data-shape | destructive>
+
+## Verification strategy (by tier)
+- CLI rule:   <covered / partial / gap> — <what factory-kit-check says; relevant UNCOVERED items>
+- Test:       <covered / partial / gap> — <what exists; what behavior is unexercised>
+- Agent:      <needed / not needed> — <the judgment question, if any>
+- Human gate: <needed / not needed> — <the section that must be comprehended, if any>
+
+## Gaps (what is currently unverifiable)
+1. <gap> — <why it matters> — <which tier would close it>
+
+## Rule-promotion candidates
+- <gap decidable by static inspection that should become a check/rules/ rule>, citing the factory-pitfalls.md / skill section it would enforce
+
+## Verdict
+<the minimum verification this change needs before merge, proportional to blast radius — and what's missing from it today>
+```
+
+## What you do NOT do
+
+- **Don't write or edit code, tests, or rules.** You design the strategy; another agent or the contributor implements it.
+- **Don't hunt for defects.** Bugs in the diff are `code-reviewer`'s job; if you spot one in passing, note it and hand it off — don't pivot into a line-by-line review.
+- **Don't strategize the whole repo by default.** Honor scope.
+- **Don't demand maximum verification.** Proportional to blast radius — over-verifying a cosmetic change is as wrong as under-verifying a destructive one.
+- **Don't treat green checks as sufficient.** Your value is naming what the passing checks *don't* prove.
+- **Don't run non-read-only commands.** `factory-kit-check`, `git diff`, test runs are fine; never mutate the repo or any database.
+
+## When the request is too small for this framework
+
+If the user asks "does this one-line copy fix need anything?" answer directly: no, tier 1 covers it. The framework is for behavioral-and-above changes where the verification strategy is a real decision.

bin/factory-kit-check.js

@@ -24,9 +24,26 @@ async function main() {
     return;
   }
 
-  // First positional arg is the target repo; default to cwd.
-  const target = process.argv[2] ? path.resolve(process.argv[2]) : process.cwd();
-  const exitCode = await run({ targetDir: target });
+  // Parse: one optional positional (target repo, default cwd) plus flags.
+  //   --base <ref>   diff against <ref>: adds the PR delta + new-critical gate
+  //   --diff         scope reported findings to files changed vs --base
+  //   --json         emit machine-readable score+delta
+  //   --md           emit the Markdown scorecard (for the PR comment)
+  const argv = process.argv.slice(2);
+  let target = null;
+  let base;
+  let diffOnly = false;
+  let format = "term";
+  for (let i = 0; i < argv.length; i++) {
+    const a = argv[i];
+    if (a === "--base") base = argv[++i];
+    else if (a === "--diff") diffOnly = true;
+    else if (a === "--json") format = "json";
+    else if (a === "--md") format = "md";
+    else if (!a.startsWith("--") && target === null) target = a;
+  }
+  const targetDir = target ? path.resolve(target) : process.cwd();
+  const exitCode = await run({ targetDir, base, diffOnly, format });
   process.exit(exitCode);
 }
 

bin/factory-kit.js

@@ -141,9 +141,38 @@ function main() {
   console.log("Restart Claude Code to pick up the new skills.");
 }
 
+// `add-ci [targetRepo]` — drop the Factory conformance GitHub Action into a
+// repo's .github/workflows/. The Action is a per-repo file, not a ~/.claude
+// symlink, so it needs a copy, not a link. Idempotent: skips an existing file.
+function addCi(targetArg) {
+  const targetRepo = targetArg ? path.resolve(targetArg) : process.cwd();
+  const src = path.join(KIT_ROOT, "templates", "factory-conformance.yml");
+  if (!fs.existsSync(src)) {
+    console.error("factory-kit: templates/factory-conformance.yml missing from the kit.");
+    process.exit(1);
+  }
+  const dstDir = path.join(targetRepo, ".github", "workflows");
+  const dst = path.join(dstDir, "factory-conformance.yml");
+  ensureDir(dstDir);
+  if (fs.existsSync(dst)) {
+    console.log(`  skip   ${path.relative(targetRepo, dst)} (already exists — remove it to re-add)`);
+    return;
+  }
+  fs.copyFileSync(src, dst);
+  console.log(`  add    ${path.relative(targetRepo, dst)}`);
+  console.log("");
+  console.log("Next: commit the workflow, then add 'Factory conformance' to your");
+  console.log("branch-protection required checks so the new-critical gate is load-bearing.");
+}
+
 try {
-  main();
+  const cmd = process.argv[2];
+  if (cmd === "add-ci") {
+    addCi(process.argv[3]);
+  } else {
+    main();
+  }
 } catch (err) {
-  console.error(`factory-kit install failed: ${err.message ?? err}`);
+  console.error(`factory-kit failed: ${err.message ?? err}`);
   process.exit(1);
 }

check/__tests__/config.test.ts

@@ -13,5 +13,13 @@ describe("loadConfig", () => {
   it("defaults to an empty set when no config file exists", async () => {
     const cfg = await loadConfig(fixtures("clean"));
     expect(cfg.disabledRules.size).toBe(0);
+    expect(cfg.gateOnHigh).toBe(false);
+    expect(cfg.ignorePaths).toEqual([]);
+  });
+
+  it("reads the kit's own ignorePaths (fixtures excluded)", async () => {
+    const kitRoot = fileURLToPath(new URL("../..", import.meta.url));
+    const cfg = await loadConfig(kitRoot);
+    expect(cfg.ignorePaths).toContain("**/__tests__/**");
   });
 });

check/__tests__/engine.test.ts

@@ -49,6 +49,17 @@ describe("engine over fixtures", () => {
   });
 });
 
+describe("walk honors extra ignore patterns", () => {
+  it("excludes files matching an ignorePaths glob, scans them without it", async () => {
+    const root = fixtures(""); // the fixtures/ dir, with violations/ + others under it
+    const withIgnore = await walk(root, ["**/violations/**"]);
+    expect(withIgnore.files.some((f) => f.path.includes("violations/"))).toBe(false);
+
+    const without = await walk(root);
+    expect(without.files.some((f) => f.path.includes("violations/"))).toBe(true);
+  });
+});
+
 describe("run (end-to-end, including report)", () => {
   afterEach(() => vi.restoreAllMocks());