Skip to content

README.md

Latest commit

 

History

History
120 lines (97 loc) · 4.73 KB

README.md

File metadata and controls

120 lines (97 loc) · 4.73 KB
Raw

Pipeline Eval — Published Scorecards

Per-release pipeline evaluation results for ForgeDock. Every scorecard here was produced by the eval harness (#1285) running the seeded benchmark corpus (#1284) and aggregated by scripts/bench-scorecard.mjs. Scorecards are published following the model-release upgrade playbook (docs/articles/model-release-playbook.md).

Current baseline

Field Value
Scorecard claude-sonnet-4-6-2026-07-04.json
Model claude-sonnet-4-6
Corpus seed-v1
Run date 2026-07-04
One-shot success rate — (pending harness)
Human-intervention mean — (pending harness)

Note: The baseline above is a format placeholder. The harness (#1285) and CI gate (#1286) must be operational before real scorecard values can be populated. When the first real run completes, update this table and remove this note.

Model upgrade: Default model was updated from claude-sonnet-4-5 to claude-sonnet-5 in PR #1441 (issue #1248, committed 2026-07-04). The --model flag and FORGEDOCK_MODEL env var override the default at runtime. No claude-sonnet-5 scorecard has been published yet — see the Published results table below.

Published results

Run ID Model Corpus Date One-shot success Human interventions/issue Scorecard
claude-sonnet-5-2026-07-04 claude-sonnet-5 seed-v1 2026-07-04 pending pending not yet published
claude-sonnet-4-6-2026-07-04 claude-sonnet-4-6 seed-v1 2026-07-04 pending pending scorecard

Scorecard schema

Every file in published/ is a JSON object with the following fields:

{
  // Run metadata
  "run_id": "claude-sonnet-5-2026-07-04",          // <model-slug>-<YYYY-MM-DD>
  "model": "claude-sonnet-5",                       // exact model identifier
  "claude_code_version": "1.x.x",                  // Claude Code version used
  "corpus_version": "seed-v1",                     // corpus tag from docs/eval/corpus/
  "run_date": "2026-07-04",                        // ISO 8601 date
  "n_issues": 20,                                  // total corpus issues attempted
  "n_completed": 20,                               // issues that reached a terminal state

  // Primary metrics — regression gate checks these
  "one_shot_success_rate": 0.75,                   // fraction of issues merged without human intervention
  "human_intervention_count": {
    "mean": 0.3,                                   // mean interventions per issue
    "median": 0.0,
    "p90": 1.0
  },
  "wall_clock_seconds": {
    "mean": 3420,                                  // mean wall-clock per issue (seconds)
    "median": 3180,
    "p90": 6000
  },
  "cost_per_issue_usd": {
    "mean": 0.85,
    "median": 0.72,
    "p90": 1.40
  },

  // Per-issue breakdown (array)
  "issues": [
    {
      "issue_id": "corpus-001",
      "difficulty": "easy",                        // easy | medium | hard
      "category": "bug",                           // bug | feature | refactor | perf | docs
      "outcome": "merged",                         // merged | needs-human | failed | timeout
      "human_interventions": 0,
      "wall_clock_seconds": 2980,
      "cost_usd": 0.63
    }
    // ... one entry per corpus issue
  ],

  // Comparison against previous baseline (populated at publish time)
  "vs_baseline": {
    "baseline_run_id": null,                       // null if this is the first run
    "one_shot_success_rate_delta": null,
    "human_intervention_delta": null,
    "regressions": []
  }
}

Directory layout

docs/eval/
  README.md                     # this file — published results index
  corpus/
    seed-v1.json                # seeded benchmark corpus (20 issues, graded difficulty)
  published/
    <run-id>.json               # published scorecard per model release
  runs/
    <run-id>/
      raw.json                  # raw per-issue outcome data from the harness
      scorecard.json            # aggregated scorecard (bench-scorecard.mjs output)

Raw run data in runs/ is the source of truth. Published scorecards in published/ are the canonical record referenced in the playbook and README.

Upgrade decision table

One-shot success rate delta Human intervention delta Decision
> -5pp <= +0.5 Adopt
-5pp to -10pp any WARN — investigate
< -10pp any BLOCK — regression issue required
any > +0.5 BLOCK — regression issue required

Full decision procedure: docs/articles/model-release-playbook.md.