feat(runner): verbs-only bench run with --shape and --run-key

[devin-ai-integration]

Summary

The old surface duplicated a benchmark's identity across four places — the .bench.ts file, package.json, the CI workflow, and imperative bench create benchmark / bench create run commands — and drifted between them. This makes bench verbs-only: the benchmark is declared in the file and materialized when you run it, and a run is opened as a side effect. There is no create.

# before
bench create benchmark sandbox-burst-local --name 'Sandbox burst TTI (local)' --kind sandbox
ID=$(bench create run --benchmark sandbox-burst-local)
bench run tti.bench.ts --benchmark sandbox-burst-local --provider e2b --run-id "$ID"

# after
bench run benchmarks/sandbox/tti.bench.ts --shape burst --provider e2b --run-key "$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT"

Two new flags on bench run:

• --shape <name> — a bench file declares named shapes, each swapping in its own platform identity (slug/name, optional kind) and a stable knob (staggerDelayMs) while reusing the same task and participants. Scale knobs (--iterations/--concurrency) stay on the CLI, so no shape sets a value only to have it overridden. This is what collapses the per-shape slug/name/knob triple out of package.json and CI.

  shapes: {
    burst:     { slug: 'sandbox-burst-local',     name: 'Sandbox burst TTI (local)' },
    staggered: { slug: 'sandbox-staggered-local', name: 'Sandbox staggered TTI (local)', staggerDelayMs: 200 },
  }

• --run-key <key> — sibling processes passing the same key get-or-create one shared run (per org + benchmark) instead of each opening its own, so parallel provider jobs land in one directly-comparable run. Each process registers only the participants it actually runs. Different shapes are different slugs, so one key value yields one run per shape — no per-shape key juggling. The binding is permanent, so a workflow re-run varies the key via GITHUB_RUN_ATTEMPT.

  if (args.runKey) {
    const { run } = await client.createRun(slug, { name, runKey: args.runKey }); // get-or-create
    for (const p of available) await client.upsertParticipant(slug, run.id, p.name, { totalTasks });
  } else { /* unchanged fresh-run path */ }

Benchmark materialization stays honest: a bare --benchmark X that merely retargets reporting at a benchmark this file doesn't name does not upsert (which would rename it); --shape/--name/the file's own slug do.

Call sites collapsed

• package.json: every --slug … --name … [--stagger-delay-ms 200] triple → --shape burst / --shape staggered.
• .github/workflows/sandbox-tti-benchmarks.yml: dropped the whole create-runs job; the provider matrix now runs bench run … --shape … --run-key "$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT" directly.

--slug remains a working alias for --benchmark.

Depends on

computesdk/benchmarks-platform#53 — POST /runs get-or-create by runKey.

Checks

pnpm typecheck clean; runner suite 65 passing, client suite 118 passing; all packages build.

Link to Devin session: https://app.devin.ai/sessions/82de2e18fab54e7bba63034865f0cb6c
Requested by: @HeyGarrison

---

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[open-cla]

Contributor License Agreement

All contributors are covered by a CLA.

[superagent-security]

Superagent found 2 supply chain issue(s)

[superagent-security]

Superagent found 1 security concern(s).

[devin-ai-integration]

Devin Review found 1 new potential issue.

View 2 additional findings in Devin Review.

[devin-ai-integration]

Devin is archived and cannot be woken up. Please unarchive Devin if you want to continue using it.

[devin-ai-integration]

Devin is archived and cannot be woken up. Please unarchive Devin if you want to continue using it.