feat(runner): verbs-only `bench run` with --shape and --run-key by devin-ai-integration[bot] · Pull Request #264 · computesdk/benchmarks

devin-ai-integration · 2026-07-30T20:29:25Z

Summary

The old surface duplicated a benchmark's identity across four places — the .bench.ts file, package.json, the CI workflow, and imperative bench create benchmark / bench create run commands — and drifted between them. This makes bench verbs-only: the benchmark is declared in the file and materialized when you run it, and a run is opened as a side effect. There is no create.

# before
bench create benchmark sandbox-burst-local --name 'Sandbox burst TTI (local)' --kind sandbox
ID=$(bench create run --benchmark sandbox-burst-local)
bench run tti.bench.ts --benchmark sandbox-burst-local --provider e2b --run-id "$ID"

# after
bench run benchmarks/sandbox/tti.bench.ts --shape burst --provider e2b --run-key "$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT"

Two new flags on bench run:

--shape <name> — a bench file declares named shapes, each swapping in its own platform identity (slug/name, optional kind) and a stable knob (staggerDelayMs) while reusing the same task and participants. Scale knobs (--iterations/--concurrency) stay on the CLI, so no shape sets a value only to have it overridden. This is what collapses the per-shape slug/name/knob triple out of package.json and CI.
```
shapes: {
  burst:     { slug: 'sandbox-burst-local',     name: 'Sandbox burst TTI (local)' },
  staggered: { slug: 'sandbox-staggered-local', name: 'Sandbox staggered TTI (local)', staggerDelayMs: 200 },
}
```
--run-key <key> — sibling processes passing the same key get-or-create one shared run (per org + benchmark) instead of each opening its own, so parallel provider jobs land in one directly-comparable run. Each process registers only the participants it actually runs. Different shapes are different slugs, so one key value yields one run per shape — no per-shape key juggling. The binding is permanent, so a workflow re-run varies the key via GITHUB_RUN_ATTEMPT.
```
if (args.runKey) {
  const { run } = await client.createRun(slug, { name, runKey: args.runKey }); // get-or-create
  for (const p of available) await client.upsertParticipant(slug, run.id, p.name, { totalTasks });
} else { /* unchanged fresh-run path */ }
```

Benchmark materialization stays honest: a bare --benchmark X that merely retargets reporting at a benchmark this file doesn't name does not upsert (which would rename it); --shape/--name/the file's own slug do.

Call sites collapsed

package.json: every --slug … --name … [--stagger-delay-ms 200] triple → --shape burst / --shape staggered.
.github/workflows/sandbox-tti-benchmarks.yml: dropped the whole create-runs job; the provider matrix now runs bench run … --shape … --run-key "$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT" directly.

--slug remains a working alias for --benchmark.

Depends on

computesdk/benchmarks-platform#53 — POST /runs get-or-create by runKey.

Checks

pnpm typecheck clean; runner suite 65 passing, client suite 118 passing; all packages build.

Link to Devin session: https://app.devin.ai/sessions/82de2e18fab54e7bba63034865f0cb6c
Requested by: @HeyGarrison

devin-ai-integration · 2026-07-30T20:29:28Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment, CI, and merge conflict monitoring

open-cla · 2026-07-30T20:29:31Z

Contributor License Agreement

All contributors are covered by a CLA.

superagent-security

Superagent found 2 supply chain issue(s)

superagent-security

Superagent found 1 security concern(s).

…nto one run Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

… itself Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

… the run own its size Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

…tion count Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration

Devin Review found 1 new potential issue.

View 2 additional findings in Devin Review.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Retire the imperative create commands: the benchmark is declared in the .bench.ts file (via optional named shapes) and materialized on run, and a run is opened as a side effect. --shape selects a named variant (swapping platform identity + stable knobs); --run-key get-or-creates a shared run so sibling provider jobs converge on one comparable run. Collapse the per-shape slug/name/knob triple out of package scripts and the sandbox TTI workflow (drop the create-runs job; providers pass a shared run key including GITHUB_RUN_ATTEMPT). Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration · 2026-07-31T20:36:57Z

Devin is archived and cannot be woken up. Please unarchive Devin if you want to continue using it.

devin-ai-integration · 2026-07-31T20:36:57Z

Devin is archived and cannot be woken up. Please unarchive Devin if you want to continue using it.

…json bench scripts (union of providers in --shape form) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

github-actions · 2026-08-05T18:18:57Z

Storage Benchmark Results

1MB Files

#	Provider	Score	Download	Throughput	Upload	Status
1	Azure Blob Storage	96.5	0.02s	339.7 Mbps	0.06s	10/10
2	Archil	95.2	0.05s	181.1 Mbps	0.40s	10/10
3	Vercel Blob	95.2	0.07s	130.2 Mbps	0.24s	10/10
4	AWS S3	94.5	0.11s	75.3 Mbps	0.20s	10/10
5	Tensorlake	94.3	0.06s	129.3 Mbps	0.69s	10/10
6	Cloudflare R2	94.2	0.15s	56.0 Mbps	0.49s	10/10
7	Tigris	93.1	0.38s	22.5 Mbps	0.55s	10/10
8	Google Cloud Storage	0.0	0.00s	0.0 Mbps	0.00s	0/10

View full run · SVGs available as build artifacts

github-actions · 2026-08-05T18:19:15Z

Browser Benchmark Results

#	Provider	Score	Create	Connect	Navigate	Release	Total	Status
1	Kernel	98.1	0.04s	0.05s	0.09s	0.05s	0.22s	10/10
2	Tilion	96.4	0.10s	0.04s	0.10s	0.02s	0.29s	10/10
3	Browseruse	96.0	0.16s	0.12s	0.08s	0.04s	0.43s	10/10
4	Hyperbrowser	94.6	0.26s	0.14s	0.11s	0.10s	0.62s	10/10
5	Browserbase	93.3	0.15s	0.30s	0.17s	0.13s	0.78s	10/10
6	Notte	60.0	0.51s	1.29s	0.37s	0.29s	2.68s	10/10
7	Steel	53.3	0.47s	1.84s	0.16s	0.37s	3.84s	10/10

View full run · SVG available as build artifact

github-actions · 2026-08-05T18:20:18Z

Browser Throughput Benchmark Results

#	Provider	Score	APS (med)	Task (med)	Task (p95)	Screenshot	Status
1	Kernel	84.8	6.83/s	1.46s	1.79s	209ms	3/3
2	Browserbase	74.8	4.57/s	2.19s	2.23s	282ms	3/3
3	Tilion	71.2	4.03/s	2.48s	4.15s	237ms	3/3
4	Browseruse	70.3	3.70/s	2.70s	3.09s	380ms	3/3
5	Hyperbrowser	63.6	2.48/s	4.04s	4.11s	454ms	3/3
6	Steel	51.6	1.51/s	6.60s	12.87s	770ms	3/3
7	Notte	36.0	0.73/s	13.70s	22.03s	1753ms	3/3

View full run · SVG available as build artifact

github-actions · 2026-08-05T18:20:25Z

Snapshot/Fork Benchmark Results

small dataset

#	Provider	Score	Snapshot create	Fork (snapshot)	Fork (live)	First read	Status
1	Tensorlake	99.6	0.05s	0.35s	0.40s	0.15s	1/1
2	Tigris	99.5	0.10s	0.52s	0.27s	0.55s	1/1
3	Azure-blob	98.4	0.92s	0.97s	1.68s	0.02s	1/1
4	Cloudflare R2	83.5	9.76s	10.04s	16.51s	0.25s	1/1
5	Archil	0.0	0.00s	0.00s	0.00s	0.00s	0/1
6	AWS S3	0.0	0.00s	0.00s	0.00s	0.00s	0/1

View full run

github-actions · 2026-08-05T18:20:44Z

Sandbox Benchmark Results

Sequential

#	Provider	Score	Median TTI	P95	P99	Status
1	isorun	99.8	0.02s	0.03s	0.03s	10/10
2	arker	99.0	0.08s	0.13s	0.13s	10/10
3	createos	98.9	0.08s	0.15s	0.15s	10/10
4	declaw	98.8	0.02s	0.29s	0.29s	10/10
5	northflank	98.1	0.15s	0.25s	0.25s	10/10
6	archil	98.0	0.19s	0.22s	0.22s	10/10
7	lightning	97.1	0.18s	0.45s	0.45s	10/10
8	blaxel	96.8	0.27s	0.41s	0.41s	10/10
9	superserve	96.0	0.30s	0.56s	0.56s	10/10
10	upstash	95.2	0.44s	0.54s	0.54s	10/10
11	vercel	94.6	0.45s	0.67s	0.67s	10/10
12	opencomputer	93.8	0.22s	1.21s	1.21s	10/10
13	tensorlake	93.5	0.54s	0.81s	0.81s	10/10
14	runloop	93.5	0.58s	0.77s	0.77s	10/10
15	cloud-run	93.3	0.58s	0.80s	0.80s	10/10
16	modal	93.3	0.57s	0.84s	0.84s	10/10
17	tenki	91.8	0.77s	0.89s	0.89s	10/10
18	e2b	91.7	0.73s	0.97s	0.97s	10/10
19	sail	88.4	0.78s	1.73s	1.73s	10/10
20	daytona	88.3	0.88s	1.60s	1.60s	10/10
21	beam	76.1	0.09s	5.82s	5.82s	10/10
22	sandbox0	66.5	1.51s	6.10s	6.10s	10/10
23	cloudflare	65.5	1.94s	3.90s	3.90s	9/10
24	codesandbox	44.7	2.55s	20.48s	20.48s	10/10
25	hopx	0.0	0.00s	0.00s	0.00s	0/10

Staggered

#	Provider	Score	Median TTI	P95	P99	Status
1	isorun	99.8	0.02s	0.02s	0.02s	10/10
2	declaw	99.7	0.01s	0.06s	0.06s	10/10
3	beam	99.0	0.10s	0.11s	0.11s	10/10
4	arker	98.9	0.07s	0.17s	0.17s	10/10
5	createos	98.2	0.08s	0.33s	0.33s	10/10
6	archil	98.0	0.18s	0.23s	0.23s	10/10
7	lightning	97.9	0.18s	0.26s	0.26s	10/10
8	northflank	97.8	0.18s	0.29s	0.29s	10/10
9	blaxel	97.3	0.25s	0.29s	0.29s	10/10
10	superserve	97.1	0.26s	0.33s	0.33s	10/10
11	opencomputer	96.6	0.13s	0.66s	0.66s	10/10
12	upstash	95.0	0.46s	0.57s	0.57s	10/10
13	tensorlake	94.3	0.50s	0.67s	0.67s	10/10
14	runloop	93.6	0.54s	0.79s	0.79s	10/10
15	cloud-run	93.4	0.57s	0.80s	0.80s	10/10
16	modal	92.8	0.62s	0.88s	0.88s	10/10
17	vercel	91.7	0.51s	1.30s	1.30s	10/10
18	tenki	91.7	0.69s	1.05s	1.05s	10/10
19	sail	86.1	1.14s	1.78s	1.78s	10/10
20	e2b	85.6	0.94s	2.20s	2.20s	10/10
21	codesandbox	74.8	2.29s	2.88s	2.88s	10/10
22	cloudflare	72.0	2.29s	3.57s	3.57s	10/10
23	sandbox0	70.9	2.62s	3.34s	3.34s	10/10
24	daytona	51.0	1.49s	15.47s	15.47s	10/10
25	hopx	0.0	0.00s	0.00s	0.00s	0/10

Burst

#	Provider	Score	Median TTI	P95	P99	Status
1	isorun	99.6	0.04s	0.04s	0.04s	10/10
2	arker	98.5	0.13s	0.16s	0.16s	10/10
3	createos	98.5	0.14s	0.15s	0.15s	10/10
4	beam	98.5	0.14s	0.16s	0.16s	10/10
5	lightning	98.0	0.19s	0.21s	0.21s	10/10
6	declaw	97.9	0.13s	0.34s	0.34s	10/10
7	northflank	97.7	0.23s	0.24s	0.24s	10/10
8	archil	97.5	0.23s	0.28s	0.28s	10/10
9	superserve	96.8	0.31s	0.33s	0.33s	10/10
10	blaxel	96.5	0.34s	0.37s	0.37s	10/10
11	tensorlake	94.7	0.52s	0.56s	0.56s	10/10
12	modal	94.0	0.56s	0.65s	0.65s	10/10
13	upstash	93.9	0.60s	0.63s	0.63s	10/10
14	cloud-run	93.8	0.57s	0.69s	0.69s	10/10
15	opencomputer	92.3	0.27s	1.52s	1.52s	10/10
16	vercel	91.2	0.59s	1.31s	1.31s	10/10
17	tenki	90.9	0.83s	1.04s	1.04s	10/10
18	e2b	89.9	0.93s	1.12s	1.12s	10/10
19	sail	82.6	1.62s	1.91s	1.91s	10/10
20	runloop	82.4	0.62s	3.46s	3.46s	10/10
21	daytona	81.2	1.06s	3.10s	3.10s	10/10
22	codesandbox	69.3	2.79s	3.48s	3.48s	10/10
23	cloudflare	67.8	2.74s	3.93s	3.93s	10/10
24	sandbox0	54.4	4.08s	5.27s	5.27s	10/10
25	hopx	0.0	0.00s	0.00s	0.00s	0/10

View full run · SVGs available as build artifacts

github-actions · 2026-08-05T18:23:00Z

Sandbox Dax Benchmark Results

Provider	Phases	Total	Prepare	Clone	Install	Typecheck	Status
archil	0/7	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
arker	7/7	61.01s	4.22s	2.50s	13.45s	35.20s	1/1 OK
beam	0/7	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
blaxel	7/7	56.67s	1.97s	3.36s	12.09s	33.01s	1/1 OK
cloud-run	2/7	1.40s	0.41s	0.00s	0.00s	0.00s	0/1 OK
cloudflare	0/7	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
codesandbox	0/7	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
createos	7/7	46.78s	2.85s	2.35s	12.33s	24.06s	1/1 OK
daytona	7/7	104.95s	3.71s	9.00s	16.88s	57.24s	1/1 OK
declaw	0/7	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
e2b	7/7	75.42s	9.36s	3.38s	16.97s	36.36s	1/1 OK
hopx	--	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
isorun	7/7	113.26s	1.01s	5.88s	10.74s	28.45s	1/1 OK
lightning	7/7	39.26s	3.93s	2.36s	12.17s	15.04s	1/1 OK
modal	7/7	105.72s	4.31s	13.81s	14.80s	64.35s	1/1 OK
namespace	7/7	48.61s	5.09s	2.50s	8.33s	24.11s	1/1 OK
northflank	--	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
opencomputer	6/7	24.96s	4.11s	1.89s	16.93s	0.00s	0/1 OK
runloop	2/7	116.58s	0.00s	0.00s	26.07s	50.93s	1/1 OK
sail	6/7	82.78s	5.91s	9.52s	39.20s	0.00s	0/1 OK
sandbox0	7/7	233.19s	6.08s	8.05s	74.65s	114.08s	1/1 OK
superserve	7/7	96.52s	5.91s	5.44s	21.95s	58.85s	1/1 OK
tenki	7/7	71.43s	5.99s	4.62s	18.68s	35.74s	1/1 OK
tensorlake	7/7	67.14s	12.85s	10.47s	12.00s	26.80s	1/1 OK
upstash	7/7	60.89s	3.74s	11.05s	11.06s	24.85s	1/1 OK
vercel	7/7	93.66s	23.64s	2.10s	13.97s	45.84s	1/1 OK

View full run

github-actions · 2026-08-05T18:26:45Z

AI Gateway Benchmark Results

#	Gateway	Score	Cold E2E	Warm TTFT	Tok/sec	Status
1	Pydantic AI Gateway	90.8	734ms	729ms	92.3	20/20
2	Cloudflare AI Gateway	90.3	714ms	679ms	84.7	20/20
3	Vercel AI Gateway	90.0	818ms	890ms	90.9	20/20
4	Concentrate AI	89.9	947ms	785ms	89.0	20/20
5	LLM Gateway	89.1	782ms	807ms	85.4	20/20
6	Anthropic (direct)	88.6	690ms	696ms	84.5	20/20
7	OpenRouter	87.4	1012ms	790ms	95.0	20/20

View full run · SVG available as a build artifact

* feat(runner): bench create-run + --run-id so every provider reports into one run Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(runner): create the shared run empty and let each joiner register itself Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(ci): drop the no-op --provider from create-run Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(runner): verb-first CLI (bench create benchmark|run) and let the run own its size Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs(bench): show the shared-run invocation in the TTI header Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(runner): open runs without a size; joiners bring their own iteration count Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(runner): don't rename a benchmark that --benchmark merely retargets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(runner): verbs-only bench run with --shape and --run-key Retire the imperative create commands: the benchmark is declared in the .bench.ts file (via optional named shapes) and materialized on run, and a run is opened as a side effect. --shape selects a named variant (swapping platform identity + stable knobs); --run-key get-or-creates a shared run so sibling provider jobs converge on one comparable run. Collapse the per-shape slug/name/knob triple out of package scripts and the sandbox TTI workflow (drop the create-runs job; providers pass a shared run key including GITHUB_RUN_ATTEMPT). Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: garrison <garrison@computesdk.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

HeyGarrison self-assigned this Jul 30, 2026

superagent-security Bot reviewed Jul 30, 2026

View reviewed changes

Base automatically changed from devin/1785442068-tti-one-file to master July 30, 2026 20:30

This comment was marked as resolved.

Sign in to view

superagent-security Bot reviewed Jul 30, 2026

View reviewed changes

Comment thread .github/workflows/sandbox-tti-benchmarks.yml Outdated

HeyGarrison and others added 5 commits July 30, 2026 21:01

feat(runner): bench create-run + --run-id so every provider reports i…

9d6ac9c

…nto one run Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

fix(runner): create the shared run empty and let each joiner register…

8285687

… itself Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

chore(ci): drop the no-op --provider from create-run

1d16585

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

refactor(runner): verb-first CLI (bench create benchmark|run) and let…

4414920

… the run own its size Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

docs(bench): show the shared-run invocation in the TTI header

4680cc4

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration Bot force-pushed the devin/1785443347-shared-run branch from 592ed93 to 4680cc4 Compare July 30, 2026 21:01

devin-ai-integration Bot changed the title ~~feat(runner): bench create-run + --run-id so every provider reports into one run~~ feat(runner): verb-first CLI + --run-id so every provider reports into one run Jul 30, 2026

feat(runner): open runs without a size; joiners bring their own itera…

e0309ba

…tion count Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration Bot commented Jul 30, 2026

View reviewed changes

Comment thread benchmarks/sandbox/tti.bench.ts Outdated

HeyGarrison and others added 2 commits July 30, 2026 21:41

fix(runner): don't rename a benchmark that --benchmark merely retargets

b055aea

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration Bot changed the title ~~feat(runner): verb-first CLI + --run-id so every provider reports into one run~~ feat(runner): verbs-only bench run with --shape and --run-key Jul 31, 2026

Merge origin/master into devin/1785529209-run-key... resolve package.…

e5b4f21

…json bench scripts (union of providers in --shape form) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

HeyGarrison merged commit 3c42cdf into master Aug 5, 2026
6 checks passed

Uh oh!

Conversation

devin-ai-integration Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Call sites collapsed

Depends on

Checks

Uh oh!

devin-ai-integration Bot commented Jul 30, 2026

🤖 Devin AI Engineer

Uh oh!

open-cla Bot commented Jul 30, 2026

Contributor License Agreement

Uh oh!

superagent-security Bot left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

superagent-security Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

devin-ai-integration Bot commented Jul 31, 2026

Uh oh!

devin-ai-integration Bot commented Jul 31, 2026

Uh oh!

Uh oh!

github-actions Bot commented Aug 5, 2026

Storage Benchmark Results

1MB Files

Uh oh!

github-actions Bot commented Aug 5, 2026

Browser Benchmark Results

Uh oh!

github-actions Bot commented Aug 5, 2026

Browser Throughput Benchmark Results

Uh oh!

github-actions Bot commented Aug 5, 2026

Snapshot/Fork Benchmark Results

small dataset

Uh oh!

github-actions Bot commented Aug 5, 2026

Sandbox Benchmark Results

Sequential

Staggered

Burst

Uh oh!

github-actions Bot commented Aug 5, 2026

Sandbox Dax Benchmark Results

Uh oh!

github-actions Bot commented Aug 5, 2026

AI Gateway Benchmark Results

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

devin-ai-integration Bot commented Jul 30, 2026 •

edited

Loading