refactor(benchmarks): inline workloads into their .bench.ts entrypoints by devin-ai-integration[bot] · Pull Request #261 · computesdk/benchmarks

devin-ai-integration · 2026-07-30T19:51:59Z

Summary

Each benchmark's workload lived in a sibling *-task.ts that the .bench.ts imported through a make*Task() factory whose only job was holding a per-participant cache closure. That indirection is gone: every entrypoint is now the whole benchmark — config plus a defineTask whose body is the ctx.step sequence.

-export const task = defineTask(makeStorageTask(testData, fileSizeBytes));
+const storageCache = new Map<string, Storage>();
+
+export const task = defineTask<StorageProviderConfig>(async (ctx) => {
+  await ctx.step('upload', ...);
+  await ctx.step('download', async () => { ...; ctx.measure({ throughputMbps }); });
+  await ctx.step('delete', ..., { reportConcurrency: false });
+});

The factory closures became module-level Maps (same lifetime: one process = one run), and each defineTask is now explicitly typed (defineTask<ProviderConfig>) since the participant type no longer arrives via the factory's return type.

Deleted (bodies moved verbatim into the entrypoint): browser/browser-task.ts, browser/browser-throughput-task.ts, storage/storage-task.ts, storage/snapshot-fork-task.ts.

Two shared modules were split along the workload/reporting line, leaving only the reporting half:

browser/throughput-benchmark.ts → keeps summarizeIterations / emptySummary / writeThroughputResultsJson; the action loop (runThroughputIteration, runActionLoop, timeAction, nav-URL handling) moved into browser-throughput.bench.ts.
sandbox/dax.ts → keeps the result types, summarize, writeDaxResultsJson, and now-exported BENCH_SCRIPT_PATH; runDaxIteration (renamed runDaxBuild) and DAX_RESOURCE_OPTIONS moved into dax.bench.ts.

Left alone on purpose: sandbox/tti-task.ts (shared by sequential/staggered/burst, so inlining would triplicate it), ai-gateway/phase-probe.ts (shared with the legacy JSON writer), and all providers.ts / types.ts / scoring.ts / legacy-results.ts / generate-*svg.ts modules.

Also folds in the doc fix from the now-closed #260: header comments that still said "built on @benchsdk/runner's runBenchmark" now describe the declarative config + task model.

No behavior change — same steps, same names, same measure payloads, same TaskError codes/data, same slugs and results paths. pnpm typecheck and both package test suites pass.

Link to Devin session: https://app.devin.ai/sessions/e0be51dd272f40d5997335584c1270b3
Requested by: @HeyGarrison

…ders Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

…h.ts entrypoints Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration · 2026-07-30T19:52:02Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment, CI, and merge conflict monitoring

open-cla · 2026-07-30T19:52:04Z

Contributor License Agreement

All contributors are covered by a CLA.

devin-ai-integration

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

github-actions · 2026-07-30T20:15:03Z

Storage Benchmark Results

1MB Files

#	Provider	Score	Download	Throughput	Upload	Status
1	Azure Blob Storage	96.6	0.02s	371.1 Mbps	0.08s	10/10
2	Tigris	95.0	0.05s	181.4 Mbps	0.56s	10/10
3	AWS S3	94.9	0.10s	87.8 Mbps	0.21s	10/10
4	Tensorlake	94.9	0.05s	160.6 Mbps	0.51s	10/10
5	Vercel Blob	94.6	0.13s	67.9 Mbps	0.27s	10/10
6	Cloudflare R2	94.0	0.18s	47.0 Mbps	0.59s	10/10
7	Google Cloud Storage	0.0	0.00s	0.0 Mbps	0.00s	0/10

View full run · SVGs available as build artifacts

github-actions · 2026-07-30T20:15:23Z

Snapshot/Fork Benchmark Results

small dataset

#	Provider	Score	Snapshot create	Fork (snapshot)	Fork (live)	First read	Status
1	Tensorlake	99.7	0.03s	0.31s	0.34s	0.15s	1/1
2	Tigris	99.6	0.14s	0.35s	0.32s	0.26s	1/1
3	Azure-blob	97.4	1.65s	1.65s	1.97s	0.04s	1/1
4	Cloudflare R2	89.0	5.01s	5.43s	17.62s	0.24s	1/1
5	AWS S3	0.0	0.00s	0.00s	0.00s	0.00s	0/1

View full run

github-actions · 2026-07-30T20:15:30Z

Browser Benchmark Results

#	Provider	Score	Create	Connect	Navigate	Release	Total	Status
1	Kernel	97.0	0.03s	0.08s	0.11s	0.04s	0.26s	10/10
2	Browserbase	93.9	0.21s	0.11s	0.08s	0.13s	0.53s	10/10
3	Browseruse	93.3	0.17s	0.15s	0.07s	0.04s	0.46s	10/10
4	Hyperbrowser	81.5	1.18s	0.14s	0.13s	0.09s	1.63s	10/10
5	Steel	79.4	0.68s	0.36s	0.12s	0.56s	1.94s	10/10
6	Tilion	69.3	0.01s	0.04s	0.08s	0.01s	0.15s	10/10
7	Notte	42.2	0.66s	2.48s	0.59s	0.58s	6.45s	10/10

View full run · SVG available as build artifact

github-actions · 2026-07-30T20:17:45Z

Browser Throughput Benchmark Results

#	Provider	Score	APS (med)	Task (med)	Task (p95)	Screenshot	Status
1	Kernel	80.0	5.74/s	1.74s	2.10s	273ms	3/3
2	Tilion	76.0	4.86/s	2.06s	2.37s	222ms	3/3
3	Browseruse	69.1	3.42/s	2.93s	3.02s	248ms	3/3
4	Browserbase	67.7	3.21/s	3.12s	3.53s	306ms	3/3
5	Hyperbrowser	66.1	2.92/s	3.42s	3.76s	452ms	3/3
6	Steel	56.4	1.57/s	6.39s	6.42s	628ms	3/3
7	Notte	12.7	0.17/s	59.02s	66.71s	5958ms	3/3

View full run · SVG available as build artifact

github-actions · 2026-07-30T20:17:47Z

Sandbox Dax Benchmark Results

Provider	Phases	Total	Prepare	Clone	Install	Typecheck	Status
archil	0/7	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
beam	0/7	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
blaxel	6/7	47.91s	2.04s	2.21s	10.59s	0.00s	0/1 OK
cloud-run	2/7	1.18s	0.36s	0.00s	0.00s	0.00s	0/1 OK
cloudflare	0/7	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
codesandbox	0/7	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
createos	7/7	48.05s	1.99s	2.74s	12.62s	25.42s	1/1 OK
daytona	7/7	69.18s	3.11s	2.13s	15.18s	35.05s	1/1 OK
declaw	7/7	188.38s	6.58s	4.62s	18.54s	147.61s	1/1 OK
e2b	7/7	85.58s	10.16s	4.15s	20.36s	42.37s	1/1 OK
hopx	--	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
isorun	--	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
lightning	7/7	39.61s	3.90s	2.55s	12.62s	14.48s	1/1 OK
modal	6/7	24.06s	4.05s	2.70s	12.24s	0.00s	0/1 OK
namespace	7/7	41.94s	4.55s	1.21s	8.13s	21.72s	1/1 OK
northflank	--	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
opencomputer	6/7	25.41s	3.95s	1.98s	16.62s	0.00s	0/1 OK
runloop	2/7	115.34s	0.00s	0.00s	30.31s	51.34s	1/1 OK
sandbox0	--	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
superserve	7/7	103.06s	5.43s	4.69s	22.66s	66.33s	1/1 OK
tenki	7/7	68.20s	3.57s	4.82s	16.29s	36.29s	1/1 OK
tensorlake	7/7	54.49s	12.22s	1.39s	9.61s	27.02s	1/1 OK
upstash	7/7	51.91s	3.73s	1.79s	10.11s	25.82s	1/1 OK
vercel	7/7	73.40s	16.59s	2.03s	11.47s	36.76s	1/1 OK

View full run

github-actions · 2026-07-30T20:22:28Z

AI Gateway Benchmark Results

#	Gateway	Score	Cold E2E	Warm TTFT	Tok/sec	Status
1	LLM Gateway	90.7	666ms	657ms	87.8	20/20
2	Pydantic AI Gateway	89.9	660ms	702ms	86.4	20/20
3	Vercel AI Gateway	89.8	760ms	680ms	92.6	20/20
4	Anthropic (direct)	89.3	808ms	594ms	86.9	20/20
5	OpenRouter	87.3	906ms	813ms	88.8	20/20
6	Cloudflare AI Gateway	86.8	891ms	613ms	90.0	20/20
7	Concentrate AI	84.8	1069ms	869ms	0.0	20/20

View full run · SVG available as a build artifact

github-actions · 2026-07-30T20:24:30Z

Sandbox Benchmark Results

Sequential

#	Provider	Score	Median TTI	P95	P99	Status
1	isorun	99.8	0.02s	0.02s	0.02s	10/10
2	northflank	98.8	0.10s	0.14s	0.14s	10/10
3	archil	98.5	0.14s	0.17s	0.17s	10/10
4	createos	98.4	0.08s	0.29s	0.29s	10/10
5	lightning	98.2	0.16s	0.22s	0.22s	10/10
6	declaw	98.1	0.05s	0.40s	0.40s	10/10
7	blaxel	97.6	0.21s	0.29s	0.29s	10/10
8	tensorlake	96.2	0.33s	0.46s	0.46s	10/10
9	superserve	95.9	0.36s	0.48s	0.48s	10/10
10	upstash	95.0	0.36s	0.70s	0.70s	10/10
11	beam	94.9	0.10s	1.13s	1.13s	10/10
12	cloud-run	93.9	0.55s	0.70s	0.70s	10/10
13	runloop	93.9	0.55s	0.70s	0.70s	10/10
14	modal	93.3	0.55s	0.85s	0.85s	10/10
15	vercel	91.2	0.43s	1.54s	1.54s	10/10
16	e2b	88.6	0.87s	1.55s	1.55s	10/10
17	tenki	87.5	0.92s	1.75s	1.75s	10/10
18	opencomputer	83.8	1.46s	1.85s	1.85s	10/10
19	daytona	81.9	1.52s	2.23s	2.23s	10/10
20	codesandbox	72.2	2.57s	3.09s	3.09s	10/10
21	cloudflare	65.7	2.72s	4.51s	4.51s	10/10
22	sandbox0	7.4	2.58s	2.58s	2.58s	1/10
23	hopx	0.0	0.00s	0.00s	0.00s	0/10

Staggered

#	Provider	Score	Median TTI	P95	P99	Status
1	declaw	99.2	0.04s	0.14s	0.14s	10/10
2	createos	99.1	0.08s	0.12s	0.12s	10/10
3	beam	98.8	0.10s	0.16s	0.16s	10/10
4	northflank	98.7	0.09s	0.18s	0.18s	10/10
5	archil	98.4	0.14s	0.19s	0.19s	10/10
6	blaxel	97.9	0.20s	0.24s	0.24s	10/10
7	superserve	96.7	0.30s	0.38s	0.38s	10/10
8	tensorlake	96.3	0.33s	0.41s	0.41s	10/10
9	upstash	96.1	0.37s	0.41s	0.41s	10/10
10	lightning	95.8	0.29s	0.61s	0.61s	10/10
11	runloop	93.6	0.57s	0.76s	0.76s	10/10
12	vercel	92.2	0.45s	1.27s	1.27s	10/10
13	modal	91.5	0.56s	1.28s	1.28s	10/10
14	e2b	89.3	0.89s	1.36s	1.36s	10/10
15	tenki	88.4	0.92s	1.52s	1.52s	10/10
16	opencomputer	82.8	1.61s	1.88s	1.88s	10/10
17	codesandbox	76.3	2.29s	2.50s	2.50s	10/10
18	cloud-run	75.2	1.94s	3.29s	3.29s	10/10
19	sandbox0	68.3	3.01s	3.42s	3.42s	10/10
20	cloudflare	62.4	3.18s	4.61s	4.61s	10/10
21	isorun	59.9	0.02s	20.60s	20.60s	10/10
22	daytona	46.5	2.27s	9.98s	9.98s	10/10
23	hopx	0.0	0.00s	0.00s	0.00s	0/10

Burst

#	Provider	Score	Median TTI	P95	P99	Status
1	isorun	99.5	0.05s	0.05s	0.05s	10/10
2	declaw	99.1	0.08s	0.09s	0.09s	10/10
3	beam	98.7	0.13s	0.14s	0.14s	10/10
4	createos	98.6	0.14s	0.14s	0.14s	10/10
5	northflank	98.5	0.15s	0.15s	0.15s	10/10
6	archil	98.0	0.19s	0.21s	0.21s	10/10
7	blaxel	97.1	0.26s	0.33s	0.33s	10/10
8	lightning	96.5	0.31s	0.41s	0.41s	10/10
9	superserve	96.4	0.35s	0.36s	0.36s	10/10
10	tensorlake	96.4	0.36s	0.36s	0.36s	10/10
11	vercel	95.5	0.43s	0.49s	0.49s	10/10
12	upstash	94.8	0.46s	0.62s	0.62s	10/10
13	modal	94.2	0.57s	0.60s	0.60s	10/10
14	e2b	89.6	0.91s	1.23s	1.23s	10/10
15	runloop	86.1	0.64s	2.50s	2.50s	10/10
16	cloud-run	86.0	1.19s	1.73s	1.73s	10/10
17	tenki	85.4	1.44s	1.50s	1.50s	10/10
18	codesandbox	69.5	2.81s	3.41s	3.41s	10/10
19	daytona	68.0	2.34s	4.49s	4.49s	10/10
20	sandbox0	67.2	2.65s	4.23s	4.23s	10/10
21	cloudflare	63.9	2.88s	4.71s	4.71s	10/10
22	opencomputer	62.1	1.89s	6.63s	6.63s	10/10
23	hopx	0.0	0.00s	0.00s	0.00s	0/10

View full run · SVGs available as build artifacts

HeyGarrison and others added 2 commits July 30, 2026 19:12

docs(benchmarks): correct stale runBenchmark wording in .bench.ts hea…

ab752d2

…ders Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

refactor(benchmarks): inline per-benchmark workloads into their .benc…

e330d4d

…h.ts entrypoints Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

HeyGarrison self-assigned this Jul 30, 2026

devin-ai-integration Bot commented Jul 30, 2026

View reviewed changes

devin-ai-integration Bot mentioned this pull request Jul 30, 2026

refactor(benchmarks): collapse the 3 sandbox TTI benches into one tti.bench.ts #262

Merged

HeyGarrison merged commit 27957b3 into master Jul 30, 2026
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

refactor(benchmarks): inline workloads into their .bench.ts entrypoints - #261

refactor(benchmarks): inline workloads into their .bench.ts entrypoints#261
HeyGarrison merged 2 commits into
masterfrom
devin/1785441088-bench-inline-workloads

devin-ai-integration Bot commented Jul 30, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot commented Jul 30, 2026

Uh oh!

open-cla Bot commented Jul 30, 2026

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

devin-ai-integration Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

devin-ai-integration Bot commented Jul 30, 2026

🤖 Devin AI Engineer

Uh oh!

open-cla Bot commented Jul 30, 2026

Contributor License Agreement

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

✅ Devin Review: No Issues Found

Uh oh!

Uh oh!

github-actions Bot commented Jul 30, 2026

Storage Benchmark Results

1MB Files

Uh oh!

github-actions Bot commented Jul 30, 2026

Snapshot/Fork Benchmark Results

small dataset

Uh oh!

github-actions Bot commented Jul 30, 2026

Browser Benchmark Results

Uh oh!

github-actions Bot commented Jul 30, 2026

Browser Throughput Benchmark Results

Uh oh!

github-actions Bot commented Jul 30, 2026

Sandbox Dax Benchmark Results

Uh oh!

github-actions Bot commented Jul 30, 2026

AI Gateway Benchmark Results

Uh oh!

github-actions Bot commented Jul 30, 2026

Sandbox Benchmark Results

Sequential

Staggered

Burst

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

devin-ai-integration Bot commented Jul 30, 2026 •

edited

Loading