refactor(benchmarks): collapse the 3 sandbox TTI benches into one tti.bench.ts by devin-ai-integration[bot] · Pull Request #262 · computesdk/benchmarks

devin-ai-integration · 2026-07-30T20:08:06Z

Summary

sequential/staggered/burst were three entrypoints around one shared ttiTask, differing only in {concurrency, staggerDelayMs} — knobs the framework already owns. They're now one benchmarks/sandbox/tti.bench.ts with the workload inline and no mode table: the file declares the sequential default, and the launch shape comes entirely from CLI flags.

bench run benchmarks/sandbox/tti.bench.ts --iterations 100
bench run benchmarks/sandbox/tti.bench.ts --iterations 100 --concurrency 100 \
  --slug sandbox-burst-local --name 'Sandbox burst TTI (local)'
bench run benchmarks/sandbox/tti.bench.ts --iterations 100 --concurrency 100 --stagger-delay-ms 200 \
  --slug sandbox-staggered-local --name 'Sandbox staggered TTI (local)'

The legacy results dir and mode tag are derived from the knobs the run actually used, so no shape is declared twice:

function legacyShape(run: ResolvedRunConfig) {
  if (run.staggerDelayMs > 0) return { resultsDir: 'staggered_tti', mode: 'staggered' };
  if (run.concurrency > 1) return { resultsDir: 'burst_tti', mode: 'concurrent' };
  return { resultsDir: 'sequential_tti' };
}

Slugs and results/ dir names are carried over verbatim (including burst's legacy 'concurrent' label), so the platform and the SVG/README pipeline see no rename.

Deleted: sequential.bench.ts, staggered.bench.ts, burst.bench.ts, tti-task.ts. package.json scripts keep their names (bench:burst etc.); sandbox-tti-benchmarks.yml now builds the three arg lists in one place.

`--slug` / `--name` in `@benchsdk/runner`

This layout needs a benchmark file to stop hardcoding which platform benchmark it reports to:

const args = parseCliArgs(argv);
const config = {
  ...fileConfig,
  ...(args.slug ? { benchmarkSlug: args.slug } : {}),
  ...(args.name ? { benchmarkName: args.name } : {}),
};

--slug is validated against /^[a-z0-9][a-z0-9-]*$/; both flags are covered by tests, changeset included.

Note (not changed here)

Each provider still gets its own platform run because the workflow uses a matrix.provider with --provider <one>; the runner itself will run all participants in a single run. Consolidating that is a CI tradeoff (one provider's quota failure takes down the whole run, and you lose per-provider secret scoping), so it's left alone deliberately.

pnpm typecheck clean; runner suite 56 passing.

Link to Devin session: https://app.devin.ai/sessions/e0be51dd272f40d5997335584c1270b3
Requested by: @HeyGarrison

devin-ai-integration · 2026-07-30T20:08:10Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment, CI, and merge conflict monitoring

open-cla · 2026-07-30T20:08:12Z

Contributor License Agreement

All contributors are covered by a CLA.

devin-ai-integration

Devin Review found 1 potential issue.

View 1 additional finding in Devin Review.

…r --slug override Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

…ug/--name from CI Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

github-actions · 2026-07-30T20:32:35Z

Storage Benchmark Results

1MB Files

#	Provider	Score	Download	Throughput	Upload	Status
1	Azure Blob Storage	96.5	0.02s	359.7 Mbps	0.09s	10/10
2	Tensorlake	95.1	0.04s	198.9 Mbps	0.46s	10/10
3	AWS S3	94.8	0.12s	68.8 Mbps	0.23s	10/10
4	Vercel Blob	94.6	0.12s	67.4 Mbps	0.22s	10/10
5	Cloudflare R2	94.0	0.19s	43.1 Mbps	0.61s	10/10
6	Tigris	92.8	0.05s	172.3 Mbps	0.81s	10/10
7	Google Cloud Storage	0.0	0.00s	0.0 Mbps	0.00s	0/10

View full run · SVGs available as build artifacts

github-actions · 2026-07-30T20:32:51Z

Snapshot/Fork Benchmark Results

small dataset

#	Provider	Score	Snapshot create	Fork (snapshot)	Fork (live)	First read	Status
1	Tensorlake	99.7	0.04s	0.24s	0.30s	0.13s	1/1
2	Tigris	99.6	0.17s	0.23s	0.23s	0.77s	1/1
3	Azure-blob	98.5	0.85s	1.03s	1.48s	0.02s	1/1
4	Cloudflare R2	88.6	5.46s	5.58s	17.87s	0.29s	1/1
5	AWS S3	0.0	0.00s	0.00s	0.00s	0.00s	0/1

View full run

github-actions · 2026-07-30T20:33:33Z

Browser Benchmark Results

#	Provider	Score	Create	Connect	Navigate	Release	Total	Status
1	Kernel	97.3	0.03s	0.07s	0.10s	0.05s	0.25s	10/10
2	Browseruse	95.5	0.18s	0.16s	0.07s	0.04s	0.49s	10/10
3	Browserbase	94.5	0.23s	0.14s	0.09s	0.13s	0.60s	10/10
4	Tilion	94.4	0.02s	0.04s	0.10s	0.01s	0.16s	10/10
5	Hyperbrowser	93.9	0.27s	0.14s	0.15s	0.09s	0.66s	10/10
6	Steel	80.1	0.68s	0.46s	0.12s	0.56s	1.94s	10/10
7	Notte	60.1	0.60s	2.53s	0.62s	0.51s	4.81s	10/10

View full run · SVG available as build artifact

github-actions · 2026-07-30T20:34:18Z

Sandbox Benchmark Results

Sequential

#	Provider	Score	Median TTI	P95	P99	Status
1	isorun	99.7	0.02s	0.05s	0.05s	10/10
2	declaw	99.2	0.05s	0.12s	0.12s	10/10
3	northflank	98.7	0.10s	0.16s	0.16s	10/10
4	archil	98.5	0.13s	0.17s	0.17s	10/10
5	createos	98.4	0.08s	0.28s	0.28s	10/10
6	lightning	97.5	0.16s	0.37s	0.37s	10/10
7	upstash	96.4	0.34s	0.40s	0.40s	10/10
8	tensorlake	96.3	0.33s	0.44s	0.44s	10/10
9	superserve	96.2	0.35s	0.43s	0.43s	10/10
10	blaxel	95.8	0.32s	0.56s	0.56s	10/10
11	vercel	95.5	0.35s	0.61s	0.61s	10/10
12	beam	94.5	0.09s	1.24s	1.24s	10/10
13	modal	93.7	0.56s	0.74s	0.74s	10/10
14	runloop	93.3	0.58s	0.80s	0.80s	10/10
15	e2b	90.8	0.86s	1.01s	1.01s	10/10
16	cloud-run	90.3	0.55s	1.58s	1.58s	10/10
17	tenki	88.1	0.82s	1.74s	1.74s	10/10
18	opencomputer	80.7	1.56s	2.49s	2.49s	10/10
19	codesandbox	72.7	2.53s	3.03s	3.03s	10/10
20	cloudflare	71.5	2.30s	3.67s	3.67s	10/10
21	sandbox0	68.6	2.57s	4.00s	4.00s	10/10
22	daytona	53.8	2.04s	8.50s	8.50s	10/10
23	hopx	0.0	0.00s	0.00s	0.00s	0/10

Staggered

#	Provider	Score	Median TTI	P95	P99	Status
1	isorun	99.8	0.02s	0.03s	0.03s	10/10
2	declaw	99.3	0.05s	0.11s	0.11s	10/10
3	createos	99.0	0.08s	0.12s	0.12s	10/10
4	beam	99.0	0.09s	0.11s	0.11s	10/10
5	northflank	98.9	0.10s	0.14s	0.14s	10/10
6	archil	98.6	0.13s	0.16s	0.16s	10/10
7	lightning	97.6	0.22s	0.27s	0.27s	10/10
8	blaxel	97.4	0.21s	0.34s	0.34s	10/10
9	superserve	96.7	0.32s	0.34s	0.34s	10/10
10	tensorlake	96.2	0.33s	0.46s	0.46s	10/10
11	upstash	95.0	0.41s	0.64s	0.64s	10/10
12	vercel	94.8	0.37s	0.74s	0.74s	10/10
13	runloop	94.2	0.56s	0.61s	0.61s	10/10
14	modal	93.8	0.55s	0.73s	0.73s	10/10
15	cloud-run	91.5	0.67s	1.13s	1.13s	10/10
16	tenki	91.2	0.80s	1.02s	1.02s	10/10
17	e2b	90.3	0.80s	1.21s	1.21s	10/10
18	daytona	84.9	1.19s	2.00s	2.00s	10/10
19	cloudflare	73.9	2.52s	2.75s	2.75s	10/10
20	codesandbox	72.0	2.74s	2.89s	2.89s	10/10
21	opencomputer	71.8	1.73s	4.45s	4.45s	10/10
22	sandbox0	67.5	2.91s	3.76s	3.76s	10/10
23	hopx	0.0	0.00s	0.00s	0.00s	0/10

Burst

#	Provider	Score	Median TTI	P95	P99	Status
1	isorun	99.0	0.05s	0.19s	0.19s	10/10
2	createos	98.7	0.13s	0.13s	0.13s	10/10
3	beam	98.7	0.13s	0.15s	0.15s	10/10
4	declaw	98.3	0.10s	0.29s	0.29s	10/10
5	northflank	98.1	0.16s	0.22s	0.22s	10/10
6	lightning	97.4	0.23s	0.32s	0.32s	10/10
7	archil	97.4	0.18s	0.39s	0.39s	10/10
8	blaxel	97.2	0.27s	0.31s	0.31s	10/10
9	tensorlake	96.0	0.40s	0.40s	0.40s	10/10
10	superserve	95.9	0.39s	0.43s	0.43s	10/10
11	vercel	94.4	0.50s	0.65s	0.65s	10/10
12	upstash	94.2	0.47s	0.76s	0.76s	10/10
13	modal	93.3	0.58s	0.80s	0.80s	10/10
14	e2b	90.1	0.89s	1.15s	1.15s	10/10
15	cloud-run	89.1	0.93s	1.32s	1.32s	10/10
16	tenki	85.8	1.38s	1.48s	1.48s	10/10
17	daytona	82.8	1.46s	2.12s	2.12s	10/10
18	runloop	80.3	0.65s	3.96s	3.96s	10/10
19	opencomputer	80.2	1.90s	2.12s	2.12s	10/10
20	codesandbox	69.7	2.76s	3.44s	3.44s	10/10
21	sandbox0	66.9	2.77s	4.12s	4.12s	10/10
22	cloudflare	59.0	3.28s	5.33s	5.33s	10/10
23	hopx	0.0	0.00s	0.00s	0.00s	0/10

View full run · SVGs available as build artifacts

github-actions · 2026-07-30T20:34:24Z

Browser Throughput Benchmark Results

#	Provider	Score	APS (med)	Task (med)	Task (p95)	Screenshot	Status
1	Kernel	85.6	7.05/s	1.42s	1.90s	237ms	3/3
2	Tilion	73.1	4.23/s	2.36s	2.59s	245ms	3/3
3	Browseruse	71.6	3.90/s	2.56s	2.58s	250ms	3/3
4	Browserbase	68.8	3.39/s	2.95s	3.23s	246ms	3/3
5	Hyperbrowser	68.6	3.35/s	2.99s	3.14s	447ms	3/3
6	Steel	57.9	1.99/s	5.03s	8.25s	720ms	3/3
7	Notte	17.6	0.38/s	26.59s	43.35s	3607ms	3/3

View full run · SVG available as build artifact

github-actions · 2026-07-30T20:35:19Z

Sandbox Dax Benchmark Results

Provider	Phases	Total	Prepare	Clone	Install	Typecheck	Status
archil	0/7	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
beam	0/7	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
blaxel	6/7	42.25s	2.01s	2.13s	10.89s	0.00s	0/1 OK
cloud-run	2/7	1.54s	0.19s	0.00s	0.00s	0.00s	0/1 OK
cloudflare	0/7	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
codesandbox	0/7	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
createos	7/7	47.79s	2.05s	2.77s	11.55s	26.22s	1/1 OK
daytona	7/7	79.62s	3.11s	4.01s	16.43s	39.22s	1/1 OK
declaw	7/7	144.83s	5.75s	4.51s	21.16s	102.14s	1/1 OK
e2b	7/7	77.35s	9.25s	2.94s	15.68s	40.71s	1/1 OK
hopx	--	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
isorun	--	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
lightning	7/7	43.54s	5.40s	2.49s	13.54s	16.29s	1/1 OK
modal	7/7	98.49s	3.98s	2.32s	20.48s	62.99s	1/1 OK
namespace	7/7	40.83s	4.94s	1.01s	8.21s	20.68s	1/1 OK
northflank	--	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
opencomputer	6/7	25.86s	4.14s	1.94s	16.62s	0.00s	0/1 OK
runloop	2/7	110.27s	0.00s	0.00s	26.77s	50.01s	1/1 OK
sandbox0	7/7	183.26s	3.47s	6.62s	73.82s	78.94s	1/1 OK
superserve	7/7	92.45s	5.67s	5.69s	19.32s	58.21s	1/1 OK
tenki	7/7	67.29s	3.04s	4.29s	18.09s	35.65s	1/1 OK
tensorlake	7/7	57.20s	12.24s	1.67s	11.69s	27.39s	1/1 OK
upstash	7/7	51.19s	3.71s	1.77s	10.43s	24.75s	1/1 OK
vercel	7/7	75.64s	17.05s	1.93s	11.85s	38.06s	1/1 OK

View full run

github-actions · 2026-07-30T20:39:43Z

AI Gateway Benchmark Results

#	Gateway	Score	Cold E2E	Warm TTFT	Tok/sec	Status
1	LLM Gateway	90.5	713ms	711ms	84.5	20/20
2	Cloudflare AI Gateway	90.4	668ms	616ms	90.5	20/20
3	Anthropic (direct)	90.3	698ms	708ms	89.4	20/20
4	Vercel AI Gateway	90.2	722ms	690ms	85.7	20/20
5	Pydantic AI Gateway	89.5	716ms	645ms	88.1	20/20
6	OpenRouter	87.7	824ms	741ms	92.8	20/20
7	Concentrate AI	83.6	1090ms	833ms	0.0	20/20

View full run · SVG available as a build artifact

HeyGarrison self-assigned this Jul 30, 2026

devin-ai-integration Bot commented Jul 30, 2026

View reviewed changes

Comment thread benchmarks/sandbox/tti.bench.ts Outdated

Base automatically changed from devin/1785441088-bench-inline-workloads to master July 30, 2026 20:13

HeyGarrison and others added 2 commits July 30, 2026 20:15

refactor(benchmarks): one tti.bench.ts with --mode presets; add runne…

8b11235

…r --slug override Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

fix(benchmarks): accept --mode=value and own-property mode lookup

8df6b31

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration Bot force-pushed the devin/1785442068-tti-one-file branch from faa6b0c to 8df6b31 Compare July 30, 2026 20:15

refactor(benchmarks): drop the --mode preset table; pass knobs + --sl…

acf76a6

…ug/--name from CI Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

HeyGarrison merged commit 930d99b into master Jul 30, 2026
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

refactor(benchmarks): collapse the 3 sandbox TTI benches into one tti.bench.ts - #262

refactor(benchmarks): collapse the 3 sandbox TTI benches into one tti.bench.ts#262
HeyGarrison merged 3 commits into
masterfrom
devin/1785442068-tti-one-file

devin-ai-integration Bot commented Jul 30, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot commented Jul 30, 2026

Uh oh!

open-cla Bot commented Jul 30, 2026

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

devin-ai-integration Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

--slug / --name in @benchsdk/runner

Note (not changed here)

Uh oh!

devin-ai-integration Bot commented Jul 30, 2026

🤖 Devin AI Engineer

Uh oh!

open-cla Bot commented Jul 30, 2026

Contributor License Agreement

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 30, 2026

Storage Benchmark Results

1MB Files

Uh oh!

github-actions Bot commented Jul 30, 2026

Snapshot/Fork Benchmark Results

small dataset

Uh oh!

github-actions Bot commented Jul 30, 2026

Browser Benchmark Results

Uh oh!

github-actions Bot commented Jul 30, 2026

Sandbox Benchmark Results

Sequential

Staggered

Burst

Uh oh!

github-actions Bot commented Jul 30, 2026

Browser Throughput Benchmark Results

Uh oh!

github-actions Bot commented Jul 30, 2026

Sandbox Dax Benchmark Results

Uh oh!

github-actions Bot commented Jul 30, 2026

AI Gateway Benchmark Results

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

devin-ai-integration Bot commented Jul 30, 2026 •

edited

Loading

`--slug` / `--name` in `@benchsdk/runner`