feat(storage): benchmark Tensorlake Cloud Volumes (conflict-free rebase of #229) by devin-ai-integration[bot] · Pull Request #257 · computesdk/benchmarks

devin-ai-integration · 2026-07-30T16:21:38Z

Summary

Supersedes #229, which had merge conflicts and lives on a fork with maintainer_can_modify: false (so it couldn't be updated in place). This branch is diptanu's two commits merged with current master, conflicts resolved.

Adds Tensorlake Cloud Volumes to the storage and snapshot/fork matrices via the first-party @storagesdk/adapters/tensorlake adapter — no Tensorlake-specific code in this repo; snapshots/forks are native metadata calls.

Conflict resolution

master diverged in three ways since #229 branched:

benchmark scripts moved off benchmarks/src/run.ts --mode storage, and then again (in Split @benchsdk/client (REST) and @benchsdk/runner (framework); rework benchmark authoring to config + task #253) behind the @benchsdk/runner bin, so the new scripts follow master's current form:

"bench:storage:tensorlake":       "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/storage/storage.bench.ts --provider tensorlake"
"bench:snapshot-fork:tensorlake": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/storage/snapshot-fork.bench.ts --provider tensorlake"

both workflows dropped env: X: ${{ secrets.X }} blocks in favor of load-vault-secrets.sh '<regex>', so feat(storage): benchmark Tensorlake Cloud Volumes #229's TENSORLAKE_* env entries became TENSORLAKE_API_KEY|TENSORLAKE_API_URL|TENSORLAKE_ORGANIZATION_ID|TENSORLAKE_PROJECT_ID|TENSORLAKE_FILESYSTEM additions to each workflow's allowlist regex instead. Those five entries are now in the Namespace vault (confirmed by the smoke run below).
benchmarks/scripts/provider-vars.json is generated from requiredEnvVars; regenerated so storage and snapshot-fork list the TENSORLAKE_* vars.

pnpm-lock.yaml was regenerated from master's lockfile rather than merged. The StorageSDK release #229 was waiting on is out, so the dependency bumps (@storagesdk/adapters@^0.10.0, @storagesdk/core@^0.5.0, tensorlake@0.5.89) resolve against the registry.

benchmarks/storage/providers.ts merged cleanly (unchanged from #229).

Validation

pnpm install --frozen-lockfile, pnpm typecheck
Live 1 MiB storage smoke test on this branch against the vault credentials — 3/3 succeeded (run, dry_run=true, whole 7-provider matrix green):

iter upload download throughput

1 491.4 ms 95.6 ms 87.8 Mbps

2 122.3 ms 47.6 ms 176.2 Mbps

3 136.1 ms 99.0 ms 84.8 Mbps

#229's quoted percentiles were measured on an arm64 macOS runner and don't reflect what CI will record.

Open item

The Superagent supply-chain gate is red on 6 heuristic findings across the three dependency bumps. I reviewed each against the package contents and believe all are false positives (details in the two review threads: the threat-process-download-exec hit is a curl | sh string inside an error message; threat-network-outbound-shady-links is Backblaze's own domain). Needs a human sign-off; I have not touched the scanner config.

Link to Devin session: https://app.devin.ai/sessions/6ad3ed1fd503463482618cd1497d11c4

Resolve conflicts: adopt master's per-bench script paths and vault secret loading, keeping the Tensorlake matrix entries, scripts, and dependencies. Co-Authored-By: Noah Kiser <noah@computesdk.com>

devin-ai-integration · 2026-07-30T16:21:40Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment, CI, and merge conflict monitoring

open-cla · 2026-07-30T16:21:44Z

Contributor License Agreement

All contributors are covered by a CLA.

superagent-security

Superagent found 2 supply chain issue(s)

devin-ai-integration

Devin Review found 1 potential issue.

View 2 additional findings in Devin Review.

Co-Authored-By: Noah Kiser <noah@computesdk.com>

Resolve package.json conflict: master routes bench scripts through packages/benchsdk-runner/dist/bin.js, so the tensorlake storage and snapshot-fork scripts adopt that form. Co-Authored-By: Noah Kiser <noah@computesdk.com>

github-actions · 2026-07-30T16:54:08Z

Storage Benchmark Results

1MB Files

#	Provider	Score	Download	Throughput	Upload	Status
1	Azure Blob Storage	96.5	0.03s	327.1 Mbps	0.06s	10/10
2	Tigris	95.2	0.04s	239.3 Mbps	0.51s	10/10
3	Tensorlake	95.0	0.05s	179.1 Mbps	0.49s	10/10
4	AWS S3	94.8	0.10s	85.1 Mbps	0.21s	10/10
5	Vercel Blob	94.4	0.21s	39.3 Mbps	0.24s	10/10
6	Cloudflare R2	94.2	0.16s	52.8 Mbps	0.41s	10/10
7	Google Cloud Storage	0.0	0.00s	0.0 Mbps	0.00s	0/10

View full run · SVGs available as build artifacts

github-actions · 2026-07-30T16:54:08Z

Snapshot/Fork Benchmark Results

small dataset

#	Provider	Score	Snapshot create	Fork (snapshot)	Fork (live)	First read	Status
1	Tensorlake	99.8	0.03s	0.20s	0.29s	0.11s	1/1
2	Tigris	99.5	0.22s	0.35s	0.46s	0.33s	1/1
3	Azure-blob	98.7	0.90s	0.68s	1.32s	0.02s	1/1
4	Cloudflare R2	94.8	2.88s	3.16s	5.65s	0.30s	1/1
5	AWS S3	0.0	0.00s	0.00s	0.00s	0.00s	0/1

View full run

github-actions · 2026-07-30T16:54:29Z

Browser Benchmark Results

#	Provider	Score	Create	Connect	Navigate	Release	Total	Status
1	Kernel	98.0	0.02s	0.08s	0.12s	0.03s	0.26s	10/10
2	Tilion	97.2	0.02s	0.04s	0.10s	0.01s	0.18s	10/10
3	Browseruse	95.1	0.20s	0.12s	0.06s	0.05s	0.45s	10/10
4	Browserbase	92.9	0.23s	0.11s	0.09s	0.13s	0.58s	10/10
5	Steel	82.3	0.65s	0.43s	0.12s	0.44s	1.69s	10/10
6	Notte	70.4	0.37s	1.45s	0.39s	0.36s	2.93s	10/10
7	Hyperbrowser	61.5	0.39s	0.54s	0.72s	0.10s	1.83s	10/10

View full run · SVG available as build artifact

github-actions · 2026-07-30T16:54:45Z

Browser Throughput Benchmark Results

#	Provider	Score	APS (med)	Task (med)	Task (p95)	Screenshot	Status
1	Kernel	79.9	5.68/s	1.76s	1.87s	224ms	3/3
2	Tilion	70.1	3.61/s	2.77s	2.81s	302ms	3/3
3	Browseruse	66.4	2.96/s	3.38s	3.68s	315ms	3/3
4	Browserbase	66.3	2.99/s	3.35s	4.06s	310ms	3/3
5	Steel	57.8	1.86/s	5.38s	7.30s	596ms	3/3
6	Hyperbrowser	50.4	1.27/s	7.89s	11.40s	934ms	3/3
7	Notte	34.9	0.63/s	15.91s	20.16s	1916ms	3/3

View full run · SVG available as build artifact

github-actions · 2026-07-30T16:55:37Z

Sandbox Benchmark Results

Sequential

#	Provider	Score	Median TTI	P95	P99	Status
1	isorun	99.6	0.02s	0.07s	0.07s	10/10
2	declaw	99.2	0.04s	0.14s	0.14s	10/10
3	createos	99.0	0.08s	0.13s	0.13s	10/10
4	northflank	98.8	0.10s	0.16s	0.16s	10/10
5	archil	98.0	0.15s	0.28s	0.28s	10/10
6	blaxel	97.5	0.24s	0.26s	0.26s	10/10
7	lightning	97.4	0.17s	0.39s	0.39s	10/10
8	superserve	96.4	0.34s	0.39s	0.39s	10/10
9	tensorlake	96.2	0.34s	0.43s	0.43s	10/10
10	upstash	95.4	0.37s	0.59s	0.59s	10/10
11	beam	95.3	0.14s	0.95s	0.95s	10/10
12	cloud-run	94.7	0.52s	0.55s	0.55s	10/10
13	runloop	94.0	0.47s	0.79s	0.79s	10/10
14	modal	93.1	0.56s	0.89s	0.89s	10/10
15	e2b	90.7	0.80s	1.13s	1.13s	10/10
16	vercel	89.4	0.59s	1.77s	1.77s	10/10
17	tenki	88.8	0.84s	1.55s	1.55s	10/10
18	daytona	85.8	1.25s	1.69s	1.69s	10/10
19	opencomputer	80.5	1.85s	2.11s	2.11s	10/10
20	sandbox0	78.0	1.85s	2.73s	2.73s	10/10
21	cloudflare	74.5	2.36s	2.85s	2.85s	10/10
22	codesandbox	72.8	2.54s	3.00s	3.00s	10/10
23	hopx	0.0	0.00s	0.00s	0.00s	0/10

Staggered

#	Provider	Score	Median TTI	P95	P99	Status
1	isorun	99.8	0.02s	0.02s	0.02s	10/10
2	declaw	99.3	0.05s	0.11s	0.11s	10/10
3	createos	99.1	0.08s	0.12s	0.12s	10/10
4	northflank	98.9	0.09s	0.15s	0.15s	10/10
5	archil	98.3	0.14s	0.21s	0.21s	10/10
6	lightning	98.1	0.18s	0.20s	0.20s	10/10
7	blaxel	97.5	0.24s	0.27s	0.27s	10/10
8	superserve	97.0	0.28s	0.33s	0.33s	10/10
9	tensorlake	96.8	0.27s	0.40s	0.40s	10/10
10	upstash	95.5	0.42s	0.49s	0.49s	10/10
11	vercel	95.5	0.41s	0.52s	0.52s	10/10
12	runloop	94.4	0.53s	0.61s	0.61s	10/10
13	beam	93.8	0.27s	1.14s	1.14s	10/10
14	modal	93.6	0.57s	0.75s	0.75s	10/10
15	e2b	91.5	0.76s	0.97s	0.97s	10/10
16	tenki	91.4	0.77s	1.00s	1.00s	10/10
17	daytona	89.7	0.83s	1.33s	1.33s	10/10
18	cloud-run	85.2	0.62s	2.79s	2.79s	10/10
19	codesandbox	76.7	2.26s	2.43s	2.43s	10/10
20	opencomputer	74.9	1.84s	3.51s	3.51s	10/10
21	sandbox0	66.6	3.12s	3.67s	3.67s	10/10
22	cloudflare	64.4	3.17s	4.15s	4.15s	10/10
23	hopx	0.0	0.00s	0.00s	0.00s	0/10

Burst

#	Provider	Score	Median TTI	P95	P99	Status
1	declaw	98.9	0.11s	0.12s	0.12s	10/10
2	createos	98.6	0.14s	0.14s	0.14s	10/10
3	northflank	98.5	0.13s	0.18s	0.18s	10/10
4	lightning	98.2	0.18s	0.19s	0.19s	10/10
5	archil	98.0	0.18s	0.22s	0.22s	10/10
6	blaxel	97.0	0.29s	0.31s	0.31s	10/10
7	tensorlake	96.6	0.33s	0.35s	0.35s	10/10
8	superserve	96.6	0.32s	0.37s	0.37s	10/10
9	upstash	94.2	0.57s	0.59s	0.59s	10/10
10	runloop	91.9	0.79s	0.84s	0.84s	10/10
11	e2b	91.6	0.78s	0.92s	0.92s	10/10
12	modal	90.5	0.56s	1.55s	1.55s	10/10
13	tenki	86.2	1.36s	1.42s	1.42s	10/10
14	beam	84.7	1.52s	1.55s	1.55s	10/10
15	vercel	80.0	0.48s	4.28s	4.28s	10/10
16	isorun	77.6	0.05s	5.54s	5.54s	10/10
17	cloud-run	73.5	0.59s	5.72s	5.72s	10/10
18	codesandbox	72.0	2.58s	3.11s	3.11s	10/10
19	opencomputer	69.7	2.58s	3.71s	3.71s	10/10
20	cloudflare	60.5	2.55s	6.04s	6.04s	10/10
21	sandbox0	55.8	3.84s	5.29s	5.29s	10/10
22	daytona	42.2	5.19s	6.66s	6.66s	10/10
23	hopx	0.0	0.00s	0.00s	0.00s	0/10

View full run · SVGs available as build artifacts

github-actions · 2026-07-30T16:56:24Z

Sandbox Dax Benchmark Results

Provider	Phases	Total	Prepare	Clone	Install	Typecheck	Status
archil	0/7	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
beam	0/7	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
blaxel	6/7	42.25s	2.25s	2.11s	10.89s	0.00s	0/1 OK
cloud-run	2/7	0.93s	0.15s	0.00s	0.00s	0.00s	0/1 OK
cloudflare	0/7	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
codesandbox	0/7	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
createos	7/7	50.65s	2.77s	2.07s	10.74s	26.16s	1/1 OK
daytona	7/7	69.87s	3.30s	2.24s	14.28s	36.66s	1/1 OK
declaw	7/7	143.21s	5.96s	4.76s	10.69s	111.27s	1/1 OK
e2b	7/7	75.80s	9.40s	4.15s	9.23s	42.46s	1/1 OK
hopx	--	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
isorun	--	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
lightning	7/7	43.36s	5.36s	2.60s	12.60s	16.89s	1/1 OK
modal	7/7	100.54s	4.09s	4.29s	20.43s	62.30s	1/1 OK
namespace	6/7	54.40s	4.36s	1.32s	8.35s	0.00s	0/1 OK
northflank	--	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
opencomputer	6/7	24.76s	4.14s	1.89s	16.44s	0.00s	0/1 OK
runloop	2/7	80.20s	0.00s	0.00s	16.59s	38.20s	1/1 OK
sandbox0	7/7	184.77s	3.71s	7.66s	73.26s	80.27s	1/1 OK
superserve	7/7	90.35s	5.07s	5.06s	17.47s	59.14s	1/1 OK
tenki	7/7	71.53s	3.53s	4.11s	15.71s	41.45s	1/1 OK
tensorlake	7/7	57.69s	12.43s	1.56s	11.07s	28.40s	1/1 OK
upstash	0/7	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
vercel	7/7	83.45s	17.38s	2.20s	12.41s	44.12s	1/1 OK

View full run

github-actions · 2026-07-30T17:01:33Z

AI Gateway Benchmark Results

#	Gateway	Score	Cold E2E	Warm TTFT	Tok/sec	Status
1	Pydantic AI Gateway	90.5	682ms	647ms	89.9	20/20
2	Anthropic (direct)	90.5	592ms	601ms	94.3	20/20
3	Cloudflare AI Gateway	90.5	706ms	677ms	93.4	20/20
4	OpenRouter	90.4	730ms	682ms	89.3	20/20
5	Vercel AI Gateway	90.3	720ms	757ms	88.4	20/20
6	LLM Gateway	90.2	778ms	679ms	84.2	20/20
7	Concentrate AI	85.5	960ms	808ms	0.0	20/20

View full run · SVG available as a build artifact

diptanu and others added 3 commits July 26, 2026 17:54

feat(storage): benchmark Tensorlake filesystems

5be7605

refactor(storage): use the official Tensorlake adapter

85c95d4

merge master into agent/add-tensorlake-storage

aa8206a

Resolve conflicts: adopt master's per-bench script paths and vault secret loading, keeping the Tensorlake matrix entries, scripts, and dependencies. Co-Authored-By: Noah Kiser <noah@computesdk.com>

superagent-security Bot reviewed Jul 30, 2026

View reviewed changes

Comment thread pnpm-lock.yaml

Comment thread pnpm-lock.yaml

devin-ai-integration Bot commented Jul 30, 2026

View reviewed changes

Comment thread benchmarks/storage/providers.ts

devin-ai-integration Bot and others added 2 commits July 30, 2026 16:23

chore(storage): regenerate provider-vars manifest for tensorlake

9fc7c1f

Co-Authored-By: Noah Kiser <noah@computesdk.com>

merge master into devin/1785428500-tensorlake-storage-merge

b7a3df1

Resolve package.json conflict: master routes bench scripts through packages/benchsdk-runner/dist/bin.js, so the tensorlake storage and snapshot-fork scripts adopt that form. Co-Authored-By: Noah Kiser <noah@computesdk.com>

kisernl merged commit 6f6eaea into master Jul 30, 2026
5 of 6 checks passed

iter	upload	download	throughput
1	491.4 ms	95.6 ms	87.8 Mbps
2	122.3 ms	47.6 ms	176.2 Mbps
3	136.1 ms	99.0 ms	84.8 Mbps

Uh oh!

Conversation

devin-ai-integration Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Conflict resolution

Validation

Open item

Uh oh!

devin-ai-integration Bot commented Jul 30, 2026

🤖 Devin AI Engineer

Uh oh!

open-cla Bot commented Jul 30, 2026

Contributor License Agreement

Uh oh!

superagent-security Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 30, 2026

Storage Benchmark Results

1MB Files

Uh oh!

github-actions Bot commented Jul 30, 2026

Snapshot/Fork Benchmark Results

small dataset

Uh oh!

github-actions Bot commented Jul 30, 2026

Browser Benchmark Results

Uh oh!

github-actions Bot commented Jul 30, 2026

Browser Throughput Benchmark Results

Uh oh!

github-actions Bot commented Jul 30, 2026

Sandbox Benchmark Results

Sequential

Staggered

Burst

Uh oh!

github-actions Bot commented Jul 30, 2026

Sandbox Dax Benchmark Results

Uh oh!

github-actions Bot commented Jul 30, 2026

AI Gateway Benchmark Results

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

devin-ai-integration Bot commented Jul 30, 2026 •

edited

Loading