Skip to content

feat: run lightning at cpu-8 (8 vCPU / 16 GiB) in dax benchmark - #199

Merged
kisernl merged 1 commit into
computesdk:masterfrom
gridai:dax-lightning-cpu-8
Jul 24, 2026
Merged

feat: run lightning at cpu-8 (8 vCPU / 16 GiB) in dax benchmark#199
kisernl merged 1 commit into
computesdk:masterfrom
gridai:dax-lightning-cpu-8

Conversation

@rusenask

Copy link
Copy Markdown
Contributor

What

Sizes the lightning provider to 8 vCPU / 16 GiB (cpu-8) in the dax benchmark, matching the standardized resource profile introduced in #196.

Why

Lightning was already in the dax matrix but had no resource sizing, so it ran on the SDK default cpu-1 (1 vCPU) while every other provider used 8 vCPU / 16 GiB — making its results incomparable (the same issue #196 fixed for the rest of the matrix).

How

Lightning can't be sized through DAX_RESOURCE_OPTIONS like most providers. The @computesdk/lightning wrapper applies instanceType from the provider factory config and overrides any instanceType passed to sandbox.create():

const params = { ...providerOptions, instanceType: config.instanceType ?? DEFAULT_INSTANCE_TYPE };

So sizing has to happen at createCompute time. This PR:

  • Reads LIGHTNING_INSTANCE_TYPE in createCompute (default cpu-1, the SDK default), mirroring how northflank reads its deployment plan from env with a fallback.
  • Sets LIGHTNING_INSTANCE_TYPE: cpu-8 in the dax workflow env, scoping the 8 vCPU sizing to the dax benchmark and leaving other modes (e.g. TTI) on the default.
  • Documents the mechanism in dax.ts (next to the existing E2B note) and env.example.

Verification

Ran the dax benchmark against lightning.ai with cpu-8:

lightning: 7/7 phases | total 41.93s | prepare 4.37s | clone 2.41s | install 12.91s | typecheck 16.04s (1/1 OK)

The sandbox reports 8 logical CPUs, confirming cpu-8 took effect, and all 7 dax phases complete.

🤖 Generated with Claude Code

@open-cla

open-cla Bot commented Jul 21, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

Lightning was in the dax matrix but had no resource sizing, so it ran on
the SDK default cpu-1 (1 vCPU) while other providers standardized to
8 vCPU / 16 GiB, making results incomparable.

Lightning's @computesdk/lightning wrapper sets instanceType from the
provider factory config and overrides any instanceType passed to
sandbox.create(), so it can't be sized through DAX_RESOURCE_OPTIONS like
most providers. Instead, read LIGHTNING_INSTANCE_TYPE in createCompute
(default cpu-1, the SDK default) and set it to cpu-8 in the dax workflow,
mirroring how northflank reads its plan from env with a fallback. This
scopes the 8 vCPU sizing to the dax benchmark and leaves other modes
(e.g. TTI) on the default.

Verified against lightning.ai: sandbox reports 8 logical CPUs, all 7
dax phases complete.
@rusenask
rusenask force-pushed the dax-lightning-cpu-8 branch from 33d0c60 to e7b2015 Compare July 23, 2026 18:11
@kisernl
kisernl merged commit 9a33f35 into computesdk:master Jul 24, 2026
3 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Sandbox Dax Benchmark Results

Provider Phases Total Prepare Clone Install Typecheck Status
archil 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
beam -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
blaxel 6/7 45.69s 2.09s 2.29s 10.79s 0.00s 0/1 OK
cloud-run 2/7 0.97s 0.19s 0.00s 0.00s 0.00s 0/1 OK
cloudflare 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
codesandbox 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
createos 7/7 48.61s 3.00s 2.52s 11.90s 25.84s 1/1 OK
daytona 7/7 68.22s 2.89s 1.91s 14.43s 36.24s 1/1 OK
declaw 7/7 154.82s 15.73s 9.87s 31.83s 77.92s 1/1 OK
e2b 7/7 71.84s 8.79s 2.99s 10.26s 41.02s 1/1 OK
hopx -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
isorun -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
lightning 6/7 10.80s 4.97s 2.91s 1.03s 0.00s 0/1 OK
modal 7/7 96.18s 4.26s 2.48s 15.32s 65.95s 1/1 OK
namespace 7/7 38.77s 4.54s 0.96s 7.59s 19.95s 1/1 OK
northflank -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
opencomputer 6/7 24.49s 3.47s 1.70s 16.45s 0.00s 0/1 OK
runloop 2/7 84.29s 0.00s 0.00s 17.50s 35.92s 1/1 OK
superserve 7/7 100.98s 6.30s 4.55s 21.56s 64.31s 1/1 OK
tenki 6/7 26.30s 8.82s 4.31s 9.62s 0.00s 0/1 OK
tensorlake 7/7 76.15s 12.76s 1.51s 11.96s 43.95s 1/1 OK
upstash 7/7 51.59s 3.83s 1.57s 10.56s 25.62s 1/1 OK
vercel 7/7 93.97s 23.22s 2.46s 14.07s 46.03s 1/1 OK

View full run

@github-actions

Copy link
Copy Markdown
Contributor

Sandbox Benchmark Results

Sequential

# Provider Score Median TTI P95 P99 Status
1 isorun 99.7 0.01s 0.07s 0.07s 10/10
2 createos 99.3 0.05s 0.10s 0.10s 10/10
3 declaw 99.3 0.05s 0.11s 0.11s 10/10
4 northflank 98.8 0.08s 0.17s 0.17s 10/10
5 archil 98.1 0.13s 0.28s 0.28s 10/10
6 lightning 98.0 0.17s 0.24s 0.24s 10/10
7 blaxel 97.7 0.21s 0.25s 0.25s 10/10
8 upstash 96.3 0.30s 0.47s 0.47s 10/10
9 superserve 96.3 0.29s 0.49s 0.49s 10/10
10 modal 94.7 0.46s 0.64s 0.64s 10/10
11 cloud-run 94.1 0.51s 0.71s 0.71s 10/10
12 tensorlake 92.8 0.65s 0.83s 0.83s 10/10
13 vercel 92.6 0.34s 1.33s 1.33s 10/10
14 e2b 91.5 0.76s 0.97s 0.97s 10/10
15 runloop 91.1 0.48s 1.51s 1.51s 10/10
16 tenki 88.0 1.03s 1.45s 1.45s 10/10
17 daytona 84.2 1.13s 2.25s 2.25s 10/10
18 cloudflare 77.6 1.87s 2.80s 2.80s 10/10
19 codesandbox 72.7 2.52s 3.06s 3.06s 10/10
20 opencomputer 47.8 2.03s 17.85s 17.85s 10/10
21 beam 0.0 0.00s 0.00s 0.00s 0/10
22 hopx 0.0 0.00s 0.00s 0.00s 0/10

Staggered

# Provider Score Median TTI P95 P99 Status
1 isorun 99.8 0.01s 0.02s 0.02s 10/10
2 createos 99.5 0.05s 0.06s 0.06s 10/10
3 northflank 99.2 0.07s 0.09s 0.09s 10/10
4 declaw 99.1 0.06s 0.13s 0.13s 10/10
5 lightning 98.2 0.17s 0.20s 0.20s 10/10
6 archil 97.9 0.12s 0.34s 0.34s 10/10
7 blaxel 97.7 0.22s 0.25s 0.25s 10/10
8 superserve 97.0 0.28s 0.32s 0.32s 10/10
9 runloop 94.8 0.49s 0.58s 0.58s 10/10
10 modal 94.6 0.48s 0.65s 0.65s 10/10
11 cloud-run 94.2 0.54s 0.63s 0.63s 10/10
12 tensorlake 93.5 0.62s 0.68s 0.68s 10/10
13 e2b 93.0 0.64s 0.81s 0.81s 10/10
14 tenki 90.0 0.98s 1.04s 1.04s 10/10
15 vercel 84.1 0.34s 3.47s 3.47s 10/10
16 upstash 80.8 0.36s 4.26s 4.26s 10/10
17 cloudflare 78.9 1.82s 2.54s 2.54s 10/10
18 codesandbox 74.2 2.44s 2.80s 2.80s 10/10
19 daytona 57.1 1.25s 8.85s 8.85s 10/10
20 opencomputer 34.1 4.31s 35.27s 35.27s 10/10
21 beam 0.0 0.00s 0.00s 0.00s 0/10
22 hopx 0.0 0.00s 0.00s 0.00s 0/10

Burst

# Provider Score Median TTI P95 P99 Status
1 isorun 99.8 0.02s 0.02s 0.02s 10/10
2 createos 99.2 0.08s 0.09s 0.09s 10/10
3 northflank 98.9 0.11s 0.11s 0.11s 10/10
4 declaw 98.5 0.07s 0.26s 0.26s 10/10
5 lightning 98.4 0.16s 0.17s 0.17s 10/10
6 archil 98.3 0.16s 0.20s 0.20s 10/10
7 blaxel 97.6 0.23s 0.26s 0.26s 10/10
8 superserve 96.2 0.35s 0.42s 0.42s 10/10
9 upstash 94.7 0.44s 0.68s 0.68s 10/10
10 vercel 94.7 0.39s 0.75s 0.75s 10/10
11 runloop 94.6 0.50s 0.60s 0.60s 10/10
12 cloud-run 94.3 0.51s 0.67s 0.67s 10/10
13 modal 93.8 0.58s 0.68s 0.68s 10/10
14 tensorlake 93.0 0.69s 0.71s 0.71s 10/10
15 e2b 91.6 0.76s 0.96s 0.96s 10/10
16 tenki 89.4 1.04s 1.08s 1.08s 10/10
17 codesandbox 69.8 2.80s 3.35s 3.35s 10/10
18 daytona 54.6 0.91s 25.39s 25.39s 10/10
19 cloudflare 52.9 3.46s 5.12s 5.12s 9/10
20 opencomputer 34.3 4.28s 35.77s 35.77s 10/10
21 beam 9.7 0.32s 0.32s 0.32s 1/10
22 hopx 0.0 0.00s 0.00s 0.00s 0/10

View full run · SVGs available as build artifacts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants