fix: standardize dax benchmark resources to 8 vCPU / 16 GiB - #196
Merged
Conversation
Each provider gets 8 vCPUs and 16 GiB RAM for fair comparison: - E2B: cpuCount=8, memoryMB=16384 - Modal: cpu=4 (4 physical = 8 vCPUs), memoryMiB=16384 - Tensorlake: cpus=8, memoryMb=16384 - isorun: vcpus=8, memMiB=16384 - runloop: customCpuCores=8, customMemoryGb=16 - daytona: resources.cpu=8, resources.memory=16 (GiB) - upstash: size=large (8 cores, 16 GB) - vercel: resources.vcpus=8 (no memory control) - blaxel: memory=16384 (CPU derived: 16384/2048=8 cores) Providers without CPU/memory control (archil, cloudflare, northflank, beam, declaw, hopx, codesandbox) use their defaults.
Contributor License AgreementAll contributors are covered by a CLA. |
This comment was marked as off-topic.
This comment was marked as off-topic.
Contributor
Sandbox Dax Benchmark Results
|
… time E2B's Sandbox.create() accepts SandboxOpts which does not include cpuCount/memoryMB. Those are template build options only.
Contributor
|
For declaw, the cpu, memory and disk are property of the template, you can create a custom template with the required size or we can have a node-large template with above spec |
- Fix runloop: nest resource params in launch_parameters with snake_case - Add beam: cpu=8, memory=16384 MiB - Add codesandbox: vmTier=VMTier.Small (8 CPU / 16 GiB) - Add northflank: deploymentPlan resolved via API at runtime - Add E2B template build prereq (base-8cpu-16gb with 8 vCPU / 16 GiB) - Add Northflank plan discovery prereq script
- Add scripts/provider-benchmark.sh: vendored from upstream with dnf support (RHEL/Fedora) alongside apt-get (Debian/Ubuntu) and aarch64 architecture detection for Node.js and Bun downloads - Update dax.ts to load script from local filesystem instead of curl, eliminating curl dependency for providers that don't ship it
Adds declaw to DAX_RESOURCE_OPTIONS so it targets the same 8 vCPU / 16 GiB profile as the other providers in the map. The node-large template provisions 8 vCPU / 16 GiB RAM with an 8 GiB disk overlay. Co-authored-by: shivam-declaw <272314190+shivam-declaw@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![superagent-security[bot]](https://avatars.githubusercontent.com/in/3287076?s=60&v=4){kind=small}
superagent-security
Bot
added
the
pr:flagged
…ce node image - Fix BUN_INTERNAL_PATH: remove .zip extension from zip-internal path (was bun-linux-x64-baseline.zip/bun, should be bun-linux-x64-baseline/bun) - Northflank: robust API response parsing, graceful fallback to nf-compute-50, sanitize plan ID before writing to GITHUB_ENV - Daytona: remove from DAX_RESOURCE_OPTIONS (resources not supported with snapshot-based sandbox creation) - Namespace: add sandboxOptions image node:22 (dax benchmark requires node)
superagent-security
Bot
removed
the
pr:flagged
- Northflank: pick smallest suitable plan instead of closest to reduce chance of exceeding project resource allowance - E2B: workaround for 60s command timeout by setting defaultProcessConnectionTimeout=0 via getInstance() while waiting for upstream wrapper fix to forward timeoutMs
Add image: 'node:22' to daytona sandboxOptions to switch from snapshot-based to image-based creation, which allows specifying resources. Re-add daytona to DAX_RESOURCE_OPTIONS with 8 vCPU / 16 GiB.
superagent-security
Bot
added
pr:flagged
HeyGarrison
force-pushed
the
fix/dax-standardize-resources
branch
from
July 20, 2026 15:07
2224965 to
19c21a1
Compare
superagent-security
Bot
added
the
pr:flagged
Update all @computesdk/* packages to versions from changeset PR #652: - @computesdk/e2b@1.7.52: forwards RunCommandOptions.timeout to E2B commands - @computesdk/hopx@0.2.27: forwards RunCommandOptions.timeout to hopx commands - @computesdk/runloop@1.3.54: properly merges launch_parameters - computesdk@4.1.4: typed SandboxResourceOptions Remove E2B defaultProcessConnectionTimeout workaround (no longer needed since @computesdk/e2b now forwards timeout natively). Pass timeout to sandbox.runCommand() so the native timeout forwarding kicks in for E2B and hopx.
Contributor
Snapshot/Fork Benchmark Resultssmall dataset
|
Contributor
Storage Benchmark Results1MB Files
View full run · SVGs available as build artifacts |
Contributor
Browser Benchmark Results
View full run · SVG available as build artifact |
Contributor
Browser Throughput Benchmark Results
View full run · SVG available as build artifact |
superagent-security
Bot
added
pr:flagged
Upstash: remove ephemeral: true from sandboxOptions. The ComputeSDK wrapper's ephemeral path doesn't forward 'size' to EphemeralBox.create(), so size: 'large' was silently dropped and the box ran on 'small'. Using a regular Box ensures the resource sizing is applied. Superserve: add vcpu: 8, memoryMib: 16384 to DAX_RESOURCE_OPTIONS. Previously had no resource sizing at all - relied on template defaults which may be undersized. The superserve SDK accepts these as overrides on sandbox creation.
Omit explicit image from namespace sandboxOptions so the provider defaults to its builtin:base image, as intended by the namespace team.
superagent-security
Bot
removed
the
pr:flagged
E2B default sandbox lifespan is 5 minutes (300s). The benchmark can take longer on cold starts or under resource contention, causing the sandbox to be killed mid-execution. Pass timeout: 600_000 (10 min) to Sandbox.create() via the timeoutMs option.
The E2B ComputeSDK provider destructures 'templateId' from sandbox options, not 'template'. Passing 'template' caused it to be treated as an unknown passthrough option, and the sandbox was created from E2B's default template instead of our custom base-8cpu-16gb template.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Standardizes sandbox sizing across providers for fair comparison. Previously Modal was bursting to 17 vCPUs while Tensorlake had 1 vCPU, making results incomparable.
Sized providers (8 vCPU / 16 GiB)
Providers without CPU/memory control (use defaults)
archil, cloudflare, northflank, beam, declaw, hopx, codesandbox