Skip to content

fix: use stateful mode for cloud-run and handle missing packages - #202

Open
HeyGarrison wants to merge 2 commits into
masterfrom
fix/dax-cloud-run-stateful
Open

fix: use stateful mode for cloud-run and handle missing packages#202
HeyGarrison wants to merge 2 commits into
masterfrom
fix/dax-cloud-run-stateful

Conversation

@HeyGarrison

@HeyGarrison HeyGarrison commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

Gets cloud-run past 2/7 phases in the dax benchmark. Two independent causes:

  1. No filesystem persistence. The default ephemeral mode runs each command via sandbox do in a fresh sandbox, so nothing the script writes survives. cloud-run now uses executionMode: 'stateful' (sandbox run/exec/delete) plus write: true and allowEgress: true for the bun/node downloads and git clone.

  2. apt-get is all-or-nothing. cloud-run's gVisor rootfs has no python3-setuptools or unzip, and a single unknown argument makes apt install nothing — which is why the log showed curl: command not found after the apt failure. The fix filters unavailable optionals out of one install invocation rather than adding a second (prepare is a timed phase):

-apt-get install -y -qq bash build-essential ca-certificates curl git python3 python3-setuptools unzip
+packages=(bash build-essential ca-certificates curl git python3)
+for optional in python3-setuptools unzip; do
+  apt-cache show "$optional" >/dev/null 2>&1 && packages+=("$optional")
+done
+apt-get install -y -qq "${packages[@]}"

dnf installs the same two optionals one at a time with || true (same all-or-nothing problem, but dnf providers are rare enough that the extra round trip doesn't matter). apk is unchanged — Alpine ships both.

unpack_bun() falls back to python3 + zipfile when unzip is absent, via a quoted heredoc with paths passed as sys.argv so nothing is interpolated into the Python source. Verified locally against a synthetic zip with unzip removed from PATH: byte-identical output to the unzip -j path.

Note for reviewers

benchmarks/sandbox/providers.ts is shared by every sandbox mode (benchmarks/src/run.ts), so the stateful switch also affects cloud-run's sequential/staggered TTI numbers (currently 91.5, 10/10) — sandbox do and sandbox run/exec are different code paths. Worth sanity-checking the benchmark comment on this PR for a TTI regression before merging.

Rebased onto the post-#207 layout (scripts/benchmarks/scripts/, src/sandbox/benchmarks/sandbox/); the previous merge conflict is gone. pnpm typecheck passes.

Link to Devin session: https://app.devin.ai/sessions/5bbf1eb8873f49b5b41a78351c86c70a
Requested by: @HeyGarrison

@open-cla

open-cla Bot commented Jul 21, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Sandbox Benchmark Results

Sequential

# Provider Score Median TTI P95 P99 Status
1 isorun 99.7 0.01s 0.05s 0.05s 10/10
2 createos 99.2 0.07s 0.11s 0.11s 10/10
3 northflank 98.7 0.09s 0.19s 0.19s 10/10
4 declaw 98.6 0.09s 0.22s 0.22s 10/10
5 blaxel 97.9 0.20s 0.23s 0.23s 10/10
6 lightning 97.1 0.22s 0.39s 0.39s 10/10
7 vercel 95.6 0.35s 0.58s 0.58s 10/10
8 archil 95.1 0.13s 1.03s 1.03s 10/10
9 modal 93.4 0.50s 0.90s 0.90s 10/10
10 superserve 93.3 0.59s 0.80s 0.80s 10/10
11 runloop 92.8 0.62s 0.88s 0.88s 10/10
12 tensorlake 91.5 0.78s 0.96s 0.96s 10/10
13 cloud-run 91.5 0.53s 1.34s 1.34s 10/10
14 e2b 90.0 0.87s 1.19s 1.19s 10/10
15 beam 86.2 0.40s 2.85s 2.85s 10/10
16 tenki 84.4 1.34s 1.89s 1.89s 10/10
17 cloudflare 83.1 1.62s 1.81s 1.81s 10/10
18 upstash 77.9 2.13s 2.33s 2.33s 10/10
19 codesandbox 74.7 2.42s 2.70s 2.70s 10/10
20 daytona 54.6 0.90s 12.02s 12.02s 10/10
21 hopx 0.0 0.00s 0.00s 0.00s 0/10

Staggered

# Provider Score Median TTI P95 P99 Status
1 isorun 99.8 0.01s 0.02s 0.02s 10/10
2 createos 99.3 0.07s 0.08s 0.08s 10/10
3 northflank 99.0 0.09s 0.13s 0.13s 10/10
4 declaw 98.9 0.05s 0.19s 0.19s 10/10
5 archil 97.7 0.11s 0.40s 0.40s 10/10
6 blaxel 97.6 0.22s 0.27s 0.27s 10/10
7 lightning 97.5 0.22s 0.28s 0.28s 10/10
8 beam 96.1 0.38s 0.40s 0.40s 10/10
9 vercel 95.5 0.37s 0.57s 0.57s 10/10
10 modal 94.0 0.53s 0.70s 0.70s 10/10
11 superserve 93.9 0.52s 0.73s 0.73s 10/10
12 runloop 93.9 0.59s 0.64s 0.64s 10/10
13 tensorlake 92.1 0.74s 0.86s 0.86s 10/10
14 e2b 90.8 0.84s 1.04s 1.04s 10/10
15 daytona 86.9 0.98s 1.82s 1.82s 10/10
16 tenki 86.4 1.30s 1.45s 1.45s 10/10
17 cloud-run 82.6 0.83s 3.11s 3.11s 10/10
18 cloudflare 78.6 1.66s 2.86s 2.86s 10/10
19 upstash 78.3 2.12s 2.25s 2.25s 10/10
20 codesandbox 75.2 2.33s 2.71s 2.71s 10/10
21 hopx 0.0 0.00s 0.00s 0.00s 0/10

Burst

# Provider Score Median TTI P95 P99 Status
1 createos 99.0 0.09s 0.11s 0.11s 10/10
2 declaw 98.9 0.07s 0.17s 0.17s 10/10
3 northflank 98.7 0.12s 0.15s 0.15s 10/10
4 archil 98.3 0.14s 0.20s 0.20s 10/10
5 blaxel 97.8 0.21s 0.24s 0.24s 10/10
6 lightning 97.6 0.22s 0.27s 0.27s 10/10
7 superserve 94.6 0.53s 0.55s 0.55s 10/10
8 beam 94.6 0.37s 0.79s 0.79s 10/10
9 modal 94.3 0.54s 0.60s 0.60s 10/10
10 runloop 93.6 0.60s 0.69s 0.69s 10/10
11 vercel 92.9 0.47s 1.06s 1.06s 10/10
12 cloud-run 91.3 0.73s 1.09s 1.09s 10/10
13 tensorlake 91.2 0.86s 0.92s 0.92s 10/10
14 e2b 89.9 0.91s 1.15s 1.15s 10/10
15 daytona 89.8 0.83s 1.30s 1.30s 10/10
16 tenki 85.8 1.36s 1.51s 1.51s 10/10
17 upstash 78.5 2.14s 2.15s 2.15s 10/10
18 isorun 70.8 0.02s 7.25s 7.25s 10/10
19 codesandbox 69.8 2.71s 3.49s 3.49s 10/10
20 cloudflare 64.4 3.43s 3.76s 3.76s 10/10
21 hopx 0.0 0.00s 0.00s 0.00s 0/10

View full run · SVGs available as build artifacts

@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Sandbox Dax Benchmark Results

Provider Phases Total Prepare Clone Install Typecheck Status
archil 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
beam 6/7 59.30s 20.26s 1.64s 5.89s 0.00s 0/1 OK
blaxel 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
cloud-run 2/7 1.38s 0.60s 0.00s 0.00s 0.00s 0/1 OK
cloudflare 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
codesandbox 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
createos 6/7 7.83s 2.41s 2.43s 1.94s 0.00s 0/1 OK
daytona -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
declaw 7/7 105.33s 12.08s 8.38s 26.52s 38.76s 1/1 OK
e2b 7/7 69.62s 8.65s 3.39s 8.94s 40.76s 1/1 OK
hopx -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
isorun -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
lightning 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
modal 7/7 99.80s 5.01s 2.54s 15.35s 68.58s 1/1 OK
namespace 7/7 52.35s 6.55s 2.10s 8.86s 23.05s 1/1 OK
northflank -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
runloop 2/7 120.52s 0.00s 0.00s 32.45s 49.77s 1/1 OK
superserve 6/7 30.88s 22.09s 4.02s 2.80s 0.00s 0/1 OK
tenki 6/7 22.85s 4.53s 4.13s 10.28s 0.00s 0/1 OK
tensorlake 7/7 87.81s 12.93s 1.77s 13.18s 53.27s 1/1 OK
upstash 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
vercel 7/7 77.71s 17.65s 1.86s 11.63s 38.32s 1/1 OK

View full run

@HeyGarrison
HeyGarrison force-pushed the fix/dax-cloud-run-stateful branch from 6e4de59 to ef25190 Compare July 21, 2026 19:07
Comment thread scripts/setup-cloud-run-gateway.sh Outdated
--region "$REGION" \
--project "$PROJECT_ID" \
--sandbox-launcher \
--allow-unauthenticated \

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Cloud Run service deployed with --allow-unauthenticated allowing public unauthenticated access

Cloud Run deployed without IAM auth, allowing public access to sandbox gateway.
Remove --allow-unauthenticated and enforce Cloud Run IAM invocation permissions.

AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name="scripts/setup-cloud-run-gateway.sh">
<violation number="1" location="scripts/setup-cloud-run-gateway.sh:64">
<priority>P2</priority>
<title>Cloud Run service deployed with --allow-unauthenticated allowing public unauthenticated access</title>
<evidence>The gcloud beta run deploy command includes --allow-unauthenticated, which disables Cloud Run IAM authentication and allows any unauthenticated user to invoke the service. Combined with --sandbox-launcher (a beta feature providing arbitrary code execution) and a container containing build tools, this exposes the sandbox to internet-wide reconnaissance and potential exploitation if the application-level SANDBOX_SECRET is bypassed or the gateway has vulnerabilities.</evidence>
<recommendation>Remove --allow-unauthenticated and configure Cloud Run IAM to restrict invocation to authorized service accounts. If public access is architecturally required, add Cloud Armor for DDoS protection and IP allowlisting, and ensure the SANDBOX_SECRET is complemented with infrastructure-layer authentication.</recommendation>
</violation>
</file>

@superagent-security superagent-security Bot added the pr:flagged PR flagged for review by security analysis. label Jul 21, 2026
@HeyGarrison
HeyGarrison force-pushed the fix/dax-cloud-run-stateful branch from ef25190 to 6e4de59 Compare July 21, 2026 20:17
@superagent-security superagent-security Bot removed the pr:flagged PR flagged for review by security analysis. label Jul 21, 2026

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 1 additional finding.

Open in Devin Review

Comment on lines +59 to +65
createCompute: () => cloudRun({
sandboxUrl: process.env.CLOUD_RUN_SANDBOX_URL!,
sandboxSecret: process.env.CLOUD_RUN_SANDBOX_SECRET!,
executionMode: 'stateful',
write: true,
allowEgress: true,
}),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 cloud-run provider options depend on unverified package types

The new executionMode: 'stateful', write: true, and allowEgress: true options passed to cloudRun(...) at src/sandbox/providers.ts:59-65 could not be validated against the @computesdk/cloud-run@^0.1.7 type definitions since node_modules is not installed in this checkout. If any of these keys are not part of the provider's options contract, TypeScript would flag it in CI (so this is not reported as a bug), but a reviewer should confirm the option names/values are accepted and actually honored by the installed cloud-run version.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

@HeyGarrison HeyGarrison reopened this Jul 27, 2026
HeyGarrison and others added 2 commits July 27, 2026 21:05
Cloud Run's default ephemeral mode (sandbox do) creates a fresh sandbox
for each command with no filesystem persistence. Switch to stateful mode
(sandbox run/exec/delete) with write=true and allowEgress=true for
persistent sessions that support the benchmark's clone+install+typecheck
workload.

Also fix the benchmark script to handle minimal package lists (e.g.
cloud-run gVisor images that lack python3-setuptools and unzip):
- Split apt/dnf install into required (must succeed) and optional
  (can fail) package groups
- Add Python zipfile fallback for unpacking bun when unzip is unavailable
apt-get installs nothing when any argument is unknown, so filter unavailable optional packages instead of splitting them into a second install pass, and extract bun via a heredoc-quoted Python fallback when unzip is absent.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration
devin-ai-integration Bot force-pushed the fix/dax-cloud-run-stateful branch from 6e4de59 to e2330dd Compare July 27, 2026 21:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant