feat(core): experimental in-process VM continuation for inline replay#2966
feat(core): experimental in-process VM continuation for inline replay#2966VaguelySerious wants to merge 2 commits into
Conversation
🦋 Changeset detectedLatest commit: 8467bdf The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed Summary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
|
Deployment failed with the following error: |
📊 Workflow Benchmarkscommit Backend:
📜 Previous results (2)ff3959bFri, 17 Jul 2026 17:06:35 GMT · run logs
ff3959bFri, 17 Jul 2026 16:25:18 GMT · run logs
Avg deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) 🟢/🔴 mark percentiles within/above target. Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
Combines the retained-session architecture from #2984 with the env kill switch and loop-level single-VM test from #2966. - executeWorkflow with discriminated request/result types and a WorkflowSession state machine (running/suspended/failed/replay/completed) - EventsConsumer.append: only newly durable events feed the live VM - WORKFLOW_RETAINED_VM=0 kill switch (default on) - retained-vm-loop.test.ts: proves one VM per run and byte-identical output vs the from-scratch replay path
Add an opt-in path (WORKFLOW_VM_CONTINUATION=1, off by default) that keeps a suspended workflow VM alive across inline-loop iterations instead of rebuilding the vm.Context and replaying the event log from scratch each pass. On a step-only suspension, runWorkflow attaches a `continuation` handle to the WorkflowSuspension. The inline loop, after appending the step's terminal events, feeds them into the SAME live VM via EventsConsumer.resume(), resolving the pending step promise so the workflow body advances to its next checkpoint. Scope and safety: - Only within a single invocation's inline loop (same process owns the writes, so the durable event log stays authoritative). - Only for step-only suspensions; hooks/waits/attribute writes fall back to a full replay (they involve out-of-band invocations and delivery-barrier ordering the replay path handles specially). - resume() prefix-checks the consumed events against the authoritative log by eventId and throws ReplayDivergenceError on any mismatch; the loop swallows it and falls back to a fresh replay. Worst case == today's behavior. Verified: full core suite green with the flag both off and on (1488 passed); a loop-level test proves the VM is constructed once (resumed per step) with the flag on vs rebuilt per replay off, producing byte-identical output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Flip the flag to default-on so CI and benchmarks exercise and measure the continuation path. `WORKFLOW_VM_CONTINUATION=0` reverts to rebuilding the VM and replaying from scratch. Updates the loop test to drive the baseline via the kill switch and the ON case via the (now default) unset env. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ff3959b to
8467bdf
Compare
pranaygp
left a comment
pranaygp
left a comment
There was a problem hiding this comment.
Trace-grounded context (durabench prod, SDK 5.0.0-beta.34, iad1, 2026-07-20): warm STSO is ~150-270 ms/step (mean ~190) vs Temporal 45-50. Decomposing Datadog flow trace 2f55f784951f4eba33987bf54463b247 (run wrun_41KXZ0W77G0GJEPSCQEKGE823D), each warm step is ~65-92 ms of step_started POST + ~50-70 ms of step_completed POST + 28-78 ms of workflow.run full replay (O(journal) per step, so O(steps^2) per run) + 40-130 ms of CPU. Keeping the VM alive removes the 28-78 ms replay slice (~30-40% of the warm gap) and most of the per-step CPU; the two serialized POSTs (~120-160 ms) remain, and are targeted by the complementary batched-transition work in vercel/workflow-server@pgp/batch-step-transitions + vercel/workflow@pgp/batch-transition-client. So the mechanism here goes after the right slice.
The implementation is clean and the fallback is well-scoped, but the safety argument is incomplete relative to the competing #2990: it covers synchronous RNG/clock/correlation-id determinism but not host-timed async progress, which a live (non-quiesced) VM can observe across the inline step's real wall-clock gap. That is the one thing that makes this unsafe to run default-on as written -- details inline.
Convergence: #2990 implements the same idea with the sandbox quiescence this PR is missing, and should be the base. I'd close this in favor of it, or keep this branch only as a minimal default-off variant if the team wants a smaller first step. What is worth carrying from here into #2990 is exactly what it already credits absorbing: the minimal EventsConsumer.resume mechanic and the createContext-called-once A/B test.
Head-to-head durabench benchmark cells for this branch and #2990 are being added; numbers to follow as a comment.
| // delivery-barrier ordering that the from-scratch replay path handles | ||
| // specially. Anything else leaves `continuation` unset so the caller falls | ||
| // back to a fresh replay. | ||
| const isContinuationEligible = (s: WorkflowSuspension): boolean => |
There was a problem hiding this comment.
This is the core soundness gap versus #2990. Eligibility here is purely structural (no hooks/waits/attrs/disposals/aborts) but says nothing about whether the suspended VM can still make progress on host timing. The sandbox on this branch still exposes Atomics.waitAsync, async WebAssembly.compile/instantiate, WeakRef/FinalizationRegistry, and a threadpool-backed async crypto.subtle.digest. A live VM kept parked across the inline step's real wall-clock gap (the two POSTs are ~120-160 ms) can observe those settle, whereas a from-scratch replay consumes the log with ~no wall-clock elapsed. Concretely:
const ia = new Int32Array(new SharedArrayBuffer(8));
const timeout = Atomics.waitAsync(ia, 0, 0, 50).value.then(() => 'timeout');
const r = await Promise.race([someStep(), timeout]);Cold replay always resolves someStep() first (its result is already in the log) -> deterministic. Retained across a ~150 ms inline step, the 50 ms waitAsync fires during the gap and the race picks 'timeout' -- and because retained-mode writes are authoritative, the run commits a branch no replay can reproduce. The summary's "RNG/clock/correlation-id advance naturally, exactly matching a from-scratch replay" is true for those synchronous draws but does not cover this. #2990 closes it by removing exactly these APIs from the sandbox and making digest synchronous; that hardening (or an equivalent quiescence guarantee) is the prerequisite for this to be safe default-on.
| * so the caller falls back to a full replay in a fresh VM. Matching is by | ||
| * `eventId`, which is immutable per event across reads. | ||
| */ | ||
| resume(events: Event[]): void { |
There was a problem hiding this comment.
The prefix guard is necessary but not sufficient. It validates that the already-consumed prefix (0..eventIndex) is byte-stable, which catches a rewritten log -- but it cannot detect a VM that took a different future branch after the cursor because it advanced on host timing during the gap (see the isContinuationEligible comment). The consumed prefix still matches; the divergence is in the events the retained VM is about to produce. #2990 adds a boundary-stability check (isSameSuspensionBoundary, demoting to replay if the VM re-suspends on anything other than the same steps) precisely for this; this PR trusts a non-quiesced advance silently. At minimum, assert the VM re-suspends on the same step set before adopting the appended events.
Minor, same method: dropping readonly and doing this.events = events makes the consumer alias the caller's array rather than own a copy (#2990 copies in the ctor and append()s only the delta). It works today because the loop happens to pass the same array, but it couples the consumer's internal invariant to the caller's array handling; copy-and-append is less fragile.
| */ | ||
| export function isVmContinuationEnabled(): boolean { | ||
| const raw = process.env.WORKFLOW_VM_CONTINUATION; | ||
| if (raw === undefined || raw === '') return true; |
There was a problem hiding this comment.
Default-on posture: this flag is read at runtime in production too (as the PR notes), and combined with the open quiescence gap above, the draft/POC status, and no execution-mode telemetry, that is a lot to enable by default. I'd flip the default to off here until the sandbox is quiesced (or this converges onto #2990's hardened sandbox), and keep default-on only in the CI/benchmark lanes via an explicit env in those workflows. As written, the first production workflow that races a host-timed primitive against a step gets a silently corrupted run with no signal -- the worst failure mode for a durability product.
|
Head-to-head durabench benchmark (follows up the review): this branch vs #2990 vs
Takeaways: (1) −30% warm STSO and −83% per-step CPU vs baseline, with the win growing with step count (replay was O(journal)/step); (2) #2990 and #2966 are statistically indistinguishable on latency — the decision between them is purely the correctness/architecture question from the reviews; (3) the remaining ~120-135ms/step is the two serialized event POSTs, which retention can't touch — that's the batch-transition track ( 🤖 Generated with Claude Code |
2 participants
Summary
Adds in-process VM continuation — on by default, kill switch
WORKFLOW_VM_CONTINUATION=0. When active, the inline replay loop keeps a suspended workflow VM alive across iterations and feeds newly-appended events into it, instead of rebuilding thevm.Contextand replaying the event log from event 0 on every pass.Default-on so CI lanes and benchmarks actually exercise and measure the new path. The kill switch restores the previous rebuild-and-replay behavior exactly.
This is the "cache the VM" idea, scoped to the only place it's sound: within a single invocation's inline loop, where the same process owns the appended writes so the durable event log stays authoritative. It does not cache a VM across process/invocation boundaries (sleeps, hooks, redelivery) — there is no live VM to resume there and a snapshot would drift from the log.
How it works
runWorkflowattaches acontinuationhandle to the thrownWorkflowSuspension.runWorkflow, it callscontinuation.resume(events):EventsConsumer.resume()adopts the authoritative event array and re-drives the still-registered step consumers, resolving the pending step promise so the same live VM advances to its next checkpoint.The step consumer already stays registered after hitting end-of-events (it only removes itself on terminal events), so resuming is just feeding it the events a fresh replay would have consumed next — the same deterministic call sequence, not restarted. RNG/clock/correlation-id state advance naturally in the live VM, exactly matching what a from-scratch replay reconstructs.
Safety (why this can't corrupt a run)
continuationunset → the loop falls back to a full replay. Those paths involve out-of-band resume/parallel invocations and delivery-barrier ordering that the replay path handles specially and that aren't validated for continuation here.resume()checks that the events already consumed by the live VM are a byte-stable prefix (byeventId) of the authoritative log. Any mismatch throwsReplayDivergenceError; the loop swallows it and falls back to a fresh replay.WORKFLOW_VM_CONTINUATION=0disables the continuation attach entirely; the suspend/complete/error control flow is byte-identical to before (verified by the full suite passing under the kill switch).Scope / limitations (intentionally not in this PR)
Verification
packages/corefull suite green with the default (on) and under the kill switch (=0): 1488 passed, 3 pre-existing expected-fails.vm-continuation.test.ts: provesresume()advances a live two-step workflow across appended events in one VM, plus the divergence guard.vm-continuation-loop.test.ts: drives a 2-step sequential workflow through the realworkflowEntrypointloop and assertscreateContext(VM construction) is called once by default (resumed per step) vs >1 under the kill switch (rebuilt per replay), and that the dehydratedrun_completedoutput is byte-identical between the two modes.pnpm --filter @workflow/core build+typecheckclean.Notes
WORKFLOW_VM_CONTINUATION=0is the per-deployment kill switch.world-localpaths.🤖 Generated with Claude Code