[miniflare] Share local storage across processes via a single owner (experimental)#14449
Draft
penalosa wants to merge 9 commits into
Draft
[miniflare] Share local storage across processes via a single owner (experimental)#14449penalosa wants to merge 9 commits into
penalosa wants to merge 9 commits into
Conversation
… props Move the per-binding configuration for storage and remote (mixed-mode) bindings out of per-resource workerd services and into runtime `ctx.props`, so a single shared service can serve any number of bindings. - Local KV namespaces now share one entry service; the namespace id is passed via props and resolved in object-entry.worker.ts (idFromName). - remoteProxyClientWorker() is now script-only; the connection string, binding name and trace id travel via props (buildRemoteProxyProps), read in remote-proxy-client.worker.ts and the dispatch-namespace proxy. - All remote-binding plugins emit one shared remote-proxy service instead of one per resource. - explorer.ts reads the KV namespace id from binding props rather than parsing it out of the service name.
…torage owner, stage 0) Scaffolding for the central storage owner feature: a per-persist-root filesystem registry so exactly one process can own the storage backing files and others can discover it. - New `StorageOwnerRegistry` primitives in src/shared/storage-owner.ts: owner-definition read/write/clear (atomic, with mtime heartbeat + pid liveness staleness), an election spawn-lock with dead-pid/stale reclaim, and a client-presence registry for lifecycle teardown decisions. - New experimental `unsafeSharedStorageOwner` core option (no-op for now). No behavioural change yet; routing and owner spawning land in later stages.
…ner, stage 1) When `unsafeSharedStorageOwner` is enabled and a persist root is set, each Miniflare instance now records its role once the runtime (and thus the workerd debug port) is available: - owner role (`unsafeStorageOwnerRole: "owner"`, set on the detached owner process): writes the owner definition — pid + debug-port address — to the persist root and heartbeats it, so clients can discover and route to it. Cleared on dispose. - client role (default): registers a heartbeated presence file so the owner can later tell when no clients remain. Also carries the pre-existing per-persist-root startup lock that serialises `runtime.updateConfig` across instances sharing a persist root. No routing yet (clients still use local storage); that lands next. Validated by unit + integration tests in test/storage-owner.spec.ts.
…e owner, stage 2) A Miniflare instance running as a shared-storage client now routes its local KV namespaces to the owner process instead of opening the SQLite/ blob files itself: - New client-side `StorageOwnerProxy` worker (src/workers/core/ storage-owner-proxy.worker.ts) forwards each request to the owner's shared `kv:ns:entry` service over the workerd debug port, passing the namespace id through as `ctx.props`. The owner's object-entry worker resolves the Durable Object locally, so the owner performs all I/O and attaches `cf.miniflare.name` itself (avoiding the actor-RPC cf problem). - When an owner is published, `Miniflare` bakes its debug-port address into the proxy service, rewrites local KV bindings to the proxy, and skips standing up local KV storage (disk/DO/migrations) via a new `storageOwnerRoutePlugins` plugin-context flag. Remote (mixed-mode) KV bindings are left untouched. Proven by a two-instance integration test: a client KV `put` is visible to the owner and readable back through the proxy. R2/D1/Cache and the detached owner spawn + lifecycle guards land next.
…entral storage owner, stage 1b) The feature is now self-driving: a shared-storage client with no owner published elects a single spawner (via the owner spawn-lock) and launches a detached owner process before assembling, then routes to it. - `runStorageOwnerProcess()` is the entry for the detached owner: it runs the same built miniflare module (path passed via env, so no second build entry), starts a headless owner-role Miniflare hosting the union of local KV ids, and self-terminates once no clients remain for a debounce window past a startup grace (minimal lifecycle guard). - `#ensureStorageOwner()` performs election + spawn + wait-for-publish, and degrades to local storage if the dev registry (needed for the debug port) isn't configured or the owner doesn't come up. Proven by an integration test: a lone client auto-spawns a separate owner process, KV round-trips through it, and the owner exits after the client disposes (with a pid safety-net kill to avoid leaking in CI).
… storage owner, stage 3) Extends shared-storage client routing from KV to R2 and D1, reusing the same object-entry + props mechanism: - `rewriteStorageOwnerBinding` now also repoints R2 (`r2Bucket`), D1 pre-3.3 (`service`) and D1 post-3.3 (`wrapped` inner fetcher service) bindings at the storage-owner proxy. - R2 and D1 plugins skip standing up local storage when routed; the R2 public-bucket service is skipped too (it binds the local entry service). - The spawned owner now hosts the union of local KV / R2 / D1 ids. Cache is intentionally not routed yet: it keys its DO by a per-request header rather than the object-entry props model, and cache evictions are recoverable, so it's lower priority. Proven by an integration test: a client's R2 object and a D1 SQL row are written through the owner and read back by the owner instance.
Demonstrates the feature resolves the original cross-process SQLITE_BUSY: three client instances sharing one persist root issue 60 concurrent D1 inserts; with a single shared owner opening the files, every request succeeds (no 500s) and the final row count is exactly correct.
…traction Adds `buildObjectEntryProps` (the shared object-entry props builder used by the KV/R2/D1 plugins) and teaches the local explorer to read R2/D1 resource ids from binding props (falling back to service-name parsing for remote bindings), mirroring the existing KV handling. These complete the shared-object-entry props model the storage plugins already rely on.
🦋 Changeset detectedLatest commit: c8f6d2d The changes in this PR will be included in the next version bump. This PR includes changesets to release 6 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
|
UnknownError: ProviderInitError |
Contributor
|
@penalosa Bonk workflow failed. Check the logs for details. View workflow run · To retry, trigger Bonk again. |
Contributor
|
|
@cloudflare/autoconfig
create-cloudflare
@cloudflare/deploy-helpers
@cloudflare/kv-asset-handler
miniflare
@cloudflare/pages-shared
@cloudflare/unenv-preset
@cloudflare/vite-plugin
@cloudflare/vitest-pool-workers
@cloudflare/workers-auth
@cloudflare/workers-editor-shared
@cloudflare/workers-utils
wrangler
commit: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Adds an experimental
unsafeSharedStorageOwneroption to Miniflare that makes a single process own local storage for a given persist root, so multiple Miniflare instances (e.g. severalwrangler dev/vite devsessions) no longer each open the same SQLite/blob files — the root cause of cross-processSQLITE_BUSYerrors under concurrent access.When enabled (and
unsafeDevRegistryPathis set):getEntrypoint+props), and skips standing up its own local storage. Only the owner performs storage I/O.This also lands a supporting refactor: storage and remote (mixed-mode) bindings now carry their per-resource config (namespace/bucket/database id, remote connection string) via
ctx.propsagainst a single shared object-entry / proxy service, instead of one workerd service per resource.Design note
Routing happens at the shared
object-entryworker layer viagetEntrypoint+props(not at the Durable Object actor layer). This is deliberate: the storage DO asserts oncf.miniflare.nameand uses it to namespace the blob store, which would not survive actor RPC. Forwarding to the owner's ownobject-entryworker lets the owner attachcfand resolve the DO locally.Validation
index.spec+ an explicit "feature off" test).Status — draft
Known follow-ups before this is review-ready (tracked in the PR discussion):
unsafe*flag not intended for general use yet; user-facing docs will follow once the design stabilises and it is opted into by Wrangler.