spike(deps): measure bun as the fix for per-worktree node_modules duplication - #7501
Draft
koala73 wants to merge 1 commit into
Draft
spike(deps): measure bun as the fix for per-worktree node_modules duplication#7501koala73 wants to merge 1 commit into
koala73 wants to merge 1 commit into
Conversation
…lication DRAFT SPIKE. Nothing here changes how the repo installs. package-lock.json stays the single source of truth, npm ci stays the install path, and every change is inert on a checkout with no bun.lock. The problem: one machine carries 125 worktrees / 181.8 GB, of which 135.5 GB is node_modules across 70 worktrees, on a volume with 47 GB free. A 7-day dormancy filter reclaims 2.0 GB — the pile is generated faster than cleanup can sweep it, so this is a duplication problem, not a stale-junk problem. npm cannot fix it, verified rather than assumed. --install-strategy=linked stores per project in node_modules/.store and shares nothing between checkouts. npm's cache is content-addressable but holds gzipped tarballs, so extraction must write new bytes — every installed file reports links=1. nodejs/node#26489 (reflink) is closed and npm/rfcs#912 (content-addressable store) has sat at zero comments since 2026-07-11. bun's default install backend on macOS is clonefile, which produces real files rather than symlinks — pnpm's global virtual store is symlink-based and would trip the pre-push guard that exists because a later cleanup can delete a symlink's target. Measured on this repo in two throwaway worktrees off origin/main: bun install (1517 packages) exit 0, 28s cold / 9s warm npm run typecheck exit 0 npx vite build --production exit 0, 30.7s (maplibre, GlobeMap, deck-stack, PWA) npm run test:sidecar 447 pass / 0 fail npm run test:data 29,142 pass / 0 fail / 35 skipped, 7m8s worktree #1 (cold cache) 2274 MB real worktree #2 (warm cache) 341 MB real, du reports 2.1 GB of which blog-site 142 MB (npm installs it; no sharing) bun's own tree ~199 MB -> ~10x sharing shared bun cache 2.1 GB, paid once Roughly 135 GB -> ~25 GB at the current worktree count. Three changes, each guarded: 1. trustedDependencies in package.json. bun blocks dependency lifecycle scripts by default and blocked 4 here (browser-tabs-lock, core-js, es5-ext, protobufjs). protobufjs is called out because of the generated sebuf clients. npm ignores the field entirely. 2. A completed-install marker for bun. This is the load-bearing fix. bun leaves NO marker inside node_modules — only .bin, which an interrupted install also creates — so .husky/pre-push would find no node_modules/.package-lock.json and run a full npm ci on every push, silently undoing the win. Running `bun install --frozen-lockfile` unconditionally is not an alternative: it costs ~11s on an installed tree (measured twice) against a cached-green re-push of ~0.5s. scripts/bun-install.mjs therefore stamps node_modules/.wm-bun-install with the sha256 of the bun.lock it installed from, and both the shell gate and shouldInstallDependencies() verify it. That is stronger than npm's marker, which proves completion but not freshness — a stale tree after a dependency bump is invisible to .package-lock.json and is caught here. The node-side and shell-side digests were confirmed identical on a real tree. 3. Lockfile policy. bun.lock is gitignored so the spike cannot ship a second source of truth. Removing those lines is the deliberate act that would make bun authoritative. Existing gates stay green: tests/bootstrap-worktree.test.mjs 29 pass, tests/prepush-hook-gate.test.mjs 31 pass. New coverage in tests/bun-install-marker.test.mjs (7 tests) pins that a bare node_modules/.bin is NOT trusted and that a stale marker is rejected. Still unproven and deliberately out of scope: CI (every workflow still runs npm ci), pro-test/node_modules and the built-output tests, Linux/CI filesystems where bun falls back to hardlink instead of clonefile, and the Tauri desktop and Railway seeder images.
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
One machine currently carries 125 worktrees / 181.8 GB, of which 135.5 GB is
node_modulesacross 70 worktrees, on a volume with 47 GB free.Applying a 7-day dormancy filter reclaims 2.0 GB. Only 1 of the 70 worktrees carrying
node_modulesis untouched — the pile is generated faster than cleanup can sweep it. This is a duplication problem, not a stale-junk problem, so cleanup automation cannot fix it.npm has no answer, verified rather than assumed
--install-strategy=linkedexists (npm 11.12.1 supports it) but its store isnode_modules/.storeinside each project. It is an isolation feature for catching phantom dependencies; it shares nothing between checkouts.~/.npm/_cacache/content-v2), so extraction must write new bytes. Confirmed on a real install: every file reportslinks=1.Why bun and not pnpm
Both solve it. bun is proposed for two repo-specific reasons:
clonefile, which produces real files. pnpm's global virtual store is symlink-based and would trip.husky/pre-push'sERROR: node_modules is a symlinkguard — a guard that exists because a later cleanup can delete the symlink's target.virtualStoreType: globalplus a store on the same filesystem.Ecosystem context, measured 2026-09-01 by classifying the root lockfile of the top 100 TypeScript repos via the GitHub API:
pnpm leads the established cohort (
vitejs/viteitself, angular, supabase, tailwindcss, shadcn-ui, n8n, immich). bun is the fastest riser among new projects.What was measured on this repo
Two throwaway worktrees off
origin/main, since removed.bun install(1517 packages)npm run typechecknpx vite build --mode productionnpm run test:sidecarnpm run test:data(387 files)Disk, measured as real free-space delta rather than
du(which counts cloned blocks that cost nothing):Roughly 135 GB → ~25 GB at the current worktree count.
The three changes
1.
trustedDependenciesinpackage.json. bun blocks dependency lifecycle scripts by default and blocked 4 here:browser-tabs-lock,core-js,es5-ext,protobufjs. Build and all tests passed anyway, butprotobufjsis listed explicitly because of the generated sebuf clients. npm ignores the field.2. A completed-install marker for bun — the load-bearing fix. bun leaves no marker inside
node_modules, only.bin, which an interrupted install also creates. So.husky/pre-pushwould find nonode_modules/.package-lock.jsonand run a fullnpm cion every push, silently undoing the entire win.Running
bun install --frozen-lockfileunconditionally is not an alternative — it costs ~11s on an already-installed tree (measured twice) against a cached-green re-push of ~0.5s.So
scripts/bun-install.mjsstampsnode_modules/.wm-bun-installwith the sha256 of thebun.lockit installed from, and both the shell gate andshouldInstallDependencies()verify it. That is stronger than npm's marker, which proves completion but not freshness: a stale tree after a dependency bump is invisible to.package-lock.jsonand is caught here. Node-side and shell-side digests were confirmed byte-identical on a real tree.3. Lockfile policy.
bun.lockis gitignored so the spike cannot ship a second source of truth. Deleting those.gitignorelines is the deliberate act that would make bun authoritative.Verification
tests/bootstrap-worktree.test.mjs— 29 pass (unchanged npm behaviour)tests/prepush-hook-gate.test.mjs— 31 passtests/bun-install-marker.test.mjs— 7 pass (new). Pins that a barenode_modules/.binis NOT trusted, that a stale marker is rejected, and thatforceInstallstill wins..package-lock.json, and the gate passed every stage including typecheck without falling back tonpm ci.Still unproven — deliberately out of scope
npm ci. Nothing here touches them, so CI on this PR exercises the npm path, which is the point: the changes must be inert.pro-test/node_modulesis not installed by the postinstall chain (it belongs tobuild:pro), so the built-output tests were not exercised under bun.clonefile. bun falls back tohardlinkthere, which shares differently. Unmeasured.What I'd want decided
Whether the ~110 GB is worth a package-manager change at all, versus capping worktree count — the ecosystem's actual default answer is 4-10 worktrees per developer, and no guide is written for 125.
Full write-up with reproduction steps:
docs/solutions/performance-issues/worktree-node-modules-duplication-bun-spike.mdhttps://claude.ai/code/session_01BmN9DZ8ZP6SaZ5BnbgqSkj