Skip to content

spike(deps): measure bun as the fix for per-worktree node_modules duplication - #7501

Draft
koala73 wants to merge 1 commit into
mainfrom
spike/bun-package-manager
Draft

spike(deps): measure bun as the fix for per-worktree node_modules duplication#7501
koala73 wants to merge 1 commit into
mainfrom
spike/bun-package-manager

Conversation

@koala73

@koala73 koala73 commented Sep 1, 2026

Copy link
Copy Markdown
Owner

Draft spike — do not merge. This is a decision aid, not a migration. package-lock.json remains the single source of truth, npm ci remains the install path, and every change here is inert on a checkout with no bun.lock.

Why

One machine currently carries 125 worktrees / 181.8 GB, of which 135.5 GB is node_modules across 70 worktrees, on a volume with 47 GB free.

Applying a 7-day dormancy filter reclaims 2.0 GB. Only 1 of the 70 worktrees carrying node_modules is untouched — the pile is generated faster than cleanup can sweep it. This is a duplication problem, not a stale-junk problem, so cleanup automation cannot fix it.

npm has no answer, verified rather than assumed

  • --install-strategy=linked exists (npm 11.12.1 supports it) but its store is node_modules/.store inside each project. It is an isolation feature for catching phantom dependencies; it shares nothing between checkouts.
  • npm's cache cannot be linked from. It is content-addressable but stores gzipped tarballs (~/.npm/_cacache/content-v2), so extraction must write new bytes. Confirmed on a real install: every file reports links=1.
  • Upstream is not moving. nodejs/node#26489 (reflink support) is closed; npm/rfcs#912 ("Content-Addressable Store") has sat at zero comments since 2026-07-11.

Why bun and not pnpm

Both solve it. bun is proposed for two repo-specific reasons:

  1. bun's default backend on macOS is clonefile, which produces real files. pnpm's global virtual store is symlink-based and would trip .husky/pre-push's ERROR: node_modules is a symlink guard — a guard that exists because a later cleanup can delete the symlink's target.
  2. bun needs no configuration; pnpm needs virtualStoreType: global plus a store on the same filesystem.

Ecosystem context, measured 2026-09-01 by classifying the root lockfile of the top 100 TypeScript repos via the GitHub API:

>15k ⭐, pushed since 2026-08-01 created in 2026, >2k ⭐
pnpm 48 33
npm 21 31
bun 8 24
yarn 18 2

pnpm leads the established cohort (vitejs/vite itself, angular, supabase, tailwindcss, shadcn-ui, n8n, immich). bun is the fastest riser among new projects.

What was measured on this repo

Two throwaway worktrees off origin/main, since removed.

check result
bun install (1517 packages) exit 0, 28s cold / 9s warm
npm run typecheck exit 0
npx vite build --mode production exit 0, 30.7s — maplibre 1.05 MB, GlobeMap 1.83 MB, deck-stack 765 kB, PWA SW generated
npm run test:sidecar 447 pass / 0 fail
npm run test:data (387 files) 29,142 pass / 0 fail / 35 skipped, 7m8s

Disk, measured as real free-space delta rather than du (which counts cloned blocks that cost nothing):

worktree #1 (cold cache):  2274 MB real
worktree #2 (warm cache):   341 MB real   <- du reports 2.1 GB
   of which blog-site:      142 MB  (npm installs it; no sharing)
   bun's own tree:         ~199 MB  -> ~10x sharing
shared ~/.bun/install/cache: 2.1 GB, paid ONCE

Roughly 135 GB → ~25 GB at the current worktree count.

The three changes

1. trustedDependencies in package.json. bun blocks dependency lifecycle scripts by default and blocked 4 here: browser-tabs-lock, core-js, es5-ext, protobufjs. Build and all tests passed anyway, but protobufjs is listed explicitly because of the generated sebuf clients. npm ignores the field.

2. A completed-install marker for bun — the load-bearing fix. bun leaves no marker inside node_modules, only .bin, which an interrupted install also creates. So .husky/pre-push would find no node_modules/.package-lock.json and run a full npm ci on every push, silently undoing the entire win.

Running bun install --frozen-lockfile unconditionally is not an alternative — it costs ~11s on an already-installed tree (measured twice) against a cached-green re-push of ~0.5s.

So scripts/bun-install.mjs stamps node_modules/.wm-bun-install with the sha256 of the bun.lock it installed from, and both the shell gate and shouldInstallDependencies() verify it. That is stronger than npm's marker, which proves completion but not freshness: a stale tree after a dependency bump is invisible to .package-lock.json and is caught here. Node-side and shell-side digests were confirmed byte-identical on a real tree.

3. Lockfile policy. bun.lock is gitignored so the spike cannot ship a second source of truth. Deleting those .gitignore lines is the deliberate act that would make bun authoritative.

Verification

  • tests/bootstrap-worktree.test.mjs29 pass (unchanged npm behaviour)
  • tests/prepush-hook-gate.test.mjs31 pass
  • tests/bun-install-marker.test.mjs7 pass (new). Pins that a bare node_modules/.bin is NOT trusted, that a stale marker is rejected, and that forceInstall still wins.
  • The push that created this branch is itself the proof: it ran from a bun-installed worktree with no .package-lock.json, and the gate passed every stage including typecheck without falling back to npm ci.

Still unproven — deliberately out of scope

  • CI. Every workflow still runs npm ci. Nothing here touches them, so CI on this PR exercises the npm path, which is the point: the changes must be inert.
  • pro-test/node_modules is not installed by the postinstall chain (it belongs to build:pro), so the built-output tests were not exercised under bun.
  • Linux/CI filesystems have no APFS clonefile. bun falls back to hardlink there, which shares differently. Unmeasured.
  • Tauri desktop and Railway seeder images were not exercised.

What I'd want decided

Whether the ~110 GB is worth a package-manager change at all, versus capping worktree count — the ecosystem's actual default answer is 4-10 worktrees per developer, and no guide is written for 125.

Full write-up with reproduction steps: docs/solutions/performance-issues/worktree-node-modules-duplication-bun-spike.md

https://claude.ai/code/session_01BmN9DZ8ZP6SaZ5BnbgqSkj

…lication

DRAFT SPIKE. Nothing here changes how the repo installs. package-lock.json
stays the single source of truth, npm ci stays the install path, and every
change is inert on a checkout with no bun.lock.

The problem: one machine carries 125 worktrees / 181.8 GB, of which 135.5 GB
is node_modules across 70 worktrees, on a volume with 47 GB free. A 7-day
dormancy filter reclaims 2.0 GB — the pile is generated faster than cleanup
can sweep it, so this is a duplication problem, not a stale-junk problem.

npm cannot fix it, verified rather than assumed. --install-strategy=linked
stores per project in node_modules/.store and shares nothing between
checkouts. npm's cache is content-addressable but holds gzipped tarballs, so
extraction must write new bytes — every installed file reports links=1.
nodejs/node#26489 (reflink) is closed and npm/rfcs#912 (content-addressable
store) has sat at zero comments since 2026-07-11.

bun's default install backend on macOS is clonefile, which produces real
files rather than symlinks — pnpm's global virtual store is symlink-based and
would trip the pre-push guard that exists because a later cleanup can delete
a symlink's target.

Measured on this repo in two throwaway worktrees off origin/main:

  bun install (1517 packages)   exit 0, 28s cold / 9s warm
  npm run typecheck             exit 0
  npx vite build --production   exit 0, 30.7s (maplibre, GlobeMap, deck-stack, PWA)
  npm run test:sidecar          447 pass / 0 fail
  npm run test:data             29,142 pass / 0 fail / 35 skipped, 7m8s

  worktree #1 (cold cache)      2274 MB real
  worktree #2 (warm cache)       341 MB real, du reports 2.1 GB
    of which blog-site           142 MB (npm installs it; no sharing)
    bun's own tree              ~199 MB  -> ~10x sharing
  shared bun cache               2.1 GB, paid once

Roughly 135 GB -> ~25 GB at the current worktree count.

Three changes, each guarded:

1. trustedDependencies in package.json. bun blocks dependency lifecycle
   scripts by default and blocked 4 here (browser-tabs-lock, core-js,
   es5-ext, protobufjs). protobufjs is called out because of the generated
   sebuf clients. npm ignores the field entirely.

2. A completed-install marker for bun. This is the load-bearing fix. bun
   leaves NO marker inside node_modules — only .bin, which an interrupted
   install also creates — so .husky/pre-push would find no
   node_modules/.package-lock.json and run a full npm ci on every push,
   silently undoing the win. Running `bun install --frozen-lockfile`
   unconditionally is not an alternative: it costs ~11s on an installed tree
   (measured twice) against a cached-green re-push of ~0.5s.

   scripts/bun-install.mjs therefore stamps node_modules/.wm-bun-install with
   the sha256 of the bun.lock it installed from, and both the shell gate and
   shouldInstallDependencies() verify it. That is stronger than npm's marker,
   which proves completion but not freshness — a stale tree after a dependency
   bump is invisible to .package-lock.json and is caught here. The node-side
   and shell-side digests were confirmed identical on a real tree.

3. Lockfile policy. bun.lock is gitignored so the spike cannot ship a second
   source of truth. Removing those lines is the deliberate act that would
   make bun authoritative.

Existing gates stay green: tests/bootstrap-worktree.test.mjs 29 pass,
tests/prepush-hook-gate.test.mjs 31 pass. New coverage in
tests/bun-install-marker.test.mjs (7 tests) pins that a bare node_modules/.bin
is NOT trusted and that a stale marker is rejected.

Still unproven and deliberately out of scope: CI (every workflow still runs
npm ci), pro-test/node_modules and the built-output tests, Linux/CI
filesystems where bun falls back to hardlink instead of clonefile, and the
Tauri desktop and Railway seeder images.
@mintlify

mintlify Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
WorldMonitor 🟢 Ready View Preview Sep 1, 2026, 1:10 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@vercel

vercel Bot commented Sep 1, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
worldmonitor Ready Ready Preview Sep 1, 2026 1:14pm UTC

Request Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant