Skip to content

fix(deploy): anchor rsync excludes to transfer root to prevent exit 23#3787

Merged
ryota-murakami merged 1 commit into
mainfrom
fix/deploy-rsync-anchor-excludes
May 31, 2026
Merged

fix(deploy): anchor rsync excludes to transfer root to prevent exit 23#3787
ryota-murakami merged 1 commit into
mainfrom
fix/deploy-rsync-anchor-excludes

Conversation

@ryota-murakami
Copy link
Copy Markdown
Collaborator

@ryota-murakami ryota-murakami commented May 31, 2026

Problem

Every push-to-main deploy was aborting at the artifact-sync rsync step with exit 23 (cannot delete non-empty directory: node_modules/@sentry/.../node_modules/@sentry/core/build/...), before the pm2 restart. Net effect of a "failed" deploy: the new bundle (server_build/server/index.js) was synced to disk and prisma migrate deploy was applied, but PM2 kept running the old code. Surfaced on the PR #3786 deploy after a Dependabot @sentry bump.

Root cause

All three deploy rsyncs used unanchored excludes:

rsync -a --delete --exclude='.env' --exclude='logs/' --exclude='run/' ...

An unanchored pattern like logs/ matches a directory of that name at any depth, and rsync protects excluded paths on the receiver from --delete. The hoisted node_modules tree contains deep dirs literally named logs/:

node_modules/@sentry/node/node_modules/@sentry/core/build/esm/logs
node_modules/@sentry/node/node_modules/@sentry/core/build/cjs/logs
node_modules/@sentry/opentelemetry/node_modules/@sentry/core/build/esm/logs
...

When a version bump hoists @sentry/core to the top level, the stale nested @sentry/*/node_modules/@sentry/core/ trees should be deleted — but their build/*/logs/ subdirs were protected by --exclude='logs/', leaving the parents non-empty. rsync cannot remove a non-empty directory → exit 23 → fail_deployment → restart skipped.

Fix

Anchor every exclude to the transfer root with a leading /, on all three rsyncs (artifact sync + both rollback rsyncs):

rsync -a --delete --exclude='/.env' --exclude='/logs/' --exclude='/run/' ...

/logs/ now protects only "$remote_dir"/logs/ (the intended top-level runtime dir), not deep node_modules/.../logs/, so --delete can prune stale nested node_modules and the sync completes through to pm2 restart. Top-level /.env, /logs/, /run/ stay protected exactly as before (preserves the production .env, PM2 logs, and the live mysqld socket dir — the CLAUDE.md "must not rm -rf the top-level run/ tree" gotcha still holds).

Verification (empirical, on the production host — rsync 3.2.7)

Reproduced the exact failing structure in /tmp (touches no real paths):

exclude nested @sentry tree top-level logs/
--exclude='logs/' (old) survivescannot delete non-empty directory: @sentry/core/build/esm protected
--exclude='/logs/' (new) removedNo such file or directory preserved (app.log kept)

This proves both that the unanchored exclude is the cause and that anchoring preserves the intended top-level protection. (CI does not exercise the deploy SSH path, so this host-level repro is the real test, not green CI.)

Notes

  • Self-healing. The first deploy carrying this fix deletes the stale nested @sentry/core dirs (dated before the bump) automatically — no manual production cleanup needed.
  • scripts/deploy (manual path) is unaffected — its rsyncs use neither --delete nor these excludes and never sync node_modules. The only other rsync in the repo (scripts/backup) already uses an anchored --exclude='/tmp/'.
  • No --force / --delete-after. With the protect rule corrected there are no un-deletable dirs left; adding those flags would only mask a future protect-rule regression instead of surfacing it as a loud exit 23.

Risk

Low — CI-workflow change only. Production is untouched until the next deploy, which this change makes complete through pm2 restart.

Summary by CodeRabbit

  • Chores
    • Enhanced deployment safety by refining file protection patterns during rollback and synchronization operations.

The three artifact-sync / rollback rsyncs used unanchored excludes
(--exclude='logs/' --exclude='run/'), which match a directory of that
name at ANY depth. rsync protects excluded paths on the receiver from
--delete, so deep node_modules dirs named logs/ (e.g.
node_modules/@sentry/core/build/esm/logs) shielded stale nested @sentry
trees from deletion. rsync could then not remove the now-non-empty
parents and aborted with exit 23 before the PM2 restart — the new bundle
synced to disk but the old code kept serving.

Anchor all three excludes with a leading '/' so they protect ONLY the
top-level runtime state (/.env, /logs/, /run/) and let --delete prune
stale nested node_modules.

Verified on the production host (rsync 3.2.7): the unanchored pattern
leaves the nested @sentry tree undeletable ("cannot delete non-empty
directory"); the anchored pattern removes it while preserving the
top-level logs/ directory.
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 31, 2026

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 8edf1b98-55e9-4439-a456-79c0c10e26f3

📥 Commits

Reviewing files that changed from the base of the PR and between 8db4608 and e149e2e.

📒 Files selected for processing (1)
  • .github/workflows/deploy.yml

📝 Walkthrough

Walkthrough

The deployment workflow's rsync commands are updated to use path-anchored exclude patterns. Rollback artifact capture and execution switch to /.env, /logs/, /run/ excludes; the final staging→remote sync after Prisma migrations receives the same anchored-pattern treatment to protect top-level runtime directories.

Changes

rsync exclude pattern anchoring

Layer / File(s) Summary
Rollback artifact rsync exclude patterns
.github/workflows/deploy.yml
Rollback artifact capture and rollback execution rsync commands update exclude patterns from unanchored (logs/, run/, .env) to path-anchored (/.env, /logs/, /run/), constraining protection scope to top-level directories.
Deployment rsync exclude patterns
.github/workflows/deploy.yml
The final rsync from staging to remote after Prisma migrations is updated to use path-anchored excludes (/.env, /logs/, /run/) for consistency with rollback pattern behavior during deletion.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Possibly related PRs

  • laststance/nsx#3773: Modifies the same .github/workflows/deploy.yml deployment rsync logic around Prisma migrations and rollback artifact handling.
  • laststance/nsx#3774: Modifies rollback-artifact rsync logic in the same workflow, refactoring rollback capture and its surrounding steps.
  • laststance/nsx#3769: Updates the same deployment workflow's rsync rollback and deploy logic for .env, run/, and logs/ directory handling.

Poem

🐰 Paths now anchored, folders safe from the broom,
Top-level treasures won't leave the room,
Three syncs aligned in their exclude refrain,
Rollback and deploy, a cleaner domain!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: anchoring rsync excludes to prevent exit 23 errors during deployment, which directly corresponds to the workflow changes made.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/deploy-rsync-anchor-excludes

Comment @coderabbitai help to get the list of available commands and usage tips.

@ryota-murakami ryota-murakami merged commit f2b33bc into main May 31, 2026
8 checks passed
@ryota-murakami ryota-murakami deleted the fix/deploy-rsync-anchor-excludes branch May 31, 2026 08:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant