Skip to content

Fix Celery worker crash on macOS due to unpicklable local function#62655

Merged
dheerajturaga merged 2 commits intoapache:mainfrom
Vamsi-klu:fix/m4-celery-bundle-cleanup-pickle-macos
Mar 2, 2026
Merged

Fix Celery worker crash on macOS due to unpicklable local function#62655
dheerajturaga merged 2 commits intoapache:mainfrom
Vamsi-klu:fix/m4-celery-bundle-cleanup-pickle-macos

Conversation

@Vamsi-klu
Copy link
Contributor

@Vamsi-klu Vamsi-klu commented Mar 1, 2026

Summary

  • Move bundle_cleanup_main() from a local function inside _run_stale_bundle_cleanup() to module-level _bundle_cleanup_main(check_interval)
  • On macOS, multiprocessing uses spawn instead of fork, and local functions cannot be pickled for spawning across processes
  • Follows the existing pattern of serve_logs (module-level function used with Process(target=serve_logs))

Co-contributors : @codingrealitylabs @girlcoder-gaming

Test plan

  • Verify Celery worker starts without error on macOS
  • Verify stale bundle cleanup process spawns correctly
  • Run on both macOS (spawn) and Linux (fork) to confirm compatibility

Closes: #62512

🤖 Generated with Claude Code

@Vamsi-klu
Copy link
Contributor Author

cc @vincbeck @o-nikolas @potiuk — Would appreciate your review. This fixes a Celery worker crash on macOS where spawn multiprocessing can't pickle a local function used as a Process target.

Copy link
Member

@dheerajturaga dheerajturaga left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@o-nikolas o-nikolas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks reasonable to me. Any way we can add a test to avoid future regression?

Move bundle_cleanup_main() from a local function inside
_run_stale_bundle_cleanup() to module-level _bundle_cleanup_main().
On macOS, multiprocessing uses spawn instead of fork, and local
functions cannot be pickled for spawning across processes.

Closes: apache#62512

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@Vamsi-klu Vamsi-klu force-pushed the fix/m4-celery-bundle-cleanup-pickle-macos branch from df54e40 to 223df87 Compare March 1, 2026 17:41
@Vamsi-klu
Copy link
Contributor Author

Code looks reasonable to me. Any way we can add a test to avoid future regression?

@o-nikolas I have added the test for that, regression. Could you please give me feedback for the test that I have added? If everything looks good to you, could you please give me a stamp for my PR? Thank you.

@Vamsi-klu
Copy link
Contributor Author

@codex could you give the detailed review feedback for this PR?

@bluczko
Copy link

bluczko commented Mar 1, 2026

I made the exact same change a few days ago: #62528 . What a coincidence.

Copy link
Member

@dheerajturaga dheerajturaga left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! For catching this 😃

@dheerajturaga dheerajturaga merged commit 722fb25 into apache:main Mar 2, 2026
86 checks passed
@potiuk
Copy link
Member

potiuk commented Mar 2, 2026

Nice!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Celery worker fails to perform a stale Dag bundle cleanup

5 participants