Fix BatchNorm crash under Mooncake in eval mode#740
Open
AstitvaAggarwal wants to merge 2 commits into
Open
Conversation
AstitvaAggarwal
marked this pull request as draft
July 22, 2026 00:40
cudnnBNForward! only fills in a passed-in cache's mean/ivar when training=true; in eval mode it leaves them unset, but cudnnBNBackward! reads the cache anyway, crashing on the stale nothing. The Mooncake batchnorm rule always built a live cache regardless of mode, so any call that turned out to be eval mode crashed with MethodError converting nothing to a CuPtr. This is easy to hit via Flux.BatchNorm, whose auto mode-detection (NNlib.within_gradient) never resolves true under Mooncake, since it's implemented purely as a ChainRulesCore.rrule trick that only Zygote-style AD triggers. Fix: only build the cache when training is actually happening.
AstitvaAggarwal
force-pushed
the
fix/batchnorm-cache-eval-mode
branch
from
July 22, 2026 00:46
a1af08e to
bf330b9
Compare
BatchNorm cache crash when training=false under Mooncake
The rule always replaced whatever `cache` value the caller passed (e.g. via Flux.BatchNorm's own `cache` argument) with its own _BNFwdCache. If a caller ever supplied their own cache object, it was silently discarded rather than being written to, breaking any state they expected to read back afterward. Now only fall back to constructing a fresh cache when the caller didn't supply one; otherwise use theirs.
BatchNorm cache crash when training=false under Mooncake
AstitvaAggarwal
marked this pull request as ready for review
July 22, 2026 10:53
Contributor
Author
|
@CarloLucibello I think this is good from my side, please let me know if this needs anything more. Thank you |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
While running BatchNorm through Mooncake in eval mode, the backward pass crashes with:
The cause:
cudnnBNForward!only fills in the cache'smean/ivarwhentraining=true— in eval mode it just leaves them unset. ButcudnnBNBackward!doesn't check for that; it assumes any non-nothingcache is already populated and hands whatever's in it straight to cuDNN.The Mooncake rule for
batchnormalways builds a live cache, whether it's training or not, so eval-mode calls hit this every time. This is easy to trigger viaFlux.BatchNorm, since its training/eval auto-detection (NNlib.within_gradient) never returnstrueunder Mooncake — that trick only works for Zygote.This PR makes two small fixes:
I tested this directly on GPU: reproduced the crash on the released NNlib, confirmed it's gone with the patch, checked the gradient still matches Zygote, and ran through all four combinations of
training/track_statsto make sure nothing else was missed.