Does BatchNorm have a wrong moving average? #2083
-
|
The |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
Arguably this just comes down to whether or not you wish to presume a prior for the mean and variance of the running averages. Since BatchNorm running statistics are exponentially weighted in the expectation of calculating them over a large number of batches, the choice of a zero-mean / unity-variance prior doesn't in practice matter much at all, and keeps the implementation much simpler. |
Beta Was this translation helpful? Give feedback.
Arguably this just comes down to whether or not you wish to presume a prior for the mean and variance of the running averages. Since BatchNorm running statistics are exponentially weighted in the expectation of calculating them over a large number of batches, the choice of a zero-mean / unity-variance prior doesn't in practice matter much at all, and keeps the implementation much simpler.