Correct for gradient bias in GRPO style reward centering.

[maitchison]

Motivation

GRPO style centering introduces bias into the gradient estimate. The bias introduced is equal to $$\frac{G-1}{G}$$, where $$G$$ is the group size. While this factor is generally small, and can be incorporated into the learning rate, it would be prefered to not have learning rate dependant on group size in this way.

References

• See https://arxiv.org/pdf/2503.20783 (Pg.14)

Changes

• Added a correction to the advantages to adjust for the bias introduced by GRPO reward centering.