grpo 多任务训练 奖励函数设置返回None 这样的话，如果想要查看单个任务的reward曲线，在tensorboard中会出现nan的情况

[XQZZK]

grpo 多任务训练 奖励函数设置返回None 这样的话，如果想要查看单个任务的reward曲线，在tensorboard中会出现nan的情况

具体来说，是在
swift/trainers/rlhf_trainer/grpo_trainer.py(1522)
metrics = {key: sum(val) / len(val) for key, val in self._metrics[mode].items()} # average the metrics

如果val中存在一个nan的话，那么sum(val)就会为nan

请问如何处理这种情况

[hjh0119]

What version of swift are you using?

In swift 3.5, I believe that NaN values have already been excluded by nanstd and nanmean, right?

https://github.com/modelscope/ms-swift/blob/v3.5.1/swift/trainers/rlhf_trainer/grpo_trainer.py#L1050-L1054

[XQZZK]

我使用的是3.6.0，是的
https://github.com/modelscope/ms-swift/blob/v3.5.1/swift/trainers/rlhf_trainer/grpo_trainer.py#L1050-L1054
会排除掉nan，但是考虑一种极端情况，

这种情况会将nan这个值给带到swift/trainers/rlhf_trainer/grpo_trainer.py(1522)这里来

[XQZZK]

What version of swift are you using?

In swift 3.5, I believe that NaN values have already been excluded by nanstd and nanmean, right?

https://github.com/modelscope/ms-swift/blob/v3.5.1/swift/trainers/rlhf_trainer/grpo_trainer.py#L1050-L1054

#4653 (comment)

[hjh0119]

So we need to exclude NaN values from the calculations in the log as well, right?

https://github.com/modelscope/ms-swift/blob/v3.5.1/swift/trainers/rlhf_trainer/grpo_trainer.py#L1512-L1518

[XQZZK]

So we need to exclude NaN values from the calculations in the log as well, right?

https://github.com/modelscope/ms-swift/blob/v3.5.1/swift/trainers/rlhf_trainer/grpo_trainer.py#L1512-L1518

我认为是这样的，可能需要修改1512到1518行这几个地方，我目前是这样修改的，但是不知道对不对