The Question about Cross-Policy Sampling Strategy

[supermancmk]

How to implement Cross-Policy Sampling Strategy and the number of model rollout engines.
Thanks~

[zhc7]

Hi, this is implemented here

AgentRL/examples/training/agentrl_trainer.py

Line 95 in 1aaf72b

rollout_stale = spawn(AsyncSglangWorker, rollout_stale_placement, num_gpus=rollout_tp)(rollout_config) if rollout_stale_placement else None

[supermancmk]

Thank you very much. I have another question. According to the code (lines 271-306, in examples/training/agentrl_trainer.py), it seems that you didn't normalize the advantage based on tasks. You have already computed the PPO loss before calculating adv_norm. I'm not sure if I missed something. https://github.com/THUDM/AgentRL/blob/1aaf72b8cb260c1842fb44d4c2c7fe59319e37c4/examples/training/agentrl_trainer.py#L301

[zhc7]

Thanks, that was a mistake. the code was refactored before releasing. It was misplaced. I'll update the code.