RuntimeError: normal expects all elements of std >= 0.0 due to unknown cause

[NaCl-1374]

Bug Report: RuntimeError: normal expects all elements of std >= 0.0 during PPO update on quadruped robot

Description

Training crashes during PPO update with RuntimeError: normal expects all elements of std >= 0.0 in the policy's distribution sampling. Despite enabling nan_guard, no NaN dump is generated (no /tmp/mjlab/nan_dumps directory created), suggesting the invalid values are negative or zero std rather than NaN/Inf.

Robot & Training Configuration

Parameter	Value
Robot Type	Quadruped (wheeled-legged, 四轮足)
Max Velocity	lin 2 m/s , ang 3rad/s
Terrain	Flat ground (平地)
Algorithm	PPO

Error Trace

Traceback (most recent call last):
  File "/home/me/unitree_rl/mjlab/src/mjlab/scripts/train.py", line 256, in <module>
    main()
  File "/home/me/unitree_rl/mjlab/src/mjlab/scripts/train.py", line 252, in main
    launch_training(task_id=chosen_task, args=args)
  File "/home/me/unitree_rl/mjlab/src/mjlab/scripts/train.py", line 203, in launch_training
    run_train(task_id, args, log_dir)
  File "/home/me/unitree_rl/mjlab/src/mjlab/scripts/train.py", line 173, in run_train
    runner.learn(
  File "/home/me/unitree_rl/rsl_rl/rsl_rl/runners/on_policy_runner.py", line 108, in learn
    loss_dict = self.alg.update()
                ^^^^^^^^^^^^^^^^^
  File "/home/me/unitree_rl/rsl_rl/rsl_rl/algorithms/ppo.py", line 256, in update
    self.actor(
  File ".../torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/me/unitree_rl/rsl_rl/rsl_rl/models/mlp_model.py", line 106, in forward
    return self.distribution.sample()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/unitree_rl/rsl_rl/rsl_rl/modules/distribution.py", line 180, in sample
    return self._distribution.sample()  # type: ignore
  File ".../torch/distributions/normal.py", line 81, in sample
    return torch.normal(self.loc.expand(shape), self.scale.expand(shape))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: normal expects all elements of std >= 0.0

Key Observations

1. Crash location: Normal.sample() receives std <= 0 (likely from policy log_std exp or softplus)
2. nan_guard ineffective: No NaN dump produced → values are not NaN/Inf, but negative or zero std
3. Potential trigger: action_clip is set to None, removing bounds protection that might prevent extreme policy outputs

Environment

Item	Version
OS	Ubuntu 20.04
Python	3.11.14
PyTorch	(conda env: rl_mjlab)
Library	mjlab + rsl_rl

Steps to Reproduce

1. Configure quadruped robot with max velocity 2 m/s on flat terrain
2. Set action_clip: None (disable action clipping)
3. Enable nan_guard in config
4. Start training with PPO
5. Wait for crash during runner.learn() → alg.update()

Actual Behavior

• Training crashes with std <= 0
• nan_guard silent (no dump directory created)
• Crash occurs mid-training (not at initialization, random crash)

Suggested Investigation

1. Root cause: Why does log_std collapse to -inf or very negative values?

   • Missing action_clip and obs_clip allows unbounded actions → unstable gradients?

2. Defensive programming: Clamp std to min=1e-6 before sampling as safeguard

---

[NaCl-1374]

I added the clip of action and obs, and there is basically no training crash problem, but the training indicators are not as good as the legged gym with the same settings

[kevinzakka]

@NaCl-1374 What are the feet geometry of the quadruped? Spheres?

[kevinzakka]

Can you try leggedrobotics/rsl_rl#190 btw? Use "log" for the std and make sure min clipping is set to something reasonable.

[NaCl-1374]

@NaCl-1374 What are the feet geometry of the quadruped? Spheres?

@kevinzakka My robot is a wheeled robot, similar to unitree b2w.
I want to migrate my robot from legged gym+rsl_rl 2.3 to mjlab. Under the same configuration, the ppo loss of legged gym will be obviously smoother.

Sorry, the following uses 'ActionTermCfg.clip' but the clip takes effect after scale and offset, which causes the training process below to be wrong.
But this often happens during training:

Implementation conditions in the figure:
clip=20 applied to all obs_term
ObservationGroupCfg.nan_policy="error", no Raise ValueError was generated

################################################################################
                         Learning iteration 1923/50000                           

                            Total steps: 189136896 
                       Steps per second: 86488 
                        Collection time: 0.971s 
                          Learning time: 0.166s 
                        Mean value loss: 0.0332
                    Mean surrogate loss: -0.0006
                      Mean entropy loss: 2.8291
                     Mean symmetry loss: 0.4158
                            Mean reward: 55.61
                    Mean episode length: 1000.00
                        Mean action std: 0.31

.....

################################################################################
                         Learning iteration 1925/50000                           

                            Total steps: 189333504 
                       Steps per second: 84230 
                        Collection time: 1.001s 
                          Learning time: 0.166s 
                        Mean value loss: 1447.2889
                    Mean surrogate loss: -0.0039
                      Mean entropy loss: 2.8627
                     Mean symmetry loss: 694.1513
                            Mean reward: 54.66
                    Mean episode length: 1000.00
                        Mean action std: 0.31

.....
################################################################################
                         Learning iteration 1926/50000                           

                            Total steps: 189431808 
                       Steps per second: 84414 
                        Collection time: 0.999s 
                          Learning time: 0.166s 
                        Mean value loss: 553594.8672
                    Mean surrogate loss: -0.0039
                      Mean entropy loss: 2.8840
                     Mean symmetry loss: 54209.8759
                            Mean reward: 54.93
                    Mean episode length: 1000.00
                        Mean action std: 0.31

[NaCl-1374]

Can you try leggedrobotics/rsl_rl#190 btw? Use "log" for the std and make sure min clipping is set to something reasonable.

@kevinzakka When I used it before in Legged Gym, there was no clip std and it didn't crash suddenly