static loss scale with stage 0 occured error

[zoyC-0a]

DeepSpeed/deepspeed/runtime/engine.py

Line 1571 in d095b18

optimizer = FP16_Optimizer(

Config:
`
deepspeed_config = {
"fp16": {
"enabled": "auto",
"loss_scale": 4,
"loss_scale_window": 1000,
"initial_scale_power": 16,
"hysteresis": 2,
"min_loss_scale": 0.00001,
"auto_cast": True
},

"zero_optimization": {
    "stage": 0
},

"optimizer": {
    "type": "AdamW",
    "params": {
        "lr": "auto",
        "betas": "auto",
        "eps": "auto",
        "weight_decay": "auto"
    }
},

"scheduler": {
    "type": "WarmupLR",
    "params": {
        "warmup_min_lr": "auto",
        "warmup_max_lr": "auto",
        "warmup_num_steps": "auto"
    }
},
"gradient_clipping": 'auto',
"train_batch_size": 'auto',
"train_micro_batch_size_per_gpu": 'auto'

}
`

Error be like :
File "/usr/local/miniconda3/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 2119, in _take_model_step self.optimizer.step() File "/usr/local/miniconda3/lib/python3.10/site-packages/deepspeed/runtime/fp16/fused_optimizer.py", line 255, in step self.timers(OVERFLOW_CHECK_TIMER).start() TypeError: 'NoneType' object is not callable

So may be we should initialize timer before line 1571?

Then I changed zero_optimization to stage 2, and the error didn't show up again. Or are there any other solutions?

[tjruwase]

@zoyC-0a, thanks for reporting this error. Do you mind creating a PR to pass timers=timers just like

DeepSpeed/deepspeed/runtime/engine.py

Line 1566 in d095b18

timers=timers,