how to finetune PLM with Deepspeed or other method to reduce GPU mem usage

[idejie]

I find that , for apps.plm.train, PLM has been applied FSDP. It can not be fine-tuned with two 80G GPU mem (OOM error), setting tp_size=2 and batch_size=1 . the code can be trained with Deepspeed?

[shuminghu]

Sorry just saw this. I can update with a TP fix tomorrow.