train slowly，gpu-util 100% ，but pow 99w/450w

[qwerdf4]

accelerate launch .\train_dreambooth_lora_sdxl.py , my 4090 gpu-util 100% ，but pow only 99w/450w, gpu fan: 30%, 3000steps need almost 40hours.

[asomoza]

you'll need to provide more information (full command for training), if I have to guess you probably have a high image batch number and you're using all the VRAM and started using the RAM.

[asomoza]

yeah, you're using all the GPU VRAM and using shared RAM.

With 24 GB VRAM you'll need a couple of params to start and then you can try changing them:

• you're missing --gradient_checkpointing
• use --use_8bit_adam
• use --train_batch_size=1 and then start going higher.

Always watch the VRAM of your GPU, in your case never go higher than 23.4 GB or it will start using RAM and it's going to be slow.

[qwerdf4]

it seems that bitsandbytes not friendly for Windows ，can't use 8 bit adam

[asomoza]

I forgot, yes, the bitsandbytes is not that easy to install but also not that complicated. you can try with this command:

pip install bitsandbytes --prefer-binary --extra-index-url=https://jllllll.github.io/bitsandbytes-windows-webui

if that doesn't work, you can manually download the wheel from here:

https://github.com/jllllll/bitsandbytes-windows-webui/releases/tag/wheels

I have tested this one:

https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.1-py3-none-win_amd64.whl

ant then install it directly, like this:

pip install bitsandbytes-0.41.1-py3-none-win_amd64.whl

[qwerdf4]

Thank you for your reply! which helped me solve this problem! my model is training quickly now!

[qwerdf4]

Okay, I'll try it !