Bad train loss and good generate image result when use train_dreambooth_lora_flux script.py

[vagitablebirdcode]

Hi everyone,

As the title says, I'm trying to fine-tune using the Flux script with the dog example dataset. Surprisingly, even though the training loss fluctuates heavily, the fine-tuned results are quite good!

This confuses me a bit — I was expecting the training loss to decrease smoothly (ideally in an exponential-like curve), but instead it's very noisy. Still, the final outputs look fine. So I'm not sure how meaningful the training loss actually is in this context.

For comparison, I tried fine-tuning on other platforms like ModelScope, and their results are also good — but their training loss curves (e.g., see Figure 2 above) look much smoother and more in line with intuition.

I'd love to understand why our fine-tuning script results in such unstable training loss (despite good performance), and if there's anything we can do to make the loss values more interpretable or stable like on other platforms.

Thanks in advance!

[asomoza]

Hi, there's a difference between showing the steps and the epochs, depending on how big your dataset, this makes a big difference, in my understanding the way we show the loss is the correct one (and it is raw data), since the training depends on noise, we get some variations but if you display the epochs you will probably see a much better curve, overall the values should still go down.

Do you have access to the modelscope loss function? Do they smooth the curve? Are you also training a dreambooth lora with the same hyperparameters?

I think this article provides helpful insights on the loss curve.

It seems to me they're doing something similar to what that article points in the EveryDream2 trainer.

[vagitablebirdcode]

Thanks a lot for the suggestions and reference! After applying a smoothing window to the loss curve, I found that the overall trend is indeed decreasing. With the smoothed plot, the loss curve is now much more interpretable and definitely useful for monitoring.