Padding free feature

[VietDunghacker]

Padding free seems to be an excellent idea, but how do you prevent the information leakage? Since the batch is flatten, the later item in the batch may attend to the information from previous items, which I don't think it is supposed to in any case, even in pretraining step.
I have not thoroughly test the feature, but I did a finetune on Qwen2.5VL and found out that padding-free will negatively affect the performance, causing the model to hallucinate information that is not even in the image.

[Jintao-Huang]

flash_attn will help with this to prevent information leakage.

[Jintao-Huang]

GRPO or SFT

[Jintao-Huang]

In theory, is padding_free completely equivalent? Could you provide a shell script?

[VietDunghacker]

I cannot provide the script, but I can tell that I did SFT and the only difference in my 2 runs is "--padding_free true". The 30% performance drop in padding_free run clearly indicates that something is wrong with the padding_free implementation. Moreover, the loss in padding_free run is lowered by half, so it is definitely not equivalent.
And from your answer, I think that flash attention has causal mask that prevents any token from seeing the token after it, but how about before it, in this case? To be clear, after flattened, the batch of sequences will become a single sequence: sequence1 + sequence2 + sequence3. How do you ensure that the tokens in sequence2 will not attend to the tokens in sequence1, which might contain the information that sequence2 is not supposed to see?

[Jintao-Huang]

#4444

[Jintao-Huang]

The main branch has already fixed this issue. Thank you for your feedback.

[VietDunghacker]

I tested the new code with Lora + GRPO, now there is new issue with gradient checkpointing (a gradient was computed twice). Can you check it?

[Jintao-Huang]

@hjh0119

[Jintao-Huang]

I tested the new code with Lora + GRPO, now there is new issue with gradient checkpointing (a gradient was computed twice). Can you check it?

If the issue persists, please open a separate issue. Thanks.

[wizyoung]

Wierd, it seems that the issue persists... The loss curves of qwen2.5vl following sft under padding_free and non_padding_free conditions differ and the padding_free one has a little bit higer loss. I think they should be exact the same? My running commit: 88be385