[Question] Enable GDN Packed Sequence Support for Context Parallelism in Qwen 3.5 series

[plmsmile]

Core

Finetuning Qwen3.5 long context commonly needs Context Parallelism (CP).

The Megatron-LM dev branch has supported CP already. But CP needs PackedSequence and the dev branch has not support it yet.

• So No GDN PackedSequence, No Qwen3.5 training.

• So for training Qwen3.5, we need GDN PackedSequence support.

And I'm not sure whether you will have a plan for GDN PackedSequence?

Details

Megatron Repo Code
megatron/core/ssm/gated_delta_net.py

if packed_seq_params is not None:
      # TODO: support packed sequence
      raise NotImplementedError("GDN does not support packed sequence for now.")

A related issue has said:
#3881

Verl mcore code: verl/models/mcore/util.py

cp_size = mpu.get_context_parallel_world_size()
assert cp_size == 1, "Context parallel size without seq_pack is not supported"

[yuzhongw-nvidia]

Hi @plmsmile , thanks for your report.

But CP needs PackedSequence

Do you mean CP needs packed sequence in Megatron-Core or in veRL?

• If the former, this is out of expectation. Could you please share your error message and reproduction with us?
• If the latter, we're going to support packed sequence + CP in [dev] feat(moe): Support packed sequence for gated delta net (GDN) #2644. We're now working on some E2E numerical issues and will try to merge ASAP. Thanks.