Deployment of DeepseekV3 on 2*H100 nodes is getting stuck.

[abhay-tih]

Hi all,
I am trying to deploy DeepSeek-V3 on two H100 nodes, but the deployment gets stuck at a certain point. I have already set the required environment variables, including:

• NCCL_SOCKET_IFNAME
• NCCL_IB_GID_INDEX=3

When I tried to look into the error using NCCL_DEBUG env variable , the last lines shows this

[Node-A]:[PID]:[Thread] [0] NCCL WARN NET/IB : Got completion from peer [PRIVATE_IP] with error 12, opcode 0, len 0, vendor err 129 (Recv)  
localGid ::ffff:[MASKED_IP] remoteGid ::ffff:[MASKED_IP]  

[Node-A]:[PID]:[Thread] [0] NCCL INFO transport/net.cc:1302 -> 6  
[Node-A]:[PID]:[Thread] [0] NCCL INFO proxy.cc:698 -> 6  
[Node-A]:[PID]:[Thread] [0] NCCL INFO proxy.cc:878 -> 6 [Progress Thread]  

[Node-A]:[PID]:[Thread] [0] NCCL WARN NET/IB : Got completion from peer [PRIVATE_IP] with error 5, opcode 29989, len 29993, vendor err 244 (Recv)  
localGid ::ffff:[MASKED_IP] remoteGid ::ffff:[MASKED_IP]  

[Node-A]:[PID]:[Thread] [0] NCCL INFO transport/net.cc:1302 -> 6  
[Node-A]:[PID]:[Thread] [0] NCCL INFO proxy.cc:698 -> 6  
[Node-A]:[PID]:[Thread] [0] NCCL INFO proxy.cc:878 -> 6 [Progress Thread]  

I am using NCCL version 2.21.5

Has anyone encountered a similar issue, or does anyone have insights on resolving this? Any guidance would be appreciated!

[fzyzcjy]

Hi, could you please use py-spy dump -s -p PID_OF_YOUR_PARENT_PROCESS to have a quick check about where it gets stuck?

[HermitSun]

After upgrading to v0.4.4, I also encountered this issue. It returned to normal after rolling back to v0.4.3.

[HermitSun]

And after I disabled cuda graph, it also returned to normal.

[cynial]

I've encountered a similar situation before, and I resolved it by setting:

export NCCL_SOCKET_IFNAME=bond1  
export NCCL_IB_GID_INDEX=3  

I'm not sure what your NCCL_SOCKET_IFNAME is currently set to, but perhaps you could try changing it?

[nannaer]

I've encountered a similar situation before, and I resolved it by setting:

export NCCL_SOCKET_IFNAME=bond1  
export NCCL_IB_GID_INDEX=3  

I'm not sure what your NCCL_SOCKET_IFNAME is currently set to, but perhaps you could try changing it?

Thanks for your great help! It solved my problem. #6763
This has been a frustrating experience. I've spent almost a week trying to figure this out.