Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: VLLLm crash when running Qwen/Qwen2.5-Coder-32B-Instruct on two H100 GPUs #10296

Open
1 task done
noamwies opened this issue Nov 13, 2024 · 0 comments
Open
1 task done

[Bug]: VLLLm crash when running Qwen/Qwen2.5-Coder-32B-Instruct on two H100 GPUs #10296

noamwies opened this issue Nov 13, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@noamwies
Copy link

Your current environment

The output of `python collect_env.py` 
PyTorch version: 2.4.0+cu121                                                                                                                                                                                                                                                    
Is debug build: False                                                                                                                                                                                                                                                           
CUDA used to build PyTorch: 12.1                                                                                                                                                                                                                                                
ROCM used to build PyTorch: N/A                                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                                
OS: Ubuntu 22.04.4 LTS (x86_64)                                                                                                                                                                                                                                                 
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0                                                                                                                                                                                                                              
Clang version: Could not collect                                                                                                                                                                                                                                                
CMake version: Could not collect                                                                                                                                                                                                                                                
Libc version: glibc-2.35                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                
Python version: 3.12.7 (main, Oct  1 2024, 08:52:12) [GCC 11.4.0] (64-bit runtime)                                                                                                                                                                                              
Python platform: Linux-6.5.0-1023-aws-x86_64-with-glibc2.35                                                                                                                                                                                                                     
Is CUDA available: True                                                                                                                                                                                                                                                         
CUDA runtime version: Could not collect                                                                                                                                                                                                                                         
CUDA_MODULE_LOADING set to: LAZY                                                                                                                                                                                                                                                
GPU models and configuration:                                                                                                                                                                                                                                                   
GPU 0: NVIDIA H100 80GB HBM3                                                                                                                                                                                                                                                    
GPU 1: NVIDIA H100 80GB HBM3                                                                                                                                                                                                                                                    
GPU 2: NVIDIA H100 80GB HBM3                                                                                                                                                                                                                                                    
GPU 3: NVIDIA H100 80GB HBM3 
GPU 4: NVIDIA H100 80GB HBM3                                                                                                                                                                                                                                                    
GPU 5: NVIDIA H100 80GB HBM3                                                                                                                                                                                                                                                    
GPU 6: NVIDIA H100 80GB HBM3                                                                                                                                                                                                                                                    
GPU 7: NVIDIA H100 80GB HBM3                                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                
Nvidia driver version: 535.183.01                                                                                                                                                                                                                                               
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      46 bits physical, 57 bits virtual
Byte Order:                         Little Endian
CPU(s):                             192
On-line CPU(s) list:                0-191
Vendor ID:                          GenuineIntel
Model name:                         Intel Xeon Processor (SapphireRapids)
CPU family:                         6
Model:                              143
Thread(s) per core:                 1
Core(s) per socket:                 192
Socket(s):                          1
Stepping:                           4
BogoMIPS:                           4200.00
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 
sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f 
avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd arat vnmi avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57
 rdpid bus_lock_detect cldemote movdiri movdir64b fsrm md_clear serialize tsxldtrk avx512_fp16 arch_capabilities
Virtualization:                     VT-x
Hypervisor vendor:                  KVM
Virtualization type:                full
L1d cache:                          6 MiB (192 instances)
L1i cache:                          6 MiB (192 instances)
L2 cache:                           768 MiB (192 instances)
L3 cache:                           16 MiB (1 instance)
NUMA node(s):                       1
NUMA node0 CPU(s):                  0-191
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Unknown: No mitigations
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI Syscall hardening, KVM SW loop
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Mitigation; TSX disabled

Versions of relevant libraries:
[pip3] flashinfer==0.1.6+cu121torch2.4
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.1.3.1
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.0.2.54
[pip3] nvidia-curand-cu12==10.3.2.106
[pip3] nvidia-cusolver-cu12==11.4.5.107
[pip3] nvidia-cusparse-cu12==12.1.0.106
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] nvidia-nvjitlink-cu12==12.6.77
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] pyzmq==26.2.0
[pip3] torch==2.4.0
[pip3] torchvision==0.19.0
[pip3] transformers==4.45.2
[pip3] triton==3.0.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.3.post1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     0-191   0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     0-191   0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     0-191   0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     0-191   0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     0-191   0               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     0-191   0               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     0-191   0               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     0-191   0               N/A
NIC0    PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB      X      PHB     PHB     PHB     PHB     PHB     PHB     PHB
NIC1    PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB      X      PHB     PHB     PHB     PHB     PHB     PHB
NIC2    PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB      X      PHB     PHB     PHB     PHB     PHB
NIC3    PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB      X      PHB     PHB     PHB     PHB
NIC4    PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB      X      PHB     PHB     PHB
NIC5    PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB      X      PHB     PHB
NIC6    PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB      X      PHB
NIC7    PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB      X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7

NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driv
er>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,drive
r<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=
unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,dr
iver>=535,driver<536 brand=titanrtx,driver>=535,driver<536
NVIDIA_DRIVER_CAPABILITIES=compute,utility
VLLM_USAGE_SOURCE=production-docker-image
CUDA_VERSION=12.4.1
CUDA_VISIBLE_DEVICES=2,3
CUDA_VISIBLE_DEVICES=2,3
LD_LIBRARY_PATH=/usr/local/lib/python3.12/dist-packages/cv2/../../lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
CUDA_MODULE_LOADING=LAZY

Model Input Dumps

No response

🐛 Describe the bug

VLLLm crash when running Qwen/Qwen2.5-Coder-32B-Instruct on two H100 GPUs

Command for reprduce:

sudo docker run --gpus all -e CUDA_VISIBLE_DEVICES=0,1  -e HUGGING_FACE_HUB_TOKEN=$HUGGING_FACE_HUB_TOKEN --ipc=host -p 8000:8000  -v /dev/shm:/dev/shm -v /nvme/.tritoncache/:/root/.triton/cache -v /nvme:/nvme -v $HF_HOME:/root/.cache/huggingface vllm/vllm-openai:v0.6.3.post1 --model Qwen/Qwen2.5-Coder-32B-Instruct --tensor-parallel-size 2 --disable-log-stats --dtype bfloat16 --max-model-len 8192 --swap-space 128 --disable-log-requests --enable-chunked-prefill=false

Error message:

(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] Exception in worker VllmWorkerProcess while processing method start_worker_execution_loop.                                                                                                      
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] Traceback (most recent call last):                                                                                                                                                              
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/multiproc_worker_utils.py", line 223, in _run_worker_process                                                                      
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]     output = executor(*args, **kwargs)                                                                                                                                                          
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]              ^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                          
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context                                                                                      
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]     return func(*args, **kwargs)                                                                                                                                                                
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]            ^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 69, in start_worker_execution_loop                                                                            
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]     output = self.execute_model(execute_model_req=None)                                                                                                                                         
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                         
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 303, in execute_model                                                                                         
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]     inputs = self.prepare_input(execute_model_req)                                                                                                                                              
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                              
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 293, in prepare_input                                                                                         
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]     return self._get_worker_input_from_broadcast()                                                                                                                                              
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                              
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 237, in _get_worker_input_from_broadcast                                                                      
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]     self.model_runner.make_model_input_from_broadcasted_tensor_dict(
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1561, in make_model_input_from_broadcasted_tensor_dict
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]     ModelInputForGPUWithSamplingMetadata.from_broadcasted_tensor_dict(
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 175, in from_broadcasted_tensor_dict
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]     tensor_dict = _init_attn_metadata_from_tensor_dict(
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner_base.py", line 53, in _init_attn_metadata_from_tensor_dict
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]     attn_metadata = attn_backend.make_metadata(**valid_attn_kwargs)
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/attention/backends/abstract.py", line 47, in make_metadata
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]     return cls.get_metadata_cls()(*args, **kwargs)
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] TypeError: FlashAttentionMetadata.__init__() missing 15 required positional arguments: 'num_prefills', 'num_prefill_tokens', 'num_decode_tokens', 'slot_mapping', 'seq_lens', 'seq_lens_tensor', 'max_query_len', 'max_decode_query_len', 'max_prefill_seq_len', 'max_decode_seq_len', 'query_start_loc', 'seq_start_loc', 'context_lens_tensor', 'block_tables', and 'use_cuda_graph'

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
@noamwies noamwies added the bug Something isn't working label Nov 13, 2024
@noamwies noamwies changed the title [Bug]: [Bug]: VLLLm crash when running Qwen/Qwen2.5-Coder-32B-Instruct on two H100 GPUs Nov 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@noamwies