--n-gpu-layers '-1' is not auto calculating amount of max available layers

[andrew-aladjev]

I am using the most recent version of llama.cpp (using docker):

docker run \
  --rm \
  --network 'host' \
  --group-add 'video' \
  --ipc 'host' \
  --cap-add 'SYS_PTRACE' \
  --security-opt 'seccomp=unconfined' \
  --device '/dev/kfd' \
  --device '/dev/dri' \
  --volume '/home/llama.cpp:/models' \
  'local/llama.cpp:light-rocm' \
  --model '/models/Qwen3-32B-Q4_K_M.gguf' \
  --ctx-size 0 \
  --n-gpu-layers 24 \
  --no-conversation \
  --prompt 'hello, how are you doing?'

The output is:

load_tensors: offloading 24 repeating layers to GPU                                                     
load_tensors: offloaded 24/65 layers to GPU                                                                                                                                                                      
load_tensors:        ROCm0 model buffer size =  6747.68 MiB                                             
load_tensors:   CPU_Mapped model buffer size = 12093.28 MiB

Now let's try to use the 25 value for --n-gpu-layers:

The output is:

ggml_backend_cuda_buffer_type_alloc_buffer: allocating 5476.00 MiB on device 0: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n: failed to allocate ROCm0 buffer of size 5742004736
graph_reserve: failed to allocate compute buffers

Ok, then 24 layers is the max possible value. Now let's try to use auto calculation (--n-gpu-layers is -1).

The output is:

load_tensors: offloading 0 repeating layers to GPU                                                      
load_tensors: offloaded 0/65 layers to GPU
load_tensors:   CPU_Mapped model buffer size = 18840.96 MiB

Why is the auto calculation of the number of layers for the GPU not working?

[am17an]

I think the correct thing to do is --ngl 99 or some very high number. AFAIK, -1 is not the "auto" calculation as that doesn't exist

[andrew-aladjev]

-ngl 99 tries to offload 99 layers to the gpu and raises cudaMalloc failed: out of memory. Is there any sort of automatic layer offloading mechanism?

[am17an]

There's a PR for this #14067. For now I think you need to find the correct number of layers

[andrew-aladjev]

Thank you.