Building llama-cpp-python with CUDA support fails due to GLIBC version incompatibility

[ObaOzai]

Hello,

Issue Summary

Building llama-cpp-python with CUDA support fails due to GLIBC version incompatibility

Environment Details

OS: Ubuntu Linux
GPU: NVIDIA (Driver 570.158.01, CUDA 12.8)
Python Environment: Miniconda3 with Python 3.13
PyTorch: 2.8.0.dev20250622+cu128
Compiler: conda’s x86_64-conda-linux-gnu-c++ (GCC 11.2.0)
/home/oba/miniconda3/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /usr/local/cuda/lib64/libcublasLt.so.12: undefined reference to log2f@GLIBC_2.27' /home/oba/miniconda3/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /usr/local/cuda/lib64/libcublasLt.so.12: undefined reference to __cxa_thread_atexit_impl@GLIBC_2.18’
collect2: error: ld returned 1 exit status

Root Cause

The CUDA library libcublasLt.so.12 requires GLIBC symbols:

log2f@GLIBC_2.27 (from GLIBC 2.27+)
__cxa_thread_atexit_impl@GLIBC_2.18 (from GLIBC 2.18+)
But the system apparently has an older GLIBC version that doesn’t provide these symbols.

Build Command Used CMAKE_ARGS=“-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=native” pip install -e . --verbose

What’s Been Tried

Using system CUDA instead of conda CUDA - same error
Both conda CUDA libraries and system CUDA libraries show the same GLIBC dependency issue
The error occurs specifically when linking vision tools (llava, mtmd) that depend on CUDA libraries
Questions for Forum

How to resolve GLIBC version conflicts when building llama-cpp-python with CUDA?
Is there a way to use older/compatible CUDA libraries that don’t require GLIBC 2.27+?
Can the build be configured to skip problematic vision components while keeping core CUDA functionality?
Should I use system compiler instead of conda compiler, or create a different conda environment?
Additional Context

The build progresses successfully until the final linking stage for vision tools
Core CUDA libraries (libggml-cuda.so) appear to build successfully
Only fails when linking final executables that use cuBLAS
This gives forum responders all the technical details they need to help diagnose the specific GLIBC/CUDA library compatibility issue.

Regards,

[thedarkbird]

I made a post with a guide on compiling/installing for (Fedora) linux, might be helpful.

[ObaOzai]

Greetings, Pls. post link bellow. Thanks.

[thedarkbird]

Link to my post: #2043

[marybary0000]

Ugh I actually ran into this exact issue a while back while playing around with llama-cpp-python and trying to enable CUDA support. I didn’t realize how tricky the GLIBC version stuff can get on some distros. Did anyone here manage to work around it without upgrading the whole system? I’d love to hear what worked for others.

[ObaOzai]

I just stoped trying to use GPU with llama-cpp-python. Seems some other people have made it work for their env. im not sure how they made it work or if their soultion would fix the issue on my end.

6.14.0-27-generic #27~24.04.1-Ubuntu

[thedarkbird]

Try my solution, should work for any distro since I'm using a Conda env with the right tools: #2043

[ObaOzai]

same error

(tf) oba@mail:/llama-cpp/llama-cpp-python$ export VERBOSE=1
export CMAKE_ARGS="-DGGML_CUDA=ON -DLLAVA_BUILD=OFF -DCMAKE_CUDA_ARCHITECTURES=all -DGGML_CUDA_FORCE_MMQ=OFF -DGGML_AVX2=OFF -DGGML_FMA=OFF -DGGML_F16C=OFF"
(tf) oba@mail:/llama-cpp/llama-cpp-python$ pip install -e . --verbose
Using pip 25.1 from /home/oba/miniconda3/envs/tf/lib/python3.10/site-packages/pip (python 3.10)
Obtaining file:///home/oba/llama-cpp/llama-cpp-python
Running command pip subprocess to install build dependencies
Using pip 25.1 from /home/oba/miniconda3/envs/tf/lib/python3.10/site-packages/pip (python 3.10)
Collecting scikit-build-core>=0.9.2 (from scikit-build-core[pyproject]>=0.9.2)
Obtaining dependency information for scikit-build-core>=0.9.2 from https://files.pythonhosted.org/packages/45/23/0ffa0df7550ca0535f6e03b9a9ab2bf0495ac62e15fd322544c98321a10c/scikit_build_core-0.11.5-py3-none-any.whl.metadata
Using cached scikit_build_core-0.11.5-py3-none-any.whl.metadata (18 kB)
Collecting exceptiongroup>=1.0 (from scikit-build-core>=0.9.2->scikit-build-core[pyproject]>=0.9.2)
Obtaining dependency information for exceptiongroup>=1.0 from https://files.pythonhosted.org/packages/36/f4/c6e662dade71f56cd2f3735141b265c3c79293c109549c1e6933b0651ffc/exceptiongroup-1.3.0-py3-none-any.whl.metadata
Using cached exceptiongroup-1.3.0-py3-none-any.whl.metadata (6.7 kB)
Collecting packaging>=23.2 (from scikit-build-core>=0.9.2->scikit-build-core[pyproject]>=0.9.2)
Obtaining dependency information for packaging>=23.2 from https://files.pythonhosted.org/packages/20/12/38679034af332785aac8774540895e234f4d07f7545804097de4b666afd8/packaging-25.0-py3-none-any.whl.metadata
Using cached packaging-25.0-py3-none-any.whl.metadata (3.3 kB)
Collecting pathspec>=0.10.1 (from scikit-build-core>=0.9.2->scikit-build-core[pyproject]>=0.9.2)
Obtaining dependency information for pathspec>=0.10.1 from https://files.pythonhosted.org/packages/cc/20/ff623b09d963f88bfde16306a54e12ee5ea43e9b597108672ff3a408aad6/pathspec-0.12.1-py3-none-any.whl.metadata
Using cached pathspec-0.12.1-py3-none-any.whl.metadata (21 kB)
Collecting tomli>=1.2.2 (from scikit-build-core>=0.9.2->scikit-build-core[pyproject]>=0.9.2)
Obtaining dependency information for tomli>=1.2.2 from https://files.pythonhosted.org/packages/6e/c2/61d3e0f47e2b74ef40a68b9e6ad5984f6241a942f7cd3bbfbdbd03861ea9/tomli-2.2.1-py3-none-any.whl.metadata
Using cached tomli-2.2.1-py3-none-any.whl.metadata (10 kB)
Collecting typing-extensions>=4.6.0 (from exceptiongroup>=1.0->scikit-build-core>=0.9.2->scikit-build-core[pyproject]>=0.9.2)
Obtaining dependency information for typing-extensions>=4.6.0 from https://files.pythonhosted.org/packages/b5/00/d631e67a838026495268c2f6884f3711a15a9a2a96cd244fdaea53b823fb/typing_extensions-4.14.1-py3-none-any.whl.metadata
Using cached typing_extensions-4.14.1-py3-none-any.whl.metadata (3.0 kB)
Using cached scikit_build_core-0.11.5-py3-none-any.whl (185 kB)
Using cached exceptiongroup-1.3.0-py3-none-any.whl (16 kB)
Using cached packaging-25.0-py3-none-any.whl (66 kB)
Using cached pathspec-0.12.1-py3-none-any.whl (31 kB)
Using cached tomli-2.2.1-py3-none-any.whl (14 kB)
Using cached typing_extensions-4.14.1-py3-none-any.whl (43 kB)
Installing collected packages: typing-extensions, tomli, pathspec, packaging, exceptiongroup, scikit-build-core

Successfully installed exceptiongroup-1.3.0 packaging-25.0 pathspec-0.12.1 scikit-build-core-0.11.5 tomli-2.2.1 typing-extensions-4.14.1
Installing build dependencies ... done
Running command Checking if build backend supports build_editable
Checking if build backend supports build_editable ... done
Running command Getting requirements to build editable
Getting requirements to build editable ... done
Running command Preparing editable metadata (pyproject.toml)
*** scikit-build-core 0.11.5 using CMake 3.28.3 (metadata_editable)
Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: typing-extensions>=4.5.0 in /home/oba/miniconda3/envs/tf/lib/python3.10/site-packages (from llama_cpp_python==0.3.14) (4.14.1)
Requirement already satisfied: numpy>=1.20.0 in /home/oba/miniconda3/envs/tf/lib/python3.10/site-packages (from llama_cpp_python==0.3.14) (1.26.4)
Collecting diskcache>=5.6.1 (from llama_cpp_python==0.3.14)
Obtaining dependency information for diskcache>=5.6.1 from https://files.pythonhosted.org/packages/3f/27/4570e78fc0bf5ea0ca45eb1de3818a23787af9b390c0b0a0033a1b8236f9/diskcache-5.6.3-py3-none-any.whl.metadata
Using cached diskcache-5.6.3-py3-none-any.whl.metadata (20 kB)
Requirement already satisfied: jinja2>=2.11.3 in /home/oba/miniconda3/envs/tf/lib/python3.10/site-packages (from llama_cpp_python==0.3.14) (3.1.6)
Requirement already satisfied: MarkupSafe>=2.0 in /home/oba/miniconda3/envs/tf/lib/python3.10/site-packages (from jinja2>=2.11.3->llama_cpp_python==0.3.14) (2.1.1)
Using cached diskcache-5.6.3-py3-none-any.whl (45 kB)
Building wheels for collected packages: llama_cpp_python
Running command Building editable for llama_cpp_python (pyproject.toml)
*** scikit-build-core 0.11.5 using CMake 3.28.3 (editable)
*** Configuring CMake...
loading initial cache file /tmp/tmpcf8naay4/build/CMakeInit.txt
Re-run cmake no build system arguments
-- The C compiler identification is GNU 13.3.0
-- The CXX compiler identification is GNU 13.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: /home/oba/miniconda3/envs/tf/bin/git (found version "2.45.2")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Warning: ccache not found - consider installing it for faster compilation or disable this warning with GGML_CCACHE=OFF
-- CMAKE_SYSTEM_PROCESSOR: x86_64
-- GGML_SYSTEM_ARCH: x86
-- Including CPU backend
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- x86 detected
-- Adding CPU backend variant ggml-cpu: -march=native
-- Found CUDAToolkit: /usr/include (found version "12.0.140")
-- CUDA Toolkit found
-- Using CUDA architectures: all
CMake Error at /usr/share/cmake-3.28/Modules/CMakeDetermineCompilerId.cmake:780 (message):
Compiling the CUDA compiler identification source file
"CMakeCUDACompilerId.cu" failed.

Compiler: /usr/bin/nvcc

Build flags:

Id flags: --keep;--keep-dir;tmp -v



The output was:

1

#$ NVCC_PREPEND_FLAGS="
-ccbin=/home/oba/miniconda3/bin/x86_64-conda-linux-gnu-c++"

#$ _NVVM_BRANCH_=nvvm

#$ _SPACE_=

#$ _CUDART_=cudart

#$ _HERE_=/usr/lib/nvidia-cuda-toolkit/bin

#$ _THERE_=/usr/lib/nvidia-cuda-toolkit/bin

#$ _TARGET_SIZE_=

#$ _TARGET_DIR_=

#$ _TARGET_SIZE_=64

#$ NVVMIR_LIBRARY_DIR=/usr/lib/nvidia-cuda-toolkit/libdevice

#$
PATH=/usr/lib/nvidia-cuda-toolkit/bin:/tmp/pip-build-env-fwu3_8_s/overlay/bin:/tmp/pip-build-env-fwu3_8_s/normal/bin:/home/oba/miniconda3/envs/tf/bin:/home/oba/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin


#$ LIBRARIES= -L/usr/lib/x86_64-linux-gnu/stubs -L/usr/lib/x86_64-linux-gnu

#$ rm tmp/a_dlink.reg.c

#$ "/home/oba/miniconda3/bin"/x86_64-conda-linux-gnu-c++
-D__CUDA_ARCH_LIST__=520 -E -x c++ -D__CUDACC__ -D__NVCC__
-D__CUDACC_VER_MAJOR__=12 -D__CUDACC_VER_MINOR__=0
-D__CUDACC_VER_BUILD__=140 -D__CUDA_API_VER_MAJOR__=12
-D__CUDA_API_VER_MINOR__=0 -D__NVCC_DIAG_PRAGMA_SUPPORT__=1 -include
"cuda_runtime.h" -m64 "CMakeCUDACompilerId.cu" -o
"tmp/CMakeCUDACompilerId.cpp4.ii"

<command-line>: fatal error: cuda_runtime.h: No such file or directory

compilation terminated.

# --error 0x1 --

Call Stack (most recent call first):
/usr/share/cmake-3.28/Modules/CMakeDetermineCompilerId.cmake:8 (CMAKE_DETERMINE_COMPILER_ID_BUILD)
/usr/share/cmake-3.28/Modules/CMakeDetermineCompilerId.cmake:53 (__determine_compiler_id_test)
/usr/share/cmake-3.28/Modules/CMakeDetermineCUDACompiler.cmake:135 (CMAKE_DETERMINE_COMPILER_ID)
vendor/llama.cpp/ggml/src/ggml-cuda/CMakeLists.txt:43 (enable_language)

-- Configuring incomplete, errors occurred!

*** CMake configuration failed
error: subprocess-exited-with-error

× Building editable for llama_cpp_python (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /home/oba/miniconda3/envs/tf/bin/python3.10 /home/oba/miniconda3/envs/tf/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_editable /tmp/tmp_ghueols
cwd: /home/oba/llama-cpp/llama-cpp-python
Building editable for llama_cpp_python (pyproject.toml) ... error
ERROR: Failed building editable for llama_cpp_python
Failed to build llama_cpp_python
ERROR: Failed to build installable wheels for some pyproject.toml based projects (llama_cpp_python)

[ObaOzai]

(tf) oba@mail:~/llama-cpp$ git clone --recursive https://github.com/JamePeng/llama-cpp-python.git
cd llama-cpp-python
git checkout tags/v0.3.14-cu128-AVX2-linux-20250801
Cloning into 'llama-cpp-python'...
remote: Enumerating objects: 9034, done.
remote: Counting objects: 100% (2702/2702), done.
remote: Compressing objects: 100% (207/207), done.
remote: Total 9034 (delta 2599), reused 2498 (delta 2495), pack-reused 6332 (from 2)
Receiving objects: 100% (9034/9034), 2.39 MiB | 5.89 MiB/s, done.
Resolving deltas: 100% (5928/5928), done.
Submodule 'vendor/llama.cpp' (https://github.com/ggerganov/llama.cpp.git) registered for path 'vendor/llama.cpp'
Cloning into '/home/oba/llama-cpp/llama-cpp-python/vendor/llama.cpp'...