ABI builds fails due wheel repair for dynamic libraries

[petrasvestartas]

I am building a binding with referenced dynamic libraries:
https://github.com/petrasvestartas/compas_occt/blob/af2e828963c952795103d1ad58f7c86403743b14/pyproject.toml

cibuildwheel succeeds for less than Python 3.12 version, when it comes to ABI it fails: 3.12, 3.13, 3.14 it fails.

                                                      ✕ 24.18s
cibuildwheel: error: Command ['sh', '-c', "python -m zipfile -e /tmp/cibuildwheel/built_wheel/compas_occt-0.1.15-cp312-abi3-linux_x86_64.whl /tmp/wheel_extract && find /tmp/wheel_extract -name '*.so*' -path '*/compas_occt/lib/*' -exec cp {} /usr/local/lib/ \\; && ldconfig && auditwheel repair --plat manylinux2014_x86_64 -w /tmp/cibuildwheel/repaired_wheel /tmp/cibuildwheel/built_wheel/compas_occt-0.1.15-cp312-abi3-linux_x86_64.whl"] failed with code 2. 

I am using repair operation that make this error, because when you build a Python wheel that uses external shared libraries (like OCCT), the wheel initially contains:

• Your Python extension modules (.so files)
• References to external shared libraries (like libTKernel.so.7.9)
• But the external libraries themselves are NOT included in the wheel by default.

How do you normally include externally build .dll without repair?

[LecrisUT]

How do you normally include externally build .dll without repair?

With pain :D. Note that .dll and .so inclusion are two different worlds, but given the issue you describe, I guess you mean the second.

What would be helpful for diagnostic are:

• the CMake targets that build python modules/shared libraries and how they are linked to the python library/other dependency
• how it works without the custom commands (can't tell which step is failing)
• a compiled wheel before the wheel repair to diagnose

[petrasvestartas]

I think it is related to ABI settings.

In CMake ABI is set like this:
find_package(Python 3.8 REQUIRED COMPONENTS Interpreter Development.Module Development.SABIModule)

What is in pyproject.toml setting ABI on and off ?
What is happening there is that multiple wheel 3.12 and 3.13 are built, but not ABI, and I cannot find what is causing that.

[LecrisUT]

The find_package(Python) is most likely a red-herring. The important part is how and what you consume in the CMake targets which may/may not use the SABI.

The building of ABI is however irrelevant (apart from the fact that you disabled the python 3.12 build) when it comes to the wheel repair. It may cause an issue if the wheel tag is inconsistent with the .so file names, and if other components link to a versioned python library.

Please try to provide a compiled, wheel, it would be straightfoward to point the issue there. I will look in the repo briefly to see if I notice anything fast.

[LecrisUT]

You have an inconsistency between the SABI usage in the wheel tag and nanobind here:
https://github.com/petrasvestartas/compas_occt/blob/af2e828963c952795103d1ad58f7c86403743b14/CMakeLists.txt#L176

Please add the STABLE_ABI keyword in there.

[petrasvestartas]

Thank you, I am rebuilding (fingers crossed):

Meanwhile this is 3.9 python compiled version:
https://mega.nz/file/M052mBaY#RfSkFQEtXSmXbQ5Y29KdTawhmG4WX3Vf9qAtkXzYT_M

[petrasvestartas]

✅ Build successful: The wheel compas_occt-0.1.15-cp312-abi3-linux_x86_64.whl was created successfully (75MB)

❌ But even when disabled, it tries to be repaired, no idea why but i get the error the following error, I will try with this pyproject.toml flag repair-wheel-command = "" since cmake must take care of coyping files:

  *** Making wheel...
  *** Created compas_occt-0.1.15-cp312-abi3-linux_x86_64.whl
  Building wheel for compas_occt (pyproject.toml): finished with status 'done'
  Created wheel for compas_occt: filename=compas_occt-0.1.15-cp312-abi3-linux_x86_64.whl size=75888603 sha256=8818b7d507dede38af5ace5bb41bd85c500592cf47ca918ba84244d0a47af7e4
  Stored in directory: /tmp/pip-ephem-wheel-cache-m30ysrk5/wheels/f6/39/82/45e2c09277865359b68039be206ec16e174831886dd6d206c0
Successfully built compas_occt
Removed build tracker: '/tmp/pip-build-tracker-l4elfqcz'
    + /opt/python/cp39-cp39/bin/python -c 'import sys, json, glob; json.dump(glob.glob('"'"'/tmp/cibuildwheel/built_wheel/*.whl'"'"'), sys.stdout)'
    + rm -rf /tmp/cibuildwheel/repaired_wheel
    + mkdir -p /tmp/cibuildwheel/repaired_wheel

                                                            ✓ 675.17s
Repairing wheel...

    + sh -c 'auditwheel repair -w /tmp/cibuildwheel/repaired_wheel /tmp/cibuildwheel/built_wheel/compas_occt-0.1.15-cp312-abi3-linux_x86_64.whl'
INFO:auditwheel.main_repair:Repairing compas_occt-0.1.15-cp312-abi3-linux_x86_64.whl
Traceback (most recent call last):
  File "/usr/local/bin/auditwheel", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/_internal/pipx/venvs/auditwheel/lib/python3.12/site-packages/auditwheel/main.py", line 53, in main
    result: int | None = args.func(args, p)
                         ^^^^^^^^^^^^^^^^^^
  File "/opt/_internal/pipx/venvs/auditwheel/lib/python3.12/site-packages/auditwheel/main_repair.py", line 255, in execute
    out_wheel = repair_wheel(
                ^^^^^^^^^^^^^
  File "/opt/_internal/pipx/venvs/auditwheel/lib/python3.12/site-packages/auditwheel/repair.py", line 88, in repair_wheel
    raise ValueError(msg)
ValueError: Cannot repair wheel, because required library "libTKernel.so.7.9" could not be located

                                                             ✕ 22.30s
cibuildwheel: error: Command ['sh', '-c', 'auditwheel repair -w /tmp/cibuildwheel/repaired_wheel /tmp/cibuildwheel/built_wheel/compas_occt-0.1.15-cp312-abi3-linux_x86_64.whl'] failed with code 1. 

[petrasvestartas]

After the repair flag removal I get this, shouldn't the making wheel stop with the ABI version ?

cibuildwheel: error: 
Build failed because a wheel named compas_occt-0.1.15-cp312-abi3-linux_x86_64.whl was already generated in the current run.

[tool.cibuildwheel]
build-verbosity = 3
test-requires = ["numpy", "compas", "pytest", "build"]
test-command = "pip install numpy compas && pip list && pytest {project}/tests"
build-frontend = "pip"
manylinux-x86_64-image = "manylinux2014"
skip = ["*_i686", "*-musllinux_*", "*-win32", "pp*", "cp39-*", "cp310-*", "cp311-*"]
macos.environment.MACOSX_DEPLOYMENT_TARGET = "11.00"

[LecrisUT]

cibuildwheel: error: 
Build failed because a wheel named compas_occt-0.1.15-cp312-abi3-linux_x86_64.whl was already generated in the current run.

This is expected, you only build once and reuse it when testing for the other versions. Why it is reported as failure though, I am not sure.

ValueError: Cannot repair wheel, because required library "libTKernel.so.7.9" could not be located

Great, this is informative. I suspect you have this dependency installed somewhere else in the image. If you pass LD_LIBRARY_PATH to where it is installed, you can get that part to work as well.

[petrasvestartas]

I cleaned up the code, but I still have issues with abi, 3.12 builds correctly and on 3.13 it says that 3.12 is already present:

  *** Making wheel...
  *** Created compas_occt-0.1.15-cp312-abi3-linux_x86_64.whl
  Building wheel for compas_occt (pyproject.toml): finished with status 'done'
  Created wheel for compas_occt: filename=compas_occt-0.1.15-cp312-abi3-linux_x86_64.whl size=75888616 sha256=4fc5b0e13749d8ca038f28792e44941a23cdde9f8b1f1351179d9469d4a03921
  Stored in directory: /tmp/pip-ephem-wheel-cache-x3hzgba0/wheels/e9/ce/9e/9b39fcc56a822657045087060eac6c7fadd2b3a33df04ed633
Successfully built compas_occt
Removed build tracker: '/tmp/pip-build-tracker-vl_5uga4'
    + /opt/python/cp39-cp39/bin/python -c 'import sys, json, glob; json.dump(glob.glob('"'"'/tmp/cibuildwheel/built_wheel/*.whl'"'"'), sys.stdout)'
    + rm -rf /tmp/cibuildwheel/repaired_wheel
    + mkdir -p /tmp/cibuildwheel/repaired_wheel
    + mv /tmp/cibuildwheel/built_wheel/compas_occt-0.1.15-cp312-abi3-linux_x86_64.whl /tmp/cibuildwheel/repaired_wheel
    + /opt/python/cp39-cp39/bin/python -c 'import sys, json, glob; json.dump(glob.glob('"'"'/tmp/cibuildwheel/repaired_wheel/*.whl'"'"'), sys.stdout)'

                                                             ✕ 24.45s
cibuildwheel: error: 
Build failed because a wheel named compas_occt-0.1.15-cp312-abi3-linux_x86_64.whl was already generated in the current run.

If you expected another wheel to be generated, check your project configuration, or run
cibuildwheel with CIBW_BUILD_VERBOSITY=1 to view build logs.

Repository:
https://github.com/petrasvestartas/compas_occt/tree/b1c43df2e300808484569c53a07447cb81a41e6e

Log:
log.txt

[petrasvestartas]

I still get the error now without the missing libraries, as declared above.

[LecrisUT]

Not sure what's going on there. It shouldn't fail if a compatible wheel is already created, but I see in the log that it is failing when building for python 3.9 which it should discard and try to build again. I would wait for @henryiii to take a look and see if he know what's going on.