Daily GitHub Actions build fail on Fri, March 28th 2025

[github-actions]

See run for more details:
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/14126265619

[ryan-williams]

Attempt 1 error: bookworm gcc=13 v=1.16.0 (ARM)

#7 21.31         See also
#7 21.31       
#7 21.31           /tmp/pip-install-rg5phnfs/tiledbsoma_29069b6d20da4355aa0e418602f7ffc6/dist_links/build/externals/src/ep_tiledb-stamp/ep_tiledb-configure-*.log
#7 21.31       
#7 21.31       
#7 21.31       gmake[2]: *** [CMakeFiles/ep_tiledb.dir/build.make:92: externals/src/ep_tiledb-stamp/ep_tiledb-configure] Error 1
#7 21.31       gmake[1]: *** [CMakeFiles/Makefile2:95: CMakeFiles/ep_tiledb.dir/all] Error 2

Attempt 2 error: bookworm gcc=12 v=1.15.7 (ARM):

#7 29.64         See also
#7 29.64       
#7 29.64           /tmp/pip-install-5e21zvtz/tiledbsoma_a930e2666ea248439a6084bdb44fb891/dist_links/build/externals/src/ep_tiledb-stamp/ep_tiledb-configure-*.log
#7 29.64       
#7 29.64       
#7 29.64       gmake[2]: *** [CMakeFiles/ep_tiledb.dir/build.make:92: externals/src/ep_tiledb-stamp/ep_tiledb-configure] Error 1
#7 29.64       gmake[1]: *** [CMakeFiles/Makefile2:95: CMakeFiles/ep_tiledb.dir/all] Error 2

I'll attempt to repro locally.

As an aside, we should instrument the build to automatically cat …/build/externals/src/ep_tiledb-stamp/ep_tiledb-configure-*.log on failure (cf. Debugging tips in docker/README.md).

[ryan-williams]

Repro'd failure locally, ep_tiledb-configure-err.log points to ep_tiledb-build/vcpkg-bootstrap.log in the usual way, where I see:

# cat /tmp/pip-install-tjf6q7xb/tiledbsoma_3eea8803fb69441cbd0bcf7fda164ff9/dist_links/build/externals/src/ep_tiledb-build/vcpkg-bootstrap.log
Unable to determine a binary release of vcpkg; attempting to build from source.
Downloading vcpkg tool sources
Building vcpkg-tool...
-- The C compiler identification is GNU 13.3.0
-- The CXX compiler identification is GNU 13.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/local/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting the C++ compiler in use
-- Detecting the C++ compiler in use - gcc
-- Version: 10.1.0
-- Build type: Release
CMake Deprecation Warning at /tmp/pip-install-tjf6q7xb/tiledbsoma_3eea8803fb69441cbd0bcf7fda164ff9/dist_links/build/externals/src/ep_tiledb-build/_deps/vcpkg-src/buildtrees/_vcpkg/build/_deps/cmakerc-src/CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.10 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
  to tell CMake that the project requires at least <min> but has been updated
  to work with policies introduced by <max> or earlier.


CMake Error at /tmp/pip-install-tjf6q7xb/tiledbsoma_3eea8803fb69441cbd0bcf7fda164ff9/dist_links/build/externals/src/ep_tiledb-build/_deps/vcpkg-src/buildtrees/_vcpkg/build/_deps/cmakerc-src/CMakeRC.cmake:37 (cmake_minimum_required):
  Compatibility with CMake < 3.5 has been removed from CMake.

  Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
  to tell CMake that the project requires at least <min> but has been updated
  to work with policies introduced by <max> or earlier.

  Or, add -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to try configuring anyway.
Call Stack (most recent call first):
  /tmp/pip-install-tjf6q7xb/tiledbsoma_3eea8803fb69441cbd0bcf7fda164ff9/dist_links/build/externals/src/ep_tiledb-build/_deps/vcpkg-src/buildtrees/_vcpkg/build/_deps/cmakerc-src/CMakeLists.txt:7 (include)


-- Configuring incomplete, errors occurred!

Seems to not like Compatibility with CMake < 3.5, but I have 3.25 installed:

root@6d37b4927a02:/# cmake --version
cmake version 3.25.1

CMake suite maintained and supported by Kitware (kitware.com/cmake).

Maybe it's just the fact of saying we support a version that's now too old?

[jp-dark]

It looks like the issue is the build of vcpkg-tool not directly TileDB or TileDB-SOMA.

[ryan-williams]

Several factors are contributing to this failure, on ARM Linux:

1. tiledbsoma doesn't publish ARM Linux wheels
2. We are building vcpkg from source (instead of using binaries)
3. Our vcpkg source build is not working (since CMake 4 release Friday)

1. tiledbsoma doesn't publish ARM Linux wheels

See:

• Formalize linux-aarch64 support #2482
• [python/ci] Provisional/evaluation Linux + arm64 builds #2413
• @jdblischak’s experimental conda channel with manylinux ARM wheels

That means pip install tiledbsoma builds from source on ARM Linux, which was taking 30-40mins in CI:

GHA timings screenshot, ARM vs. x86

2. We are building vcpkg from source (instead of using binaries)

@teo-tsirpanis thinks we should be getting vcpkg on ARM Linux, not source-building in the first place.

3. Our vcpkg source build is not working

Since CMake 4 release (Friday 3/28), we get this failure when installing vcpkg from source on ARM Linux:

CMake Error at /tmp/pip-install-92o1f3x5/tiledbsoma_121d1aaba58e477bb371d1db8f019c01/dist_links/build/externals/src/ep_tiledb-build/_deps/vcpkg-src/buildtrees/_vcpkg/build/_deps/cmakerc-src/CMakeRC.cmake:37 (cmake_minimum_required):
  Compatibility with CMake < 3.5 has been removed from CMake.

That comes from [email protected] CMakeRC.cmake:37:

cmake_minimum_required(VERSION 3.3)

We should be getting more recent cmrc though, and not hitting this issue:

• TileDB pins vcpkg to 7aeffc91
• That should pin cmrc to 952ffdd / cmrc#48, which fixes the offending line

Instead, I'm observing vcpkg to install the latest cmrc release, 2.0.1 (Sept 2018, has the error), instead of 952ffdd (July 2023, has the error fix).

Other notes

• cmrc: bump cmake_minimum_required Homebrew/homebrew-core#217419 bumps cmrc to avoid the same cmrc error we're seeing.
• I tried installing CMake<4 on the system before pip install tiledbsoma (which source builds tiledbsoma → tiledb → vcpkg → cmrc), but vcpkg appears to install and use its own instance of CMake 4.

[ryan-williams]

The immediate failure here should be resolved as of #3891, will close this issue once the daily succeeds tmrw AM.

Follow-on debugging / root-causing can happen in #3890 / #2482.

[johnkerl]

Thanks @ryan-williams !!!

[teo-tsirpanis]

Today's builds succeeded: https://github.com/single-cell-data/TileDB-SOMA/actions/runs/14240027891

[ryan-williams]

Great, closing this now, am experimenting separately with publishing manylinux_aarch64 wheels (#3909), because those builds are still 25-40mins 😬