Skip to content

(build) fix some compile and conan problems#774

Open
yangzhg wants to merge 1 commit into
mainfrom
fix_compile
Open

(build) fix some compile and conan problems#774
yangzhg wants to merge 1 commit into
mainfrom
fix_compile

Conversation

@yangzhg

@yangzhg yangzhg commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Bolt builds in strict or isolated toolchain environments exposed several build and dependency-management issues:

  • Supported GCC versions no longer require a separate libstdc++fs, but Bolt still linked against -lstdc++fs, causing linker failures in environments where that library is unavailable.
  • Absolute source and build paths could be embedded in generated artifacts.
  • The Makefile did not automatically consume externally supplied Conan host and build profiles.
  • Several pinned Conan recipes had incompatible option names, CMake targets, or compiler settings.
  • GNU and Apache source downloads lacked preferred mirrors.
  • The Volcengine TOS C++ SDK did not have a local Conan recipe based on upstream release archives.

Issue Number: N/A

Type of Change

  • 🐛 Bug fix (non-breaking change which fixes an issue)
  • ✨ New feature (non-breaking change which adds functionality)
  • 🚀 Performance improvement (optimization)
  • ⚠️ Breaking change (fix or feature that would cause existing functionality to change)
  • 🔨 Refactoring (no logic changes)
  • 🔧 Build/CI or Infrastructure changes
  • 📝 Documentation only

Description

This PR improves Bolt build compatibility across different compiler, sysroot, and Conan environments:

  • Add configurable compiler prefix mapping for GCC and Clang to replace absolute source and build paths embedded in binaries, debug information, predefined macros, and coverage metadata.
  • Automatically consume externally supplied Conan host and build profiles while preserving the existing profile fallback.
  • Align conan graph info arguments with the actual conan install invocation.
  • Standardize the minimum supported GCC version as GCC 10 and remove obsolete stdc++fs linkage from Bolt and its test targets.
  • Align Arrow options with the pinned Conan recipe, including acero and the legacy hdfs_bridgs option.
  • Fix Arrow integration for RapidJSON and Snappy CMake target detection.
  • Build pkgconf with GNU99 extensions required by its source.
  • Fix Snappy target naming and Conan dependency discovery for Paimon and its Avro external project.
  • Prepend Kernel.org mirrors for the required GNU sources while retaining all original URLs as fallbacks.
  • Prepend Huawei Cloud mirrors for Arrow 15.0.1, ORC 2.0.0, and Thrift 0.17.0 while retaining their original Apache URLs.
  • Add a Conan recipe for all published Volcengine TOS C++ SDK releases from 2.6.0 through 2.6.28:
    • Source URLs and SHA-256 checksums are managed through conandata.yml.
    • Sources are downloaded from upstream GitHub tag archives.
    • CMake package metadata and a Conan test_package are included.

Validation performed:

  • Ran scripts/install-bolt-deps.sh with an isolated Conan home and confirmed that all Conan Center patches apply successfully.
  • Downloaded the added Apache mirror artifacts and verified their SHA-256 checksums.
  • Exported and fetched the TOS recipes for versions 2.6.0 and 2.6.28.

Performance Impact

  • No Impact: This change only affects the build system and dependency recipes and does not modify runtime critical paths.
  • Positive Impact: I have run benchmarks.
  • Negative Impact: Explained below (e.g., trade-off for correctness).

Release Note

Release Note:

Release Note:

  • Improved Bolt build compatibility across strict and isolated toolchain environments.
  • Added compiler path mapping to avoid embedding absolute source and build paths.
  • Added fallback mirrors for pinned GNU and Apache dependencies.
  • Added Conan recipes for Volcengine TOS C++ SDK releases from 2.6.0 through 2.6.28.

Checklist (For Author)

  • I have added/updated unit tests (ctest).
  • I have verified the code with local build (Release/Debug).
  • I have run clang-format / linters.
  • (Optional) I have run Sanitizers (ASAN/TSAN) locally for complex C++ changes.
  • No need to test or manual test.

Breaking Changes

  • No
  • Yes (Description: ...)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves Bolt’s build reliability in strict/isolated toolchain environments by cleaning up obsolete linkage, aligning Conan integration and pinned recipes, adding source mirrors, and introducing a local Conan recipe for the Volcengine TOS C++ SDK.

Changes:

  • Add compiler prefix/path mapping flags in CMake to avoid embedding absolute source/build paths in artifacts.
  • Improve Conan profile handling/graph info generation in the Makefile; align Arrow/Snappy/RapidJSON-related recipe patches and add Apache/GNU mirror fallbacks.
  • Add a full Conan recipe + test_package for tos_client across multiple upstream SDK versions.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
scripts/conan/recipes/tos_client/config.yml Registers multiple tos_client versions for the local recipe.
scripts/conan/recipes/tos_client/all/conanfile.py Adds the Conan recipe for building/packaging the Volcengine TOS C++ SDK.
scripts/conan/recipes/tos_client/all/conandata.yml Defines per-version source URLs and checksums for the TOS SDK archives.
scripts/conan/recipes/tos_client/all/test_package/conanfile.py Adds Conan test_package to validate the exported package via CMake.
scripts/conan/recipes/tos_client/all/test_package/CMakeLists.txt Builds and links a minimal consumer against tos_client::tos_client.
scripts/conan/recipes/tos_client/all/test_package/test_package.cpp Minimal runtime check that exercises a symbol from the SDK.
scripts/conan/recipes/paimon-cpp/all/patches/paimon-cpp-conan-deps.patch Adjusts Snappy discovery/targets and improves Conan/CMakeDeps integration for ExternalProject builds.
scripts/conan/patches/source-mirrors-apache.patch Prepends Huawei Cloud mirrors for selected Apache sources.
scripts/conan/patches/gnu-source-mirrors.patch Prepends kernel.org mirrors for required GNU sources.
scripts/conan/patches/pkgconf-2.2.0-fix-gnu99.patch Ensures pkgconf is built with GNU99 extensions as required by its sources.
scripts/conan/patches/arrow.15.0.1-csv-support.patch Aligns Arrow 15.0.1 options/targets and fixes Snappy/RapidJSON-related integration details.
Makefile Consumes externally supplied Conan host/build profiles and aligns conan graph info args with install.
conanfile.py Aligns Arrow options with the pinned Arrow recipe (e.g., acero, legacy hdfs_bridgs).
CMakeLists.txt Adds file-prefix-map support and removes obsolete stdc++fs linkage while standardizing GCC>=10.
bolt/substrait/tests/CMakeLists.txt Removes ${FILESYSTEM} linkage from tests.
bolt/jit/tests/CMakeLists.txt Removes ${FILESYSTEM} linkage from tests.
bolt/dwio/txt/tests/CMakeLists.txt Removes ${FILESYSTEM} linkage from tests.
bolt/dwio/parquet/tests/CMakeLists.txt Removes ${FILESYSTEM} linkage from tests.
bolt/dwio/dwrf/test/utils/CMakeLists.txt Removes legacy GCC<9 stdc++fs linkage workaround.
bolt/dwio/dwrf/test/CMakeLists.txt Removes ${FILESYSTEM} linkage from tests.
bolt/dwio/common/tests/utils/CMakeLists.txt Removes legacy GCC<9 stdc++fs linkage workaround.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread CMakeLists.txt
Comment thread scripts/conan/recipes/tos_client/all/conandata.yml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants