GH-44607: [C++][Dev] Update bundled Thrift, update mirrors to use CDN #44685

amoeba · 2024-11-09T01:34:09Z

Rationale for this change

Builds with bundled Thrift could fail because of permanently offline download mirror URLs. ASF now provides a CDN.

What changes are included in this PR?

Updates the pinned version for bundled Thrift from 0.16.0 to 0.20.0. We couldn't just update the URLs because the ASF CDN only hosts the last two versions of Thrift (0.20, 0.21.0). We didn't upgrade to Thrift 0.21.0 in this PR because it requires Boost 1.86.0 and we found incompatibilities with Boost 1.86.0 and cpp/build-support/trim-boost.sh which we use to publish a trimmed-down distribution of Boost for our build system to use.
Removes the set of URLs we were using to download Thrift in favor of just using the main CDN URL. We'll see how this works in practice.

Are these changes tested?

Yes. I configured a build with ...-DThrift_SOURCE=BUNDLED... and ran the full C++ test suite. I think further testing can be done in CI.

Are there any user-facing changes?

No.

Closes #44607

GitHub Issue: [C++] Thrift mirrors are outdated #44607

amoeba · 2024-11-09T01:40:10Z

@github-actions crossbow submit test-r-linux-as-cran

github-actions · 2024-11-09T01:42:26Z

Revision: 7173f31

Submitted crossbow builds: ursacomputing/crossbow @ actions-c90acec2ee

Task	Status
test-r-linux-as-cran

amoeba · 2024-11-09T01:43:12Z

@github-actions crossbow submit -g cpp

github-actions · 2024-11-09T01:45:39Z

Revision: 77d7c31

Submitted crossbow builds: ursacomputing/crossbow @ actions-da415e2d2d

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp-ubuntu-20.04-cuda-11.2.2
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-bundled
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

amoeba · 2024-11-09T16:04:07Z

@github-actions crossbow submit -g cpp

github-actions · 2024-11-09T16:06:42Z

Revision: 2bbe2aa

Submitted crossbow builds: ursacomputing/crossbow @ actions-4c1b909295

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp-ubuntu-20.04-cuda-11.2.2
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-bundled
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

amoeba · 2024-11-09T16:15:06Z

Windows MSVC is failing with :

D:\a\arrow\arrow\build\cpp\thrift_ep-prefix\src\thrift_ep\lib\cpp\src\thrift\TUuid.cpp(22): fatal error C1083: Cannot open include file: 'boost/uuid/string_generator.hpp': No such file or directory

rhub/ubuntu-gcc-12 is failing with:

/tmp/Rtmpa7sH8K/file73956317ada/thrift_ep-prefix/src/thrift_ep/lib/cpp/src/thrift/TUuid.cpp:22:10: fatal error: boost/uuid/string_generator.hpp: No such file or directory
22 | #include <boost/uuid/string_generator.hpp>
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

amoeba · 2024-11-09T16:51:42Z

The two failing builds were both bundling Boost and Thrift and I was able to reproduce the failure locally. From looking around, it looks like you have to upgrade to Boost 1.86.0 if you use Thrift 0.21.0. To test that hypothesis, I lowered Thrift down to 0.20.0 in 4886ad7 to test. I think we can get away with bumping Boost to 1.86.0 and Thrift to 0.21.0 in this PR so I'll try that once CI finishes.

amoeba · 2024-11-09T18:04:31Z

With Boost 1.86.0 and Thrift 0.21.0, the MSVC build fails with:

2024-11-09T17:43:07.4309013Z [550/608] Linking CXX shared library debug\parquet.dll
2024-11-09T17:43:07.4310713Z FAILED: debug/parquet.dll debug/parquet.lib 
2024-11-09T17:43:07.4323359Z C:\Windows\system32\cmd.exe /C "cd . && "C:\Program Files\CMake\bin\cmake.exe" -E vs_link_dll --intdir=src\parquet\CMakeFiles\parquet_shared.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\mt.exe --manifests  -- C:\PROGRA~2\MICROS~2\2019\ENTERP~1\VC\Tools\MSVC\1429~1.301\bin\Hostx64\x64\link.exe /nologo src\parquet\CMakeFiles\parquet_shared.dir\Unity\unity_5_cxx.cxx.obj src\parquet\CMakeFiles\parquet_shared.dir\Unity\unity_4_cxx.cxx.obj src\parquet\CMakeFiles\parquet_shared.dir\Unity\unity_3_cxx.cxx.obj src\parquet\CMakeFiles\parquet_shared.dir\Unity\unity_2_cxx.cxx.obj src\parquet\CMakeFiles\parquet_shared.dir\Unity\unity_1_cxx.cxx.obj src\parquet\CMakeFiles\parquet_shared.dir\Unity\unity_0_cxx.cxx.obj src\parquet\CMakeFiles\parquet_shared.dir\__\generated\parquet_constants.cpp.obj src\parquet\CMakeFiles\parquet_shared.dir\__\generated\parquet_types.cpp.obj src\parquet\CMakeFiles\parquet_shared.dir\level_comparison_avx2.cc.obj src\parquet\CMakeFiles\parquet_shared.dir\level_conversion_bmi2.cc.obj  /out:debug\parquet.dll /implib:debug\parquet.lib /pdb:debug\parquet.pdb /dll /version:1900.0 /machine:x64  /NODEFAULTLIB:LIBCMT /debug /INCREMENTAL -Wl,--version-script=D:/a/arrow/arrow/cpp/src/parquet/symbols.map  debug\arrow.lib  "C:\Program Files\OpenSSL\lib\VC\libcrypto64MDd.lib"  "C:\Program Files\OpenSSL\lib\VC\libssl64MDd.lib"  thrift_ep-install\bin\thriftmd.lib  "C:\Program Files\OpenSSL\lib\VC\libcrypto64MDd.lib"  kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && cd ."
2024-11-09T17:43:07.4347738Z LINK Pass 1: command "C:\PROGRA~2\MICROS~2\2019\ENTERP~1\VC\Tools\MSVC\1429~1.301\bin\Hostx64\x64\link.exe /nologo src\parquet\CMakeFiles\parquet_shared.dir\Unity\unity_5_cxx.cxx.obj src\parquet\CMakeFiles\parquet_shared.dir\Unity\unity_4_cxx.cxx.obj src\parquet\CMakeFiles\parquet_shared.dir\Unity\unity_3_cxx.cxx.obj src\parquet\CMakeFiles\parquet_shared.dir\Unity\unity_2_cxx.cxx.obj src\parquet\CMakeFiles\parquet_shared.dir\Unity\unity_1_cxx.cxx.obj src\parquet\CMakeFiles\parquet_shared.dir\Unity\unity_0_cxx.cxx.obj src\parquet\CMakeFiles\parquet_shared.dir\__\generated\parquet_constants.cpp.obj src\parquet\CMakeFiles\parquet_shared.dir\__\generated\parquet_types.cpp.obj src\parquet\CMakeFiles\parquet_shared.dir\level_comparison_avx2.cc.obj src\parquet\CMakeFiles\parquet_shared.dir\level_conversion_bmi2.cc.obj /out:debug\parquet.dll /implib:debug\parquet.lib /pdb:debug\parquet.pdb /dll /version:1900.0 /machine:x64 /NODEFAULTLIB:LIBCMT /debug /INCREMENTAL -Wl,--version-script=D:/a/arrow/arrow/cpp/src/parquet/symbols.map debug\arrow.lib C:\Program Files\OpenSSL\lib\VC\libcrypto64MDd.lib C:\Program Files\OpenSSL\lib\VC\libssl64MDd.lib thrift_ep-install\bin\thriftmd.lib C:\Program Files\OpenSSL\lib\VC\libcrypto64MDd.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFESTFILE:src\parquet\CMakeFiles\parquet_shared.dir/intermediate.manifest src\parquet\CMakeFiles\parquet_shared.dir/manifest.res" failed (exit code 1104) with the following output:
2024-11-09T17:43:07.4362016Z LINK : warning LNK4044: unrecognized option '/Wl,--version-script=D:/a/arrow/arrow/cpp/src/parquet/symbols.map'; ignored
2024-11-09T17:43:07.4363542Z LINK : fatal error LNK1104: cannot open file 'thrift_ep-install\bin\thriftmd.lib'

amoeba · 2024-11-09T18:20:51Z

It looks like our build may be conflicting with a change in Thrift that appeared in 0.21.0:

https://github.com/apache/thrift/blob/b65ec607db09f2efdb7397acde88fd53cfb97f6b/build/cmake/DefineInstallationPaths.cmake#L23-L29

# For MSVC builds, install shared libs to bin/, while keeping the install
# dir for static libs as lib/.
if(MSVC AND BUILD_SHARED_LIBS)
    set(LIB_INSTALL_DIR "bin${LIB_SUFFIX}" CACHE PATH "The library install dir (default: bin${LIB_SUFFIX})")
else()
    set(LIB_INSTALL_DIR "lib${LIB_SUFFIX}" CACHE PATH "The library install dir (default: lib${LIB_SUFFIX})")
endif()

amoeba · 2024-11-09T19:59:52Z

@github-actions crossbow submit -g cpp

github-actions · 2024-11-09T20:02:34Z

Revision: 88601bc

Submitted crossbow builds: ursacomputing/crossbow @ actions-46a48b959a

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp-ubuntu-20.04-cuda-11.2.2
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-bundled
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

amoeba · 2024-11-09T20:38:18Z

@assignUser this is ready for a review. Three of the four failed crossbow jobs looks like test flakiness but another set of eyes would be good. The test-debian-12-cpp-amd64 fails due to a related issue but for ORC. I think I could do a follow-up PR to update those URLs after this one.

Edit: On the last point, I think maybe we can hold off on updating the ORC mirror URLs. I looked closer and it looks like the runner VM had a more fundamental networking issue at the time so maybe we can just keep an eye on this.

assignUser · 2024-11-10T16:00:27Z

cpp/thirdparty/versions.txt

-  "ARROW_BOOST_URL boost-${ARROW_BOOST_BUILD_VERSION}.tar.gz https://apache.jfrog.io/artifactory/arrow/thirdparty/7.0.0/boost_${ARROW_BOOST_BUILD_VERSION//./_}.tar.gz"
+  "ARROW_BOOST_URL boost-${ARROW_BOOST_BUILD_VERSION}.tar.gz https://archives.boost.io/release/1.86.0/source/boost_${ARROW_BOOST_BUILD_VERSION//./_}.tar.gz"


We use a trimmed down version of boost (10mb vs 142mb) to speed up downloads. Could you create the new archive and upload it to the artifactory (as a committer you can do that) and use that version here?

Will do. I just tried to run the script and am getting failures on the bootstrap/b2 step. I'll take a look later today.

I didn't have any luck and it appears to be caused by Boost 1.86.0 releasing with a broken bcp. See comment and really the whole thread at boostorg/bcp#18 (comment).

I think we could probably update the trim-boost script to work around this but we could also just switch back to Thrift 0.20.0 and Boost 1.81.0 until Boost 1.87.0.

amoeba · 2024-11-11T18:22:44Z

Due to the bcp issue with boost 1.86.0, I bumped Thrift down to 0.20.0 and Boost back to 1.81.0. We can update to Thrift 0.21.0 once Boost 1.87.0 is out (and we can confirm the included bcp works).

amoeba · 2024-11-11T20:16:22Z

CI passed and PR description updated. @assignUser I think we can merge this without re-running all of the crossbow jobs and just be mindful during the 18.1.0 release process to look out for any issues.

assignUser

Agreed 👍 feel free to merge

assignUser · 2024-11-11T20:27:53Z

cpp/cmake_modules/ThirdpartyToolchain.cmake

@@ -1775,6 +1763,7 @@ macro(build_thrift)
      set(THRIFT_LIB_SUFFIX "md")
      list(APPEND THRIFT_CMAKE_ARGS "-DWITH_MT=OFF")
    endif()
+    # NOTE(amoeba): When you bump Thrift to >=0.21.0, change bin to lib


amoeba · 2024-11-11T20:29:48Z

Merged, thanks.

…#44685) ### Rationale for this change Builds with bundled Thrift could fail because of permanently offline download mirror URLs. ASF now provides [a CDN](https://dlcdn.apache.org/). ### What changes are included in this PR? 1. Updates the pinned version for bundled Thrift from 0.16.0 to 0.20.0. We couldn't just update the URLs because the ASF CDN only hosts the last two versions of Thrift (0.20, 0.21.0). We didn't upgrade to Thrift 0.21.0 in this PR because it requires Boost 1.86.0 and we found incompatibilities with Boost 1.86.0 and `cpp/build-support/trim-boost.sh` which we use to publish a trimmed-down distribution of Boost for our build system to use. 2. Removes the set of URLs we were using to download Thrift in favor of just using the main CDN URL. We'll see how this works in practice. ### Are these changes tested? Yes. I configured a build with `...-DThrift_SOURCE=BUNDLED...` and ran the full C++ test suite. I think further testing can be done in CI. ### Are there any user-facing changes? No. Closes #44607 * GitHub Issue: #44607 Authored-by: Bryce Mecum <[email protected]> Signed-off-by: Bryce Mecum <[email protected]>

conbench-apache-arrow · 2024-11-12T03:50:47Z

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit d7e982c.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 4 possible false positives for unstable benchmarks that are known to sometimes produce them.

amoeba added 2 commits November 8, 2024 17:19

Update bundled Thrift from 0.16.0 to 0.21.0

8a2f392

Replace mirrors for bundled Thrift to just Apache CDN

7173f31

amoeba requested a review from assignUser November 9, 2024 01:34

github-actions bot added the awaiting review Awaiting review label Nov 9, 2024

cmake-lint

77d7c31

github-actions bot added the Component: C++ label Nov 9, 2024

paste goof

2bbe2aa

Downgrade Thrift from 0.21.0 to 0.20.1

4886ad7

Bundle Boost 1.86.0 & Thrift 0.21.0

1c9dd88

Update Thrift static lib path prefix

88601bc

assignUser reviewed Nov 10, 2024

View reviewed changes

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Nov 10, 2024

amoeba added 2 commits November 11, 2024 10:18

Bump versions back down

00eb4c3

Update versions.txt

89e1421

Fix lib/bin thing

925a828

assignUser approved these changes Nov 11, 2024

View reviewed changes

amoeba merged commit d7e982c into apache:main Nov 11, 2024
40 checks passed

amoeba removed the awaiting committer review Awaiting committer review label Nov 11, 2024

amoeba mentioned this pull request Nov 11, 2024

[C++] Thrift mirrors are outdated #44607

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-44607: [C++][Dev] Update bundled Thrift, update mirrors to use CDN #44685

GH-44607: [C++][Dev] Update bundled Thrift, update mirrors to use CDN #44685

amoeba commented Nov 9, 2024 •

edited

Loading

amoeba commented Nov 9, 2024

github-actions bot commented Nov 9, 2024

amoeba commented Nov 9, 2024

github-actions bot commented Nov 9, 2024

amoeba commented Nov 9, 2024

github-actions bot commented Nov 9, 2024

amoeba commented Nov 9, 2024

amoeba commented Nov 9, 2024

amoeba commented Nov 9, 2024

amoeba commented Nov 9, 2024

amoeba commented Nov 9, 2024

github-actions bot commented Nov 9, 2024

amoeba commented Nov 9, 2024 •

edited

Loading

assignUser Nov 10, 2024

amoeba Nov 10, 2024

amoeba Nov 11, 2024

amoeba commented Nov 11, 2024

amoeba commented Nov 11, 2024

assignUser left a comment

assignUser Nov 11, 2024

amoeba commented Nov 11, 2024

conbench-apache-arrow bot commented Nov 12, 2024

		"ARROW_BOOST_URL boost-${ARROW_BOOST_BUILD_VERSION}.tar.gz https://apache.jfrog.io/artifactory/arrow/thirdparty/7.0.0/boost_${ARROW_BOOST_BUILD_VERSION//./_}.tar.gz"
		"ARROW_BOOST_URL boost-${ARROW_BOOST_BUILD_VERSION}.tar.gz https://archives.boost.io/release/1.86.0/source/boost_${ARROW_BOOST_BUILD_VERSION//./_}.tar.gz"

GH-44607: [C++][Dev] Update bundled Thrift, update mirrors to use CDN #44685

GH-44607: [C++][Dev] Update bundled Thrift, update mirrors to use CDN #44685

Conversation

amoeba commented Nov 9, 2024 • edited Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

amoeba commented Nov 9, 2024

github-actions bot commented Nov 9, 2024

amoeba commented Nov 9, 2024

github-actions bot commented Nov 9, 2024

amoeba commented Nov 9, 2024

github-actions bot commented Nov 9, 2024

amoeba commented Nov 9, 2024

amoeba commented Nov 9, 2024

amoeba commented Nov 9, 2024

amoeba commented Nov 9, 2024

amoeba commented Nov 9, 2024

github-actions bot commented Nov 9, 2024

amoeba commented Nov 9, 2024 • edited Loading

assignUser Nov 10, 2024

Choose a reason for hiding this comment

amoeba Nov 10, 2024

Choose a reason for hiding this comment

amoeba Nov 11, 2024

Choose a reason for hiding this comment

amoeba commented Nov 11, 2024

amoeba commented Nov 11, 2024

assignUser left a comment

Choose a reason for hiding this comment

assignUser Nov 11, 2024

Choose a reason for hiding this comment

amoeba commented Nov 11, 2024

conbench-apache-arrow bot commented Nov 12, 2024

amoeba commented Nov 9, 2024 •

edited

Loading

amoeba commented Nov 9, 2024 •

edited

Loading