[C++] Segmentation fault at `SparseUnionType` destructor when arrow.dll is unloaded via ODBC driver manager

[alinaliBQ]

Describe the bug, including details regarding any error messages, version, and platform.

We found that when arrow.dll is unloaded on Windows MinGW Msys and attached the back trace below. There is a segmentation fault occurring at SparseUnionType deallocation. However, our code does not use SparseUnionType so I am not sure what is the cause. Could it have to do with arrow.dll being loaded and unloaded too quickly? Because the lesser the tests that use arrow.dll, the more likely the segfault seems to occur.

Context: we develop a Flight SQL ODBC driver dll #47971 which has a dependency on the arrow.dll. The ODBC driver manager unloads the driver after use, and it also unloads all dependency dlls including arrow.dll. When arrow.dll is unloaded, we get segmentation fault from the SparseUnionType destructor.

Environment: Windows MinGW CLANG64 Msys Shell.

Stack Trace from gdb:

Thread 1 received signal SIGSEGV, Segmentation fault.
0x00007ff891c5cf31 in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
(gdb) bt
#0  0x00007ff891c5cf31 in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#1  0x00007ff89252d364 in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#2  0x00007ff89252be44 in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#3  0x00007ff8925ab11b in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#4  0x00007ff8925abe68 in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#5  0x00007ff8925a933b in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#6  0x00007ff8925a736d in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#7  0x00007ff8925a71fc in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#8  0x00007ff8925b0b30 in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#9  0x00007ff8925ab06a in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#10 0x00007ff8925aab8c in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#11 0x00007ff8925ab244 in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#12 0x00007ff8925ab28e in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#13 0x00007ff8925abf08 in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#14 0x00007ff8925abdd0 in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#15 0x00007ff8925ab1e3 in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#16 0x00007ff8925a3eb7 in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#17 0x00007ff8925abeb8 in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#18 0x00007ff8925abd70 in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#19 0x00007ff8925ab183 in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#20 0x00007ff89369d782 in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#21 0x00007ff8b51a2b99 in ucrtbased!_initterm_e () from C:\WINDOWS\SYSTEM32\ucrtbased.dll
#22 0x00007ff8b51a25c5 in ucrtbased!_initterm_e () from C:\WINDOWS\SYSTEM32\ucrtbased.dll
#23 0x00007ff8b51a2717 in ucrtbased!_initterm_e () from C:\WINDOWS\SYSTEM32\ucrtbased.dll
#24 0x00007ff8b51a2d44 in ucrtbased!_execute_onexit_table () from C:\WINDOWS\SYSTEM32\ucrtbased.dll
#25 0x00007ff89339aae9 in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#26 0x00007ff89339c2dd in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#27 0x00007ff89339c0b7 in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#28 0x00007ff89339c44f in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#29 0x00007ff89339c5c2 in arrow!??1SparseUnionType@arrow@@UEAA@XZ () from C:\Users\path-to\cpp\debug-build\debug\Debug\arrow.dll
#30 0x00007ff9c44ff89e in ntdll!RtlEncodeRemotePointer () from C:\WINDOWS\SYSTEM32\ntdll.dll
#31 0x00007ff9c43abcae in ntdll!RtlRaiseException () from C:\WINDOWS\SYSTEM32\ntdll.dll
#32 0x00007ff9c4437a48 in ntdll!LdrGetProcedureAddressForCaller () from C:\WINDOWS\SYSTEM32\ntdll.dll
#33 0x00007ff9c443679e in ntdll!LdrGetProcedureAddress () from C:\WINDOWS\SYSTEM32\ntdll.dll
#34 0x00007ff9c4436942 in ntdll!LdrGetProcedureAddress () from C:\WINDOWS\SYSTEM32\ntdll.dll
#35 0x00007ff9c44365e4 in ntdll!LdrGetProcedureAddress () from C:\WINDOWS\SYSTEM32\ntdll.dll
#36 0x00007ff9c43b4730 in ntdll!LdrUnloadDll () from C:\WINDOWS\SYSTEM32\ntdll.dll
#37 0x00007ff9c1b67f19 in KERNELBASE!FreeLibrary () from C:\WINDOWS\System32\KernelBase.dll
#38 0x00007ff9aca41e9e in ValidateErrorQueue () from C:\WINDOWS\SYSTEM32\odbc32.dll
#39 0x00007ff9aca5588c in ODBCGetTryWaitValue () from C:\WINDOWS\SYSTEM32\odbc32.dll
#40 0x00007ff9aca019a7 in SQLFreeHandle () from C:\WINDOWS\SYSTEM32\odbc32.dll
#41 0x00007ff659b92004 in arrow::flight::sql::odbc::ODBCRemoteTestBase::FreeEnvConnHandles (this=0x6abc10)
    at C:/path-to/cpp/src/arrow/flight/sql/odbc/tests/odbc_test_suite.cc:85
#42 0x00007ff659b91e98 in arrow::flight::sql::odbc::ODBCRemoteTestBase::Disconnect (this=0x6abc10)
    at C:/path-to/cpp/src/arrow/flight/sql/odbc/tests/odbc_test_suite.cc:76
#43 0x00007ff659b9462c in arrow::flight::sql::odbc::FlightSQLODBCMockTestBase::TearDown (this=0x6abc10)
    at C:/path-to/cpp/src/arrow/flight/sql/odbc/tests/odbc_test_suite.cc:417
#44 0x00007ff92287a475 in ?? () from C:\msys64\ucrt64\bin\libgtest.dll
#45 0x00007ff922860a1b in ?? () from C:\msys64\ucrt64\bin\libgtest.dll
#46 0x00007ff922860fec in ?? () from C:\msys64\ucrt64\bin\libgtest.dll
#47 0x00007ff922865ca1 in ?? () from C:\msys64\ucrt64\bin\libgtest.dll
#48 0x00007ff92286c610 in ?? () from C:\msys64\ucrt64\bin\libgtest.dll
#49 0x00007ff92286cf7a in ?? () from C:\msys64\ucrt64\bin\libgtest.dll
#50 0x00007ff9adc22304 in ?? () from C:\msys64\ucrt64\bin\libgmock_main.dll
#51 0x00007ff659b910c9 in __tmainCRTStartup () at D:/W/B/src/mingw-w64/mingw-w64-crt/crt/crtexe.c:236
--Type <RET> for more, q to quit, c to continue without paging--
#52 0x00007ff659b91416 in mainCRTStartup () at D:/W/B/src/mingw-w64/mingw-w64-crt/crt/crtexe.c:122

Component(s)

C++, FlightRPC

[alinaliBQ]

I recall that unloading is not supported by the Arrow dll files in general, and it might be the reason why we are seeing this issue.

[alinaliBQ]

ODBC tests are failing due to segmentation fault at teardown on MinGW Windows (CI run), while the tests pass on MSVC from my local testing. The good news is that MSVC is the main platform for ODBC builds so most end users will not be impacted by this segmentation fault error. (Though it is worth noting we currently don't have MSVC build in the CI as we do for C++ MinGW Windows.)

According to stack trace, the segmentation fault is from Arrow library DLLs. This is understandable as Arrow DLLs don't support unloading.

I propose to ignore the segmentation fault at ODBC test teardown in MinGW CI, so the CI isn't broken by the ODBC tests. Please let me know your thoughts (and feel free to tag more ppl to join this conversation, as I plan to bring this up in the Arrow community meeting this week as well). @lidavidm @kou

[kou]

How about disabling ODBC in MinGW CI?

[alinaliBQ]

I am okay with disabling ODBC in MinGW CI for the time being. Our team will test ODBC on MSVC locally. And when the tests stop breaking the CI, we can re-enable ODBC

[kou]

How about enabling ODBC in MSVC CI?

[alinaliBQ]

You're referring to the MSVC CI at cpp_windows.yml, right? Sure my team can look into enabling ODBC in MSVC

[kou]

Yes. If we focus on only MSVC support, it's better that we use MSVC CI job instead of MinGW CI job.