How do we make sure that our current system dependencies are supported by velox and if we can upgrade our os versions?

[jaystarshot]

Our environment
gcc (GCC) 12.3.0
Our os Debian GNU/Linux 10 (buster)

We noticed that there were a lot of crashes in prestissimo in our setup. Some of these couldn't be explained. Also, they were only happening in release build and not debug build. While fixing one of them (stack attached below) I noticed that the issue was in libunwind system library.

E20240413 12:02:06.788125   237 PrestoMain.cpp:40] *** SIGSEGV (@0x293) received by PID 55 (TID 0x7e1c7ffff700) from PID 659; stack trace: ***
E20240413 12:02:06.789721   237 PrestoMain.cpp:40]     @     0x7fb499e2c21e google::(anonymous namespace)::FailureSignalHandler()
E20240413 12:02:06.791419   237 PrestoMain.cpp:40]     @     0x7fb49a0ea730 (unknown)
E20240413 12:02:06.792943   237 PrestoMain.cpp:40]     @     0x7fb499bcd23f (unknown)
E20240413 12:02:06.794430   237 PrestoMain.cpp:40]     @     0x7fb499bcdb47 (unknown)
E20240413 12:02:06.795902   237 PrestoMain.cpp:40]     @     0x7fb499bcdc8a _ULx86_64_resume
E20240413 12:02:06.797418   237 PrestoMain.cpp:40]     @     0x7fb499bcce06 __libunwind_Unwind_Resume
E20240413 12:02:06.797569   237 PrestoMain.cpp:40]     @           0x4af18d _ZN8facebook5velox7parquet10PageReader10seekToPageEl.cold

Our os had libunwind version 1.2.1 and on upgrading this library 1.8.1 we fixed 99% of our crashes.https://github.com/libunwind/libunwind/releases. Now, we need to ensure that all our system dependencies are compatible with Velox. Is there a way to audit and document all the dependencies required by Velox, for example document them, as tested in Meta's production environment?

[jaystarshot]

cc: @mbasmanova @pedroerp

[pedroerp]

@jaystarshot we have the list of dependencies documented here, but admittedly it may still be a work in progress. As far as I remember libunwind was an optional folly dependency, but if not, we should also add if to the list with the respective minimum version.

We can then enforce that a compatible version is found by tweaking the CMake files. There are a few examples in the codebase of how minimum dependency versions can be enforced. Could you help add anything missing to that list?

Cc: @assignUser @majetideepak @kgpai

[assignUser]

Yes as you say libunwind is optional and folly builds without it. we don't look for it at all. I don't know what folly does with it and if we would lose any useful functionality by omitting it. I am guessing it's for better stack traces or something?

We could remove it from the install script if it's used for something we don't rely on. We could look for it when building folly and produce an error if to low a version is found (as it clearly produces segfaults?

[pedroerp]

Yes, it's related to unwinding the stack to capture stack traces. In theory it's optional, but I'm not sure we have tested things without it.

[jaystarshot]

Makes sense, could we also introduce an additional column (perhaps called 'tested version' lets say from meta's production) that might differ from the 'minimum version' column and could contain values when the 'minimum version' column is set to default?

[majetideepak]

I think this is more of a Folly library issue than Velox. Folly should mandate the right version?

[assignUser]

1.8.0 was released this year and as mentioned above it seems lot's of distros are behind so it's likely that folly hasn't gotten a report about it. But in general I would agree.

[kgpai]

@assignUser Would using conda as a dependency manager solve issues like this ? We could freeze and list out all stable transitive dependencies with an approach like that.

[jaystarshot]

Not sure if 1.8.1 is needed actually, 1.2.1 had that issue and we directly updated to 1.8.1

[assignUser]

Oh 1.2.1 is from 2017 so that changes things a bit. In that case I am against adding this detection unless it's demonstrated that this bug comes from a recent'ish widely used version. Again Buster is EOL.

@kgpai Yes it could but pinning dependencies to old version can lead to resolution hell really quickly (pinning minimum versions should be ok though).

[jaystarshot]

Our concern is regarding version mismatch in all other system dependencies too because there can be such hidden bugs and they are difficult to find.