[C++] S3FS can sometimes send sigpipe signals to the application

[mderoy]

Our application uses many instances running arrow in parallel on linux. Occasionally we will get a SIGPIPE signal from our calls to the s3 filesystem (for both reads and writes). We believe these are coming from the S3 api thats being embedded by arrow. Since arrow is multithreaded, handling these signals around the scope of our arrow api calls is ineffective.

AWS exposes an option to install their own handler
in SDKOptions.HttpOptions.installSigPipeHandler
https://github.com/aws/aws-sdk-cpp/blob/a154acd5893e2b2c913844e312648149d12a12d1/src/aws-cpp-sdk-core/include/aws/core/Aws.h#L103
but arrow has no way to set this. S3GlobalOptions only exposes the s3 log level and number of event loop threads, and the rest of the options are kept private within arrow.

Arrow needs a way to pass this (and probably other AWS options). So I'm opening this bug report.

if the community is okay with adding the installSigPipeHandler option to S3GlobalOptions I'd be happy to contribute.... though I wonder why we don't open the floodgates to let the user set s3 options directly at will?

version: 13.0.0
platform: rhel 8

Component(s)

C++

[assignUser]

Hey, thanks for your report!

if the community is okay with adding the installSigPipeHandler option to S3GlobalOptions I'd be happy to contribute.... though I wonder why we don't open the floodgates to let the user set s3 options directly at will?

I looked at the original PR and I don't think there was a motivating reason outside of scope in reducing the number of exposed global options. Maybe @pitrou can comment? A PR would be appreciated either way but I think you can go ahead and open the flood gates :D

[pitrou]

I'm curious, why don't you simply ignore SIGPIPE signals by yourself? It should be as easy as:

signal(SIGPIPE, SIG_IGN);

IMHO, it doesn't seem useful to expose an option for something you can trivially do yourself. Except perhaps to document the existence of this issue.

[assignUser]

IMHO, it doesn't seem useful to expose an option for something you can trivially do yourself.

Is there a reason not to expose the options? I doubt that they change constantly and create a lot of maintenance work?

[pitrou]

Is there a reason not to expose the options? I doubt that they change constantly and create a lot of maintenance work?

Do you mean expose additional individual options in S3GlobalOptions, or let people pass a SDKOptions directly?

[assignUser]

hm I guess allowing SDKOptions to be passed would be the solution that requires minimal maintenance and provides maximum features to users? Or do we gain anything from wrapping it in our own abstraction with limited features? It should work with forward declaration similar to how we do it with AWSCredentialsProvider without the need to expose aws.h?

[pitrou]

hm I guess allowing SDKOptions to be passed would be the solution that requires minimal maintenance and provides maximum features to users? Or do we gain anything from wrapping it in our own abstraction with limited features?

Well, the problem is that allowing SDKOptions needs the user to install the right version of the SDK headers, and include them in their project.

So, if there are any important options there, we should provide an abstraction around them.