Segmentation fault (core dumped) while running Matrix Multiplication

[AjinkyaBankar]

I am running a simple matrix multiplication code on Ubuntu 20.04 with CUDA 11. It keeps showing the following line for several times:

GPGPU-Sim PTX: CUDA API function "unsigned int __cudaPushCallConfiguration(dim3, dim3, size_t, CUstream_st*)" has been called.

Then it says:

Segmentation fault (core dumped)

Kindly help to solve the problem. Thank you.

[mkhairy]

@tgrogers I think we have fixed that, right?

[PsFreedom]

I have this problem as well in Ubuntu 20.04.
Then I tried several GPGPU-Sim, GCC, and CUDA versions but the problem still persists.

Decided to change back to Ubuntu 18.04, it works now, the problem disappeared. Tried on CUDA 9.1 and the latest 11.3. So, you can just change to Ubuntu 18.04.

[electricSamarth]

I am facing the exact same issue, I am running Ubuntu 20.04.2 LTS, I don't wanna shift to 18.04? Any sort of help will be useful.

[mattsinc]

Have any of you tried running the debug version of GPGPU-Sim with this and running the application in gdb? Curious what the backtrace shows.

[electricSamarth]

Hi @mattsinc. As it turns out, it works if I use the debug version instead of the release version. I did this after reading your comment, thinking maybe before running gdb, I should just try the debug version.
Thanks for your time :)

[mattsinc]

Well I'm glad that is working, but it does make it difficult to debug. If you run release with gdb, does it give you any information about where the failure is happening?

[electricSamarth]

Hi @mattsinc , sorry for the late reply but here is a screenshot of the release environment running the program in GDB. The documentation says that the debug version is slower. So I need to get the release version working.

Does this have anything with me running Ubuntu 20.04 ?

[mattsinc]

Not sure about Ubuntu 20, what does the backtrace show?

Matt

[electricSamarth]

This is how the backtrace looks

[mattsinc]

Interesting. Just speculating, my guess is the version of gcc in Ubuntu 20 is using the __my_func__ variable differently than prior versions have assumed. I don't know exactly why that might be, but I will note that the line where your failure is coming from:

gpgpu-sim_distribution/libcuda/cuda_runtime_api.cc

Line 3596 in 5e8e849

announce_call(__my_func__);

is only be used when you set the debug level is >= 3. I'm assuming you are setting PTX_SIM_DEBUG to 3 or higher on the command line? If you don't need that level of debug information, you could reduce the debug level and this problem in theory should go away. Alternatively, you could just change the debug level for the prints in cuda_runtime_api if you want debug information elsewhere but don't want these specific prints.

Matt

[electricSamarth]

I have explicitly not set the environment variable you are talking about. I did try setting it to 2, I am still getting the same error and same gdb output. Also, here is the end of that backtrace

[mattsinc]

Interesting -- if you didn't set the flag to anything, then the code from the first backtrace shouldn't have been triggered :)

Not sure what else to try, hopefully one of the main maintainers can chime in. @mkhairy above you said this was fixed. Is there a fix to push in for it?

Matt

[electricSamarth]

What confuses me the most is that when I run the set_environment to debug it works perfectly (it gives me the output as it should) but I get a SEGSEV when I source set_environment default i.e. release
Also, @mattsinc thanks for your time and advice :)

[electricSamarth]

@AjinkyaBankar @tgrogers @mkhairy can you guys help?

[electricSamarth]

@mattsinc I used gcc 7 and it worked, the release version works

[mkhairy]

Hi, we have fixed a weird memory stomp recently in gpgpu-sim. Could you please pull this fix below and see if this fixes the error in gcc 7? Maybe, changing the gcc has altered the data structure allocation, and thus triggered the stomp. accel-sim@c526262

…

On Mon, May 17, 2021 at 7:01 AM Samarth Mathur ***@***.***> wrote: @mattsinc <https://github.com/mattsinc> I used gcc 7 and it worked, the release version works — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#211 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPVJ4PAUW5JPILGHKXCUALTODZQJANCNFSM4WZ5WKWA> .

-- Thanks! -Mahmoud

[jiashenC]

I have tried this fix, but the problem still exists. Following other posts, the debug version works fine but the release version fails. It first outputs many lines of

GPGPU-Sim PTX: CUDA API function "unsigned int __cudaPushCallConfiguration(dim3, dim3, size_t, CUstream_st*)" has been called.

and then seg faults.

I am on Ubuntu 16.04 LTS with gcc 8.

[electricSamarth]

Try gcc7, that worked for me

…

On Thu, 13 Jan, 2022, 8:30 PM Jiashen Cao, ***@***.***> wrote: I have tried this fix, but the problem still exists. Following other posts, the debug version works fine but the release version fails. It first outputs many lines of GPGPU-Sim PTX: CUDA API function "unsigned int __cudaPushCallConfiguration(dim3, dim3, size_t, CUstream_st*)" has been called. and then seg faults. I am on Ubuntu 16.04 LTS with gcc 8. — Reply to this email directly, view it on GitHub <#211 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKT55UJBOLSG4SMROIIN7QTUV3SKVANCNFSM4WZ5WKWA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[jiashenC]

I am still getting the same error with gcc-7.5.0