Skip to content

AddressSanitizer: unknown-crash in umfMemoryTrackerAdd() #1233

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ldorau opened this issue Apr 2, 2025 · 2 comments · May be fixed by #1270
Open

AddressSanitizer: unknown-crash in umfMemoryTrackerAdd() #1233

ldorau opened this issue Apr 2, 2025 · 2 comments · May be fixed by #1270
Assignees
Labels
bug Something isn't working high priority
Milestone

Comments

@ldorau
Copy link
Contributor

ldorau commented Apr 2, 2025

ERROR: AddressSanitizer: unknown-crash on address :
https://github.com/ldorau/unified-memory-framework/actions/runs/14172971689/job/39700871936

[ RUN      ] disjointPoolTests/umfPoolTest.multiThreadedpow2AlignedAlloc/0
=================================================================
==1159214==ERROR: AddressSanitizer: unknown-crash on address 0x7f3e9feac248 at pc 0x5621d13ce718 bp 0x7f3e9a9fea00 sp 0x7f3e9a9fe9f0
READ of size 8 at 0x7f3e9feac248 thread T9
    #0 0x5621d13ce717 in utils_atomic_load_acquire_u64 /home/runner/work/unified-memory-framework/unified-memory-framework/src/utils/utils_concurrency.h:166
    #1 0x5621d13cfe1c in umfMemoryTrackerAdd /home/runner/work/unified-memory-framework/unified-memory-framework/src/provider/provider_tracking.c:202
    #2 0x5621d13d29b0 in trackingAlloc /home/runner/work/unified-memory-framework/unified-memory-framework/src/provider/provider_tracking.c:481
    #3 0x5621d13caa1e in umfMemoryProviderAlloc /home/runner/work/unified-memory-framework/unified-memory-framework/src/memory_provider.c:245
    #4 0x5621d13eaf23 in disjoint_pool_aligned_malloc /home/runner/work/unified-memory-framework/unified-memory-framework/src/pool/pool_disjoint.c:745
    #5 0x5621d13c7ea2 in umfPoolAlignedMalloc /home/runner/work/unified-memory-framework/unified-memory-framework/src/memory_pool.c:195
    #6 0x5621d12b6d99 in pow2AlignedAllocHelper(umf_memory_pool_t*) /home/runner/work/unified-memory-framework/unified-memory-framework/test/poolFixtures.hpp:190
    #7 0x5621d12b8cf1 in operator() /home/runner/work/unified-memory-framework/unified-memory-framework/test/poolFixtures.hpp:250
    #8 0x5621d12f1c43 in __invoke_impl<void, umfPoolTest_multiThreadedpow2AlignedAlloc_Test::TestBody()::<lambda(umf_memory_pool_handle_t)>, umf_memory_pool_t*> /usr/include/c++/11/bits/invoke.h:61
    #9 0x5621d12f11d0 in __invoke<umfPoolTest_multiThreadedpow2AlignedAlloc_Test::TestBody()::<lambda(umf_memory_pool_handle_t)>, umf_memory_pool_t*> /usr/include/c++/11/bits/invoke.h:96
    #10 0x5621d12f0270 in _M_invoke<0, 1> /usr/include/c++/11/bits/std_thread.h:259
    #11 0x5621d12ef8f1 in operator() /usr/include/c++/11/bits/std_thread.h:266
    #12 0x5621d12ef74e in _M_run /usr/include/c++/11/bits/std_thread.h:211
    #13 0x7f3ea00dc252  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc252)
    #14 0x7f3e9f494ac2  (/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
    #15 0x7f3e9f52684f  (/lib/x86_64-linux-gnu/libc.so.6+0x12684f)
2025-03-31T17:41:35.3852388Z
Address 0x7f3e9feac248 is a wild pointer.
SUMMARY: AddressSanitizer: unknown-crash /home/runner/work/unified-memory-framework/unified-memory-framework/src/utils/utils_concurrency.h:166 in utils_atomic_load_acquire_u64
Shadow bytes around the buggy address:
  0x0fe853fcd7f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe853fcd800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f7
  0x0fe853fcd810: f7 f7 f7 f7 f7 00 00 00 00 00 00 00 00 00 00 00
  0x0fe853fcd820: 00 00 00 00 00 00 00 00 00 00 f7 f7 f7 f7 f7 f7
  0x0fe853fcd830: 00 00 00 00 00 00 f7 f7 f7 f7 f7 f7 00 00 00 f7
=>0x0fe853fcd840: f7 f7 00 00 00 f7 f7 f7 f7[f7]f7 f7 f7 f7 f7 f7
  0x0fe853fcd850: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0fe853fcd860: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0fe853fcd870: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0fe853fcd880: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0fe853fcd890: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7

or ERROR: AddressSanitizer: use-after-poison:
https://github.com/ldorau/unified-memory-framework/actions/runs/14173082582/job/39701246433

[ RUN      ] mallocPoolTest/umfPoolTest.multiThreadedMallocFree/1
=================================================================
==1562263==ERROR: AddressSanitizer: use-after-poison on address 0x7fc68a62e218 at pc 0x5587ac5a19b8 bp 0x7fc67e7feb30 sp 0x7fc67e7feb20
READ of size 8 at 0x7fc68a62e218 thread T19
    #0 0x5587ac5a19b7 in utils_atomic_load_acquire_u64 /home/runner/work/unified-memory-framework/unified-memory-framework/src/utils/utils_concurrency.h:166
    #1 0x5587ac5a30bc in umfMemoryTrackerAdd /home/runner/work/unified-memory-framework/unified-memory-framework/src/provider/provider_tracking.c:202
    #2 0x5587ac5a5c50 in trackingAlloc /home/runner/work/unified-memory-framework/unified-memory-framework/src/provider/provider_tracking.c:481
    #3 0x5587ac59dcbe in umfMemoryProviderAlloc /home/runner/work/unified-memory-framework/unified-memory-framework/src/memory_provider.c:245
    #4 0x5587ac5c0f10 in proxy_aligned_malloc /home/runner/work/unified-memory-framework/unified-memory-framework/src/pool/pool_proxy.c:51
    #5 0x5587ac5c1036 in proxy_malloc /home/runner/work/unified-memory-framework/unified-memory-framework/src/pool/pool_proxy.c:64
    #6 0x5587ac59b010 in umfPoolMalloc /home/runner/work/unified-memory-framework/unified-memory-framework/src/memory_pool.c:189
    #7 0x5587ac409e51 in operator() /home/runner/work/unified-memory-framework/unified-memory-framework/test/poolFixtures.hpp:223
    #8 0x5587ac453b77 in __invoke_impl<void, umfPoolTest_multiThreadedMallocFree_Test::TestBody()::<lambda(size_t, umf_memory_pool_handle_t)>, long unsigned int, umf_memory_pool_t*> /usr/include/c++/11/bits/invoke.h:61
    #9 0x5587ac45309c in __invoke<umfPoolTest_multiThreadedMallocFree_Test::TestBody()::<lambda(size_t, umf_memory_pool_handle_t)>, long unsigned int, umf_memory_pool_t*> /usr/include/c++/11/bits/invoke.h:96
    #10 0x5587ac451e4d in _M_invoke<0, 1, 2> /usr/include/c++/11/bits/std_thread.h:259
    #11 0x5587ac4511ed in operator() /usr/include/c++/11/bits/std_thread.h:266
    #12 0x5587ac450fc2 in _M_run /usr/include/c++/11/bits/std_thread.h:211
    #13 0x7fc68a4dc252  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc252)
    #14 0x7fc689894ac2  (/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
    #15 0x7fc68992684f  (/lib/x86_64-linux-gnu/libc.so.6+0x12684f)
2025-03-31T19:11:14.9259311Z
Address 0x7fc68a62e218 is a wild pointer.
SUMMARY: AddressSanitizer: use-after-poison /home/runner/work/unified-memory-framework/unified-memory-framework/src/utils/utils_concurrency.h:166 in utils_atomic_load_acquire_u64
Shadow bytes around the buggy address:
  0x0ff9514bdbf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ff9514bdc00: 00 00 00 00 00 00 00 00 00 00 00 00 f7 f7 f7 f7
  0x0ff9514bdc10: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 00 00 00 00 00
  0x0ff9514bdc20: 00 f7 f7 f7 f7 f7 f7 f7 f7 f7 00 00 00 f7 f7 f7
  0x0ff9514bdc30: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
=>0x0ff9514bdc40: f7 f7 f7[f7]f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0ff9514bdc50: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0ff9514bdc60: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0ff9514bdc70: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0ff9514bdc80: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0ff9514bdc90: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7

Environment Information

  • UMF version (hash commit or a tag): commit e3d2929
  • OS(es) version(s): Linux

Please provide a reproduction of the bug:

AddressSanitizer: unknown-crash:
https://github.com/ldorau/unified-memory-framework/actions/runs/14172971689/job/39700871936
https://github.com/ldorau/unified-memory-framework/actions/runs/14173082582/job/39701246468
https://github.com/ldorau/unified-memory-framework/actions/runs/14172971689/job/39700871892
https://github.com/ldorau/unified-memory-framework/actions/runs/14172971689/job/39700871926

or AddressSanitizer: use-after-poison:

https://github.com/ldorau/unified-memory-framework/actions/runs/14173082582/job/39701246433
https://github.com/ldorau/unified-memory-framework/actions/runs/14173082582/job/39701246427

How often bug is revealed:

very rare (when tests are looped)

Details

Additional information about Priority and Help Requested:

Requested priority: Medium

@ldorau ldorau added the bug Something isn't working label Apr 2, 2025
@ldorau ldorau changed the title AddressSanitizer: unknown-crash in umfMemoryTrackerAdd (in utils_atomic_load_acquire_u64()) AddressSanitizer: unknown-crash in umfMemoryTrackerAdd() Apr 2, 2025
@bratpiorka bratpiorka added this to the v0.12.x milestone Apr 8, 2025
@ldorau ldorau self-assigned this Apr 15, 2025
ldorau added a commit to ldorau/unified-memory-framework that referenced this issue Apr 15, 2025
Postpone freeing a tracker entry until it is really removed from tracker.

Ref: oneapi-src#1233

Signed-off-by: Lukasz Dorau <[email protected]>
@ldorau ldorau linked a pull request Apr 15, 2025 that will close this issue
3 tasks
ldorau added a commit to ldorau/unified-memory-framework that referenced this issue Apr 15, 2025
Postpone freeing a tracker entry until it is really removed from tracker.

Ref: oneapi-src#1233

Signed-off-by: Lukasz Dorau <[email protected]>
ldorau added a commit to ldorau/unified-memory-framework that referenced this issue Apr 17, 2025
Postpone freeing a tracker entry until it is really removed from tracker.

Ref: oneapi-src#1233

Signed-off-by: Lukasz Dorau <[email protected]>
ldorau added a commit to ldorau/unified-memory-framework that referenced this issue Apr 17, 2025
Postpone freeing a tracker entry until it is really removed from tracker.

Ref: oneapi-src#1233

Signed-off-by: Lukasz Dorau <[email protected]>
@ldorau
Copy link
Contributor Author

ldorau commented May 6, 2025

This is the real scenario (see also: #1270 (comment)):
a) thread T2 allocates memory region R (for example: 0x100-0x300 - address 0x100 and size 0x200) and adds it to the tracker (this is the only entry in the tracker yet) and gets preempted ...
b) thread T1 allocates memory of size 0x500 and receives pointer P (address 0x400, size 0x500 - range 0x400-0x900) and tries to add it to the tracker, so it calls umfMemoryTrackerAdd(), so critnib_find(FIND_LE) finds and returns the only region R existing in the tracker R: 0x100-0x300 and T1 gets preempted ...
c) thread T2 removes the region R from the tracker, frees its memory and T2 gets preempted ...
d) thread T1 tries to read a size of the already freed region R and it crashes ...

@ldorau
Copy link
Contributor Author

ldorau commented May 8, 2025

Fix for this issue: #1270

ldorau added a commit to ldorau/unified-memory-framework that referenced this issue May 9, 2025
Add a reference counter and critnib_release() function.

The following 4 functions:
- critnib_remove(),
- critnib_get(),
- critnib_find_le() and
- critnib_find()

returns a reference (void *ref) to the key,
that MUST be released by calling critnib_release()
when the returned value is no longer used
and can be freed using the cb_free_leaf() callback.

Fixes: oneapi-src#1233

Signed-off-by: Lukasz Dorau <[email protected]>
ldorau added a commit to ldorau/unified-memory-framework that referenced this issue May 9, 2025
Add a reference counter and critnib_release() function.

The following 4 functions:
- critnib_remove(),
- critnib_get(),
- critnib_find_le() and
- critnib_find()

returns a reference (void *ref) to the key,
that MUST be released by calling critnib_release()
when the returned value is no longer used
and can be freed using the cb_free_leaf() callback.

Fixes: oneapi-src#1233

Signed-off-by: Lukasz Dorau <[email protected]>
ldorau added a commit to ldorau/unified-memory-framework that referenced this issue May 12, 2025
Add a reference counter and critnib_release() function.

The following 4 functions:
- critnib_remove(),
- critnib_get(),
- critnib_find_le() and
- critnib_find()

returns a reference (void *ref) to the key,
that MUST be released by calling critnib_release()
when the returned value is no longer used
and can be freed using the cb_free_leaf() callback.

Fixes: oneapi-src#1233

Signed-off-by: Lukasz Dorau <[email protected]>
ldorau added a commit to ldorau/unified-memory-framework that referenced this issue May 12, 2025
Add a reference counter and critnib_release() function.

The following 4 functions:
- critnib_remove(),
- critnib_get(),
- critnib_find_le() and
- critnib_find()

returns a reference (void *ref) to the key,
that MUST be released by calling critnib_release()
when the returned value is no longer used
and can be freed using the cb_free_leaf() callback.

Fixes: oneapi-src#1233

Signed-off-by: Lukasz Dorau <[email protected]>
ldorau added a commit to ldorau/unified-memory-framework that referenced this issue May 13, 2025
Add a reference counter and critnib_release() function.

The following 4 functions:
- critnib_remove(),
- critnib_get(),
- critnib_find_le() and
- critnib_find()

return a reference (void *ref) to the returned value,
that MUST be released by calling critnib_release()
when it is no longer used and can be freed
using the cb_free_leaf() callback.

Fixes: oneapi-src#1233

Signed-off-by: Lukasz Dorau <[email protected]>
ldorau added a commit to ldorau/unified-memory-framework that referenced this issue May 13, 2025
Add a reference counter and critnib_release() function.

The following 4 functions:
- critnib_remove(),
- critnib_get(),
- critnib_find_le() and
- critnib_find()

return a reference (void *ref) to the returned value,
that MUST be released by calling critnib_release()
when it is no longer used and can be freed
using the cb_free_leaf() callback.

Fixes: oneapi-src#1233

Signed-off-by: Lukasz Dorau <[email protected]>
ldorau added a commit to ldorau/unified-memory-framework that referenced this issue May 13, 2025
Add a reference counter and critnib_release() function.

The following 4 functions:
- critnib_remove(),
- critnib_get(),
- critnib_find_le() and
- critnib_find()

return a reference (void *ref) to the returned value,
that MUST be released by calling critnib_release()
when it is no longer used and can be freed
using the cb_free_leaf() callback.

Fixes: oneapi-src#1233

Signed-off-by: Lukasz Dorau <[email protected]>
ldorau added a commit to ldorau/unified-memory-framework that referenced this issue May 13, 2025
Add a reference counter and critnib_release() function.

The following 4 functions:
- critnib_remove(),
- critnib_get(),
- critnib_find_le() and
- critnib_find()

return a reference (void *ref) to the returned value,
that MUST be released by calling critnib_release()
when it is no longer used and can be freed
using the cb_free_leaf() callback.

Fixes: oneapi-src#1233

Signed-off-by: Lukasz Dorau <[email protected]>
ldorau added a commit to ldorau/unified-memory-framework that referenced this issue May 13, 2025
Add a reference counter and critnib_release() function.

The following 4 functions:
- critnib_remove(),
- critnib_get(),
- critnib_find_le() and
- critnib_find()

return a reference (void *ref) to the returned value,
that MUST be released by calling critnib_release()
when it is no longer used and can be freed
using the cb_free_leaf() callback.

Fixes: oneapi-src#1233

Signed-off-by: Lukasz Dorau <[email protected]>
ldorau added a commit to ldorau/unified-memory-framework that referenced this issue May 13, 2025
Add a reference counter and critnib_release() function.

The following 4 functions:
- critnib_remove(),
- critnib_get(),
- critnib_find_le() and
- critnib_find()

return a reference (void *ref) to the returned value,
that MUST be released by calling critnib_release()
when it is no longer used and can be freed
using the cb_free_leaf() callback.

Fixes: oneapi-src#1233

Signed-off-by: Lukasz Dorau <[email protected]>
ldorau added a commit to ldorau/unified-memory-framework that referenced this issue May 13, 2025
Add a reference counter and critnib_release() function.

The following 4 functions:
- critnib_remove(),
- critnib_get(),
- critnib_find_le() and
- critnib_find()

return a reference (void *ref) to the returned value,
that MUST be released by calling critnib_release()
when it is no longer used and can be freed
using the cb_free_leaf() callback.

Fixes: oneapi-src#1233

Signed-off-by: Lukasz Dorau <[email protected]>
ldorau added a commit to ldorau/unified-memory-framework that referenced this issue May 13, 2025
Add a reference counter and critnib_release() function.

The following 4 functions:
- critnib_remove(),
- critnib_get(),
- critnib_find_le() and
- critnib_find()

return a reference (void *ref) to the returned value,
that MUST be released by calling critnib_release()
when it is no longer used and can be freed
using the cb_free_leaf() callback.

Fixes: oneapi-src#1233

Signed-off-by: Lukasz Dorau <[email protected]>
ldorau added a commit to ldorau/unified-memory-framework that referenced this issue May 14, 2025
Add a reference counter and critnib_release() function.

The following 4 functions:
- critnib_remove(),
- critnib_get(),
- critnib_find_le() and
- critnib_find()

return a reference (void *ref) to the returned value,
that MUST be released by calling critnib_release()
when it is no longer used and can be freed
using the cb_free_leaf() callback.

Fixes: oneapi-src#1233

Signed-off-by: Lukasz Dorau <[email protected]>
ldorau added a commit to ldorau/unified-memory-framework that referenced this issue May 19, 2025
Add a reference counter and critnib_release() function.

The following 4 functions:
- critnib_remove(),
- critnib_get(),
- critnib_find_le() and
- critnib_find()

return a reference (void *ref) to the returned value,
that MUST be released by calling critnib_release()
when it is no longer used and can be freed
using the cb_free_leaf() callback.

Fixes: oneapi-src#1233

Signed-off-by: Lukasz Dorau <[email protected]>
ldorau added a commit to ldorau/unified-memory-framework that referenced this issue May 21, 2025
Add a reference counter and critnib_release() function.

When cb_free_leaf() is SET in critnib_new() the following 4 functions:
- critnib_remove(),
- critnib_get(),
- critnib_find_le() and
- critnib_find()

return a reference (void *ref) to the returned value,
that MUST be released by calling critnib_release()
when it is no longer used and can be freed
using the cb_free_leaf() callback.

Fixes: oneapi-src#1233

Signed-off-by: Lukasz Dorau <[email protected]>

Fixes
ldorau added a commit to ldorau/unified-memory-framework that referenced this issue May 21, 2025
Add a reference counter and critnib_release() function.

When cb_free_leaf() is SET in critnib_new() the following 4 functions:
- critnib_remove(),
- critnib_get(),
- critnib_find_le() and
- critnib_find()

return a reference (void *ref) to the returned value,
that MUST be released by calling critnib_release()
when it is no longer used and can be freed
using the cb_free_leaf() callback.

Fixes: oneapi-src#1233

Signed-off-by: Lukasz Dorau <[email protected]>
ldorau added a commit to ldorau/unified-memory-framework that referenced this issue May 21, 2025
Add a reference counter and critnib_release() function.

When cb_free_leaf() is SET in critnib_new() the following 4 functions:
- critnib_remove(),
- critnib_get(),
- critnib_find_le() and
- critnib_find()

return a reference (void *ref) to the returned value,
that MUST be released by calling critnib_release()
when it is no longer used and can be freed
using the cb_free_leaf() callback.

Fixes: oneapi-src#1233

Signed-off-by: Lukasz Dorau <[email protected]>
ldorau added a commit to ldorau/unified-memory-framework that referenced this issue May 22, 2025
Add a reference counter and critnib_release() function.

When cb_free_leaf() is SET in critnib_new() the following 4 functions:
- critnib_remove(),
- critnib_get(),
- critnib_find_le() and
- critnib_find()

return a reference (void *ref) to the returned value,
that MUST be released by calling critnib_release()
when it is no longer used and can be freed
using the cb_free_leaf() callback.

Fixes: oneapi-src#1233

Signed-off-by: Lukasz Dorau <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working high priority
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants