Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky test MockSharedArbitrationTest.concurrentArbitrations #11467

Open
jinchengchenghh opened this issue Nov 7, 2024 · 2 comments
Open

Flaky test MockSharedArbitrationTest.concurrentArbitrations #11467

jinchengchenghh opened this issue Nov 7, 2024 · 2 comments
Assignees
@tanjialiang
Labels
bug Something isn't working triage Newly created issue that needs attention.

Comments

@jinchengchenghh
Copy link
Contributor

Bug description

Unexpected exception Memory arbitration timed out on memory pool: RootPool-30 after running 5m 0s.

325/379 Test #252: MockSharedArbitrationTest.concurrentArbitrations .....................***Failed  300.06 sec
Note: Google Test filter = MockSharedArbitrationTest.concurrentArbitrations
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from MockSharedArbitrationTest
[ RUN      ] MockSharedArbitrationTest.concurrentArbitrations
E20241106 02:53:42.650642 30352 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/tests/MockSharedArbitratorTest.cpp:4030, Function:operator(), Expression:  Inject enter arbitration failure, Source: RUNTIME, ErrorCode: INVALID_STATE
E20241106 02:53:42.651927 30356 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/tests/MockSharedArbitratorTest.cpp:4030, Function:operator(), Expression:  Inject enter arbitration failure, Source: RUNTIME, ErrorCode: INVALID_STATE
E20241106 02:53:42.652761 30343 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/tests/MockSharedArbitratorTest.cpp:4030, Function:operator(), Expression:  Inject enter arbitration failure, Source: RUNTIME, ErrorCode: INVALID_STATE
E20241106 02:53:42.653581 30323 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/tests/MockSharedArbitratorTest.cpp:4030, Function:operator(), Expression:  Inject enter arbitration failure, Source: RUNTIME, ErrorCode: INVALID_STATE
E20241106 02:53:42.659518 30335 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/tests/MockSharedArbitratorTest.cpp:4030, Function:operator(), Expression:  Inject enter arbitration failure, Source: RUNTIME, ErrorCode: INVALID_STATE
E20241106 02:56:12.651465 30315 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:1159, Function:reclaimUsedMemoryByAbort, Expression:  Memory pool aborted to reclaim used memory, current capacity 32.00MB, memory pool stats:
Memory Pool[RootPool-48 AGGREGATE root[RootPool-48] parent[null] MALLOC track-usage thread-safe]<unlimited max capacity capacity 32.00MB used 30.02MB available 0B reservation [used 0B, reserved 32.00MB, min 0B] counters [allocs 0, frees 0, reserves 0, releases 0, collisions 0])>
RootPool-48 usage 30.02MB reserved 32.00MB peak 120.00MB
    MockTask50 usage 30.02MB reserved 32.00MB peak 30.02MB

Top 1 leaf memory pool usages:
    MockTask50 usage 30.02MB reserved 32.00MB peak 30.02MB
, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.652047 30358 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-48 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.652163 30315 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:1159, Function:reclaimUsedMemoryByAbort, Expression:  Memory pool aborted to reclaim used memory, current capacity 72.00MB, memory pool stats:
Memory Pool[RootPool-42 AGGREGATE root[RootPool-42] parent[null] MALLOC track-usage thread-safe]<unlimited max capacity capacity 72.00MB used 71.77MB available 0B reservation [used 0B, reserved 72.00MB, min 0B] counters [allocs 0, frees 0, reserves 0, releases 0, collisions 0])>
RootPool-42 usage 71.77MB reserved 72.00MB peak 72.00MB
    MockTask46 usage 71.77MB reserved 72.00MB peak 71.77MB

Top 1 leaf memory pool usages:
    MockTask46 usage 71.77MB reserved 72.00MB peak 71.77MB
, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.652580 30315 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:1159, Function:reclaimUsedMemoryByAbort, Expression:  Memory pool aborted to reclaim used memory, current capacity 272.00MB, memory pool stats:
Memory Pool[RootPool-24 AGGREGATE root[RootPool-24] parent[null] MALLOC track-usage thread-safe]<unlimited max capacity capacity 272.00MB used 265.47MB available 0B reservation [used 0B, reserved 272.00MB, min 0B] counters [allocs 0, frees 0, reserves 0, releases 0, collisions 0])>
RootPool-24 usage 265.47MB reserved 272.00MB peak 288.00MB
    MockTask28 usage 265.47MB reserved 272.00MB peak 283.57MB

Top 1 leaf memory pool usages:
    MockTask28 usage 265.47MB reserved 272.00MB peak 283.57MB
, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.652689 30355 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-42 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.652953 30338 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-24 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.653612 30315 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:1159, Function:reclaimUsedMemoryByAbort, Expression:  Memory pool aborted to reclaim used memory, current capacity 4.00MB, memory pool stats:
Memory Pool[RootPool-54 AGGREGATE root[RootPool-54] parent[null] MALLOC track-usage thread-safe]<unlimited max capacity capacity 4.00MB used 3.63MB available 0B reservation [used 0B, reserved 4.00MB, min 0B] counters [allocs 0, frees 0, reserves 0, releases 0, collisions 0])>
RootPool-54 usage 3.63MB reserved 4.00MB peak 4.00MB
    MockTask58 usage 3.63MB reserved 4.00MB peak 3.63MB

Top 1 leaf memory pool usages:
    MockTask58 usage 3.63MB reserved 4.00MB peak 3.63MB
, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.653817 30360 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-48 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.653911 30315 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:1159, Function:reclaimUsedMemoryByAbort, Expression:  Memory pool aborted to reclaim used memory, current capacity 64.00MB, memory pool stats:
Memory Pool[RootPool-36 AGGREGATE root[RootPool-36] parent[null] MALLOC track-usage thread-safe]<unlimited max capacity capacity 64.00MB used 63.50MB available 0B reservation [used 0B, reserved 64.00MB, min 0B] counters [allocs 0, frees 0, reserves 0, releases 0, collisions 0])>
RootPool-36 usage 63.50MB reserved 64.00MB peak 64.00MB
    MockTask41 usage 63.50MB reserved 64.00MB peak 63.50MB

Top 1 leaf memory pool usages:
    MockTask41 usage 63.50MB reserved 64.00MB peak 63.50MB
, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.654227 30315 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:1159, Function:reclaimUsedMemoryByAbort, Expression:  Memory pool aborted to reclaim used memory, current capacity 40.00MB, memory pool stats:
Memory Pool[RootPool-18 AGGREGATE root[RootPool-18] parent[null] MALLOC track-usage thread-safe]<unlimited max capacity capacity 40.00MB used 35.83MB available 0B reservation [used 0B, reserved 40.00MB, min 0B] counters [allocs 0, frees 0, reserves 0, releases 0, collisions 0])>
RootPool-18 usage 35.83MB reserved 40.00MB peak 40.00MB
    MockTask20 usage 35.83MB reserved 40.00MB peak 38.17MB

Top 1 leaf memory pool usages:
    MockTask20 usage 35.83MB reserved 40.00MB peak 38.17MB
, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.654458 30334 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-18 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.654778 30348 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-36 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.654824 30353 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-42 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.654861 30362 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-54 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.654915 30357 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-48 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.655764 30331 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-18 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.656750 30333 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-18 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.657604 30335 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-18 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.659263 30346 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-36 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.660156 30349 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-36 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.660398 30354 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-42 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.660652 30347 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-36 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.661379 30339 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-24 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.662001 30351 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-42 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.663138 30359 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-48 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.663478 30352 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-42 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.[664](https://github.com/facebookincubator/velox/actions/runs/11695476425/job/32570960778?pr=11384#step:10:665)201 30337 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-24 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.[665](https://github.com/facebookincubator/velox/actions/runs/11695476425/job/32570960778?pr=11384#step:10:666)004 30336 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-24 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.665486 30340 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-24 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.[668](https://github.com/facebookincubator/velox/actions/runs/11695476425/job/32570960778?pr=11384#step:10:669)715 30361 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-54 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.[669](https://github.com/facebookincubator/velox/actions/runs/11695476425/job/32570960778?pr=11384#step:10:670)520 30363 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-54 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.670312 30365 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-54 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:56:12.671752 30356 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:942, Function:checkIfAborted, Expression:  Memory pool RootPool-48 aborted, Source: RUNTIME, ErrorCode: MEM_ABORTED
E20241106 02:58:42.656728 30341 SharedArbitrator.cpp:805] [MEM] RootPool-30 wait for memory arbitration timed out after running 5m 0s
E20241106 02:58:42.656852 30341 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:949, Function:checkIfTimeout, Expression:  Memory arbitration timed out on memory pool: RootPool-30 after running 5m 0s, Source: RUNTIME, ErrorCode: MEM_ARBITRATION_TIMEOUT
/home/runner/work/velox/velox/velox/velox/common/memory/tests/MockSharedArbitratorTest.cpp:4063: Failure
Value of: true
  Actual: true
Expected: false
Unexpected exception Memory arbitration timed out on memory pool: RootPool-30 after running 5m 0s
E20241106 02:58:42.657884 30345 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:949, Function:checkIfTimeout, Expression:  Memory arbitration timed out on memory pool: RootPool-30 after running 5m 0s, Source: RUNTIME, ErrorCode: MEM_ARBITRATION_TIMEOUT
/home/runner/work/velox/velox/velox/velox/common/memory/tests/MockSharedArbitratorTest.cpp:4063: Failure
Value of: true
  Actual: true
Expected: false
Unexpected exception Memory arbitration timed out on memory pool: RootPool-30 after running 5m 0s
[  FAILED  ] MockSharedArbitrationTest.concurrentArbitrations (300017 ms)
[----------] 1 test from MockSharedArbitrationTest (300017 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (300017 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] MockSharedArbitrationTest.concurrentArbitrations

 1 FAILED TEST

https://github.com/facebookincubator/velox/actions/runs/11695476425/job/32570960778?pr=11384

System information

CI

Relevant logs

No response
@jinchengchenghh jinchengchenghh added bug Something isn't working triage Newly created issue that needs attention. labels Nov 7, 2024
@jinchengchenghh jinchengchenghh changed the title Flaky test Flaky test MockSharedArbitrationTest.concurrentArbitrations Nov 7, 2024
@jinchengchenghh jinchengchenghh mentioned this issue Nov 7, 2024
Closed
@xiaoxmeng
Copy link
Contributor

@tanjialiang can you help take a look? thanks!

@tanjialiang
Copy link
Contributor

It shall be fixed in this PR #11464

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tanjialiang tanjialiang

Labels
bug Something isn't working triage Newly created issue that needs attention.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tanjialiang @jinchengchenghh @xiaoxmeng