Skip to content

Conversation

@tanmayv25
Copy link
Contributor

This test is currently failing with leaked objects

        if tensor_objects:
            print("Tensor objects")
            print(len(tensor_objects))
            print(type(tensor_objects[-1].memory_buffer.owner))
        print(Counter(new_objects))
    
>       assert len(tensor_objects) == 0, "Leaked Objects"
E       AssertionError: Leaked Objects
E       assert 100 == 0
E        +  where 100 = len([Tensor(data_type=<TRITONSERVER_DataType.FP64: 12>, shape=[134217728], memory_buffer=MemoryBuffer(data_ptr=126597480841232, memory_type=<TRITONSERVER_MemoryType.CPU: 0>, memory_type_id=0, size=1073741824, owner=array([1., 1., 1., ..., 1., 1., 1.]))), Tensor(data_type=<TRITONSERVER_DataType.FP64: 12>, shape=[134217728], memory_buffer=MemoryBuffer(data_ptr=126596405002256, memory_type=<TRITONSERVER_MemoryType.CPU: 0>, memory_type_id=0, size=1073741824, owner=array([1., 1., 1., ..., 1., 1., 1.]))), Tensor(data_type=<TRITONSERVER_DataType.FP64: 12>, shape=[134217728], memory_buffer=MemoryBuffer(data_ptr=126595329163280, memory_type=<TRITONSERVER_MemoryType.CPU: 0>, memory_type_id=0, size=1073741824, owner=array([1., 1., 1., ..., 1., 1., 1.]))), Tensor(data_type=<TRITONSERVER_DataType.FP64: 12>, shape=[134217728], memory_buffer=MemoryBuffer(data_ptr=126594253324304, memory_type=<TRITONSERVER_MemoryType.CPU: 0>, memory_type_id=0, size=1073741824, owner=array([1., 1., 1., ..., 1., 1., 1.]))), Tensor(data_type=<TRITONSERVER_DataType.FP64: 12>, shape=[134217728], memory_buffer=MemoryBuffer(data_ptr=126593177485328, memory_type=<TRITONSERVER_MemoryType.CPU: 0>, memory_type_id=0, size=1073741824, owner=array([1., 1., 1., ..., 1., 1., 1.]))), Tensor(data_type=<TRITONSERVER_DataType.FP64: 12>, shape=[134217728], memory_buffer=MemoryBuffer(data_ptr=126592101646352, memory_type=<TRITONSERVER_MemoryType.CPU: 0>, memory_type_id=0, size=1073741824, owner=array([1., 1., 1., ..., 1., 1., 1.]))), ...])

With the changes from PR #421, this test will cause segmentation fault.

Fatal Python error: Segmentation fault

Current thread 0x000078df1cc491c0 (most recent call first):
  Garbage-collecting
  File "/usr/local/lib/python3.10/dist-packages/test_tritonservrer/test_api.py", line 326 in test_cpu_memory_leak_async
  File "/usr/local/lib/python3.10/dist-packages/_pytest/python.py", line 159 in pytest_pyfunc_call
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/local/lib/python3.10/dist-packages/_pytest/python.py", line 1627 in runtest
  File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 174 in pytest_runtest_call
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 242 in <lambda>
  File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 341 in from_call

@tanmayv25 tanmayv25 requested review from GuanLuo and nnshah1 January 3, 2025 02:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants