Transient inconsitency in in-memory values causes a HTTP 500 error

[benoit74]

1772496185222	2026-03-03T00:03:05.222Z	INFO:     73.133.114.154:0 - "GET /api/v1/requests/b93ed938-6cc9-4284-b0d8-f4c69b888662 HTTP/1.1" 200 OK
1772496185330	2026-03-03T00:03:05.330Z	INFO:     73.133.114.154:0 - "GET /api/v1/requests/b93ed938-6cc9-4284-b0d8-f4c69b888662 HTTP/1.1" 200 OK
1772496185334	2026-03-03T00:03:05.334Z	INFO:     73.133.114.154:0 - "POST /api/v1/tracker_status HTTP/1.1" 200 OK
1772496185336	2026-03-03T00:03:05.336Z	ERROR: list.remove(x): x not in list
1772496185336	2026-03-03T00:03:05.336Z	Traceback (most recent call last):
1772496185336	2026-03-03T00:03:05.336Z	  File "/usr/local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
1772496185336	2026-03-03T00:03:05.336Z	    await self.app(scope, receive, _send)
1772496185336	2026-03-03T00:03:05.336Z	  File "/usr/local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 93, in __call__
1772496185336	2026-03-03T00:03:05.336Z	    await self.simple_response(scope, receive, send, request_headers=headers)
1772496185336	2026-03-03T00:03:05.336Z	  File "/usr/local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 144, in simple_response
1772496185336	2026-03-03T00:03:05.336Z	    await self.app(scope, receive, send)
1772496185336	2026-03-03T00:03:05.336Z	  File "/usr/local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
1772496185336	2026-03-03T00:03:05.336Z	    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
1772496185336	2026-03-03T00:03:05.336Z	  File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
1772496185336	2026-03-03T00:03:05.336Z	    raise exc
1772496185336	2026-03-03T00:03:05.336Z	  File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
1772496185336	2026-03-03T00:03:05.336Z	    await app(scope, receive, sender)
1772496185336	2026-03-03T00:03:05.336Z	  File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
1772496185336	2026-03-03T00:03:05.336Z	    await self.middleware_stack(scope, receive, send)
1772496185336	2026-03-03T00:03:05.336Z	  File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
1772496185336	2026-03-03T00:03:05.336Z	    await route.handle(scope, receive, send)
1772496185336	2026-03-03T00:03:05.336Z	  File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
1772496185336	2026-03-03T00:03:05.336Z	    await self.app(scope, receive, send)
1772496185336	2026-03-03T00:03:05.336Z	  File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
1772496185336	2026-03-03T00:03:05.336Z	    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
1772496185336	2026-03-03T00:03:05.336Z	  File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
1772496185336	2026-03-03T00:03:05.336Z	    raise exc
1772496185336	2026-03-03T00:03:05.336Z	  File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
1772496185336	2026-03-03T00:03:05.336Z	    await app(scope, receive, sender)
1772496185336	2026-03-03T00:03:05.336Z	  File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
1772496185336	2026-03-03T00:03:05.336Z	    response = await f(request)
1772496185336	2026-03-03T00:03:05.336Z	               ^^^^^^^^^^^^^^^^
1772496185336	2026-03-03T00:03:05.336Z	  File "/usr/local/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
1772496185336	2026-03-03T00:03:05.336Z	    raw_response = await run_endpoint_function(
1772496185336	2026-03-03T00:03:05.336Z	                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1772496185336	2026-03-03T00:03:05.336Z	  File "/usr/local/lib/python3.12/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
1772496185336	2026-03-03T00:03:05.336Z	    return await run_in_threadpool(dependant.call, **values)
1772496185336	2026-03-03T00:03:05.336Z	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1772496185336	2026-03-03T00:03:05.336Z	  File "/usr/local/lib/python3.12/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
1772496185336	2026-03-03T00:03:05.336Z	    return await anyio.to_thread.run_sync(func, *args)
1772496185336	2026-03-03T00:03:05.336Z	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1772496185336	2026-03-03T00:03:05.336Z	  File "/usr/local/lib/python3.12/site-packages/anyio/to_thread.py", line 63, in run_sync
1772496185336	2026-03-03T00:03:05.336Z	    return await get_async_backend().run_sync_in_worker_thread(
1772496185336	2026-03-03T00:03:05.336Z	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1772496185336	2026-03-03T00:03:05.336Z	  File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2502, in run_sync_in_worker_thread
1772496185336	2026-03-03T00:03:05.336Z	    return await future
1772496185336	2026-03-03T00:03:05.336Z	           ^^^^^^^^^^^^
1772496185336	2026-03-03T00:03:05.336Z	  File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 986, in run
1772496185336	2026-03-03T00:03:05.336Z	    result = context.run(func, *args)
1772496185336	2026-03-03T00:03:05.336Z	             ^^^^^^^^^^^^^^^^^^^^^^^^
1772496185336	2026-03-03T00:03:05.336Z	  File "/usr/local/lib/python3.12/site-packages/zimitfrontend/routes/tracker.py", line 34, in post_tracker_status
1772496185336	2026-03-03T00:03:05.336Z	    tracker_status = tracker.add_task(
1772496185336	2026-03-03T00:03:05.336Z	                     ^^^^^^^^^^^^^^^^^
1772496185336	2026-03-03T00:03:05.336Z	  File "/usr/local/lib/python3.12/site-packages/zimitfrontend/tracker.py", line 118, in add_task
1772496185336	2026-03-03T00:03:05.336Z	    self._refresh_ongoing_tasks(ip_address, unique_id)
1772496185336	2026-03-03T00:03:05.336Z	  File "/usr/local/lib/python3.12/site-packages/zimitfrontend/tracker.py", line 77, in _refresh_ongoing_tasks
1772496185336	2026-03-03T00:03:05.336Z	    client.ongoing_tasks.remove(task)
1772496185336	2026-03-03T00:03:05.336Z	ValueError: list.remove(x): x not in list
1772496185336	2026-03-03T00:03:05.336Z	INFO:     73.133.114.154:0 - "POST /api/v1/tracker_status HTTP/1.1" 500 Internal Server Error
1772496185338	2026-03-03T00:03:05.338Z	ERROR:    Exception in ASGI application
1772496185338	2026-03-03T00:03:05.338Z	Traceback (most recent call last):
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 416, in run_asgi
1772496185338	2026-03-03T00:03:05.338Z	    result = await app(  # type: ignore[func-returns-value]
1772496185338	2026-03-03T00:03:05.338Z	             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
1772496185338	2026-03-03T00:03:05.338Z	    return await self.app(scope, receive, send)
1772496185338	2026-03-03T00:03:05.338Z	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
1772496185338	2026-03-03T00:03:05.338Z	    await super().__call__(scope, receive, send)
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
1772496185338	2026-03-03T00:03:05.338Z	    await self.middleware_stack(scope, receive, send)
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
1772496185338	2026-03-03T00:03:05.338Z	    raise exc
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
1772496185338	2026-03-03T00:03:05.338Z	    await self.app(scope, receive, _send)
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
1772496185338	2026-03-03T00:03:05.338Z	    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
1772496185338	2026-03-03T00:03:05.338Z	    raise exc
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
1772496185338	2026-03-03T00:03:05.338Z	    await app(scope, receive, sender)
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
1772496185338	2026-03-03T00:03:05.338Z	    await self.middleware_stack(scope, receive, send)
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
1772496185338	2026-03-03T00:03:05.338Z	    await route.handle(scope, receive, send)
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 460, in handle
1772496185338	2026-03-03T00:03:05.338Z	    await self.app(scope, receive, send)
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
1772496185338	2026-03-03T00:03:05.338Z	    await super().__call__(scope, receive, send)
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
1772496185338	2026-03-03T00:03:05.338Z	    await self.middleware_stack(scope, receive, send)
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
1772496185338	2026-03-03T00:03:05.338Z	    raise exc
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
1772496185338	2026-03-03T00:03:05.338Z	    await self.app(scope, receive, _send)
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 93, in __call__
1772496185338	2026-03-03T00:03:05.338Z	    await self.simple_response(scope, receive, send, request_headers=headers)
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 144, in simple_response
1772496185338	2026-03-03T00:03:05.338Z	    await self.app(scope, receive, send)
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
1772496185338	2026-03-03T00:03:05.338Z	    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
1772496185338	2026-03-03T00:03:05.338Z	    raise exc
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
1772496185338	2026-03-03T00:03:05.338Z	    await app(scope, receive, sender)
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
1772496185338	2026-03-03T00:03:05.338Z	    await self.middleware_stack(scope, receive, send)
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
1772496185338	2026-03-03T00:03:05.338Z	    await route.handle(scope, receive, send)
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
1772496185338	2026-03-03T00:03:05.338Z	    await self.app(scope, receive, send)
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
1772496185338	2026-03-03T00:03:05.338Z	    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
1772496185338	2026-03-03T00:03:05.338Z	    raise exc
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
1772496185338	2026-03-03T00:03:05.338Z	    await app(scope, receive, sender)
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
1772496185338	2026-03-03T00:03:05.338Z	    response = await f(request)
1772496185338	2026-03-03T00:03:05.338Z	               ^^^^^^^^^^^^^^^^
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
1772496185338	2026-03-03T00:03:05.338Z	    raw_response = await run_endpoint_function(
1772496185338	2026-03-03T00:03:05.338Z	                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
1772496185338	2026-03-03T00:03:05.338Z	    return await run_in_threadpool(dependant.call, **values)
1772496185338	2026-03-03T00:03:05.338Z	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
1772496185338	2026-03-03T00:03:05.338Z	    return await anyio.to_thread.run_sync(func, *args)
1772496185338	2026-03-03T00:03:05.338Z	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/anyio/to_thread.py", line 63, in run_sync
1772496185338	2026-03-03T00:03:05.338Z	    return await get_async_backend().run_sync_in_worker_thread(
1772496185338	2026-03-03T00:03:05.338Z	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2502, in run_sync_in_worker_thread
1772496185338	2026-03-03T00:03:05.338Z	    return await future
1772496185338	2026-03-03T00:03:05.338Z	           ^^^^^^^^^^^^
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 986, in run
1772496185338	2026-03-03T00:03:05.338Z	    result = context.run(func, *args)
1772496185338	2026-03-03T00:03:05.338Z	             ^^^^^^^^^^^^^^^^^^^^^^^^
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/zimitfrontend/routes/tracker.py", line 34, in post_tracker_status
1772496185338	2026-03-03T00:03:05.338Z	    tracker_status = tracker.add_task(
1772496185338	2026-03-03T00:03:05.338Z	                     ^^^^^^^^^^^^^^^^^
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/zimitfrontend/tracker.py", line 118, in add_task
1772496185338	2026-03-03T00:03:05.338Z	    self._refresh_ongoing_tasks(ip_address, unique_id)
1772496185338	2026-03-03T00:03:05.338Z	  File "/usr/local/lib/python3.12/site-packages/zimitfrontend/tracker.py", line 77, in _refresh_ongoing_tasks
1772496185338	2026-03-03T00:03:05.338Z	    client.ongoing_tasks.remove(task)
1772496185338	2026-03-03T00:03:05.338Z	ValueError: list.remove(x): x not in list
1772496186159	2026-03-03T00:03:06.159Z	INFO:     108.28.25.49:0 - "POST /api/v1/tracker_status HTTP/1.1" 200 OK
1772496186215	2026-03-03T00:03:06.215Z	INFO:     108.28.25.49:0 - "GET /api/v1/requests/837a19ac-8cde-455a-96da-ca2fca4cdeac HTTP/1.1" 200 OK

[aaryanrajputhere]

I investigated this and can reproduce the transient 500 (ValueError: list.remove(x): x not in list) in the tracker flow.

Root cause (from my investigation)

• The tracker uses shared in-memory state (known_clients / ongoing_tasks) that can be accessed by concurrent requests.
• During refresh, the code builds a list of completed tasks and then removes them.
• If another request mutates the same list between those two steps, a stale remove is attempted and raises ValueError, which bubbles up as HTTP 500.

Proposed fix

1. Serialize tracker state mutations with a lock around add_task / refresh logic.
2. Make task cleanup idempotent (only remove if still present) so stale snapshots cannot crash the request.
3. Add a regression test that simulates stale/removal race behavior to ensure this does not regress.

I can open/attach the patch and tests for this issue.

Could you please assign this issue to me?
@benoit74

[benoit74]

LGTM, thank you, please proceed

[aaryanrajputhere]

Fix for Issue #193: Transient Inconsistency Causing HTTP 500

This patch addresses the race condition in the Tracker singleton that could cause a transient HTTP 500 due to concurrent modification of shared in-memory state.

---

Change 1: Add Thread Synchronization

File: api/src/zimitfrontend/tracker.py

Add a lock to protect shared tracker state.

Modify Tracker initialization

class Tracker:
    def __init__(self):
        self.known_clients: list[ClientInfo] = []
        self._lock = threading.Lock()  # NEW: Protect shared state
        self.ongoing_task_has_finished = self._ongoing_task_has_finished
        self.has_reached_maximum_tasks = self._has_reached_maximum_tasks

Serialize state mutations in add_task()

Wrap the entire method body with the lock.

def add_task(
    self, ip_address: str, unique_id: str | None, task_id: str | None
) -> AddTaskResponse:
    with self._lock:  # NEW: Serialize all tracker state mutations
        if unique_id and not is_valid_unique_id(unique_id):
            return AddTaskResponse(
                status=AddTaskStatus.INVALID_UNIQUE_ID,
            )

        self._refresh_ongoing_tasks(ip_address, unique_id)

        # ... rest of existing logic

This ensures that concurrent FastAPI request handlers cannot mutate known_clients or ongoing_tasks simultaneously.

---

Change 2: Make Task Removal Idempotent

File: tracker.py

Replace unsafe removal logic.

Before

for task in completed_tasks:
    client.ongoing_tasks.remove(task)

After

for task in completed_tasks:
    if task in client.ongoing_tasks:  # NEW: Check before removing
        client.ongoing_tasks.remove(task)

This prevents a ValueError when another thread has already removed the task.

---

Change 3: Add Regression Test

File: test_tracker.py

Add a regression test to simulate the race condition and verify the fix.

def test_stale_task_removal():
    """Regression test for issue #193 - ensure stale task removal doesn't crash."""
    tracker = Tracker()
    tracker.known_clients = [
        ClientInfo(
            ip_address=CLIENT_2_IP,
            unique_id=CLIENT_2_VALID_ID,
            ongoing_tasks=[TASK_ID1],
        )
    ]

    def custom_ongoing_task_has_finished(task_id: str):
        # Simulate race condition: another thread clears tasks while being checked
        tracker.known_clients[0].ongoing_tasks.clear()
        return True

    tracker.ongoing_task_has_finished = custom_ongoing_task_has_finished

    # Before fix: Would crash with ValueError
    # After fix: Handles stale state gracefully
    result = tracker.add_task(CLIENT_2_IP, None, None)
    assert result.status == AddTaskStatus.CAN_ADD_TASK

---

Testing

Run all tracker unit tests from the workspace root:

.venv/Scripts/python.exe -m pytest api/tests/unit/test_tracker.py -v

Result:

============================= test session starts =============================
collected 25 items

api/tests/unit/test_tracker.py::test_stale_task_removal PASSED          [100%]

============================= 25 passed in 1.41s ==============================

---

Summary

This fix eliminates the race condition by:

• Adding thread synchronization using threading.Lock
• Making task removal idempotent
• Adding a regression test to prevent future regressions

Together these changes prevent transient HTTP 500 errors caused by concurrent modification of ongoing_tasks.

[aaryanrajputhere]

I have linked a PR , Kindly review at your convenience @benoit74

[benoit74]

Please avoid notifying me both in issue and PR, this is just noise, one notification is enough. And please avoid copy/pasting code changes without providing any added value, I know how to read a PR by myself and should something important need to be mentioned, then mention it, here the comment is so sloppy I won't read it.