LoadScopeScheduling.remove_node leaves a dead worker in registered_collections, causing INTERNALERROR KeyError when another worker is still collecting

[nevenincs]

Summary

LoadScopeScheduling.remove_node() removes a dead worker from assigned_work but never removes its entry from registered_collections. The stale entry keeps collection_is_completed returning True while a later worker is still collecting, so schedule() proceeds and _assign_work_unit() raises KeyError on the still-collecting node — crashing the whole run with INTERNALERROR.

This affects both --dist=loadscope and --dist=loadfile, since LoadFileScheduling subclasses LoadScopeScheduling.

Mechanism

In xdist/scheduler/loadscope.py (3.8.0):

• collection_is_completed → len(self.registered_collections) >= self.numnodes
• remove_node() → workload = self.assigned_work.pop(node) … and no corresponding removal from self.registered_collections

So after a worker dies, registered_collections still holds its entry. If a different worker has not yet finished collecting, collection_is_completed is already satisfied by the dead worker's stale entry. worker_collectionfinish then calls schedule(), whose assert self.collection_is_completed passes, and _reschedule → _assign_work_unit does self.registered_collections[node] for a node that has not registered → KeyError.

Traceback path: dsession.py worker_collectionfinish → loadscope.py schedule → _reschedule → _assign_work_unit.

Observed

Two independent runs on the same repository, --dist=loadfile -n auto (24 workers on a 24-core host under heavy concurrent load):

[gw7] node down: Not properly terminated
[gw6] node down: Not properly terminated
[gw8] node down: Not properly terminated
INTERNALERROR> KeyError: <WorkerController gw14>

[gw2] node down: Not properly terminated
INTERNALERROR> KeyError: <WorkerController gw12>

Note the pattern in both: the workers that die are low-index, and the KeyError names a high-index worker — i.e. one that had not yet finished collecting. That is exactly what the stale-entry mechanism predicts.

Secondary symptom: duplicated reports for a single node

In a separate run on the same codebase, a module collecting 6 tests produced 11 reports, with one non-parametrised test listed 6 times in the FAILED block, after 5 workers went down and their work was rescheduled onto successive workers. So beyond the crash, bookkeeping desynchronisation appears able to inflate report counts — a FAILED block from a run that lost workers cannot be read as a failure inventory.

Impact

The run exits 3, so CI does fail correctly — this is not a false green. The damage is diagnostic: no FAILED summary is printed, counts are partial or inflated, and the remaining tests never execute. The only recourse is a full re-run (20–45 minutes on our suite), which is how "just re-run it" becomes institutionalised and why failure sets are not comparable between runs.

Worker deaths themselves are environmental here (heavy host contention), and we are capping worker count as mitigation — but a scheduler crash on worker loss seems worth fixing independently, since remove_node's own docstring states it is called "either when the node crashed or at shutdown time".

Suggested fix

Remove the node's registered_collections entry in remove_node() alongside the assigned_work.pop(node), so collection_is_completed reflects only live nodes. (I have not opened a PR as I have not evaluated the effect on numnodes accounting during shutdown, where remove_node is also called.)

Environment

• pytest-xdist 3.8.0 (latest on PyPI at time of filing)
• pytest 9.1.1
• Python 3.13.11, Windows