[Feature]: Detect unresponsive node-local GPU driver calls

[YQ-Wang]

Prerequisites

• I searched existing issues

Code of Conduct

• I agree to follow NVSentinel's Code of Conduct

Feature Summary

Add a watchdog to the GPU health monitor that detects DCGM operations that stop returning because the node-local NVIDIA driver is unresponsive.

Problem/Use Case

A GPU driver failure does not always return a DCGM timeout or error. A driver-facing call can block indefinitely while Kubernetes continues to report the node as Ready and its GPUs as allocatable. Because the poll loop itself is blocked, it cannot publish a health event. The liveness probe eventually restarts the monitor, but the replacement can block on the same driver operation without recording the underlying fault.

Remote DCGM endpoints require different handling: an unresponsive service, DNS failure, or network problem does not prove that the local node's GPU driver needs remediation.

Proposed Solution

• Track node-local DCGM operations from an independent watchdog thread using a configurable monotonic deadline.
• Emit GpuDriverUnresponsive with DRIVER_PROBE_HANG and a reboot recommendation only for embedded, node-local mode.
• Classify hangs in remote modes as GpuDcgmConnectivityFailure with DCGM_PROBE_HANG and no local reboot recommendation.
• Bound critical event delivery within the liveness restart budget and retry failed delivery attempts.
• Persist active local-driver state across monitor restarts to avoid duplicate events and allow a later healthy clearing event.
• Export a detection metric and expose watchdog and liveness timing through Helm values.
• Ship GpuDriverUnresponsive with STORE_ONLY processing by default so operators can validate detections before enabling remediation.

Component

Health Monitor

Related Implementation

Draft implementation: #1574

Depends on the pre-cleanup delivery fix tracked in #1575.

[lalitadithya]

Hi @YQ-Wang , I have a question, when you say "A driver-facing call can block indefinitely", do you mean DCGM will report healthy for all watches, but some NVML or CUDA call hang?

cc/ @deesharma24 , @XRFXLP

[YQ-Wang]

Hi @YQ-Wang , I have a question, when you say "A driver-facing call can block indefinitely", do you mean DCGM will report healthy for all watches, but some NVML or CUDA call hang?

cc/ @deesharma24 , @XRFXLP

Hi @lalitadithya, not quite. The failure mode is the opposite of “DCGM watches stay healthy while some other NVML/CUDA call hangs.”

What we mean is: a DCGM call that talks to the node-local driver never returns at all. There is no timeout and no error path, the probe parks forever. Because the GPU health monitor’s poll loop is blocked inside that call, it never re-evaluates watches and never publishes a health event. Kubernetes can still report the node as Ready with GPUs allocatable, so nothing else in the stack surfaces the fault either. The liveness probe eventually restarts the monitor, and the replacement can block on the same call again.

@jessicawinb and I saw this shape on the wedged nodes:

nvidia-smi -L returned in ~125 ms on a healthy peer, but hung past 120 s on the bad ones, with processes stuck in uninterruptible D state (unkillable even with SIGKILL)
new GPU pods failed with FailedCreatePodSandBox: context deadline exceeded, while the node stayed Ready with full allocatable GPUs
already-open DCGM handles could still look “fine,” which is why watch-based / metric-based checks alone missed it
So this is not “watches green + unrelated CUDA hang.” It is “the probe into DCGM/driver never comes back,” which is exactly why we need an independent watchdog thread with a monotonic deadline (#1574).

One naming clarification from review: an embedded-mode hang proves the stuck call is node-local, but it does not yet prove a kernel-driver wedge vs a DCGM userspace deadlock. That’s why #1574 now emits GpuDcgmUnresponsive / DCGM_PROBE_HANG (observe-only by default) rather than overstating it as GpuDriverUnresponsive. An independent bounded NVML/nvidia-smi probe would be the natural next step if we want to promote to a true driver-wedge signal.

Happy to walk through the watchdog flow in the PR if useful.