Skip to content

[Feature]: Add a configurable --sysfs-root for device NUMA detection (GetNumaNode) #1833

Description

@eliranw

Summary

GetNumaNode reads a device's NUMA node from a hardcoded /sys/bus/pci/devices/<busID>/numa_node, with no way to redirect the sysfs root.
This is the device-plugin counterpart of NVIDIA/k8s-test-infra#264 (same --sysfs-root request for the DRA driver); both consume the synthetic tree from #263 (render-pci-sysfs).

Problem

internal/rm/nvml_devices.go:150-162:

busID := strings.ToLower(strings.TrimPrefix(uint8Slice(info.BusId[:]).String(), "0000"))
b, err := os.ReadFile(fmt.Sprintf("/sys/bus/pci/devices/%s/numa_node", busID))
if err != nil {
    return false, 0, nil  // error swallowed -> device reported with no NUMA
}

The /sys prefix is a literal; nothing overrides it (--nvidia-driver-root / --nvidia-dev-root feed only CDI / library resolution, not this read). When NVML reports devices whose bus IDs are not backed by host sysfs (simulated/mock GPUs), the read fails, hasNuma=false, and BuildDevice (internal/rm/devices.go:93-101) leaves Device.Topology nil, so the device is advertised with no NUMA and the Topology Manager / podresources see no affinity.

Proposed solution

Flag --sysfs-root (default /sys), matching #264 and the existing --nvidia-*-root family:

os.ReadFile(filepath.Join(sysfsRoot, "bus/pci/devices", busID, "numa_node"))

In-tree precedent for the same read: go-nvlib's WithPCIDevicesRoot (pkg/nvpci/nvpci.go:187,317, mock at mock.go:104) and k8s dynamic-resource-allocation's WithFSFromRoot (which #264 builds on). Default unchanged means zero impact on real deployments.

Alternatives

  • nvmlDeviceGetNumaNodeId: doesn't help. A mock NVML returns NVML_ERROR_NOT_SUPPORTED for it, and on real hardware it signals "no NUMA" via return code (not the sysfs -1) and is absent on older drivers, so it would need return-code gating plus a sysfs fallback anyway.
  • Subpath-mount the tree over /sys/bus/pci/devices: possible (Add GOLANG_ARCH arg to Ubuntu 20.04 Dockerfile #264's fallback) but shadows the real PCI dir per-pod; a flag is cleaner.

Scoped to GetNumaNode (NVML full GPU + MIG, which delegates to the parent). A second hardcoded root at internal/vgpu/pciutil.go:42 (GFD vGPU detection) could adopt the same flag later. Happy to send a PR.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions