GPU-Sharing compatibility with MIG

[gflatters]

Scenario:

• I have H100 instances that have been split into 20GB slices with MIG (config all-1g.20gb)
• I have on-demand L4 instances
• My pods have an affinity such that they would prefer to run on the H100 instances, but can burst onto the on-demand L4 instances
• My pods specify a 0.5 GPU fraction, with the intention that they either take 0.5 of a 20GB slice of an H100 OR they take 0.5 of an L4

When a pod is scheduled onto an H100 instance, running nvidia-smi on the GPU reservation pod shows that it can see a MIG slice. However, the pod itself cannot see any GPU resources and pytorch code does not run with the device set to CUDA.

When a pod is scheduled on one of the on-demand L4 instances, everything works as expected.

Questions:

Am I trying to do something that isn't officially supported, i.e. is GPU sharing compatible with MIG?
Is there a better way of achieving the intended behaviour?

[enoodle]

Hi @gflatters 👋

I think that this MIG configuration means that there are now 7 nvidia.com/mig-1g.20gb devices on the node.
They should be assigned only to tasks that ask for exactly that resource currently. In that case they would count to the quota as the number of GPUs that are mentioned in this resource name (for example 2g.40gb is counted as 2 GPUs, 4g.80gb as 4 etc.)

The fact that a fractional pod is assigned to this node but gets no real GPU access is a bug.

The way you might use KAI to schedule your jobs on this cluster with different kinds of GPUs is to leave the MIG option on the GPU disabled and ask for the exact amount of GPU memory that you want on the pod: https://github.com/NVIDIA/KAI-Scheduler/blob/main/docs/gpu-sharing/README.md#gpu-memory-pod

We could develop in the future something that reads these devices as GPUs with this amount of memory and allows assigning pods that request this amount of memory or lower on them.

[github-actions]

Issue has been open for over 3 months with no activity. It’s being marked as stale. Please update if it’s still relevant.