Add KEP for DRA: Extended Resource

[yliaog]

• One-line PR description:
  Add new KEP for supporting extended resource requests in DRA

• Issue link:
  DRA: Handle extended resource requests via DRA Driver #5004

• Other comments:
  kubelet and scheduler for extended resource backed by DRA kubernetes#130653

[yliaog]

/assign @johnbelamaric

[johnbelamaric]

Awesome, thanks @yliaog

[mortent]

The DeviceRequestAllocationResult has a Request field that is a reference to the request in the ResourceClaim spec. But in this situation, there is no corresponding request in the spec (since the spec is empty). Drivers might use this value, among other things for looking up any device configuration. How do we plan to handle this?

[yliaog]

DRA driver has two parts:
1/ 1st part that handles DRA resource claim actuation
2/ 2nd part that handles extended resources actuation

the 1st part does not need to actuate on this special resource claim object, it is ignored there.

the 2nd part needs to read the list of allocated devices from the special resource claim object, and only use the devices in that list for extended resources actuation.

[pohly]

This pushes the responsibility of choosing a device back into the DRA driver. I don't think this is the right direction from an architectural perspective.

cc @klueska

I also very much dislike that DRA drivers need to be updated at all.

[johnbelamaric]

No, I think you misunderstood (or I did). As I read it, the allocations are made by the scheduler and stored in the special singleton resource claim, not decided by the driver.

[yliaog]

DRA driver has to be updated to advertise devices as 'extended resource'.

That said, I revised the design based on the feedback, thanks for the comments!

A cluster node install either a DRA driver, or a deivce plugin driver for a given named resource. Devices are picked at the scheudling time, and communicated to kubelet and DRA driver through the special resource claim.

[pohly]

No, I think you misunderstood (or I did). As I read it, the allocations are made by the scheduler and stored in the special singleton resource claim, not decided by the driver.

That's a bit different from what I understood. If it's still the scheduler which picks devices and tells the driver about it, then it's fine.

From #5136 (review):

When you and I talked on Friday, we discussed allowing kubelet to remain unchanged. Instead, it would call the Device Plugin grpc API for those with extended resources, but since that grpc API would be handled by the DRA driver, it would know when receiving calls on that API, it should look for the special resource claim.

That probably goes back to my comment above: "I also very much dislike that DRA drivers need to be updated at all."

Why should we put additional work on all DRA drivers, now and in the future, if we can instead do something once in the kubelet? It has implications for the graduation of this feature, but should that be a deciding factor?

I know that I've said that we want to keep the kubelet as dumb as possible, but in this case I think it's simpler overall to do this in the kubelet. I'm also a bit worried about the implications of skipping some of the usual admission checks that the kubelet does for claims, like "reserved for". Perhaps that doesn't matter, but then we need to explain in the KEP why.

[yliaog]

My initial thoughts were to minimize kubelet changes, and shift the work to device driver. But after some more thoughts, I agree it's better to do the work once, and great in kubelet, so device driver can have less work to do, and less chances to implement it wrongly.

[pohly]

/cc

[pohly]

This pushes the responsibility of choosing a device back into the DRA driver. I don't think this is the right direction from an architectural perspective.

cc @klueska

I also very much dislike that DRA drivers need to be updated at all.

[johnbelamaric]

/approve

[pohly]

/hold

For SIG Scheduling approval (currently has approval through @johnbelamaric for PRR, but not from the SIG), and for squashing into one commit before merging.

[johnbelamaric]

@pohly thanks for the hold, I should have waited on the approve :)

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johnbelamaric, yliaog

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• keps/prod-readiness/OWNERS [johnbelamaric]
• keps/sig-scheduling/OWNERS [johnbelamaric]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[yliaog]

/assign @dom4ha

[yliaog]

/assign @klueska

[yliaog]

/assign @thockin

[johnbelamaric]

/assign @liggitt

[liggitt]

Are there other references to the original design(s) for extended resources that would be helpful to reference? either KEPs or original designs if they were before KEPs?

A quick search in the KEPs folder showed these, I wasn't sure if they were related or not:

• https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2403-pod-resources-allocatable-resources
• https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/1819-scheduler-extender/README.md?plain=1#L153

[liggitt]

1. Does that mean the scheduler needs to already know which node the pod will go to before it creates the resourceclaim, to know whether the node is providing the extended resource via a legacy driver or a DRA driver?
2. Is the scheduler able to run its full node selection algorithm in the absence of this resource claim, then create it and satisfy it, without invalidating already-run parts of the scheduler pipeline?
3. Does this require relocating the existing handling of extended resource matching in the scheduler?
4. Consider how the selection / resourceclaim creation approach will work in the scheduler when there are multiple extended resources requested, some of which use DRA and some of which use legacy device plugins.
5. Describe and test the failure / recovery / cleanup path if the scheduler succeeds in creating some ResourceClaim objects for a pod's extended resources and fails to create others.

[liggitt]

plural, since there can be more than one?

Suggested change

	1. Introduce a field `ExtendedResourceClaimStatus` to pod's `Status`, such that:
	1. Introduce a field `ExtendedResourceClaimStatuses` to pod's `Status`, such that:

[liggitt]

is it correct to assume that a single resourceclaim is sufficient for the extended resources requested by a pod?

a pod requesting a GPU and a high bandwidth network device could have two extended resources, satisfied by two different DRA drivers ... is it normal to batch those requests into a single resource claim?

[liggitt]

I can't recall, are container names guaranteed unique across container types (initContainers, containers, ephemeralContainers)?

[liggitt]

Suggested change

	annotation `resource.kubernetes.io/extened-resource-claim: pod-name`.
	annotation `resource.kubernetes.io/extended-resource-claim: pod-name`.

[liggitt]

Does it do this check using its local informer-fed cache? if so, be aware that could be stale and might not have observed a resourceclaim it created in a previous loop.

Describe how the scheduler recovers from a scenario where two resourceclaims like this incorrectly created for the same pod?

[liggitt]

if the resource claim already exists, it will have been created for a specific node, right? does this step have to make sure the node the claim was for is still in the set of viable nodes here? if it isn't, what happens here?

[liggitt]

Clarify if the "create" mentioned here means "create in the API" or "create in-memory"? Since ResourceClaimSpec is immutable in the API on update, I assume it means in-memory in the scheduler. That means the scheduler bookkeeping needs to know if the claim is persisted yet or not / mutable or not

[liggitt]

Also, the scheduler can encounter scenarios where it creates the resourceclaim object, then a subsequent step fails, it has to try again later, and in the meantime, DeviceClass mappings to extended resources have changed.

Would having one resourceclaim per {pod, extended resource type} combination make this situation easier to deal with?

[liggitt]

is this manipulating a separate copy of the special claim per node (since whether to use DRA for the extended resource requests depends on whether the particular node in consideration has device plugins for those extended resources or not)?

[liggitt]

It looks like that code assumes a single extended resource referenced from a pod, which I don't think is correct. I agree it would be better to limit this to informing what is in the claimsToAllocate list handled inside Allocate(). That would need to become node-specific, but would be far more contained, I think.

Manipulating / needing to deepcopy a separate copy of a single special claim per node seems problematic (since whether to use DRA for the extended resource requests depends on whether the particular node in consideration has device plugins for a particular combination of extended resources or not)

This seems like another place where partitioning claims by extended resource type would help:

1. in PreFilter, for every DRA-backed extended resource a pod references, make an in-memory ResourceClaim for that extended resource
2. when filtering nodes in DynamicResources#Filter, compute claimsToAllocate inside Allocate() by adding in the in-memory claims for resources the node doesn't advertise legacy capacity for
3. when filtering nodes in Fit#Filter, allow fitsRequest to ignore DRA-backed extended resources the node doesn't advertise legacy capacity for
4. in Reserve, create the spec and update the status for the in-memory ResourceClaims for the extended resources the selected node did not advertise capacity for