Add DataProducer PrepareData and Admission control plugins #1796

rahulgurnani · 2025-10-31T16:22:18Z

Add PrepareData and AdmitRequest plugins based on recommendations in evolving datalayer changes

The prepare data plugins are executed in the order such that the plugin dependent on other plugins waits for the execution of other plugins to complete.
Furthermore, the dependency graph (DAG) of data producers and consumers are validated at the startup for cycles. If there is a cycle, the startup fails with error.

The PR also does some refactor of director.go code.

cc @BenjaminBraunDev

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR is needed to make it easier to implement plugins that produce data consumed by other plugins. For instance latency predictor, prefix match plugin etc.

Read the doc evolving datalayer changes
for more details

Which issue(s) this PR fixes:

Addresses #1743

Does this PR introduce a user-facing change?:

Yes, enables writing prepare data and admit request plugins for users of IGW.

Add prepare data and admit request plugins

netlify · 2025-10-31T16:22:24Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`b85113f`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/6912784e6fab4300074d2679
😎 Deploy Preview	https://deploy-preview-1796--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

k8s-ci-robot · 2025-10-31T16:22:26Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rahulgurnani
Once this PR has been reviewed and has the lgtm label, please assign nirrozenbaum for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-10-31T16:22:29Z

Hi @rahulgurnani. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

kfswain · 2025-10-31T20:50:01Z

/ok-to-test

kfswain · 2025-10-31T20:11:13Z

pkg/epp/requestcontrol/director.go

+	loggerDebug := log.FromContext(ctx).V(logutil.DEBUG)
+	for _, plugin := range d.requestControlPlugins.admitRequestPlugins {
+		loggerDebug.Info("Running AdmitRequest plugin", "plugin", plugin.TypedName())
+		if !plugin.Admit(ctx, request, pods) {


We should allow the plugin to return a string explaining the reason for rejection. We can then just treat the empty string as the allow mechanism (less opinionated on this part tho).

Updated, thanks!

pkg/epp/datalayer/attributemap.go

pkg/epp/requestcontrol/director.go

pkg/epp/requestcontrol/plugins.go

kfswain · 2025-10-31T20:45:42Z

pkg/epp/scheduling/types/types.go

+
+// Attributes provides a goroutine-safe implementation of AttributeMap.
+type Attributes struct {
+	data sync.Map // key: attribute name (string), value: attribute value (opaque, Cloneable)


Typically I am all for using prebuild libs to handle this type of complexity.

But since writes to specific attributes will lock the entire data object, we may have high lock contention here. Did we explore having a lock per attribute key?

That would let locks be at the granularity of a specific endpoint & a specific attribute, which should hopefully reduce lock contention & let our system be more performant.

Thanks for the suggestion. This attribute map is a per request copy where we take a snapshot of the attributes so that we can use them in the scheduling layer. Given the per request nature of the map, I think it won't have contention because it will take like t < microsecond to update the map. I think its reasonable to use sync map here.

Can we scale test to ensure we don't have any regression? We can consider baseline metrics what we have here: #1458

I plan to do this in next PRs since we are not actually using the map in this change. Thanks!

pkg/epp/scheduling/types/types.go

pkg/epp/requestcontrol/director_test.go

pkg/epp/requestcontrol/director.go

pkg/epp/scheduling/types/types.go

pkg/epp/requestcontrol/plugins.go

nirrozenbaum · 2025-11-03T12:14:44Z

pkg/epp/requestcontrol/director.go

 		return reqCtx, errutil.Error{Code: errutil.ServiceUnavailable, Msg: "failed to find candidate pods for serving the request"}
 	}
-
+	// TODO(rahulgurnani/lukevandrie): Perhaps, refactor/implement Admit plugin for Admission control.


this comment is important. what is the relation between admission controller Admit and the admitRequest plugins?

++

This would be a break in contract of how flow control operates. This specific plugin is for request specific semantics. We have currently do not have Flow Control considering request specific semantics, and there hasnt been a proposal suggesting this change. I think we should remove this todo until we have strong reasoning to actually do this work.

my apologize, but I don't understand the intention of this new Admission plugin.
I thought we want to have admission check pluggable, but it seems now that we have two types of admission checks, with two different interfaces.
this seems wrong.

Removed the comment. Thanks for the catch!

pkg/epp/requestcontrol/director.go

nirrozenbaum · 2025-11-03T12:24:06Z

pkg/epp/requestcontrol/director.go

-	result, err := d.scheduler.Schedule(ctx, reqCtx.SchedulingRequest, d.toSchedulerPodMetrics(candidatePods))
+	// Prepare per request data
+	// TODO(rahulgurnani): Add retries and timeout in the preparedata step.
+	d.runPrepareDataPlugins(ctx, reqCtx.SchedulingRequest, snapshotOfCandidatePods)


why do we create snapshot of candidate pods?
we should work with candidatePods and create a snapshot only when calling the scheduler.
this is true for both prep data and admit request.
helper functions in the director should not rely on the internal scheduler representation of the endpoints.

both prep data and admit request are request specific, and so if we add request specific data to the shared endpoints that could risk data corruption.

Snapshotting before these steps ensures that this data lifecycle is only in the context it is consumed in.

the intention of converting the endpoints representation to scheduler internal structure was only for the purpose of sending it to the scheduler.
PodMetrics has MetricsState behind atomic pointer and reading the metrics is an atomic operation (read all metrics in one operation).
I must be missing something although I've read the proposal doc.

pkg/epp/backend/metrics/types.go

…timeout logic back since it got removed in previous commit.

rahulgurnani added 2 commits October 30, 2025 17:47

Refactor director to split into smaller functions

9cfe65f

Add AdmitRequest and PrepareData plugins

dfea9cf

k8s-ci-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Oct 31, 2025

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Oct 31, 2025

k8s-ci-robot requested review from ahg-g and elevran October 31, 2025 16:22

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Oct 31, 2025

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 31, 2025

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 31, 2025

kfswain reviewed Oct 31, 2025

View reviewed changes

pkg/epp/requestcontrol/director.go Outdated Show resolved Hide resolved

Add unit tests and comments

bf49fb5

rahulgurnani force-pushed the preparedata-changes branch from 63208cf to bf49fb5 Compare October 31, 2025 22:56

Add comments

d6c81d9

nirrozenbaum reviewed Nov 2, 2025

View reviewed changes

pkg/epp/requestcontrol/director.go Outdated Show resolved Hide resolved

rahulgurnani force-pushed the preparedata-changes branch 2 times, most recently from e9f0b7f to 3e16069 Compare November 3, 2025 03:05

Address review comments

e3021c7

rahulgurnani force-pushed the preparedata-changes branch from 3e16069 to e3021c7 Compare November 3, 2025 03:13

nirrozenbaum reviewed Nov 3, 2025

View reviewed changes

pkg/epp/scheduling/types/types.go Outdated Show resolved Hide resolved

nirrozenbaum reviewed Nov 3, 2025

View reviewed changes

pkg/epp/requestcontrol/plugins.go Outdated Show resolved Hide resolved

nirrozenbaum reviewed Nov 3, 2025

View reviewed changes

pkg/epp/requestcontrol/plugins.go Outdated Show resolved Hide resolved

nirrozenbaum reviewed Nov 3, 2025

View reviewed changes

pkg/epp/requestcontrol/director.go Outdated Show resolved Hide resolved

nirrozenbaum reviewed Nov 3, 2025

View reviewed changes

rahulgurnani changed the title ~~Add DataProducer (PrepareData) and Admission control plugins~~ [WIP] Add DataProducer (PrepareData) and Admission control plugins Nov 7, 2025

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 7, 2025

rahulgurnani force-pushed the preparedata-changes branch 2 times, most recently from a9976c9 to cf26496 Compare November 8, 2025 00:40

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 8, 2025

rahulgurnani force-pushed the preparedata-changes branch from cf26496 to 86dd466 Compare November 8, 2025 01:13

rahulgurnani changed the title ~~[WIP] Add DataProducer (PrepareData) and Admission control plugins~~ Add DataProducer (PrepareData) and Admission control plugins Nov 8, 2025

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 8, 2025

rahulgurnani force-pushed the preparedata-changes branch 4 times, most recently from 853ef84 to 0b0b7b7 Compare November 9, 2025 01:48

rahulgurnani requested a review from nirrozenbaum November 9, 2025 02:56

rahulgurnani changed the title ~~Add DataProducer (PrepareData) and Admission control plugins~~ Add DataProducer PrepareData and Admission control plugins Nov 9, 2025

rahulgurnani force-pushed the preparedata-changes branch from 0b0b7b7 to b5b3e23 Compare November 9, 2025 03:06

Sort plugins based on dependencies

e34b49c

rahulgurnani force-pushed the preparedata-changes branch from b5b3e23 to e34b49c Compare November 10, 2025 04:42

Validate dependency graph for prepare data plugins

f89bb96

nirrozenbaum reviewed Nov 10, 2025

View reviewed changes

pkg/epp/backend/metrics/types.go Outdated Show resolved Hide resolved

rahulgurnani requested a review from nirrozenbaum November 10, 2025 16:09

rahulgurnani force-pushed the preparedata-changes branch 2 times, most recently from 15f6758 to 4cc246d Compare November 10, 2025 21:11

Cleanup dead code, also add some more comments

1be7468

rahulgurnani force-pushed the preparedata-changes branch from 4cc246d to db8f9f7 Compare November 10, 2025 23:14

Cleanup dead code, ensure all preparedata plugins execute

c8b6ba5

rahulgurnani force-pushed the preparedata-changes branch 2 times, most recently from 404a257 to f28020a Compare November 10, 2025 23:38

Implement error propagation for prepare data plugins. Also add retry …

b85113f

…timeout logic back since it got removed in previous commit.

rahulgurnani force-pushed the preparedata-changes branch from f28020a to b85113f Compare November 10, 2025 23:42

Add DataProducer PrepareData and Admission control plugins #1796

Are you sure you want to change the base?

Add DataProducer PrepareData and Admission control plugins #1796

Conversation

rahulgurnani commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

k8s-ci-robot commented Oct 31, 2025

Uh oh!

k8s-ci-robot commented Oct 31, 2025

Uh oh!

kfswain commented Oct 31, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rahulgurnani Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nirrozenbaum Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfswain Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rahulgurnani commented Oct 31, 2025 •

edited

Loading

netlify bot commented Oct 31, 2025 •

edited

Loading

rahulgurnani Nov 3, 2025 •

edited

Loading

nirrozenbaum Nov 3, 2025 •

edited

Loading

kfswain Nov 3, 2025 •

edited

Loading