Implement PrepareDataPlugin for prefix cache match plugin #1942

rahulgurnani · 2025-12-04T00:52:32Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR implements PrepareDataPlugin hook for prefix cache match plugin. The goal is to split the prefix cache match plugin into match plugin and scorer.
This is an intermediate step towards migration of the plugin. In a following PR, scorer would be added and the existing prefix cache match plugin would be deprecated.

Did some initial scale testing using vllm:

On this version

============ Serving Benchmark Result ============
Successful requests:                     1000      
Failed requests:                         0         
Benchmark duration (s):                  83.04     
Total input tokens:                      127000    
Total generated tokens:                  2048000   
Request throughput (req/s):              12.04     
Output token throughput (tok/s):         24663.03  
Peak output token throughput (tok/s):    52286.00  
Peak concurrent requests:                1000.00   
Total Token throughput (tok/s):          26192.43  
---------------Time to First Token----------------
Mean TTFT (ms):                          1875.19   
Median TTFT (ms):                        1877.09   
P99 TTFT (ms):                           2368.76   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          31.62     
Median TPOT (ms):                        29.59     
P99 TPOT (ms):                           39.16     
---------------Inter-token Latency----------------
Mean ITL (ms):                           31.62     
Median ITL (ms):                         27.45     
P99 ITL (ms):                            92.53     
==================================================

On head (latest release)

============ Serving Benchmark Result ============
Successful requests:                     1000      
Failed requests:                         0         
Benchmark duration (s):                  101.31    
Total input tokens:                      127000    
Total generated tokens:                  2048000   
Request throughput (req/s):              9.87      
Output token throughput (tok/s):         20215.75  
Peak output token throughput (tok/s):    52972.00  
Peak concurrent requests:                1000.00   
Total Token throughput (tok/s):          21469.36  
---------------Time to First Token----------------
Mean TTFT (ms):                          1885.78   
Median TTFT (ms):                        1851.98   
P99 TTFT (ms):                           2482.30   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          33.57     
Median TPOT (ms):                        32.10     
P99 TPOT (ms):                           48.02     
---------------Inter-token Latency----------------
Mean ITL (ms):                           33.57     
Median ITL (ms):                         27.89     
P99 ITL (ms):                            93.49

@kaushikmitr / @BenjaminBraunDev FYI

Which issue(s) this PR fixes:

Addresses #1924

Does this PR introduce a user-facing change?:

yes. this PR adds a new feature flag "prepareDataPlugins"

k8s-ci-robot · 2025-12-04T00:52:34Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

netlify · 2025-12-04T00:52:37Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`ab82246`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/693a372fb2aa7c0008df86b0
😎 Deploy Preview	https://deploy-preview-1942--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

liu-cong · 2025-12-04T21:26:30Z

pkg/epp/datalayer/plugins/data_types.go

+	PrefixCacheMatchPrecentKey = "PrefixCacheMatchPercentKey"
+)
+
+type PrefixCacheMatchPercent struct {


I suggest making this something more generic like PrefixCacheInfo so we can easily extend. One of things we will likely need to add is the cache tier info.

Instead (or in addition) of providing the percentage, let's provide the match length. The idea is to provide more "raw" data for consumers. Percentage is just something the scorer cares about, the latency predictor may work better with the length.

I renamed it to PrefixCacheInfo to make it easy for extension. Predictor is presently only trained to depend on prefix cache match percent. I think updating this to length may not be suitable in the short term. @BenjaminBraunDev keep me honest here. Thanks!

I think @liu-cong 's opinion carries the most weight here.

The reccomendations here seem reasonable, aggregating data for easier ergonomics of an unproven, experimental plugin, at the expense of overall usability does not seem a worth tradeoff.

I'm not sure why this comment was resolved

Updated the info to contain prefix cache related info. Thanks

rahulgurnani · 2025-12-08T15:08:35Z

/assign @kfswain

k8s-ci-robot · 2025-12-08T18:43:57Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rahulgurnani
Once this PR has been reviewed and has the lgtm label, please ask for approval from kfswain. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

cmd/epp/runner/runner.go

pkg/epp/datalayer/factory.go

pkg/epp/datalayer/plugins/approximateprefix/data_types.go

kfswain · 2025-12-08T22:17:46Z

pkg/epp/requestcontrol/request_control_config.go

 // If a cycle is detected, it returns an error.
-func (c *Config) PrepareDataPluginGraph() error {
+func (c *Config) PrepareDataPluginGraph(enablePrepareDataPlugins bool) error {
+	if !enablePrepareDataPlugins {


Related to the other comment, lets remove this check from here.

Done, thanks!

pkg/epp/scheduling/framework/plugins/multi/prefix/plugin_test.go

pkg/epp/scheduling/framework/plugins/scorer/prefix_cache_match_scorer.go

kfswain · 2025-12-08T22:22:24Z

pkg/epp/scheduling/framework/plugins/scorer/prefix_cache_match_scorer_test.go

+	return keys
+}
+
+func TestPrefixCacheScorer_Score(t *testing.T) {


Can we have a unit test that also creates the legacy prefix scorer, and test to make sure the created scores have identical output? We should be able to validate the behavior is the same in these unit tests

I removed the scorer from this PR since I am not using it yet. I plan to add the scorer back in a separate PR. Scoping this PR to just adding prepare data step so that predictor can use it. Thanks!

pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go

pkg/epp/scheduling/framework/plugins/scorer/prefix_cache_match_scorer.go

pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go

cmd/epp/runner/runner.go

kfswain · 2025-12-09T23:24:32Z

Looks good. Just a minor nit, and a comment to open the issue.

I feel strongly about the issue, however, as that is a key part of ensuring that the P/C data model is reliable. Once you tag the created issue here (and bind it to the milestone), I'm happy to stamp the PR.

Thanks! Looks great!

zetxqx · 2025-12-10T01:07:18Z

pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go

 }

+func (p *Plugin) Produces() map[string]any {
+	return map[string]any{approximateprefix.PrefixCacheMatchInfoKey: approximateprefix.PrefixCacheMatchInfo{}}


Others Looks good, just one question for my understanding. what's the Produces() used for?

here it does not populate any data in PrefixCacheMatchInfo{}. It's only used for constructing the DAG. Are we planning to populate PrefixCacheMatchInfo in produces? And when/how should we populate it?

We do it in PrepareRequestData. Produces is used for DAG validation on EPP startup.

Refer: https://docs.google.com/document/d/1EQwXL2pCuUyM1B917FUgP_8pFS3VF8F_bUfjy8IE7gM/edit?tab=t.vmaefhinvkl5#heading=h.s9mr6kynb3ls for more context. Thanks!

Yeah, Produces output a map[string]any. I get the idea of DAG validation of using the key of the output map. I'm wondering what's the usage of the value here. Currently you put a placeholder approximateprefix.PrefixCacheMatchInfo{}. I'm wondering if it will be populated in the future.

if the Produces output type is meant to be a set, should we change it to a map[string]struct{} or map[string]bool which is more go idiomatic? @rahulgurnani @kfswain

@zetxqx as far as I understand the validation is not only for key existence, but also that the producer output type correlates to the type the consumer wants.

the way it’s done here is by setting an empty struct, and then the validation code can use reflection (or alike) to validate the types match

@zetxqx as far as I understand the validation is not only for key existence, but also that the producer output type correlates to the type the consumer wants.

This is correct, the value is used in reflection on startup to ensure that they value of the key type is expected. Allowing for confidence in type usage, even in an environment where only out of tree plugins are used. So using a set would not allow for a fully reliable dag (a key name may be used with an unexpected type, or force reflection to happen on every plugin call)

I see, thank you all. Non-blocking for this PR, just two points from my understanding of the current code path:

Currently the DAG construction is only checking the Key here:

gateway-api-inference-extension/pkg/epp/requestcontrol/dag.go

Lines 41 to 43 in 31d388d

for producedKey := range plugins[i].Produces() {

// If plugin j consumes the produced key, then j depends on i. We can break after the first match.

if _, ok := plugins[j].Consumes()[producedKey]; ok {

if we only want to do pure type checking not information checking in the type. Should we consider Produces output map[string]reflect.Type or map[reflect.Type]struct{}

That should be fixed, the DAG needs more work, I have other comments in this PR suggesting as such.

You could , it wouldn't really make a material difference other than making the implementers call reflect.TypeOf(pluginProducingType). We will do that all under the hood in the DAG checker anyway. Just depends on what the preferred UX is

I added the validation in this PR itself. Please take another look at the last commit. I think data check is simpler than using reflection. We could document this behavior. Thanks!

rahulgurnani · 2025-12-10T01:14:36Z

Looks good. Just a minor nit, and a comment to open the issue.

I feel strongly about the issue, however, as that is a key part of ensuring that the P/C data model is reliable. Once you tag the created issue here (and bind it to the milestone), I'm happy to stamp the PR.

Thanks! Looks great!

Addressed it in the current PR. Let me know what you think. Thanks!

kfswain · 2025-12-10T18:48:59Z

pkg/epp/requestcontrol/request_control_config.go


 // PrepareDataPluginGraph creates data dependency graph and sorts the plugins in topological order.
 // If a cycle is detected, it returns an error.
 func (c *Config) PrepareDataPluginGraph() error {


The DAG should be built, completely agnostic of if any PrepareData plugins are in use. Will add suggestions to help clarify what I mean

kfswain · 2025-12-10T18:52:05Z

pkg/epp/requestcontrol/request_control_config.go

+	if len(c.prepareDataPlugins) == 0 {
+		return nil
+	}
 	dag := buildDAG(c.prepareDataPlugins)
 	plugins, err := sortPlugins(dag, c.prepareDataPlugins)
 	if err != nil {


Suggested change

if len(c.prepareDataPlugins) == 0 {

return nil

}

dag := buildDAG(c.prepareDataPlugins)

plugins, err := sortPlugins(dag, c.prepareDataPlugins)

if err != nil {

dag := buildDAG(c.allPlugins)

plugins, err := sortPlugins(dag, c.allPlugins)

if err != nil {

This is semi psuedocode. But I hope this kind of illustrates what I mean. We should always create the DAG, entirely agnostic of which plugins are in use.

Ohh. I understand. Added a TODO for it and address it as a fast follow in the next PR. Let me know if that's reasonable. Thanks!

… comments

rahulgurnani · 2025-12-10T20:23:45Z

/retest

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/feature Categorizes issue or PR as related to a new feature. labels Dec 4, 2025

k8s-ci-robot requested review from ahg-g and liu-cong December 4, 2025 00:52

k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Dec 4, 2025

rahulgurnani changed the title ~~Implement PreapreDataPlugin for prefix cache match plugin~~ Implement PrepareDataPlugin for prefix cache match plugin Dec 4, 2025

rahulgurnani force-pushed the pc-m branch from b8c71a8 to d6556ed Compare December 4, 2025 18:19

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 4, 2025

liu-cong reviewed Dec 4, 2025

View reviewed changes

rahulgurnani force-pushed the pc-m branch from c67979f to 5752e58 Compare December 6, 2025 05:15

rahulgurnani requested a review from liu-cong December 6, 2025 05:16

rahulgurnani marked this pull request as ready for review December 6, 2025 05:16

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 6, 2025

k8s-ci-robot requested review from elevran and robscott December 6, 2025 05:16

k8s-ci-robot assigned kfswain Dec 8, 2025

rahulgurnani requested a review from kfswain December 8, 2025 18:51

rahulgurnani force-pushed the pc-m branch from c330833 to ee9a486 Compare December 8, 2025 19:46

rahulgurnani mentioned this pull request Dec 8, 2025

Deprecate existing prefix cache match plugin in favor of new datalayer #1970

Open

kfswain reviewed Dec 8, 2025

View reviewed changes

zetxqx reviewed Dec 8, 2025

View reviewed changes

rahulgurnani force-pushed the pc-m branch from ee9a486 to 598b4b2 Compare December 9, 2025 01:23

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 9, 2025

rahulgurnani force-pushed the pc-m branch from c673022 to 74cf6a5 Compare December 9, 2025 02:03

rahulgurnani force-pushed the pc-m branch 2 times, most recently from 5e8aff8 to bf00276 Compare December 10, 2025 01:04

zetxqx reviewed Dec 10, 2025

View reviewed changes

rahulgurnani requested review from kfswain and zetxqx December 10, 2025 01:14

kfswain reviewed Dec 10, 2025

View reviewed changes

rahulgurnani added 14 commits December 10, 2025 19:53

Add PrepareRequestData method for the prefix cache plugin

6187ace

Add prefix cache match scorer

178632e

Ensure prefix cache plugin implements all preprae data plugin methods

b619766

Enable prepare data plugins behind a feature flag

e100653

Add static check to ensure plugin implements PrepareDataPlugin interface

4bd3d6b

Rename prefix cache match

e7e7eae

Add feature gate for prepare data plugin

0fce568

Update prefix cache match to have total and match length

963d164

Add more tests, register prefix cache scorer and address other review…

1145aea

… comments

Move prefix cache scorer out of this PR

c10dd35

Fix rebase errors

80e07e0

Update directory structure

4a2c02e

Update TODOs

b198efd

Validate type as well in dag validation

05d7565

rahulgurnani force-pushed the pc-m branch from 59e9016 to f4142a8 Compare December 10, 2025 19:54

rahulgurnani requested a review from kfswain December 10, 2025 20:11

rahulgurnani force-pushed the pc-m branch 3 times, most recently from 672ca5e to 1a0c28c Compare December 11, 2025 03:10

Fix rebase errors, add TODO

ab82246

rahulgurnani force-pushed the pc-m branch from 1a0c28c to ab82246 Compare December 11, 2025 03:14

	for producedKey := range plugins[i].Produces() {
	// If plugin j consumes the produced key, then j depends on i. We can break after the first match.
	if _, ok := plugins[j].Consumes()[producedKey]; ok {

Implement PrepareDataPlugin for prefix cache match plugin #1942

Are you sure you want to change the base?

Implement PrepareDataPlugin for prefix cache match plugin #1942

Conversation

rahulgurnani commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Dec 4, 2025

Uh oh!

netlify bot commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rahulgurnani commented Dec 8, 2025

Uh oh!

k8s-ci-robot commented Dec 8, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rahulgurnani Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kfswain commented Dec 9, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfswain Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rahulgurnani commented Dec 10, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rahulgurnani Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rahulgurnani commented Dec 10, 2025

Uh oh!

Reviewers

rahulgurnani commented Dec 4, 2025 •

edited

Loading

netlify bot commented Dec 4, 2025 •

edited

Loading

rahulgurnani Dec 9, 2025 •

edited

Loading

kfswain Dec 10, 2025 •

edited

Loading

rahulgurnani Dec 10, 2025 •

edited

Loading