helm support for sidecar injection in EPP

[capri-xiyue]

What type of PR is this?

What this PR does / why we need it:
see #1778

Which issue(s) this PR fixes:

Fixes #1778

Does this PR introduce a user-facing change?:

NONE

[netlify]

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	fd9cdea
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/691781c63eca46000871b2e7
😎 Deploy Preview	https://deploy-preview-1821--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[kfswain]

Thanks @capri-xiyue! Can we make this a distinct helm chart? Since we call this current chart inferencePool it would be odd to have a mode in that chart that doesn't even use inferencePool.

[capri-xiyue]

@kfswain For this stage, EPP standalone mode still uses InferencePool. That's why I didn't put it in a distinct chart. Later, if we decide to have a standalone EPP without any k8s crd, I will refactor it to a distinct helm chart.

To clarify, this PR just removes gateway api dependency, inferencepool api is still used.

Thanks @capri-xiyue! Can we make this a distinct helm chart? Since we call this current chart inferencePool it would be odd to have a mode in that chart that doesn't even use inferencePool.

[capri-xiyue]

/assign @ahg-g

[kfswain]

@kfswain For this stage, EPP standalone mode still uses InferencePool. That's why I didn't put it in a distinct chart. Later, if we decide to have a standalone EPP without any k8s crd, I will refactor it to a distinct helm chart.

Thanks @capri-xiyue! Can we make this a distinct helm chart? Since we call this current chart inferencePool it would be odd to have a mode in that chart that doesn't even use inferencePool.

That may lead to confusing UX in the case where this is deployed in a cluster that has a GW controller, as it will attempt to reconcile on the InferencePool and integrate it into the GW system, is modifying EPP to just accept a selector not a viable path forward?

[capri-xiyue]

@kfswain For this stage, EPP standalone mode still uses InferencePool. That's why I didn't put it in a distinct chart. Later, if we decide to have a standalone EPP without any k8s crd, I will refactor it to a distinct helm chart.

Thanks @capri-xiyue! Can we make this a distinct helm chart? Since we call this current chart inferencePool it would be odd to have a mode in that chart that doesn't even use inferencePool.

That may lead to confusing UX in the case where this is deployed in a cluster that has a GW controller, as it will attempt to reconcile on the InferencePool and integrate it into the GW system, is modifying EPP to just accept a selector not a viable path forward?

I talked with @ahg-g before, modifying EPP to just accept a selector needs further discussion. Therefore he suggested me finalizing EPP with envoy proxy first with helm chart.

Curious now what will happen when a inference pool deployed in a cluster with two GW controller?(for example kgateway and istio), will it cause issues here? Initially I thought each GW controller is able to handle this case.

[ahg-g]

Yeah, my suggestion is to take a gradual approach, a gateway controller should not care about an inferencePool that is not referenced by an httpRoute.

[capri-xiyue]

As an update, I'm now working on another PR to modify EPP to just accept a selector and will refactor the helm chart to have a distinct one as no inference pool is needed in that PR. EPP refactor probably takes a little while as fix bunch of ut takes time. Will let you know when I send a PR. @kfswain

[danehans]

IMHO we need to get the #1778 design gdoc that @ahg-g created into an md and added to docs/proposals.

[capri-xiyue]

With that being said, I agree with the point of splitting the vendor specific to a separate repo provided that we offer the necessary "hooks" to enable that. So my suggestion is to make the necessary changes in this chart to enable customizing it by an external one to allow those other deployment patterns.

@capri-xiyue isn't all needed here to enable the above is to allow adding a sidecar proxy to the epp deployment?

Yes. I've updated it to enable additional sidecar proxy injection.

[ahg-g]

Great!

@danehans can you please make sure this works for agentgateway?

[ahg-g]

What assumptions are we making in this configuration with regards to the EPP configuration, the configuration of the model servers and inferencePool? Put another way, is this configuration assuming that the epp/servers/infoPool is configuration in a specific way?

[capri-xiyue]

The example envoy config only assumes the communication between envoy proxy and epp communication. It has nothing to do with model server and inference pool. By the way, such envoy config can be customized by users. Users can use what ever config they want to use.

[ahg-g]

Ideally we want a config that just works, and the helm chart we offer should not even offer any customization for this config.

[capri-xiyue]

Yep. The current default config works as I just disable side car injection. By the way, originally the example envoy config was commented already. Now I just removed it as it will live in llm-d

[ahg-g]

The current default config works as I just disable side car injection

I was referring to the envoy config, that config in llm-d should just work and not be customizable.

[capri-xiyue]

Yep. The envoy config in llm-d will just work and won't be customizable. I tested it with no inference pool dependency. Will ping you when I open a PR in llm-d

[ahg-g]

/lgtm
/approve

Great!

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g, capri-xiyue

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ahg-g]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment