Add benchmark automation tool

[liu-cong]

This is a tool to automatically generate benchmark manifests deploying vllm model servers and an inference gateway or a k8s service, and runs benchmarks against them. It automatically deploys the resources in a new namespace, collects results, and tears down the resources.

This tool is experimental. I tested this tool using GKE. In theory, it should work on any cluster, though it may require minimal modifications.

EDIT:
This tool currently has many limitations and will require non-trivial improvements to become a stable tool for long term maintenance. In the meantime, I found it very useful and saved me a lot of time, so I'd like to share it out so people may find it useful. I am seeking feedback on whether we should merge it now, or keep it in a branch and start thinking about changes that benefit us in the long term..

Future improvements

• This tool currently has its own helm charts and it requires effort to keep them up to date. It will be nice to have these charts standardized across this repo.
• The tool makes heavy use of bash scripts which is hard to test/maintain long term.
• Improve code structure, readability and testing

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: liu-cong
Once this PR has been reviewed and has the lgtm label, please assign arangogutierrez for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[netlify]

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	0d5e02c
🔍 Latest deploy log	https://app.netlify.com/sites/gateway-api-inference-extension/deploys/67e19c0cc386b000085761ea
😎 Deploy Preview	https://deploy-preview-563--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[kfswain]

/hold

[liu-cong]

cc @kaushikmitr @achandrasekar

[smarterclayton]

This is a lot of bash. What's the principle underpinning how this will evolve? If you need to do analysis of the output, bash will quickly hit limits.

I kind of expected either a go code / go test like structure, or a python / colab style experimentation setup. Agree some of these things are good for bash, but not all of them are.

[liu-cong]

Bash is mostly kubectl commands to set up and tear down the benchmarks, output analysis is done with a Jupyter notebook.

But agree there is quite a lot of bash, and some of it can/should be done with python or go for better readability and testing.

I think a key principle to discuss is the benchmark config API, as well as the benchmark results API (currently implicitly defined by the LPG tool json format). With those defined people can come up with different implementations of automating the benchmark. And bash, or go, or juputer notebook is more of an implementation detail.

[kfswain]

What are we thinking with this one?

[liu-cong]

I think most folks are busy with other things and didn't have time to try this out. I know @kaushikmitr was planning to try this.
Some solid feedback will help us decide whether we want to maintain this.

I am happy to put this to a branch, and defer the decision once we have more concrete feedbacks.