-
Notifications
You must be signed in to change notification settings - Fork 1
feat(vllm_performance): Add GuideLLM experiments #459
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
…m experiments Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
|
Checks Summary Last run: 2026-01-27T15:46:31.450Z Code Risk Analyzer vulnerability scan found 2 vulnerabilities:
Mend Unified Agent vulnerability scan found 3 vulnerabilities:
|
...rformance/ado_actuators/vllm_performance/vllm_performance_test/execute_guidellm_benchmark.py
Outdated
Show resolved
Hide resolved
...rformance/ado_actuators/vllm_performance/vllm_performance_test/execute_guidellm_benchmark.py
Outdated
Show resolved
Hide resolved
...rformance/ado_actuators/vllm_performance/vllm_performance_test/execute_guidellm_benchmark.py
Outdated
Show resolved
Hide resolved
...rformance/ado_actuators/vllm_performance/vllm_performance_test/execute_guidellm_benchmark.py
Outdated
Show resolved
Hide resolved
...rformance/ado_actuators/vllm_performance/vllm_performance_test/execute_guidellm_benchmark.py
Outdated
Show resolved
Hide resolved
...rformance/ado_actuators/vllm_performance/vllm_performance_test/execute_guidellm_benchmark.py
Show resolved
Hide resolved
...rformance/ado_actuators/vllm_performance/vllm_performance_test/execute_guidellm_benchmark.py
Show resolved
Hide resolved
...rformance/ado_actuators/vllm_performance/vllm_performance_test/execute_guidellm_benchmark.py
Show resolved
Hide resolved
plugins/actuators/vllm_performance/ado_actuators/vllm_performance/experiment_executor.py
Outdated
Show resolved
Hide resolved
plugins/actuators/vllm_performance/GUIDELLM_PARAMETER_MAPPING.md
Outdated
Show resolved
Hide resolved
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
plugins/actuators/vllm_performance/ado_actuators/vllm_performance/experiment_executor.py
Outdated
Show resolved
Hide resolved
plugins/actuators/vllm_performance/ado_actuators/vllm_performance/experiment_executor.py
Outdated
Show resolved
Hide resolved
...rformance/ado_actuators/vllm_performance/vllm_performance_test/execute_guidellm_benchmark.py
Outdated
Show resolved
Hide resolved
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
…ydantic models usage and improved docs Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
|
@christian-pinto there is a fixable ruff problem in vllm_performance causing CI to fail. Can you pull and fix? |
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Just fixed it. It's weird though that the pre-commit hook missed it |
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
...rs/vllm_performance/ado_actuators/vllm_performance/vllm_performance_test/benchmark_models.py
Outdated
Show resolved
Hide resolved
...rs/vllm_performance/ado_actuators/vllm_performance/vllm_performance_test/benchmark_models.py
Outdated
Show resolved
Hide resolved
...ors/vllm_performance/ado_actuators/vllm_performance/vllm_performance_test/guidellm_models.py
Outdated
Show resolved
Hide resolved
...ors/vllm_performance/ado_actuators/vllm_performance/vllm_performance_test/guidellm_models.py
Outdated
Show resolved
Hide resolved
...ors/vllm_performance/ado_actuators/vllm_performance/vllm_performance_test/guidellm_models.py
Outdated
Show resolved
Hide resolved
...ors/vllm_performance/ado_actuators/vllm_performance/vllm_performance_test/guidellm_models.py
Outdated
Show resolved
Hide resolved
...ors/vllm_performance/ado_actuators/vllm_performance/vllm_performance_test/guidellm_models.py
Outdated
Show resolved
Hide resolved
...ors/vllm_performance/ado_actuators/vllm_performance/vllm_performance_test/guidellm_models.py
Outdated
Show resolved
Hide resolved
...ors/vllm_performance/ado_actuators/vllm_performance/vllm_performance_test/guidellm_models.py
Outdated
Show resolved
Hide resolved
plugins/actuators/vllm_performance/ado_actuators/vllm_performance/experiment_executor.py
Outdated
Show resolved
Hide resolved
…ers model Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Co-authored-by: Alessandro Pomponio <10339005+AlessandroPomponio@users.noreply.github.com> Signed-off-by: Michael Johnston <66301584+michael-johnston@users.noreply.github.com>
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
...rs/vllm_performance/ado_actuators/vllm_performance/vllm_performance_test/benchmark_models.py
Outdated
Show resolved
Hide resolved
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
This PR adds two new experiments to the vllm_performance actuator:
The new experiments are 100% compatible with the ones based on vLLM bench. Therefore, the same entity space can be used across vllm bench and GuideLLM. Also, the metrics reported are 100% matching.
At this stage, also for GuideLLM we only support a synthetic (random) dataset. Also, the vLLM experiment shave a burstiness argument the controls the distribution used for generating requests. The default value is 1 and it uses a Poisson distribution. GuideLLM does not support setting the burstiness of the requests. For the sake of guaranteeing using the same space across the two experiments I have decided to still have the
burstinessargument in the guidellm experiments and forcing a poisson distribution for the requests generation.This pr implements #457