Skip to content

Conversation

@christian-pinto
Copy link
Member

@christian-pinto christian-pinto commented Jan 23, 2026

This PR adds two new experiments to the vllm_performance actuator:

  • test-endpoint-guidellm-v1
  • test-deployment-guidellm-v1

The new experiments are 100% compatible with the ones based on vLLM bench. Therefore, the same entity space can be used across vllm bench and GuideLLM. Also, the metrics reported are 100% matching.

At this stage, also for GuideLLM we only support a synthetic (random) dataset. Also, the vLLM experiment shave a burstiness argument the controls the distribution used for generating requests. The default value is 1 and it uses a Poisson distribution. GuideLLM does not support setting the burstiness of the requests. For the sake of guaranteeing using the same space across the two experiments I have decided to still have the burstiness argument in the guidellm experiments and forcing a poisson distribution for the requests generation.

This pr implements #457

Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
…m experiments

Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
@DRL-NextGen
Copy link
Member

DRL-NextGen commented Jan 23, 2026

Checks Summary

Last run: 2026-01-27T15:46:31.450Z

Code Risk Analyzer vulnerability scan found 2 vulnerabilities:

Severity Identifier Package Details Fix
◻ Unknown CVE-2025-53000 nbconvert
nbconvert has an uncontrolled search path that leads to unauthorized code execution on WindowsGHSA-xm59-rqc7-hhvf

nbconvert:7.16.6->ado-core:1.3.3
>7.16.6
◻ Unknown CVE-2026-0994 protobuf
protobuf affected by a JSON recursion depth bypassGHSA-7gcm-g887-7qv7

protobuf:6.33.4->guidellm:0.5.3,protobuf:6.33.4,vllm:0.14.1,protobuf:6.33.4,ado-core:1.3.3
>6.33.4

Mend Unified Agent vulnerability scan found 3 vulnerabilities:

Severity Identifier Package Details Fix
❗ Critical CVE-2025-56005 ply-3.11-py2.py3-none-any.whl
An undocumented and unsafe feature in the PLY (Python Lex-Yacc) library 3.11 allows Remote Code Exec...An undocumented and unsafe feature in the PLY (Python Lex-Yacc) library 3.11 allows Remote Code Execution (RCE) via the "picklefile" parameter in the "yacc()" function. This parameter accepts a ".pkl" file that is deserialized with "pickle.load()" without validation. Because "pickle" allows execution of embedded code via "reduce()", an attacker can achieve code execution by passing a malicious pickle file. The parameter is not mentioned in official documentation or the GitHub repository, yet it is active in the PyPI version. This introduces a stealthy backdoor and persistence risk.
Not Available
🔺 High CVE-2025-53000 nbconvert-7.16.6-py3-none-any.whl
The nbconvert tool, jupyter nbconvert, converts Jupyter notebooks to various other formats via Jinja...The nbconvert tool, jupyter nbconvert, converts Jupyter notebooks to various other formats via Jinja templates. Versions of nbconvert up to and including 7.16.6 on Windows have a vulnerability in which converting a notebook containing SVG output to a PDF results in unauthorized code execution. Specifically, a third party can create a "inkscape.bat" file that defines a Windows batch script, capable of arbitrary code execution. When a user runs "jupyter nbconvert --to pdf" on a notebook containing SVG output to a PDF on a Windows platform from this directory, the "inkscape.bat" file is run unexpectedly. As of time of publication, no known patches exist.
Not Available
🔺 High CVE-2026-0994 protobuf-6.33.4-cp39-abi3-manylinux2014_x86_64.whl
A denial-of-service (DoS) vulnerability exists in google.protobuf.json_format.ParseDict() in Python,...A denial-of-service (DoS) vulnerability exists in google.protobuf.json_format.ParseDict() in Python, where the max_recursion_depth limit can be bypassed when parsing nested google.protobuf.Any messages.
Due to missing recursion depth accounting inside the internal Any-handling logic, an attacker can supply deeply nested Any structures that bypass the intended recursion limit, eventually exhausting Python’s recursion stack and causing a RecursionError.
Not Available

Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
…ydantic models usage and improved docs

Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
@michael-johnston
Copy link
Member

@christian-pinto there is a fixable ruff problem in vllm_performance causing CI to fail. Can you pull and fix?

Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
@christian-pinto
Copy link
Member Author

@christian-pinto there is a fixable ruff problem in vllm_performance causing CI to fail. Can you pull and fix?

Just fixed it. It's weird though that the pre-commit hook missed it

Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
christian-pinto and others added 4 commits January 27, 2026 13:06
…ers model

Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Co-authored-by: Alessandro Pomponio <10339005+AlessandroPomponio@users.noreply.github.com>
Signed-off-by: Michael Johnston <66301584+michael-johnston@users.noreply.github.com>
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
@michael-johnston michael-johnston added this pull request to the merge queue Jan 27, 2026
Merged via the queue into main with commit 46ffac8 Jan 27, 2026
19 checks passed
@michael-johnston michael-johnston deleted the cp_add_guidellm_experiment branch January 27, 2026 16:35
@michael-johnston michael-johnston linked an issue Jan 27, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: add guidellm experiment in vllm_performance

4 participants