Feat/sync upstream22012026 by nguyenhoangthuan99 · Pull Request #22 · janhq/verl

nguyenhoangthuan99 · 2026-01-22T07:28:58Z

What does this PR do?

Add concise overview of what this PR aims to achieve or accomplish. Reference related GitHub issues and PRs that help with the review.

Checklist Before Starting

Search for similar PRs. Paste at least one query link here: ...
Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
- {modules} include fsdp, megatron, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data, cfg, reward
- If this PR involves multiple modules, separate them with , like [megatron, fsdp, doc]
- {type} is in feat, fix, refactor, chore, test
- If this PR breaks any API (CLI arguments, config, function signature, etc.), add [BREAKING] to the beginning of the title.
- Example: [BREAKING][fsdp, megatron] feat: dynamic batching

Test

For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc.

API and Usage Example

Demonstrate how the API changes if any, and provide usage example(s) if possible.

# Add code snippet or script demonstrating how to use this

Design & Code Changes

Demonstrate the high-level design if this PR is complex, and list the specific changes.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

Read the Contribute Guide.
Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
Add / Update the documentation.
Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why: ...
Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace. (If not accessible, please try the Feishu group (飞书群).)

…erl-project#4420) ### What does this PR do? As title. ### Checklist Before Starting - [ ] Search for similar PRs. Paste at least one query link here: ... - [ ] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [ ] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).)

…#4422) ### What does this PR do? Retires the legacy SGLang SPMD rollout path and makes async/server mode the only supported backend for SGLang. The PR removes the old `SGLangRollout` class, its helpers, tests, and recipes, and updates all docs, scripts, and CI references so they speak only to the async HTTP adapter (`ServerAdapter`). ### Checklist Before Starting - [ ] Search for similar PRs. Paste at least one query link here: _N/A (follow-up to the vLLM SPMD removal)._ - [ ] Format the PR title as `[sglang, rollout, trainer, recipe, ci, doc] refactor: remove SGLang SPMD rollout` ### Test ### API and Usage Example Same as vLLM: configs/scripts must set `actor_rollout_ref.rollout.mode=async` and rely on the HTTP server. Example: ```bash python -m verl.trainer.main_ppo \ ... \ actor_rollout_ref.rollout.name=sglang \ actor_rollout_ref.rollout.mode=async \ ... ``` ### Design & Code Changes - Deleted the `SGLangRollout` class and associated helpers from `verl/workers/rollout/sglang_rollout/sglang_rollout.py`, keeping only the async `ServerAdapter`. Cleared its registry entries, configs, and guards the same way as the vLLM PR. - Removed SGLang SPMD-specific tests (`tests/workers/rollout/test_sglang_*`) and CI steps in `.github/workflows/sgl.yml`, plus any lint exclusions that referenced those files. - Updated recipes/examples/e2e scripts that referenced SGLang rollout to hardcode `rollout.mode=async`, drop sync branches, and set `return_raw_chat` (mirroring the vLLM cleanup). ### Checklist Before Submitting - [ ] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting). - [x] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [x] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: _Removed the obsolete SGLang SPMD jobs; async workflows remain covered._ - [ ] Once your PR is ready for CI, notify the `ci-request` channel (or Feishu group).

…ct#4406) ### What does this PR do? This PR optimizes Qwen3-MoE model training performance on Ascend NPU devices. It optimizes the implementation of **GMM (Grouped Matmul)** and integrates fused **permute/unpermute** kernels, achieving a 20%+ training speedup. Key changes: 1. Added NPU GMM kernel for backward `dw`. 2. Added `npu_moe_token_permute` and `npu_moe_token_unpermute` fused kernels. 3. Unified GMM function for Qwen3-VL and Qwen3-MoE. 4. Reduced transpose operators in expert weight stacking. ### Checklist Before Starting - [ ] Search for similar PRs. Paste at least one query link here: verl-project#3221 - [ ] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. Tested with Qwen3-30B-A3B (FSDP, sp=8) on 64 Ascend A2 NPUs. Baseline: ![img_v3_02sj_639d375c-ef1e-4e3f-9c3e-afa1c6696feg](https://github.com/user-attachments/assets/87615508-13c1-4cdf-96c0-e1ee09fa99d1) With optimized fusion kernels: ![img_v3_02sj_401d607a-5bb0-42d8-bf07-af47a06cfd4g](https://github.com/user-attachments/assets/6a2f2ca7-e741-416b-b892-234b75b0f809) <!DOCTYPE html><p cid="n349" mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0.8em 0px; white-space: pre-wrap; position: relative; caret-color: rgb(51, 51, 51); color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", sans-serif; font-size: 16px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration: none;">Performance results (step 1):<figure class="md-table-fig" cid="n350" mdtype="table" style="box-sizing: border-box; margin: 1.2em 0px; overflow-x: auto; max-width: calc(100% + 16px); padding: 0px; cursor: default; caret-color: rgb(51, 51, 51); color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", sans-serif; font-size: 16px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration: none;"> Experiment | gen(s) | old_log_prob(s) | update_actor(s) | step(s) -- | -- | -- | -- | -- Baseline | 1180.1 | 71.3 | 152.5 | 1406.9 Fused (This PR) | 1167.5 | 58.6 | 111.1 | 1340.9 </figure> **Precision comparison:** <img width="664" height="360" alt="image" src="https://github.com/user-attachments/assets/5e6629c4-31b5-49ec-97cf-d5e4e6beb69c" /> ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [x] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [x] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [x] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).)

### What does this PR do? As per title. ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: ... - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).)

### What does this PR do? > Add **concise** overview of what this PR aims to achieve or accomplish. Reference related GitHub issues and PRs that help with the review. With an increasing number of models on verl being adapted to Ascend NPUs, along with the upgrade of the transformers version (currently `v4.57.3`), the content in the `npu_patch.py` has grown significantly. Its organization has gradually become cluttered, lacking a unified naming format, and a small portion of the patches have already become obsolete (the original functions no longer exist). This PR aims to address the above issues, ensuring that `npu_patch.py` remains clean and well-organized. ### Checklist Before Starting - [ ] Search for similar PRs. Paste at least one query link here: ... - [ ] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. Not related. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. Not related. ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. Not related. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [ ] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).)

…-project#4430) ### What does this PR do? Prevents background event-loop leaks in the MCP tool loader. `initialize_tools_from_config` now lazily spawns the auxiliary asyncio loop only when an MCP tool is present, and always stops/closes the loop on exit so the worker process doesn’t hold onto threads or loop resources. ### Checklist Before Starting - [ ] Search for similar PRs. Paste at least one query link here: _N/A (targeted fix for the MCP tool registry)._ - [ ] Format the PR title as `[tool, misc] fix: clean up MCP tool event loop` ### Test Not applicable (behavioral fix in tool initialization; existing tool-based tests still cover the code path). ### API and Usage Example No API surface changes; the existing YAML config flow stays the same. Example usage: ```python from verl.tools.utils.tool_registry import initialize_tools_from_config tool_instances = initialize_tools_from_config("configs/tools.yaml") ``` ### Design & Code Changes - Replaced the eager `asyncio.new_event_loop()` creation with a lazy `get_mcp_event_loop()` helper so purely native tool configs no longer spawn threads. - Simplified coroutine execution to always go through the lazily-initialized loop. - Added robust cleanup in the `finally` block: stop the loop, join the thread, and call `loop.close()` to release resources (fixing the leak). ### Checklist Before Submitting - [ ] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting). - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows). If not feasible, explain why: _Tool loading already covered by existing smoke tests._ - [ ] Once your PR is ready for CI, notify the `ci-request` channel (or Feishu group).

…icy distillation recipe (verl-project#3975) ### What does this PR do? This PR provides a simple implementation of one and two step off async knowledge distillation with megatron and vllm backend. ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: ... - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [x] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) --------- Co-authored-by: funrunding <furunding@163.com>

…roject#4444) ### What does this PR do? - Optimize the performance of index_select_tensor_dict by unbind first then index. Results on chat count task <img width="360" height="253" alt="image" src="https://github.com/user-attachments/assets/cdc611e4-6006-4bc2-85a1-a49ec5f63b37" /> <img width="363" height="254" alt="image" src="https://github.com/user-attachments/assets/0a89cbc0-325f-4f00-b986-1fac75a74aa5" /> There is still a gap and we need to keep investigation ### Checklist Before Starting - [ ] Search for similar PRs. Paste at least one query link here: ... - [ ] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [ ] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).)

…#4445)

verl-project#4398) ### What does this PR do? > Add **concise** overview of what this PR aims to achieve or accomplish. Reference related GitHub issues and PRs that help with the review. During the rollout process, this configuration will be read from RolloutConfig. If it cannot be read, the default value is 1.0. However, if you try to configure this parameter in the YAML file, an error will occur because there is no such member in RolloutConfig. <img width="1040" height="749" alt="image" src="https://github.com/user-attachments/assets/921eb8ae-35b6-49db-a22c-d40f31e6f59b" /> ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: ... - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [x] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [x] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [x] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) --------- Co-authored-by: Li Zuming <lizuming@huawei.com>

…isagg mode (verl-project#4433) ### What does this PR do? This PR provides a new feature on VLA recipe, fully async communication between rollout and simulation nodes in disaggregate mode. This PR allows two overlap optimization: 1. Overlapping between simulation tool **reset** and rollout weights **update**. Now we can reset the simulation status and update actor_rollout weights simultaneously, without sync wait. *This feature saves costs close to 8-step execution time.* 2. Overlapping **communication** between simulation steps and rollout steps, aka eliminating the transfer overhead among nodes. Now in each step, different pipeline stages (see PR verl-project#3918 ) will be executed independently, thus there will be no sync waits among stages, so that the relatively longer steps (simulation steps currently) can be executed continuously. *This feature saves about 12% costs in each VLA RL step, depending on actual execution time of simulation steps.* **Details about communication overlapping** - At first our disaggregate mode on VLA show workflows as follow: ![img_v3_02sm_7be54524-8d37-4380-a4d2-fb871837d64g](https://github.com/user-attachments/assets/f21244b1-1d4e-4bac-82d3-4a40008c6ca2) - As in disaggregate mode, rollout steps are executed on local nodes and simulation steps are executed on remote nodes. - Low GPU utilization, lots of resources are wasted. - So we implement the pipeline execute mode in PR verl-project#3918 , then the workflow (2 stages for instance) shows as follows: ![img_v3_02sm_d398e0bb-ed08-4eaf-ae4d-53d6b9f1343g](https://github.com/user-attachments/assets/084e21d1-6224-4cce-8e0c-026e4f8e733c) - We can see that the rollout steps (R) and simulation steps (S) are executed partially overlapped (pipeline stage 0 and 1), and simulation steps take much longer time than rollout ones. - There are still parts of wasted time (unnecessary data transfer delays) because of the sequential workflow. The reason is shown as follows: ![img_v3_02sm_abd3887d-1220-43b6-af16-fcf357d20c3g](https://github.com/user-attachments/assets/c9493870-0934-4478-a8cf-7b9928f580f1) - The sync waits among pipeline stages cannot be eliminated due to sequential workflow: although steps are executed in various nodes, they need to be synchronized after each step, which causes communication cost and is unnecessary. - Then we implemented async communication pipelines to overlap the communication time by different execution time between rollout and simulation steps. The workflows and execution results are shown as follows: ![img_v3_02sm_60b64952-ed9c-40c5-8665-2187e21112fg](https://github.com/user-attachments/assets/074caaeb-8cfe-4b75-8de3-9e73f8c7d02e) --- ### Checklist Before Starting - [ ] Search for similar PRs. Paste at least one query link here: ... - [ ] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test ```shell # you can use `ray.timeline` tool for profiling in recipe/vla/main_ppo.py bash recipe/vla/run_simpleVLA_isaac_disagg.sh ``` Profile results of reset overlapping <img width="1420" height="232" alt="image" src="https://github.com/user-attachments/assets/dd98ea70-fa3f-4938-a02e-a44f64a0d1e9" /> Profile results of communication overlapping <img width="1058" height="228" alt="image" src="https://github.com/user-attachments/assets/a9274864-ccd4-42f9-b5c4-1277bc0c1317" />

### What does this PR do? - As title. The same as PR: verl-project#4444 ### Checklist Before Starting - [ ] Search for similar PRs. Paste at least one query link here: ... - [ ] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [ ] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).)

…#4432) ### What does this PR do? add examples/grpo_trainer/run_qwen3-4b_npu.sh ### Test The figure below shows the comparison curve of the critic_reward_mean metric. <img width="1790" height="948" alt="image" src="https://github.com/user-attachments/assets/01df9bed-f888-470d-936c-eb335acd57e9" /> ### API and Usage Example ```sh # install jemalloc sudo apt update sudo apt install libjemalloc2 # run bash bash examples/grpo_trainer/run_qwen3-4b_npu.sh ```

…ct#4396) ### What does this PR do? This PR removes the deprecated arguments during Megatron optimizer building for compatibility with the latest Megatron, see [https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/optimizer/__init__.py#L442](https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/optimizer/__init__.py#L442). These arguments are never used by verl so they can be safely removed. This solves the following exception with the latest Megatron: ``` File "/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception ^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/verl/verl/single_controller/ray/base.py", line 825, in func return getattr(self.worker_dict[key], name)(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/verl/verl/single_controller/base/decorator.py", line 451, in inner return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/root/verl/verl/utils/transferqueue_utils.py", line 187, in dummy_inner return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/root/verl/verl/workers/megatron_workers.py", line 573, in init_model ) = self._build_model_optimizer( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/verl/verl/workers/megatron_workers.py", line 464, in _build_model_optimizer actor_optimizer = get_megatron_optimizer(model=actor_module, config=optim_config_megatron) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/verl/verl/utils/megatron/optimizer.py", line 71, in get_megatron_optimizer return get_megatron_optimizer_native( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: get_megatron_optimizer() got an unexpected keyword argument 'no_weight_decay_cond' ``` ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: ... - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).)

…roject#4392) ### What does this PR do? Currently `use_distributed_optimizer` is hardcoded as `optim_args`, which is unexpected since `use_distributed_optimizer` is a config for `megatron`. ### Checklist Before Starting - [X] Search for similar PRs. Paste at least one query link here: ... - [X] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [X] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [X] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [X] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [X] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [X] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) Signed-off-by: Hollow Man <hollowman@opensuse.org>

verl-project#4448) ### What does this PR do? Reward loop deprecated batch reward manager. Fix `genrm_remote` recipe as it used batch reward manager. ### Checklist Before Starting - [ ] Search for similar PRs. Paste at least one query link here: ... - [ ] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [ ] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).)

) ### What does this PR do? This PR introduces FP8 rollout with sglang inference backend in verl. #### Experiments and Outcomes Qwen3-8B-Base Dense Model **Configuration** - DAPO recipe. AIME24 online validation. - SGLang + FSDP - Note that SPMD rollout has been deprecated, so we removed the FP8 SPMD rollout. - Prompt batch size 32, n=16. - Rollout batch size: 32\*3*16 - Train_batch_size & ppo_mini_batch_size 32 - Max response length 20K - Token-level TIS, C=2 - 8*H100 - verlai/verl:sgl055.latest **Accuracy** With TIS, FP8 rollout aligns with BF16 <img width="1460" height="782" alt="image" src="https://github.com/user-attachments/assets/c8b04c8c-2961-4ad3-9c0a-0d0bee80fd74" /> **Performance** <img width="661" height="661" alt="image" src="https://github.com/user-attachments/assets/967b6889-08b6-407b-8586-86b42a58d0b7" /> <img width="661" height="668" alt="image" src="https://github.com/user-attachments/assets/0b3f4ad1-87e2-428e-ab96-d241944a2b41" /> *purple: BF16, red: FP8 rollout* Results and observations: - FP8 rollout leads to around ~12% rollout speedup ### Checklist Before Starting - [ ] Search for similar PRs. Paste at least one query link here: ... - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [x] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [x] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) --------- Co-authored-by: Xue Huang <xueh@nvidia.com>

…ect#4403) ### What does this PR do? SFT trainer support vlm model - [x] fsdp engine - [x] megatron engine Qwen3-VL-2B-Instruct sft trainer compare on [llamafactory/pokemon-gpt4o-captions](https://huggingface.co/datasets/llamafactory/pokemon-gpt4o-captions) <img width="1550" height="620" alt="image" src="https://github.com/user-attachments/assets/40f79711-a542-4816-89e2-24184c8cb495" />

…t#4452) ### What does this PR do? Future PRs will transfer from legacy rm implementation to reward loop (in both rule-based, genrm, disrm, ...) gradually; this PR adds reward loop configs to defaults, which inherit the legacy reward model config, so it will not break any current api. Specifically, future PRs will: - align results between reward loop disrm and legacy fsdp/megatron disrm - deprecate fsdp/megatron disrm, use reward loop disrm as default - use reward loop rule-based, disrm-based, genrm-based as default - deprecate legacy reward model config ### Checklist Before Starting - [ ] Search for similar PRs. Paste at least one query link here: ... - [ ] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [ ] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).)

…roject#4453) ### What does this PR do? This PR adds abort functionality to the vLLM async rollout server, enabling users to cancel ongoing generation requests efficiently. This is particularly useful for scenarios where generation needs to be stopped early (e.g., policy updates, timeout handling, or user cancellation). **Key additions:** - `abort_all_requests()` method to cancel all active generation requests - `abort_request(request_id)` method to cancel a specific request - `stop_reason` field in `TokenOutput` to distinguish between completed, aborted, and other finish states ### Test The abort functionality has been validated through a standalone test script that: 1. Starts 8 concurrent generation requests with long prompts 2. Waits 0.5s and calls `abort_all_requests()` 3. Verifies that requests are properly aborted with partial outputs **Test Results:** - ✅ Successfully aborted 7 out of 8 requests (1 completed before abort was triggered) - ✅ Abort operation completed in **2.27ms** (very fast) - ✅ All aborted requests returned partial outputs with `stop_reason="aborted"` - ✅ Completed request had `stop_reason="completed"` - ✅ All requests finished without timeout See full test log in the PR description above. ### API and Usage Example ```python import ray from verl.workers.rollout.replica import get_rollout_replica_class # Initialize vLLM rollout server rollout_server_class = get_rollout_replica_class("vllm") server = rollout_server_class(replica_rank=0, config=rollout_config, ...) await server.init_standalone() # Start generation ref = server._server_handle.generate.remote( request_id="req_123", prompt_ids=[1, 2, 3], sampling_params={"temperature": 1.0}, ) # Abort all requests result = await server.abort_all_requests() # Returns: {"aborted_count": 1, "request_ids": ["req_123"], ...} # Or abort a specific request result = await server.abort_request("req_123") # Returns: {"aborted": True, "request_id": "req_123"} # Check stop reason in output output = ray.get(ref) print(output.stop_reason) # "aborted" or "completed" ``` ### Design & Code Changes **1. Added `stop_reason` field to `TokenOutput` protocol** (`verl/workers/rollout/replica.py`): - New optional field to track why generation stopped - Values: `"completed"`, `"aborted"`, or other finish reasons **2. Implemented abort methods in `vLLMHttpServerBase`** (`verl/workers/rollout/vllm_rollout/vllm_async_server.py`): - `abort_all_requests()`: Aborts all active requests by: - Fetching all request states from the output processor - Creating abort outputs and putting them into request queues - Calling abort on both output processor and engine core - `abort_request(request_id)`: Aborts a specific request using similar logic **3. Implemented abort methods in `vLLMReplica`**: - Distributes abort calls across all server instances - Aggregates results from multiple servers **4. Added stop reason mapping**: - Maps vLLM's `finish_reason` to verl's `stop_reason`: - `"abort"` → `"aborted"` - `"stop"` or `"length"` → `"completed"` - Other reasons pass through as-is **5. Added comprehensive test** (`tests/workers/rollout/rollout_vllm/test_vllm_abort.py`): - Standalone script to validate abort functionality - Tests concurrent request abortion and partial output handling ### Checklist Before Submitting - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [x] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: Added standalone test script in `tests/workers/rollout/rollout_vllm/test_vllm_abort.py` - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ).

### What does this PR do? As title

verl-project#4423) ### What does this PR do? Allow subclass to set reward_manager_worker ### Checklist Before Starting - [ ] Search for similar PRs. Paste at least one query link here: ... - [ ] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [ ] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) Co-authored-by: Ryan Li <rynli@amazon.com> Co-authored-by: Yuyang Ding <61647442+yyDing1@users.noreply.github.com>

### What does this PR do? - Support tensordict make iterator with nested tensor - improve TrainingWorker by setting default engineering args in the worker init - Add a unit test of TrainingWorker - fix verl-project#4004 ### Checklist Before Starting - [ ] Search for similar PRs. Paste at least one query link here: ... - [ ] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [ ] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).)

…project#4468)

### What does this PR do? MLFlow would not work with metrics that have // in its item name, it will yield error like so: ``` File "/usr/local/lib/python3.12/dist-packages/mlflow/tracking/client.py", line 2511, in log_batch return self._tracking_client.log_batch( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/mlflow/telemetry/track.py", line 30, in wrapper result = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/mlflow/tracking/_tracking_service/client.py", line 581, in log_batch self.store.log_batch(run_id=run_id, metrics=metrics_batch, params=[], tags=[]) File "/usr/local/lib/python3.12/dist-packages/mlflow/store/tracking/rest_store.py", line 906, in log_batch self._call_endpoint(LogBatch, req_body) File "/usr/local/lib/python3.12/dist-packages/mlflow/store/tracking/rest_store.py", line 208, in _call_endpoint return call_endpoint( ^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/mlflow/utils/rest_utils.py", line 596, in call_endpoint response = verify_rest_response(response, endpoint) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/mlflow/utils/rest_utils.py", line 315, in verify_rest_response raise RestException(json.loads(response.text)) mlflow.exceptions.RestException: INVALID_PARAMETER_VALUE: Invalid value "val-aux//reward/mean_at_1" for parameter 'metrics[0].name' supplied: Names may be treated as files in certain cases, and must not resolve to other names when treated as such. This name would resolve to 'val-aux/reward/mean_at_1' ``` ### Test Added testing for this behavior into `TestMlflowLoggingAdapter`. ### Design & Code Changes Used regular expression to parse and substituted multiple slashes pattern

…on (verl-project#4458) ### What does this PR do? > Add **concise** overview of what this PR aims to achieve or accomplish. Reference related GitHub issues and PRs that help with the review. ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: ... - verl-project#3978 missing the reward one ### Test only need to test in CI ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [x] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [x] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [x] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).)

compatible to vllm0.12, use `docker://verlai/verl:vllm012.exp` to test

…when merging RayResourcePool (verl-project#4949) ### What does this PR do? Fix the parameter missing issue in the `merge_resource_pool` function, which failed to pass max_colocate_count and detached arguments when instantiating the merged RayResourcePool object. If the max_colocate_count parameter is not passed, the default value of 10 will be used. Co-authored-by: weidongliang.339 <weidongliang.339@bytedance.com>

…h 3 or more dimensions (verl-project#4940) ### What does this PR do? There are 2 errors that prevent dispatching and collecting nested tensors with 3 or more dimensions. #### Dispatch When chunking a `TensorDict` with more than 1 nested tensor that has 3 or more dimensions, re-use of the variable name `td` in the function args and inner loop results in a `KeyError`: https://github.com/volcengine/verl/blob/e204cd80bd0886c75606d4b82ba88eed2658d1c7/verl/utils/tensordict_utils.py#L276-L312 #### Collection When collecting returned `TensorDict`s that have a nested tensor with 3 or more dimensions, there is an assertion enforcing that the nested tensor have exactly 2 dimensions, although the function works for tensors with an arbitrary number of dimensions: https://github.com/volcengine/verl/blob/e204cd80bd0886c75606d4b82ba88eed2658d1c7/verl/utils/tensordict_utils.py#L159-L192 ### Tests The added tests demonstrate both errors when run on `main`: ```python FAILED tests/test_protocol_v2_on_cpu.py::test_concat_nested_tensor - AssertionError: nested tensor must have 2 dimensions. Got torch.Size([2, 4, j32]) FAILED tests/test_protocol_v2_on_cpu.py::test_chunk_tensordict - KeyError: 'key "position_ids" not found in TensorDict with keys [\'attention_mask\', \'input_ids\', \'multi_modal_inputs\']' ```

…x performance issue (verl-project#4960)

…erl-project#4961) ### What does this PR do? `get_torch_device()` will return a module (e.g. `torch.cuda`) instead of the expected `torch.device` type, we shall construct it manually using `get_device_name`. Error logs: ```log File "verl/verl/trainer/main_ppo.py", line 367, in run trainer.fit() File "verl/verl/trainer/ppo/ray_trainer.py", line 1623, in fit batch = compute_advantage( ^^^^^^^^^^^^^^^^^^ File "verl/verl/trainer/ppo/ray_trainer.py", line 273, in compute_advantage advantages, returns = adv_estimator_fn(**adv_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "verl/verl/trainer/ppo/core_algos.py", line 351, in compute_grpo_vectorized_outcome_advantage mean_g, std_g, _ = group_mean_std(scores, g, eps=epsilon) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "verl/verl/utils/groupwise.py", line 191, in group_mean_std scores = scores.reshape(-1).to(device=target, dtype=torch.float32) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: to() received an invalid combination of arguments - got (dtype=torch.dtype, device=module, ), but expected one of: * (torch.device device = None, torch.dtype dtype = None, bool non_blocking = False, bool copy = False, *, torch.memory_format memory_format = None) * (torch.dtype dtype, bool non_blocking = False, bool copy = False, *, torch.memory_format memory_format = None) * (Tensor tensor, bool non_blocking = False, bool copy = False, *, torch.memory_format memory_format = None) ``` ### Checklist Before Starting - [X] Search for similar PRs. Paste at least one query link here: ... - [X] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `veomni`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data`, `cfg`, `reward` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [X] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [X] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) - [ ] If your PR is related to the `recipe` submodule, please also update the reference to the submodule commit via `git submodule update --remote` or `cd recipe && git pull origin main`. Signed-off-by: Hollow Man <hollowman@opensuse.org>

…-project#4962) ### What does this PR do? To prevent device mismatch for: ```log File "verl/trainer/main_ppo.py", line 367, in run trainer.fit() File "verl/trainer/ppo/ray_trainer.py", line 1623, in fit batch = compute_advantage( ^^^^^^^^^^^^^^^^^^ File "verl/trainer/ppo/ray_trainer.py", line 273, in compute_advantage advantages, returns = adv_estimator_fn(**adv_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "verl/trainer/ppo/core_algos.py", line 353, in compute_grpo_vectorized_outcome_advantage scalars = (scores - mean_g[g]) / (std_g[g] + epsilon) ~~~~~~~^~~~~~~~~~~ RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! ``` ### Checklist Before Starting - [X] Search for similar PRs. Paste at least one query link here: ... - [X] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `veomni`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data`, `cfg`, `reward` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [X] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [X] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) - [ ] If your PR is related to the `recipe` submodule, please also update the reference to the submodule commit via `git submodule update --remote` or `cd recipe && git pull origin main`. Signed-off-by: Hollow Man <hollowman@opensuse.org>

…erl-project#4959) ### What does this PR do? Previous PR verl-project#4929 is reverted due to CI break. This PR fixes CI break & re-add fix for attention FLOPS calculation for causal LLMs. ### Problem The current attention FLOPS calculations for all causal LLMs are missing the `/2` factor for causal (lower triangular) attention mask. This causes **2× overestimation** of attention FLOPS. Additionally, DeepSeek V3's MLA attention incorrectly uses the same dimension for both Q@K^T and attn@V operations, when `v_head_dim` differs from `q_head_dim`. ### Changes Summary | Function | Models | Change | |----------|--------|--------| | `_estimate_qwen2_flops` | qwen2, llama, qwen3, mistral, etc. | `12 *` → `6 *` | | `_estimate_qwen3_vl_flops` | qwen3_vl | `12 *` → `6 *` | | `_estimate_qwen3_vl_moe_flops` | qwen3_vl_moe | `12 *` → `6 *` | | `_estimate_qwen2_moe_flops` | qwen2_moe, qwen3_moe | `12 *` → `6 *` | | `_estimate_gemma3_flops` | gemma3_text | `12 *` → `6 *` | | `_estimate_apertus_flops` | apertus | `12 *` → `6 *` | | `_estimate_gpt_oss_flops` | gpt_oss | `12 *` → `6 *` | | `_estimate_deepseek_v3_flops` | deepseek_v3 | `12 * q` → `3 * (q + v)` | | `_estimate_qwen3_vit_flop` | ViT (vision) | **No change** (bidirectional) | For causal (autoregressive) attention, only the lower triangular portion of the attention matrix is computed: ``` Attention Matrix (causal): [✓ · · ·] [✓ ✓ · ·] [✓ ✓ ✓ ·] [✓ ✓ ✓ ✓] ``` | Model Type | Before | After | Overestimation | |------------|--------|-------|----------------| | Standard GQA/MHA | `12 * seq² * d` | `6 * seq² * d` | **2.0×** | | DeepSeek V3 MLA | `12 * seq² * q` | `3 * seq² * (q+v)` | **2.4×** | ### Reference This fix aligns with [Megatron-LM's FLOPS calculation](https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/training/training.py): - Uses `/2` for causal attention - Separately accounts for `q_head_dim` and `v_head_dim` in MLA ### Checklist Before Starting - [ ] Search for similar PRs. Paste at least one query link here: ... - [ ] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `veomni`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data`, `cfg`, `reward` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [ ] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) - [ ] If your PR is related to the `recipe` submodule, please also update the reference to the submodule commit via `git submodule update --remote` or `cd recipe && git pull origin main`.

…cs to fail has been covered by tests. (verl-project#4967) ### What does this PR do? Fixes a GRPO crash in `verl.utils.groupwise.group_mean_std` caused by passing a device *module* (e.g., `torch.cuda`) to `Tensor.to(device=...)`. The default device resolution now returns a proper `torch.device` (e.g., `cuda:0`), and a regression test is added to prevent this from coming back. ### Checklist Before Starting - [ ] Search for similar PRs. Paste at least one query link here: [Search PRs: group_mean_std device module](https://github.com/volcengine/verl/pulls?q=is%3Apr+group_mean_std+device+module) - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - Title used: `[algo, training_utils] fix: prevent device=module crash in group_mean_std` ### Test - Reproduced the original failure path in docker (no `VERL_FORCE_DEVICE`, non-pytest code path) and verified the crash is gone: - `group_mean_std(...)` now returns tensors on `cuda:0` instead of failing with `device=module`. - Added a regression test in `tests/utils/test_groupwise.py`. - Note: I could not run the full pytest suite locally because `pytest` is not installed in the current docker image; CI should cover it. ### API and Usage Example No API changes. # Minimal reproduction (used for docker smoke test) import os os.environ.pop("VERL_FORCE_DEVICE", None) os.environ.pop("PYTEST_CURRENT_TEST", None) import torch from verl.utils.groupwise import group_mean_std scores = torch.tensor([1.0, 2.0, 3.0]) gidx = torch.tensor([0, 1, 0]) mean_g, std_g, cnt_g = group_mean_std(scores, gidx) print(mean_g.device, std_g.device, cnt_g.device) --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…t#4970) ### What does this PR do? Fix gsm8k datasets and tensordict hyperlink 404 not found. ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: ... - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `veomni`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data`, `cfg`, `reward` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test N/A - This is a comment-only change ### API and Usage Example N/A - No API changes. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes - Changed 'https://huggingface.co/datasets/gsm8k' to 'https://huggingface.co/datasets/openai/gsm8k' - Changed 'https://pytorch.org/tensordict/tutorials/data_fashion' to 'https://pytorch.org/tensordict/stable/tutorials/data_fashion' - Changed 'https://pytorch.org/tensordict/overview.html' to 'https://pytorch.org/tensordict/stable/overview.html' ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) - [ ] If your PR is related to the `recipe` submodule, please also update the reference to the submodule commit via `git submodule update --remote` or `cd recipe && git pull origin main`.

…nc policy (verl-project#4191) ### What does this PR do? Extend the fully async policy recipe by adding SGLang as an alternative rollout engine to vLLM when using FSDP ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: ... - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [ ] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) --------- Co-authored-by: jsfanfanfan <2981866535@qq.com> Co-authored-by: jsfanfanfan <2981856535@qq.com> Co-authored-by: jsfanfanfan <71052636+jsfanfanfan@users.noreply.github.com>

…#4936) # Guide to Using MTP in RL Training and Inference **Author**: `https://github.com/meituan-search` **Last Updated**: 2026/01/14 # 1. Scope of Support Currently, RL training can be performed on mimo-7B-RL, Qwen-next, and Deepseek series models based on the MTP architecture. The support rules for training and inference engines are as follows: - **Training Engine**: Only supports the `mbridge + megatron` combination; other training engines are not compatible at this time; - **Inference Engine**: Compatible with all engines, but the model must be in the corresponding engine's compatibility list; - **Dependency Versions**: - mbridge: Use the specified branch: [https://github.com/ArronHZG/mbridge/tree/feature/verl_mtp](https://github.com/ArronHZG/mbridge/tree/feature/verl_mtp) (will be merged into the main branch in the future); - megatron: Use the latest dev version (commit: [23e092f41ec8bc659020e401ddac9576c1cfed7e](https://github.com/NVIDIA/Megatron-LM/tree/23e092f41ec8bc659020e401ddac9576c1cfed7e)), which supports MTP + CP training methods. # 2. MTP Training Configuration (Core Parameters) The MTP training process can be flexibly controlled through the following configurations. All configurations are based on the `actor_rollout_ref.model.mtp` prefix: | Configuration Scenario | Core Parameters | Description | |------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------| | Load MTP Parameters Only | `enable=True` | VRAM usage will increase, but the exported parameters include the MTP module and can be directly used for online deployment | | Full-Parameter MTP Training | `enable=True` `enable_train=True` `mtp_loss_scaling_factor=0.1` | MTP Loss will apply to all model parameters | | MTP Parameter-Only Training | `enable=True` `enable_train=True` `detach_encoder=True` | Freeze the Encoder layer, update only MTP module parameters, MTP Loss applies only to MTP parameters | | MTP Accelerated Rollout | 1. vLLM configuration: `enable=True` `enable_rollout=True` `method="mtp"` `num_speculative_tokens=1` 2. SGLang configuration: `enable=True` `enable_rollout=True` `speculative_algorithm="EAGLE"` `speculative_num_steps=2` `speculative_eagle_topk=2` `speculative_num_draft_tokens=4` | Achieve inference acceleration during the Rollout phase based on MTP | # 3. Experimental Results The experiment was conducted as follows: * model = mimo-7B-math * max_response_length = 8k Experiment chart: ![fully_async_policy_revenue]( https://github.com/ArronHZG/verl-community/blob/main/docs/mimo-7b-mtp.png?raw=true) **Scenarios with No Significant Effect** The following configurations will not have a noticeable impact on training results: 1. The base model does not carry MTP parameters; 2. The base model carries MTP parameters, but the MTP module is not trained; 3. The base model carries MTP parameters and trains MTP, with `mtp_loss_scaling_factor=0`; 4. The base model carries MTP parameters, trains MTP and detaches the encoder, with `mtp_loss_scaling_factor=0.1`. **Scenarios with Significant Effect** Only the following configuration will have a noticeable impact on training results: - The base model carries MTP parameters, MTP Loss applies to all model parameters, and `mtp_loss_scaling_factor=0.1`. **Recommended Training Method** It is recommended to adopt the `detach_encoder=True` approach for MTP training. # 4. Performance Notes for MTP in Rollout Inference The effectiveness of MTP-accelerated Rollout is significantly affected by **model size** and **inference hardware**. Key reference information is as follows: **Hardware Tensor Core Performance** | Hardware Model | FP16 Performance (TFLOPS) | |----------------|---------------------------| | H20 | 148 | | H800 | 1,671 | | H200 | 1,979 | **Measured Performance and Recommendations** Taking the mimo-7B model deployed separately on H20 hardware using SGLang as an example: After enabling MTP speculative decoding, the Rollout throughput decreases by approximately 50%. - Current priority recommendation: Do not enable MTP acceleration during the inference phase for now; - Future planning: Further optimization of the speculative logic in the Rollout phase will be conducted to improve throughput performance. > Add **concise** overview of what this PR aims to achieve or accomplish. Reference related GitHub issues and PRs that help with the review. ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: ... - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `veomni`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data`, `cfg`, `reward` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [x] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [x] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [x] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) - [x] If your PR is related to the `recipe` submodule, please also update the reference to the submodule commit via `git submodule update --remote` or `cd recipe && git pull origin main`.

…date mode (verl-project#4944) ### What does this PR do? > Add **concise** overview of what this PR aims to achieve or accomplish. Reference related GitHub issues and PRs that help with the review. fix user_trainer_do_validate=True run megatron+VLLM process error. In megatron case, this also can recive some improvments <img width="1452" height="1162" alt="image" src="https://github.com/user-attachments/assets/ac6c2d01-a48c-48a2-a5d6-0aab73e9d3e4" /> <img width="984" height="672" alt="image" src="https://github.com/user-attachments/assets/ee3f99ca-0264-41e5-a80f-0d31783c21e7" /> <img width="424" height="724" alt="image" src="https://github.com/user-attachments/assets/217a6090-b132-453a-805d-22f414fb2286" /> <img width="1160" height="616" alt="image" src="https://github.com/user-attachments/assets/3110b313-024f-46c5-a08e-0b81a4a0f117" /> <img width="1574" height="1268" alt="image" src="https://github.com/user-attachments/assets/3e12a53c-6dc7-4f33-b7d8-9016585d80ed" /> ### Checklist Before Starting - [ ] Search for similar PRs. Paste at least one query link here: ... - [ ] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `veomni`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data`, `cfg`, `reward` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [ ] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) - [ ] If your PR is related to the `recipe` submodule, please also update the reference to the submodule commit via `git submodule update --remote` or `cd recipe && git pull origin main`.

…vllm async server (verl-project#4976) ### What does this PR do? fix the num_preempted metrics in multi-turn,and the typo in vllm_async_server ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: ... - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `veomni`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data`, `cfg`, `reward` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [ ] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) - [ ] If your PR is related to the `recipe` submodule, please also update the reference to the submodule commit via `git submodule update --remote` or `cd recipe && git pull origin main`.

…erl-project#4665) ## What does this PR do? [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) has recently added [Ray orchestrator](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/ray_orchestrator) and essential features required for the RL workflow. This PR introduces TensorRT-LLM as a new rollout engine for VeRL. VeRL currently supports several rollout modes: - **Hybrid engine:** The training and rollout engines share the same process group. VeRL uses the `WorkerDict` class to manage multiple workers within a single process group. Communication between training and rollout workers takes place within the same process, allowing them to share the Torch GPU memory pool. - **Colocated:** Different engines use the same set of GPUs but run in separate process groups. Currently, this mode is used only by the reward model. - **Standalone:** Rollout engines use completely independent GPU resources. Unlike other rollout engines, TensorRT-LLM primarily targets the *colocated* mode. However, instead of relying purely on standard colocated mode, we introduced a mixed design combining aspects of the hybrid engine and colocated mode. The design goals are: - Clear resource separation through distinct process groups, offering maximum flexibility between training and rollout processes. - Hybrid workers that act as proxies to LLM servers. - Fully RESTful rollout API support through `TRTLLMHttpServer`. - A unified framework for both asynchronous and synchronous RL workflows. This PR aims to make the integration as minimally intrusive as possible to VeRL's infrastructure. Currently, it only invokes `RolloutReplica.init_hybrid_colocated()` when both the hybrid engine is enabled and the rollout engine is set to TensorRT-LLM. ## High Level Design Please refer to [workers/rollout/trtllm_rollout/trtllm_async_rollout.md](https://github.com/davidmlw/verl/pull/43/changes#diff-96bab8796296991333a973a5211166f45993b13d7c533732c83bcf23c5664f39) for more details. ```mermaid %%{init: {'theme':'base', 'themeVariables': { 'fontSize':'18px', 'edgeLabelBackground':'#eeeeee'}}}%% flowchart TB space1[" "] style space1 fill:none,stroke:none subgraph VERL["VERL Training Pipeline"] subgraph Workers["Training Workers"] Actor["Actor Worker"] Critic["Critic Worker"] RefModel["Ref Model Worker"] end Actor -->|Weight Updates IPC| Rollout["TensorRT-LLM Rollout"] subgraph RayCluster["Rollout Workers (Ray Cluster)"] space2[" "] style space2 fill:none,stroke:none subgraph AsyncRollout["TRTLLMAsyncRollout (per DP rank)"] DPLeader["• DP Leader coordination"] IPCMgmt["• IPC handle management"] HTTPAdapter["• HTTP adapter for server communication"] end AsyncRollout -->|HTTP/REST API| HTTPServer subgraph HTTPServer["TRTLLMHttpServer (Ray Actor per Replica)"] OpenAI["• OpenAI Server wrapper"] EngMgmt["• AsyncLLM engine management"] MemMgmt["• Memory management (resume/release)"] end HTTPServer --> AsyncLLM subgraph AsyncLLM["TensorRT-LLM AsyncLLM Engine"] GPUWorkers["• GPU workers (Tensor Parallel)"] KVCache["• KV Cache management"] CUDAGraph["• CUDA Graph optimization"] end end end space1 ~~~ VERL style VERL fill:#e1f5ff style RayCluster fill:#fff4e6 style AsyncRollout fill:#f3e5f5 style HTTPServer fill:#e8f5e9 style AsyncLLM fill:#fce4ec ``` ## Experiments results: Setup: single node with H100 * 8/slurm env. 1. FSDP/GRPO: Qwen2-7B (TP1 * 8 on 8 GPUs, launching cmd `bash examples/grpo_trainer/run_qwen2-7b_math_trtllm.sh 1`) * Convergence: <img width="563" height="352" alt="image" src="https://github.com/user-attachments/assets/5df943a7-e4ce-416f-8601-0655738bb33d" /> * Validation: <img width="1155" height="344" alt="image" src="https://github.com/user-attachments/assets/a1a203e1-a85e-46c9-a9ea-e9c0f3caf683" /> 2. FSDP/GRPO: Qwen2-7B (TP4 * 2 on 8 GPUs, launching cmd `bash examples/grpo_trainer/run_qwen2-7b_math_trtllm.sh 4`) * Convergence: <img width="553" height="354" alt="image" src="https://github.com/user-attachments/assets/dedfe3e2-498e-4d77-80bb-f1cd5d916c21" /> * Validation: <img width="1132" height="353" alt="image" src="https://github.com/user-attachments/assets/fbf6ae33-3643-466a-94e7-7edd70f53b3c" /> 3. Megatron/GRPO: Qwen2-7B (TP1 * 8 on 8 GPUs, launching cmd `bash examples/grpo_trainer/run_qwen2-7b_math_megatron_trtllm.sh 1`) * Convergence: <img width="766" height="323" alt="image" src="https://github.com/user-attachments/assets/6d9bc023-c5e7-466a-bf31-7ef9eda7b06d" /> * Validation: <img width="1546" height="338" alt="image" src="https://github.com/user-attachments/assets/ee6e263c-7779-4915-93dd-2f414370a9fc" /> 4. Megatron/GRPO: Qwen2-7B (TP2 * 2 on 8 GPUs, launching cmd `bash examples/grpo_trainer/run_qwen2-7b_math_megatron_trtllm.sh 4`) * Convergence: <img width="746" height="322" alt="image" src="https://github.com/user-attachments/assets/7a21dc39-0467-4b85-a231-8e5994b76a8a" /> * Validation: <img width="1552" height="334" alt="image" src="https://github.com/user-attachments/assets/00a3307b-a0d6-4d13-8f30-85f903f0c946" /> ## Special notes for using VeRL with TensorRT-LLM: 1. All RL required APIs for VeRL were implemented within [TensorRT-LLM 1.2.0rc6](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tensorrt-llm/containers/release?version=1.2.0rc6). To install VeRL with TensorRT-LLM, please use command `pip install -e ".[trtllm]" --extra-index-url https://pypi.nvidia.com/`. 2. All verification of integration work was primarily done in Slurm environment. 3. The current design requires `export RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1` and the following env settings before launching the Ray cluster. While these have been included in any example scripts or tests added, we will work toward removing such dependencies to improve the user experience in the near future. ``` # Clean all slurm / MPI / PMIx env to avoid pmix mismatch error for v in $(env | awk -F= '/^(PMI|PMIX|MPI|OMPI|SLURM)_/{print $1}'); do unset "$v" done # Force UCX to use only eth0; otherwise, it will attempt to use all available devices and raise warnings if any issues occur. export TRTLLM_UCX_INTERFACE=eth0 ``` ## Outstanding issues for this MR: 1. WIP on passing CI tests ## Upcoming works (in separate MRs) 1. Further performance optimization 3. Multi-node testing and functionality will be delivered in the near future. 7. The current MR focuses on and was validated wtih Qwen model variants. We'll work on validations and optimizations for MoE models as the next step. > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [ ] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) --------- Signed-off-by: Jonas Yang <joyang@nvidia.com> Co-authored-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com>

…l-project#4991) ### What does this PR do? Fix `ModuleNotFoundError` when importing `verl.trainer.sft_trainer_ray` without `qwen-vl-utils` installed. **Problem**: Importing `verl.trainer.sft_trainer_ray` fails with `ModuleNotFoundError: No module named 'qwen_vl_utils'` even when not using vision features, because `vision_utils.py` unconditionally imports `qwen_vl_utils` at module level. **Solution**: Move `qwen_vl_utils` imports from module level to function level (lazy import), following the same pattern already used in `verl/utils/dataset/rl_dataset.py:393`. Fixes verl-project#4958 ### Checklist Before Starting - [x] Search for similar PRs: https://github.com/volcengine/verl/pulls?q=is%3Apr+qwen_vl_utils - [x] Format the PR title as `[{modules}] {type}: {description}` ### Test ```bash # Before fix (fails) pip install verl==0.7.0 python -c "import verl.trainer.sft_trainer_ray" # ModuleNotFoundError: No module named 'qwen_vl_utils' # After fix (works) python -c "import verl.trainer.sft_trainer_ray" # Success - no error ``` ### API and Usage Example No API changes. This is a transparent fix - existing code using vision features will continue to work (with `qwen-vl-utils` installed), while code not using vision features will no longer require the dependency. ### Design & Code Changes **File changed**: `verl/utils/dataset/vision_utils.py` ```diff - from qwen_vl_utils import fetch_image, fetch_video def process_image(image: dict | Image.Image, image_patch_size: int = 14) -> Image.Image: + from qwen_vl_utils import fetch_image ... def process_video(...) -> torch.Tensor: + from qwen_vl_utils import fetch_video ... ``` This follows the existing pattern in `verl/utils/dataset/rl_dataset.py:393`: ```python def _extract_vision_info(...): from qwen_vl_utils import process_vision_info # existing lazy import ``` ### Checklist Before Submitting - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md) - [ ] Apply pre-commit checks - [ ] Add / Update the documentation (N/A - no user-facing doc changes) - [ ] Add unit or end-to-end test(s) (N/A - this is a fix for import behavior, tested manually) Co-authored-by: Wheels Wu <wheels.wu@flsdex.com>

verl-project#4998) ### What does this PR do? The goal of this PR is to **make the GSM8K multiturn tool quickstart actually work**.[web:69] In the official Verl GSM8K example, issue [verl-project#1569](verl-project#1569) reports an unexpected phenomenon: under the current configuration (e.g., Qwen2.5-0.5B-Instruct + PPO + GSM8K train/test), enabling multi-turn + tool (Agent Loop + `Gsm8kTool`) does **not** outperform the single-turn no-tool baseline and can even be slightly worse.[web:69] This is counter-intuitive because `Gsm8kTool` is essentially a strong oracle that evaluates answers against the ground truth, so a working quickstart should benefit from such a powerful tool rather than degrade. In my reproduction, the official multi-turn + tool configuration shows that the `turns` metric quickly collapses to 2, which means the agent almost always skips `<tool_call>` and directly guesses the final answer in a 2-turn dialogue (user + assistant). To address this, this PR does **not** modify the Agent Loop or `Gsm8kTool` implementation, but introduces a minimal reward shaping on top of the existing GSM8K correctness reward: - keep the original correctness-based outcome reward; - **add a +0.1 shaping reward if a trajectory contains at least one `<tool_call>`**. Under this setting, the multi-turn + tool validation accuracy improves from around **0.608** to about **0.717**, clearly surpassing the single-turn no-tool baseline (around **0.610**), and the agent maintains stable tool usage instead of collapsing to “no-tool guessing”. Detailed training curves and the behavior of `turns` / tool calls can be found in the SwanLab logs: <https://swanlab.cn/@huquan/gsm8k-reward-shaping/runs/f08c3iidt50caqenquv9z/chart>. --- ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: - Issue: [verl-project#1569](verl-project#1569) (unexpected degradation when multi-turn/tool is enabled on GSM8K).[web:69] - Related PR pattern: [[doc] fix: quickstart example can't work on zsh (verl-project#2509)](https://github.com/volcengine/verl/pull/2509).[web:139] - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - Title: `[recipe,tool,reward] feat: make GSM8K multiturn tool quickstart actually work` --- ### Test This PR cannot be fully validated by unit tests, so I ran end-to-end GSM8K PPO training with Qwen2.5-0.5B-Instruct under three configurations (same data split and training recipe): - **Single-turn, no tool (baseline)** - Multi-turn and tool use disabled. - Validation accuracy: ~0.610. - Behavior: number of turns is always 2 and the model never calls tools. - **Multi-turn + tool, no shaping (official agent config)** - Multi-turn + `Gsm8kTool` enabled, reward only depends on final correctness. - Validation accuracy: ~0.608. - Behavior: the agent briefly calls tools at the beginning (tool calls ~8), then quickly collapses to 2 turns (no `<tool_call>`), effectively giving up tool use. - **Multi-turn + tool + shaping (this PR)** - Same as the above, but with an additional +0.1 shaping reward if a trajectory contains at least one `<tool_call>`. - Validation accuracy: ~0.717. - Behavior: the agent maintains meaningful tool usage instead of collapsing to “no-tool guessing”. Training curves (including `turns`, tool-call statistics and accuracy) are available in the SwanLab run: <https://swanlab.cn/@huquan/gsm8k-reward-shaping/runs/f08c3iidt50caqenquv9z/chart>. --- ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python bash examples/sglang_multiturn/gsm8k_toolcall_shaping/run_gsm8k_grpo_toolcall_shaping.sh ``` ### Design & Code Changes Add a custom GSM8K reward function file that implements compute_score ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) - [ ] If your PR is related to the `recipe` submodule, please also update the reference to the submodule commit via `git submodule update --remote` or `cd recipe && git pull origin main`.

…egrate torch.profiler (verl-project#4320) ### What does this PR do? **Summary** This PR enhances the verl profiling system to support the Agent Loop architecture (decoupled inference and training processes). It integrates torch.profiler as a core backend within the DistProfiler framework, providing a unified performance tuning interface for both distributed training workers and asynchronous rollout servers. **Key Enhancements** Agent Loop Support: Implemented a coordinated control flow that allows the RayTrainer to explicitly trigger profiling sessions on remote inference servers (vLLM/SGLang) via AgentLoopManager. - For vLLM, we leverage the AsyncLLM profiling interface. - For SGLang, we utilize the TokenizerCommunicatorMixin profiling API. Unified Torch Profiler: Integrated the native PyTorch Profiler into the verl ecosystem, supporting both continuous and discrete collection modes consistent with Nsight systems and NPU. **Profiling Workflow** The provided Sequence Diagram illustrates the decoupled profiling logic: Rollout Phase: Profiling is managed through explicit RPC calls (start_profile/stop_profile) to the inference engine's server interface (AsyncLLM / TokenizerCommunicatorMixin), ensuring capture is synchronized with generation steps. <img width="3986" height="3834" alt="whiteboard_exported_image" src="https://github.com/user-attachments/assets/afe68413-c338-4eec-a843-df5a9c106d98" /> ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: ... - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [x] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) --------- Co-authored-by: tardis-key <huxiaobo@zju.edu.cn>

### What does this PR do? fix verl-project#4715 data_parallel_size_local is compute error. File "/tmp/ray/session_2026-01-16_05-13-18_263690_267/runtime_resources/working_dir_files/_ray_pkg_765c8a21e650e117/verl/workers/rollout/vllm_rollout/vllm_async_server.py", line 344, in launch_server assert len(self.workers) == data_parallel_size_local * self.config.tensor_model_parallel_size, ( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: num workers (4) should be equal to dp_size_local > Add **concise** overview of what this PR aims to achieve or accomplish. Reference related GitHub issues and PRs that help with the review. ### Checklist Before Starting - [ ] Search for similar PRs. Paste at least one query link here: ... - [ ] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `veomni`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data`, `cfg`, `reward` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [ ] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) - [ ] If your PR is related to the `recipe` submodule, please also update the reference to the submodule commit via `git submodule update --remote` or `cd recipe && git pull origin main`. Signed-off-by: Gao Shiyuan <gaoshiyuan@baidu.com>

…rl-project#5002) ### What does this PR do? Fixes verl-project#4996 This PR fixes an AttributeError when using FSDP2 with PyTorch 2.4/2.5: ``` AttributeError: 'function' object has no attribute 'FSDPModule' ``` ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: https://github.com/volcengine/verl/pulls?q=is%3Apr+FSDPModule - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) ### Test **Reproducer (before fix):** ```python import torch fully_shard_module = torch.distributed._composable.fsdp.fully_shard fully_shard_module.FSDPModule # AttributeError: 'function' object has no attribute 'FSDPModule' ``` **After fix:** ```python import torch fully_shard_module = torch.distributed._composable.fsdp fully_shard_module.FSDPModule # Works correctly - returns the FSDPModule class ``` ### API and Usage Example No API changes - this is a bug fix. ### Design & Code Changes **Root cause:** For PyTorch 2.4-2.5, `fully_shard_module` was incorrectly assigned the `fully_shard` function (line 45) instead of the containing module. When `maybe_patch_fsdp_module()` tried to access `fully_shard_module.FSDPModule`, it failed because functions don't have this attribute. **Fix:** Change line 45 from: ```python fully_shard_module = torch.distributed._composable.fsdp.fully_shard ``` to: ```python fully_shard_module = torch.distributed._composable.fsdp ``` This makes it consistent with the PyTorch >= 2.6 branch which correctly assigns the module. ### Checklist Before Submitting - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: This is a one-line fix for PyTorch version compatibility. The fix can be verified by running with PyTorch 2.4/2.5 in the existing CI workflows. - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) - [ ] If your PR is related to the `recipe` submodule, please also update the reference to the submodule commit via `git submodule update --remote` or `cd recipe && git pull origin main`.

### What does this PR do? as title ### Checklist Before Starting - [ ] Search for similar PRs. Paste at least one query link here: ... - [ ] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `veomni`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data`, `cfg`, `reward` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [ ] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) - [ ] If your PR is related to the `recipe` submodule, please also update the reference to the submodule commit via `git submodule update --remote` or `cd recipe && git pull origin main`.

### What does this PR do? Based on ckpt engine abstraction [verl-project#4775 ](verl-project#4775), in this pr, we add hccl backend to support huawei Ascend npu. TODO: - [x] Give more detailed performance testing results. - [x] Improve checkpoint engine README. In the near future, we will - Add Mooncake transfer engine to support p2p communication both for npu and GPU. - Integrate kimi ckpt engine for more complex communication. For now, the basic functions are tested, we will provide some performance results. ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: ... - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data`, `cfg`, `reward` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [x] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) - [ ] If your PR is related to the `recipe` submodule, please also update the reference to the submodule commit via `git submodule update --remote` or `cd recipe && git pull origin main`.

This reverts commit 1a66041.

FightingZhen and others added 30 commits December 5, 2025 14:19

[ci] test: Disable ReMax training test in vllm workflow (verl-project…

12b2851

…#4445)

[misc] feat: optimize rearrange_micro_batches (verl-project#4451)

b96b53a

fix conflict upstream

932a462

Merge branch 'main' of github.com:janhq/verl

1365848

[ci] chore: cleanup some ci workflow (verl-project#4459)

2319bab

### What does this PR do? As title

[recipe] feat: Modify the way of obtaining default_runtime_env (verl-…

aee5aa8

…project#4468)

[vllm] fix: compatible to vllm0.12 (verl-project#4473)

bb75788

compatible to vllm0.12, use `docker://verlai/verl:vllm012.exp` to test

wdl339 and others added 22 commits January 17, 2026 12:45

[env] fix: upgrade torch, cudnn and deps versions in vllm image to fi…

65eb5a1

…x performance issue (verl-project#4960)

Merge branch 'main' of github.com:janhq/verl

a118a28

sync upstream 22012026

2263986

nguyenhoangthuan99 requested a review from bachvudinh January 22, 2026 07:29

nguyenhoangthuan99 marked this pull request as ready for review January 22, 2026 07:29

nguyenhoangthuan99 requested a review from hiento09 March 3, 2026 05:16

hiento09 approved these changes Mar 3, 2026

View reviewed changes

nguyenhoangthuan99 merged commit 1a66041 into main Mar 3, 2026
63 of 69 checks passed

nguyenhoangthuan99 added a commit that referenced this pull request Mar 3, 2026

Revert "Feat/sync upstream22012026 (#22)"

2d73800

This reverts commit 1a66041.

nguyenhoangthuan99 mentioned this pull request Mar 3, 2026

Revert "Feat/sync upstream22012026" #25

Merged

nguyenhoangthuan99 added a commit that referenced this pull request Mar 4, 2026

Revert "Feat/sync upstream22012026 (#22)" (#25)

d6614f3

This reverts commit 1a66041.

Navigation Menu

Uh oh!

Feat/sync upstream22012026 - #22

Feat/sync upstream22012026#22
nguyenhoangthuan99 merged 301 commits into
mainfrom
feat/sync-upstream22012026

nguyenhoangthuan99 commented Jan 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Uh oh!

Conversation

nguyenhoangthuan99 commented Jan 22, 2026

What does this PR do?

Checklist Before Starting

Test

API and Usage Example

Design & Code Changes

Checklist Before Submitting

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants