Dmolinari/ab evals - #3381
Draft
d-molinari wants to merge 234 commits into
Draft
Dmolinari/ab evals#3381d-molinari wants to merge 234 commits into
d-molinari wants to merge 234 commits into
Conversation
Signed-off-by: Christian Munley <cmunley@nvidia.com>
## Summary - Follow-up to #2603. That PR added `Acquire::Retries=3` / `Acquire::http(s)::Timeout=30` to apt itself plus a 5-min step timeout. - Verified locally (Docker repro of the exact stall — TLS connection accepted, headers sent, then silence) that apt's own retry/timeout options **do** work against a stall at that layer, but production logs show the real stall doesn't even get one retry cycle logged before going fully silent for the whole step duration — suggesting the actual failure is a lower-layer network/NAT issue that apt's own idle-read timeout isn't reliably catching. - This PR removes the apt-level `Acquire::*` options and replaces them with a step-level retry loop instead: up to 4 attempts of `apt-get update && apt-get install`, each hard-bounded to 5 minutes via `timeout 5m`, for a 20-minute worst-case total per step (`timeout-minutes: 20`). Signed-off-by: Kajal Jain <kajalj@nvidia.com>
## What does this PR do? Adds `openair_congestion`, a stateful, multi-turn 5G congestion-control environment for NeMo Gym. On each turn, a policy reads cell and UE KPIs and returns exactly one bounded tool call. The resource server validates the call, deterministic synthetic replay updates the state, and the environment computes a decomposed KPI reward without an LLM judge. The contribution includes: - seven network-control actions plus `noop`; - server-side schema, topology, and safety checks; - causal synthetic `replay` for controlled training and evaluation; - diagnostic-only `dataset_replay` for recorded KPI data; - reset, step, retry, and cleanup handling; - five example tasks and scripted rollouts; - focused tests, reviewer evidence, and a Fern tutorial. This does not include live OAI/FlexRIC control or claim physical-network fidelity, current SFT/GRPO improvement, or upstream acceptance. ## Checklist - [ ] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [ ] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [ ] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Andy Vu <andrew.vu@ufl.edu> Co-authored-by: Christian Munley <cmunley@nvidia.com>
<!-- Thanks for contributing to NeMo Gym! Please fill out the sections below. --> ## What does this PR do? 1. Swap from DP > 1 to DP = 1 to match Tomer's config and fix accuracy issues: benchmarks/nemotron_3.5_super/vllm_configs/nemotron_3.5_super.sh, benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh 2. Update Super 3.5 configs and export: benchmarks/nemotron_3.5_super/eval_container_config.yaml, benchmarks/nemotron_3.5_super/export_to_csv.py 3. OpenCode each turn uses full max length: benchmarks/nemotron_3.5_super/policy_model_override.yaml, responses_api_agents/opencode_sandboxed_agent/configs/opencode_agent.yaml, responses_api_agents/opencode_sandboxed_agent/app.py, responses_api_models/vllm_model/configs/vllm_model.yaml 4. SWE Bench Multilingual fixes: resources_servers/swebench/app.py 5. Misc: nemo_gym/rollout_collection.py, responses_api_agents/opencode_sandboxed_agent/app.py, resources_servers/swebench/print_test_outputs.py ## Checklist - [ ] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [ ] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [ ] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [ ] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [ ] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Brian Yu <bxyu@nvidia.com>
## What does this PR do? Adds a `max_tokens` config option to `HermesAgentConfig` and threads it through to the underlying Hermes agent call (`responses_api_agents/hermes_agent/app.py`). This lets callers cap the per-call output token length for Hermes rollouts via YAML config instead of relying on the model server default, defaulting to `None` (no change in behavior) when unset. ## Checklist - [x] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Elisa Martinez Abad <elisam@nvidia.com> Co-authored-by: Christian Munley <cmunley@nvidia.com>
## SpartQA — spatial-reasoning benchmark Adds a native gym port of [SpartQA](https://huggingface.co/datasets/mteb/SpartQA) — a spatial-reasoning QA benchmark where the model is shown a scene-description query and must return the matching answer phrase. ### What it does Every task is **single-turn generation** — no agent fork, no judge model. `verify()` is pure, rule-based scoring (stdlib `re` / `string` only). The model is prompted to end its response with `Final answer: <phrase>`. `verify()` extracts that phrase, normalizes it, and compares against every accepted phrase in `all_targets`: | Outcome | `reward` | `exact` | |---------|----------|---------| | Extracted phrase == any accepted phrase (normalized) | 1.0 | `true` | | Extracted phrase contains any accepted phrase (normalized) | 1.0 | `false` | | No match | 0.0 | `false` | | Empty / unparsed output | 0.0 | `false` | `compute_metrics`'s `mean_reward` equals corpus-level accuracy (the headline metric). `exact_match_rate` and `parse_rate` are reported alongside for downstream inspection. ### Scoring details `_extract_answer` handles: - `<think>…</think>` reasoning-block stripping (reasoning models) - `Final answer: …` / `Selected answer: …` / `Answer: …` patterns with fallback to last non-empty line when no explicit marker is present - Markdown artifacts (`**`, `` ` ``, `*`, bullet prefixes) ### Contents - `resources_servers/spartqa/app.py` — `SpartqaResourcesServer` (`/verify` + `compute_metrics` / `get_key_metrics`) with answer extraction, normalization, and multi-target scoring. - `prepare_spartqa.py` — joins `mteb/SpartQA` (HF `datasets`) into `data/spartqa_test.jsonl`. Committed `data/example.jsonl` is the 5-row smoke set. - `configs/spartqa.yaml` — server + `simple_agent` wiring for local eval/training. - `configs/spartqa_serve.yaml` — serve-only scorer for nemo-evaluator via the `gym://` adapter (rule-based verify needs no model on the gym side). - `requirements.txt` — no extra runtime deps; `-e nemo-gym[dev]` editable install only. - `tests/test_app.py` — **25 tests** over `_extract_answer`, `_strip_reasoning`, `_normalize`, `verify()` (exact / contains / empty / multi-target / metadata-fallback), `compute_metrics`, `get_key_metrics`, `_response_text`, and acceptance parity against every `example.jsonl` row. ### Testing ```bash gym env test --resources-server spartqa ruff check resources_servers/spartqa/app.py resources_servers/spartqa/prepare_spartqa.py resources_servers/spartqa/tests/ ``` ### Notes - **No API keys required** — all scoring is rule-based. - **Reasoning models** — `verify()` strips leading `<think>…</think>` blocks before extracting the answer, so thinking-model outputs score correctly. - **`verified: false`** — not yet baselined. Reward-profiling across an open-instruct / open-thinking / closed-source model suite is a follow-up before flipping to `verified: true`. - Heavy dataset-prep deps (`datasets`) are opt-in and not in `requirements.txt` — install only when running `prepare_spartqa.py`. --------- Signed-off-by: mbagdasarova <mbagdasarova@nvidia.com>
## LongMemEval — long-term-memory QA benchmark Adds a native gym port of [LongMemEval](https://github.com/xiaowu0162/LongMemEval) — a benchmark that measures whether a model can recall, reason over, update and abstain from multi-session chat history. ### What it does Every task is **single-turn generation** — one user message carrying the rendered JSON-format session history plus the question. No agent fork is required; the gym policy model (driven by the built-in `simple_agent`) generates once and the resources server scores it. `verify()` dispatches by `question_type` (with abstention rows routed by `_abs` in `question_id`): | `question_type` | Rubric | |-----------------|--------| | `single-session-user` | contain | | `single-session-assistant` | contain | | `multi-session` | contain | | `temporal-reasoning` | temporal (off-by-one tolerant) | | `knowledge-update` | knowledge-update | | `single-session-preference` | preference | | abstention (`_abs`) | abstention | ### Metrics `compute_metrics` reports: - `accuracy` — mean reward over scored rows (excludes `judge_call_failed` and `unknown_question_type`, matching upstream's skip path) - `accuracy_strict` — mean reward over all rewarded rows (full denominator) - `question_type/<type>/accuracy` and `/count` per bucket - `abstention/accuracy` and `/count` - `n_judge_call_failed`, `n_empty_judge_output`, `n_unknown_question_type`, `n_bad_metadata`, `n_judge_errors_other`, `n_excluded`, `n_empty_response` `get_key_metrics` surfaces `accuracy`, `abstention/accuracy`, `n_excluded`. ### Contents - `resources_servers/longmemeval/app.py` — `LongMemEvalResourcesServer` - `prepare_longmemeval.py` — dataset builder (oracle / s / m splits) - `benchmarks/longmemeval/config.yaml` + `prepare.py` — gym-native benchmark - `configs/longmemeval.yaml` — server + `simple_agent` wiring - `configs/longmemeval_serve.yaml` — serve-only scorer - `data/example.jsonl` — 5-row committed smoke sample - `tests/test_app.py` — unit tests - `tests/test_acceptance.py` — upstream parity acceptance tests (AC1–AC11) ### Testing ```bash gym env test --resources-server longmemeval ruff check resources_servers/longmemeval/app.py resources_servers/longmemeval/prepare_longmemeval.py resources_servers/longmemeval/tests/ ``` ### Notes - **Judge required**: supply `+judge_base_url +judge_api_key +judge_model_name` at launch. Recommended: `gpt-4o-mini` (upstream default). - **License**: Creative Commons Attribution-NonCommercial 4.0. Add a line in `ATTRIBUTIONS.md` under the Datasets table before merging. - **`verified: false`** — not yet baselined. Reward profiling across model suite is a follow-up before flipping to `verified: true`. - **topk-context default (50)**: may drop gold evidence sessions for entries with >50 sessions. Use `--topk-context 0` to keep all sessions; `--topk-context 1000` matches upstream's run script. --------- Signed-off-by: mbagdasarova <mbagdasarova@nvidia.com> Co-authored-by: Christian Munley <cmunley@nvidia.com>
## Motivation
PinchBench drives the OpenClaw CLI agent, which calls the policy model
endpoint directly. Gateway-fronted deployments can require a routing
header such as `X-Inference-Priority: batch` on every model request;
requests without the required header may stall. Gym can add headers to
its own model connectors through `default_headers`, but that does not
affect OpenClaw's external provider configuration.
OpenClaw supports arbitrary `headers` on custom providers. This change
exposes that capability through the PinchBench agent config and writes
it into the generated `openclaw.json` provider block.
## Changes
- Add an optional `provider_headers` mapping to `PinchBenchAgentConfig`.
- Pass configured headers into the sandbox as JSON and render them into
OpenClaw's custom provider.
- Cover the configured and omitted cases, including the sandbox
environment handoff.
## Configuration
```yaml
pinchbench agent config:
provider_headers: {"X-Inference-Priority": "batch"}
```
The field is optional and defaults to `None`, so existing configurations
and generated provider blocks are unchanged unless headers are
explicitly configured.
## Validation
- `python -m pytest
responses_api_agents/pinchbench/tests/test_openclaw_timeout_config.py
-q` (12 passed)
- `python -m pytest responses_api_agents/pinchbench/tests -q` (86
passed)
- `pre-commit run --all-files`
Downstream validation against a gateway-backed PinchBench smoke run is
in progress in a separate campaign session. No cluster jobs were run
while developing this PR.
## Pinned-version note
This branch is based on current `main`. Current `main` is 144 commits
ahead of `e446e4f415b9cde0e95bb813c85e9e3e23f5d893`, and
`responses_api_agents/pinchbench/app.py` has substantial intervening
changes. A campaign pinned to that older commit should point
`install_on_the_fly` at this branch or commit; cherry-picking only this
change onto the pinned revision may require conflict resolution in
`pinchbench/app.py`.
Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
<!-- Thanks for contributing to NeMo Gym! Please fill out the sections below. --> ## What does this PR do? This PR extends the experiment tracking possibilities by adding MLFlow backend. Till now there was a W&B exporter wired in the code, that tracks configuration, metrics and rollouts. This PR adds an exporter abstraction, moves existing functionality into W&B backend and adds MLFlow backend. Both can be configured at the same time - artifacts are sent to both tracking servers. **Deprecated field:** `upload_rollouts_to_wandb` has been renamed to `upload_rollouts`. If the user provides an old key, it is mapped to the new one and a deprecation warning is emitted. ## Followup 1. Export file artifacts. The original W&B code logs rollouts only, while NEL was logging more. This PR just adds a new backend and doesn't provide full feature parity with NEL. Tracked with #2619 2. Placement of export initialization seems sub-optimal. In my opinion it belongs to CLI, not config parsing. As a side effect we don't have the user-friendly error handling on config error in exporter. Tracked with #2620 3. Clean `RewardProfiler` up. It looks like it uses `wandb.Histrogram` for no good reason and carries some other obsolete code. Tracked with #2621 ## Usage example ``` gym eval run \ --benchmark gpqa \ --model-type openai_model \ --output gpqa-results/rollouts.jsonl \ --split benchmark \ +mlflow_tracking_uri=https://mlflow.my-server.com/ \ +mlflow_experiment_name=my-eval \ +mlflow_run_name=gpqa ``` This will run the evaluation as usual and create an MLFlow experiment for the run: ``` ... INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [2090575] NeMo Gym finished! Shutting down Ray cluster spun up by NeMo Gym... 🏃 View run gpqa at: https://mlflow.my-server.com/#/experiments/1740/runs/f088e196471048c9bf20e9beb9b771f5 🧪 View experiment at: https://mlflow.my-server.com/#/experiments/1740 ``` ## Checklist - [x] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com> Co-authored-by: bxyu-nvidia <bxyu@nvidia.com>
…ns (#2512) <!-- Thanks for contributing to NeMo Gym! Please fill out the sections below. --> ## What does this PR do? - Reformat data logic changed from chat/completions to responses format - Update README with detailed IHEval metrics and usage instructions - Update requirements with explicit defusedxml version ## Checklist - [x] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: mbagdasarova <mbagdasarova@nvidia.com> Co-authored-by: Christian Munley <cmunley@nvidia.com>
## Summary - report unknown workload names with the noun requested by `gym list benchmarks` or `gym list environments` - scope fuzzy suggestions to the requested catalog kind - identify exact wrong-kind names and point users to the working command instead of suggesting an unrelated workload - reject invalid catalog kinds through the existing `RegistryError` path ## Root cause `nemo_gym.cli.env._inspect_environment()` received the unified environment and benchmark catalog, but its unknown-name path hard-coded `environment` as the error noun and built its suggestion pool from both kinds. After the noun followed `catalog_kind`, the shared suggestion pool could still recommend an environment for a benchmark lookup. Exact names from the opposite kind also fell through to misleading unknown-name behavior. The final implementation validates the kind, filters the lookup and suggestion map by that kind, and handles a name that exists under the opposite kind explicitly. ## Before the fix At base `2251ef7f7fcbe60a352b790b0bc14a4f0d522f01`: ```console $ gym list benchmarks gsm8kk Unknown environment 'gsm8kk'. Did you mean `gsm8k`? ``` The checkout-pinned regression test failed on the incorrect noun: ```text E assert "Unknown benchmark 'gsm8kk'" in "Unknown environment 'gsm8kk'. Did you mean `gsm8k`?" 1 failed ``` ## After the fix At `248f0308630715be11b47d26eb695f18a07f37e9`: ```console $ gym list benchmarks gsm8kk Unknown benchmark 'gsm8kk'. Did you mean `gsm8k`? $ gym list environments blackjak Unknown environment 'blackjak'. Did you mean `blackjack`? $ gym list benchmarks blackjak Unknown benchmark 'blackjak'. $ gym list benchmarks biomnibench_da 'biomnibench_da' is an environment, not a benchmark. Try `gym list environments biomnibench_da`. ``` The identical checkout-pinned regression test now passes: ```text 1 passed ``` ## Why existing tests did not catch it The existing coverage did not drive an unknown benchmark name through the public `gym list benchmarks <name>` router and assert the rendered noun. It also did not exercise kind-specific fuzzy suggestions or an exact name that exists under the opposite catalog kind. This PR adds hermetic catalog tests for those semantics and retains one end-to-end router smoke over the real catalog. ## Test plan - [x] checkout-pinned regression test: base `1 failed`, fixed HEAD `1 passed` - [x] scoped pre-commit hooks on all three changed files - [x] affected unit modules: `298 passed` - [x] full unit suite: `2490 passed, 41 skipped, 15 subtests passed` - [x] CI core suite: `2194 passed, 2 skipped, 336 deselected, 15 subtests passed` - [x] sandbox suite: `460 passed, 3 skipped` - [x] `coverage report --fail-under=96` exits successfully; every executable production line added by this PR is covered --------- Signed-off-by: qiaochuz <qiaochuz@nvidia.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Multi-Instance vLLM Support in `gym eval submit` ### Summary Adds support for running multiple vLLM engine replicas in `gym eval submit` using vLLM's native data-parallel multi-instance mode. Previously, `VllmServiceConfig` only supported a single vLLM engine. This change introduces two new fields: - **`number_of_instances`** — the number of engine replicas to run (default `1`, preserving existing behavior) - **`distributed_backend`** — a typed, extensible sub-config declaring how replicas are coordinated. The only type available in this PR is `vllm_service`, which maps to vLLM's `--data-parallel-size N` flag. vLLM manages load balancing internally, so no external router or extra processes are needed. When `number_of_instances > 1`, the generated sbatch script passes `--data-parallel-size N` to the single `vllm serve` command. The single endpoint, health check, and driver URL wiring are all unchanged. The `distributed_backend` field is designed to be extended in future PRs — Ray Serve and Dynamo routing backends will each add a new type to the discriminated union without touching existing configs. ### Validation rules - `number_of_instances > 1` requires `distributed_backend` to be set (fails loudly with a clear message) - `number_of_instances == 1` with `distributed_backend` set is rejected - `number_of_instances < 1` is rejected ### Also included - `--pipeline-parallel-size` is now emitted in the generated `vllm serve` command when `> 1` (it was in the config but was never passed to the CLI) - An example YAML (`examples/slurm_vllm_multi_instance.yaml`) showing a 4-instance setup with TP=2 across 8 GPUs --------- Signed-off-by: Tomasz Grzegorzek <tgrzegorzek@nvidia.com> Signed-off-by: prokotg <19536019+prokotg@users.noreply.github.com> Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com> Co-authored-by: Tomasz Grzegorzek <tgrzegorzek@nvidia.com> Co-authored-by: prokotg <19536019+prokotg@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…ruction-following (#2586) ## What does this PR do? Closes #2585 Adds `resources_servers/citation_if`, a resources server that grades **citation instruction-following**: did the model cite in the grammar it was told to use, cite only IDs that exist in its retrieved context, and cite the ones that were required — and did it actually answer. CPU-only string matching, one `/verify` endpoint, binary reward; no model calls, no GPU. Each row is a complete frozen conversation — system instruction naming a citation grammar, a question, search rounds, then a final user turn that closes retrieval and asks for synthesis. The policy answers once. ### Verification logic Pure string matching in `scorer.py` (no FastAPI dependency, importable standalone). Gate sequence in cite mode: | # | gate | fails when | |---|---|---| | 0 | `structural` | empty final text, or the policy emitted a tool call | | 0b | `no_answer` | citation markup only — no word character survives once citations are removed | | 1 | `malformed_attempt` | after stripping well-formed spans, any residual ID-shaped token remains | | 2 | `must_cite` | fewer than `min_valid_citations` valid citations | | 3 | `no_hallucination` | a parsed ID is not in `valid_id_set` | | 4 | `correctness` | a required ID was missed, or `|cited| > |expected| + expected_slack` | `mode=no_cite` inverts gates 1-4 (reward 1 iff zero citations of any shape), which is how abstention rows are expressed. 9 citation grammars; `id_regex` is authoritative per row, so random-ID and numeric-ID schemes coexist in one pool. Two further grammars are marked `holdout: True` and must never be assigned to training rows — they exist to detect grid overfitting. **The answer's content is deliberately never graded.** The reward is for instruction-following and the data carries no gold answer. But an answer must *exist* — see gate 0b. Those two properties are locked by an adjacent pair of tests so a future "fix" to one cannot silently undo the other. ### Source of prompts/tasks The committed `data/example.jsonl` (5 rows) is **synthetic — hand-written for this server**, covering four grammars, both `id_kind` values, and one `no_cite` row. It is a contract example, not a difficulty example, and the README says so. The training corpus this reward was developed against is internal and **is not included**. Nothing in the server depends on it; `configs/citation_if.yaml` points `train` at a gitignored path. ### Tests — 327 passing - `test_citation_if.py` — hand-written gate tests, one or more per gate and failure mode - `test_reward_hack_matrix.py` — generated matrix, 16 known reward-hack classes x 9 grammars x 2 ID granularities = 265 cells; pass bar is 100% - `test_residual_fuzz.py` — 3 property tests x 1,000 mutations, **both directions**: leak mutations must score 0, benign ones (whitespace, punctuation, unicode quotes) must stay 1. An over-strict gate is as much a regression as a leaky one and much harder to notice. - `test_app.py` — server layer: Responses-API extraction contract and the `/verify` response ### Licensing Code: Apache 2.0. Dependencies: `nemo_gym` (Apache 2.0). No external corpus is redistributed — the shipped example rows are synthetic. ### Notes for review - `verified: false`, per the convention that the flag is maintainer-granted. - The top-level `README.md` change is generated by the `update-readme-table` pre-commit hook, not hand-edited. ## Checklist - [x] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Kavya Nayaka <knayaka@nvidia.com> Co-authored-by: Christian Munley <cmunley@nvidia.com>
## Problem
`TrainDataProcessor.validate_samples_and_aggregate_metrics` writes the
aggregate-metrics sidecar (`<dataset>_metrics.json`) next to the dataset
**unconditionally** — even when the on-disk sidecar already exists and
matches the freshly computed metrics:
```python
maybe_conflicting = self._validate_aggregate_metrics(aggregate_metrics_dict, metrics_fpath)
if maybe_conflicting is not None:
...
# always runs, even on a clean match:
with open(metrics_fpath, "w") as f:
json.dump(aggregate_metrics_dict, f, indent=4)
```
`_validate_aggregate_metrics` already returns `None` both when the
sidecar is *missing* and when it *matches*, so a matching sidecar still
gets truncated and rewritten with identical bytes.
That redundant `open(..., "w")` breaks a common multi-user setup: when
the dataset lives in a **shared, read-only directory** — e.g. a
project-wide pre-staged benchmark workspace bind-mounted into the eval
container — a user who doesn't own the pre-staged sidecar hits:
```
PermissionError: [Errno 13] Permission denied:
'benchmarks/swebench/data/swebench_verified_benchmark_metrics.json'
at nemo_gym/train_data_utils.py:670 (validate_samples_and_aggregate_metrics)
```
The run aborts during data preparation, before any rollout. This
surfaced running SWE-bench Verified P/D on a Slurm cluster where the Gym
workspace is shared across a project group.
## Fix
Skip the write when the sidecar already exists and no conflict was
reported. Missing-file and overwrite-on-conflict paths are unchanged.
| sidecar state | before | after |
|---|---|---|
| missing | write | write |
| exists & matches | **rewrite (redundant)** | **skip** |
| exists & differs, `overwrite_metrics_conflicts=True` | overwrite |
overwrite |
| exists & differs, `overwrite_metrics_conflicts=False` | skip + raise |
skip + raise |
In the newly skipped case the write was provably a no-op (identical
bytes), so on-disk output is unchanged.
## Tests
- New
`test_validate_samples_and_aggregate_metrics_skips_rewrite_when_sidecar_matches`:
copies the bundled example dataset into a tmp dir, runs once to create
the sidecar, makes it **read-only**, and asserts a second pass neither
raises nor rewrites it. Verified this fails without the fix
(`PermissionError`) and passes with it.
- Updated `test_validate_samples_and_aggregate_metrics_sanity`: its
bundled sidecar already matches, so it is no longer rewritten — the
write assertion is now `[]`.
`pytest tests/unit_tests/test_train_data_utils.py` → 22 passed. `ruff` +
`pre-commit` clean on the changed files.
## Note for maintainers
This fixes the redundant-rewrite that breaks the shared-read-only case.
The deeper design point — the metrics sidecar is written *relative to
cwd*, which `_resolve_under_cwd_or_install` documents as needing to be
"the user's writable cwd" but which is the shared install root under a
bind-mounted workspace — is left as-is. Happy to follow up if you'd
prefer the sidecar target to move to an explicit output dir instead.
Signed-off-by: Grzegorz Chlebus <gchlebus@nvidia.com>
<!-- Thanks for contributing to NeMo Gym! Please fill out the sections below. --> ## What does this PR do? <!-- Briefly describe the change and the motivation. Link any related issue, e.g. "Closes #123". --> ## Checklist - [x] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [X] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: DanialTaheri <smohsenitahe@nvidia.com> Co-authored-by: Christian Munley <cmunley@nvidia.com>
## Summary - Add the required `README.md` for `responses_api_agents/image_tools_agent`. - Allow the server suite to include the agent and satisfy its total-module count check. ## Test plan - [x] `gym env test +entrypoint=responses_api_agents/image_tools_agent` fyi @DanialTaheri Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
## Summary - add the main Gym CI workflow with preflight, unit tests, container build, GPU test matrix, and aggregate status check - add shared Gym change classification so docs-only behavior matches the unit-test workflow without label overrides - add a simplified CPU/GPU container test action and an initial nvidia-smi E2E placeholder ## Testing - pytest tests/unit_tests/test_ci_environment.py -q - pre-commit run --files .github/actions/classify-changes/action.yml .github/actions/test-template/action.yml .github/workflows/cicd-main.yml .github/workflows/unit-tests.yml tests/e2e/gpu_e2e_test.sh tests/unit_tests/test_ci_environment.py - YAML parsing, Bash syntax validation, and git diff --check --------- Signed-off-by: Charlie Truong <chtruong@nvidia.com>
…es (#2124) Records the exact prompt token IDs, generated token IDs, and generation log probabilities for model calls made by an external agent harness. ## Control and data flow ```mermaid sequenceDiagram participant RC as Rollout collection participant AS as Agent server participant H as External harness participant MW as Model middleware participant MS as Model route participant TS as TokenSink RC->>AS: /run with rollout identity AS->>AS: select static agent capture or all_agents AS->>H: model URL with /ng-rollout/id/training-token-capture H->>MW: model request MW->>MW: mint model_call_id and CaptureContext MW->>MS: normalized model request MS->>MS: assemble exact token arrays MS->>TS: await put(TokenEntry) alt capture fails or arrays are incomplete MS->>TS: await mark_incomplete(rollout_id, model_call_id) end MS-->>H: dialect response or stream ``` Capture happens before dialect conversion or stream synthesis can discard token fields. The write is awaited before the model response returns. ## Summary - Separates run-level infrastructure enablement from static per-agent capture selection; `all_agents` selects every configured agent for a training run. - Encodes selected training intent in the explicit `/training-token-capture` path, independently of rollout correlation and evaluation observability. - Defines framework-neutral async `TokenSink` and `TokenSource` contracts with durable incomplete state, atomic frozen snapshots, lifecycle hooks, and versioned conditional retirement. - Keeps a frozen tombstone after retirement so a late writer cannot resurrect a consumed attempt; explicit pre-dispatch cleanup starts the next attempt. - Uses durable call digests and tail reconciliation so the common append path does not reread or reserialize prior token arrays. - Rejects malformed or conflicting token records and closes owned endpoints through the app lifespan. Gym imports no training-framework data plane. A framework configures sink and lineage proxy factories in Gym model-server workers and constructs its source in the rollout-consumer process. Stack base. Followed by #2125. --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…ity (#1669) NemoGymLLM built a fresh httpx.AsyncClient for every chat-completion call and closed it immediately, dropping the Gym vllm_model server's session cookie. The server's SessionMiddleware then minted a new session_id each turn and round-robined it across the data-parallel vLLM engines, so the growing conversation prefix was almost never warm on the engine handling the next turn -> prefix-cache miss -> the full context was re-prefilled every turn. With many engines this made multi-turn agentic generation much slower than single-node and inflated async-GRPO exposed_generation (training stalled waiting on rollouts). Reuse one persistent client for the lifetime of the per-episode NemoGymLLM so the session cookie persists, pinning the whole episode to a single engine and keeping its prefix cache warm (only new tokens are prefilled each turn). Close it in Terminus2NemoGym.run()'s finally so connections are not leaked across Ray-worker reuse. Timeout is applied per-request to preserve the per-call override. Sequential per-episode reuse (one connection at a time) does not hit the high-concurrency httpx pooling issue and reduces client churn versus the previous per-call creation. Signed-off-by: Puneesh Khanna <puneesh.khanna@tii.ae> Co-authored-by: bxyu-nvidia <bxyu@nvidia.com>
) Builds one trainable Responses trajectory from a rollout's unordered token-capture records. ## Reconstruction flow ```mermaid flowchart TD E[Unordered TokenEntry records] --> Z[Exclude calls with no generated tokens] Z --> O[Order calls by prompt length] O --> P[Find the earlier call whose complete tokens are the longest strict prompt prefix] P --> A{Several candidates share the longest prefix?} A -->|yes| Q[Quarantine the ambiguous call] A -->|no| L[Attach the inferred predecessor] Q --> F[Build roots and chains] L --> F F --> S{Exactly one root and one trainable chain?} S -->|no| M[mask_sample = true] S -->|yes| R[Project contiguous Responses output] R --> C[Assert prefix continuity and report metrics] ``` The predecessor relationship is inferred only during reconstruction: call B follows call A when A's complete token sequence is a strict prefix of B's prompt. ## Summary - Chains calls through strict longest-prefix inference and quarantines indistinguishable candidates instead of guessing. - Excludes empty generations so a filtered call cannot become the predecessor of its retry. - Fails closed for incomplete capture, no trainable generation, multiple roots or chains, unresolved records, or unsafe projection. - Reports roots, chains, quarantine, delivered-token fraction, empty generations, and parent-link failures. - Keeps the low-level `per_request` builder for multi-trajectory consumers, while single-response delivery rejects it explicitly. - Covers `trajectories_from_source` success, incomplete state, source failures, and unsupported single-response modes. Depends on #2124. Consumed by #2126. --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
`sbatch_external_vllm.sh` hardcoded the rollouts path and appended its
`++output_jsonl_fpath` **after** the caller's arguments. Duplicate `++`
overrides are last-wins, so a caller could not choose the path even by
passing
it explicitly.
`export_to_csv.py` derives the aggregate as
`<base>_aggregate_metrics.json`, so
that name carries the timestamp too — and anything reading the aggregate
without having watched the job run has to glob for it.
`ROLLOUTS_FPATH` overrides the path. Unset, behaviour is unchanged.
Also fixes the CSV copy beside it: `cp
results/$experiment_name_export.csv`
expanded `${experiment_name_export}`, an unset variable. It sits behind
`EXPORT_CSV_TO_MODEL_DIR` (default 0), so it had not been hit.
Tested by rendering the submitted script with a stubbed `sbatch` and
simulating
the job-side resolution:
```
default: results/<exp>/slurm_job_id_12345/date_20260821_101500_aggregate_metrics.json
overridden: results/evaluator_rollouts_aggregate_metrics.json
```
`bash -n` clean.
---------
Signed-off-by: Piotr Laszkiewicz <plaszkiewicz@nvidia.com>
Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
Attaches a safely rebuilt token trajectory to each participating rollout
and retires capture evidence only after durable handoff. This PR also
owns the durable capture lifecycle used by the later lineage and
prefix-supply PRs.
## Capture custody and delivery
```mermaid
sequenceDiagram
participant M as Model worker
participant S as TokenSink / TokenSource backend
participant C as Rollout collector or framework
participant B as Trajectory builder
participant D as Durable downstream
M->>S: begin_call(rollout_id, model_call_id) when supported
Note over M,S: Failure occurs before generation - a dangling intent identifies a lost entry
M->>M: run inference and build TokenEntry
M->>S: await put(TokenEntry)
S-->>M: entry durable
M-->>M: return model response
C->>S: freeze(rollout_id)
S-->>C: TokenCaptureSnapshot(entries, incomplete, snapshot_id, version)
C->>B: rebuild frozen snapshot
alt incomplete, malformed, ambiguous, split, or empty
B-->>C: mask_sample=true and diagnostics
Note over C,S: retain evidence
else one safe trainable chain
B-->>C: rebuilt response and metrics
C->>D: write rebuilt rollout
D-->>C: durable acknowledgement
C->>S: drop(rollout_id, snapshot_id, version)
S-->>C: true only if the frozen version is still current
end
```
A write racing `freeze` may become durable, but it advances the version
so a stale `drop` cannot erase it. Retirement keeps a tombstone until
operator-controlled garbage collection, preventing an old worker from
recreating a retired rollout.
## Summary
- Clears and finalizes records only for agents selected by static
`token_id_capture` configuration or `all_agents`; tokenless
nonparticipants remain ordinary evaluation rollouts.
- Defines `TokenSink.put` as an awaited durability boundary and supports
optional pre-dispatch `begin_call` custody. A dangling intent makes the
frozen snapshot incomplete when both `put` and `mark_incomplete` fail.
- Freezes entries and incomplete state as one versioned snapshot,
deduplicates at-least-once entries, and conditionally retires only the
consumed `snapshot_id` and version after downstream durability.
- Masks missing, malformed, incomplete, ambiguous, multi-root,
multi-chain, and empty-token builds. `max_mask_fraction` can stop a run
that produces too many masked rollouts.
- Recovers a lagging state index from the durable JSONL tail, repairs an
unacknowledged torn final line, reduces append-path state fsyncs, and
provides `sweep_retired` for explicit tombstone cleanup.
- Fsyncs Gym's JSONL result before retirement and retains failed or
masked evidence.
- Uses an installed caller-owned source without closing it, or
constructs and closes Gym's default file source.
- Fails before rollout dispatch when selected rows require rebuilding
and the collector process has no source.
- Provides transport conformance checks for publication, incomplete
state, freeze, idempotency, conditional retirement, and optional
call-intent custody.
Depends on #2125. The integration guide begins in #2341.
---------
Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
## What does this PR do? - Updates the `openclaw_agent` to use the new `openclaw onboard` CLI command instead of the deprecated `setup` command,. - Makes `context_window` / `max_output_tokens` optional config values instead of hardcoded defaults, so the model entry only sets `contextWindow`/`maxTokens` when explicitly configured. ## Checklist - [x] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Elisa Martinez Abad <elisam@nvidia.com> Co-authored-by: Christian Munley <cmunley@nvidia.com>
<!-- Thanks for contributing to NeMo Gym! Please fill out the sections below. --> ## What does this PR do? 1. DeepSWE benchmark from @sdevare-nv: benchmarks/deepswe, resources_servers/deepswe 2. Minor fixes/changes: benchmarks/nemotron_3.5_super/vllm_configs/nemotron_3.5_super.sh, benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh ## Checklist - [ ] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [ ] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [ ] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [ ] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [ ] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Brian Yu <bxyu@nvidia.com>
…t by timeout (#2615) ## What does this PR do? Salvages the partial trace when an OpenClaw rollout is cut short by timeout, instead of losing the whole transcript. Previously, the agent only read OpenClaw's session file from the completion envelope, which is absent on a cut-short run, and the `finally` block deleted the workdir before anything could recover it — so a timed-out rollout came back empty. This PR handles both cut-short paths: our own `config.timeout` (`TimeoutError`), and an outer harness/sandbox timeout that SIGTERMs this process during its grace window before SIGKILL. On either path, we stop waiting, locate OpenClaw's incrementally-written session on disk (`_find_partial_session`), parse it, and return the partial output — so the harness records a partial transcript instead of an empty rollout. Returning quickly on SIGTERM keeps the response inside the grace window, so no harness-side change is needed. Pairs with the apptainer provider's SIGTERM-grace change. ## Checklist - [x] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Elisa Martinez Abad <elisam@nvidia.com> Signed-off-by: Christian Munley <cmunley@nvidia.com> Co-authored-by: James Nolan <jnolan@nvidia.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Christian Munley <cmunley@nvidia.com>
## What does this PR do? - Adds an OpenClaw config for the Anyterminal agent (`anyterminal_openclaw.yaml`), letting Terminal Bench run via OpenClaw natively inside the task container, alongside a deps install script (`openclaw_agent_deps.sh`) that provisions portable Python, Node, and the OpenClaw CLI. - Also bumps the portable Python version (3.12.8 → 3.13.14). - Fixes `_resolve_model_base_url`/`_resolve_base_url` overrides to accept arbitrary args. - Adds the new entry to the README benchmark table. ## Checklist - [x] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Elisa Martinez Abad <elisam@nvidia.com> Signed-off-by: Christian Munley <cmunley@nvidia.com> Co-authored-by: James Nolan <j.nolan@otherwise.ch> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Christian Munley <cmunley@nvidia.com>
<!-- Thanks for contributing to NeMo Gym! Please fill out the sections below. --> ## What does this PR do? Fixes secret redaction in the Anyterminal agent: `_safe_config_json`'s redaction did a substring match on key names (`api_key`, `secret`, `password`, `token`), which also caught unrelated fields like `max_output_tokens` and mangled them in logs/output. Splits the previously combined "serialize + redact" step into `_full_config_dict` (unredacted, for internal use) and `_redact_secrets` (for logs/display only), and reworks `_responses`/`run` to redact directly from the live params object instead of round-tripping through `AnyTerminalInstanceConfig.model_validate_json(...)`, so a lossy redacted string never gets re-parsed back into a config. ## Checklist - [x] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Elisa Martinez Abad <elisam@nvidia.com> Signed-off-by: Christian Munley <cmunley@nvidia.com> Co-authored-by: Christian Munley <cmunley@nvidia.com>
Signed-off-by: Christian Munley <cmunley@nvidia.com>
<!-- Thanks for contributing to NeMo Gym! Please fill out the sections below. --> ## What does this PR do? 1. DeepSWE cleanup: benchmarks/deepswe/opencode.yaml, resources_servers/deepswe/configs/deepswe.yaml, resources_servers/deepswe/tests/test_app.py, resources_servers/deepswe/app.py 2. Add Nemotron 3.5 Lightning config to Super 3.5 evals: benchmarks/nemotron_3.5_super/vllm_configs/nemotron_3.5_lightning.sh 3. Rollout collection logs intermediate progress: nemo_gym/rollout_collection.py 4. Enable Opencode agent observability: responses_api_agents/opencode_sandboxed_agent/app.py 5. Misc: benchmarks/nemotron_3.5_super/vllm_configs/nemotron_3.5_super.sh, benchmarks/nemotron_3.5_super/eval_container_config.yaml, benchmarks/nemotron_3.5_super/eval_container_config_with_staged.yaml, benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh, nemo_gym/environment/scaffold.py, tests/unit_tests/test_environment_scaffold.py ## Checklist - [ ] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [ ] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [ ] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [ ] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [ ] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Brian Yu <bxyu@nvidia.com>
This MR unifies MMLU-ProX answer extraction with MMLU-Pro and batches xCOMET scoring for WMT24++. - **MMLU-ProX:** 70.13 (no patch version) → 72.11 (+2.0 pp). Japanese 57.95 → 69.10 (+11.2 pp). No-answer 13.2% → 10.3%. Zero correct→incorrect regressions. - **WMT24++:** `verify()` is BLEU only, xCOMET runs as batched `predict` in `compute_metrics()` (128×4), with jsonl checkpoints so scoring can resume --------- Signed-off-by: Meriem Boubdir <mboubdir@nvidia.com> Co-authored-by: Grzegorz Chlebus <gchlebus@nvidia.com>
Current RULER evaluation didn't have answer prefix as default. We add assistant message at the end and use `continue_final_message=True` to continue assistant message. There are two things we need to know in this feature: - `continue_final_message=True` can only continue `content` of the assistant message in non-reasoning mode. - the response will be in `reasoning`or `reasoning_content` since vllm by default will split response with reasoning parser. Skills on RULER 1M with Nemotron 3 Nano: 86.3. Gym on RULER 1M with Nemotron 3 Nano: 86.1. --------- Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com> Signed-off-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com> Co-authored-by: Christian Munley <cmunley@nvidia.com>
<!-- Thanks for contributing to NeMo Gym! Please fill out the sections below. --> ## What does this PR do? This PR fixes timeout issue when using lc_niah env. Change default reasoning overlap metric LCS to seq_match. <!-- Briefly describe the change and the motivation. Link any related issue, e.g. "Closes #123". --> ## Checklist - [x] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>
… AB600 panel config Add client_timeout_s / client_max_retries to VerifiersAgentConfig and pass them to AsyncOpenAI. None keeps the openai SDK defaults (600s, 2 retries). The AutomationBench config sets 3600s and 0 retries: a single long agentic turn from a large policy on a shared engine exceeds 600s under load, and the SDK otherwise retries the whole generation. config_ab600_panel.yaml registers a validation-only second instance of the same agent, automationbench_aa, with reward_fn pinned to aa_headline, so a run training on partial_credit can still report the AA-comparable number. Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
client_connect_timeout_s joins client_timeout_s / client_max_retries. The SDK's 5s connect, with no retries, permanently aborted 196 of 300 step-0 validation rollouts on a 116B policy: a validation pass fires every rollout at once into engines that have just come up, and the connections never established in time. AutomationBench sets 60s so the burst queues in the accept backlog instead. Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
…_model router spreads across engines vllm_model picks a vLLM engine per session (sha256(session_id) % len(base_urls) in _resolve_client) and mints the session id per cookie jar (server_utils.setup_session_middleware). openai.AsyncOpenAI sits on an httpx client that persists cookies, so the single cached client this agent shared across every rollout in its process was one session and therefore one engine. On CMH job 3670120 (2026-09-10) 512 concurrent rollouts ran on 6 of 48 engines while 42 sat idle; the router's outbound connection table showed exactly those six. Gym's own aiohttp client sidesteps this with a DummyCookieJar. Drop client_cache. _make_openai_client() builds a client per rollout with the same timeout/retry kwargs; the handler wraps it in NeMoRLChatCompletionsClient for vf_env.run_group and closes it in a finally. Each rollout now owns its session: all of its turns stay on one engine, which is the affinity the router is built for, and the rollouts as a whole hash across every engine. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
…sion cookie Corrects 19bfe25. A client per rollout did spread the 512 rollouts over all 48 vLLM engines (CMH 3670792: 48/48 generation nodes busy, router fan-out 48 engines), but each rollout's single pooled connection sat idle through its tool phases, the router's uvicorn closes idle connections after 30 s (server_utils timeout_keep_alive=30), and the next turn raced that close: 468 of 512 rollouts aborted with ModelError -> APIConnectionError -> ReadError(BrokenResourceError) while the two shared-client runs before it had zero aborts. The verifiers Environment never stores the client, so this was the connection pattern, not a shared-object bug. Go back to one shared AsyncOpenAI per process so the pool stays hot, and give it a CookieJar whose set_cookie is a no-op. Every request is then cookie-less, the router mints a fresh session id per request and sha256(session_id) % 48 spreads them, exactly as Gym's own aiohttp client behaves with DummyCookieJar. Pass the bare CookieJar: httpx.Cookies adopts a CookieJar instance but copies an httpx.Cookies into a fresh stdlib jar, which would silently discard the no-store behaviour (caught by the offline check, which now includes a positive control showing the default client does re-send the cookie). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
…nt contract The vllm_model router picks a vLLM engine per session cookie, so the properties a100181 relies on get tests: _get_client() returns one shared client; it carries the configured timeouts/retries; its jar is _NoStoreCookieJar (httpx.Cookies adopts a bare CookieJar but copies an httpx.Cookies into a fresh jar, which would silently drop the subclass); driven through an httpx.MockTransport that sets Set-Cookie on every reply, three consecutive chat completions send no Cookie header. A positive control shows a stock AsyncOpenAI replays the cookie from the second request on, so the suite is known to be sensitive; it fails when the cookies= argument is removed from the client construction. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Checklist
pre-commit run --all-files) (so CI lint/format/copyright pass).git commit -s) (so the DCO check passes).