gateway: linear (DAG) delta-chain token storage — RFC + Phase 1 - #767
Draft
jeffreysijuntan wants to merge 59 commits into
Draft
gateway: linear (DAG) delta-chain token storage — RFC + Phase 1#767jeffreysijuntan wants to merge 59 commits into
jeffreysijuntan wants to merge 59 commits into
Conversation
…uation mangling (#656) * fix(sandbox): add [environment].replay_dockerfile toggle + fix RUN-continuation mangling Non-docker backends (modal/daytona) replay a task's Dockerfile RUN steps on top of the booted image. Two defects (#655): 1. Multi-line `RUN ... \` steps were rejoined with "\n" instead of a space, producing invalid shell (observed: `bash: syntax error near '&&'`). 2. The replay assumes the configured image is a *base*. Terminal-bench / Harbor tasks boot a *fully-built* image, so replaying the RUN steps double-applies the build (`git clone ... already exists`, missing COPY'd files, etc.). Fixes: - `_dockerfile_run_commands` joins `\`-continuations with a space so multi-line RUN steps stay valid when re-exec'd via `bash -c`. - New `[environment].replay_dockerfile` toggle. Default `true` preserves SWE-bench base-image behavior; `false` skips replay for prebuilt task images. Honored across the resolution cold path, daytona + modal snapshot builds, and the snapshot env-key fingerprint. - Tests for the continuation join, COPY skipping, and the toggle (incl. the task.toml -> metadata path). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style: apply ruff-format to satisfy pre-commit Collapse multi-line expressions onto single lines per ruff-format (line-length=200), fixing the failing pre-commit check on PR #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
save_config() rebuilt config.json from scratch, writing only provider/model/api_keys/base_url. Any other key was dropped — so `rllm model swap`/`setup` silently wiped the `ui_api_key` written by `rllm login`, logging the user out of the rLLM UI. Merge into the existing file instead (mirroring save_ui_config). Adds a regression test. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…#658) * feat(gateway): cumulative token mode for the in-process (Tinker) path `cumulative_token_mode` (drift-free multi-turn token forwarding: turn 2+ is rewritten to a pre-tokenized prompt built by renderers.bridge_to_next_turn, avoiding decode→re-encode drift) previously only worked for the HTTP-proxy (vLLM/verl) path — the cumulative handlers always routed to an HTTP worker's /v1/completions, so backends that run in-process via `local_handler` (Tinker) fell back to re-rendering + re-tokenizing the full conversation every turn. This wires the same feature through the local_handler path, reusing the existing backend-agnostic TokenAccumulator + Prime Intellect `renderers` bridge: - tinker_adapter: the handler now detects a pre-tokenized `prompt` (list[int]) and samples straight from it via the engine's existing get_token_output_from_token_input + assemble_model_output, returning a completions-style body (prompt_token_ids + choices[].token_ids) — the shape the gateway's cumulative handler already extracts token IDs from. The chat (messages) path is unchanged. - proxy: _handle_cumulative_non_streaming routes to local_handler when present (no HTTP worker); added _handle_cumulative_streaming_local to synthesize an SSE stream from the single in-process completion (mirrors _handle_streaming_local), with the same token ingest. Net effect: Tinker rollouts get the same prefix-extension guarantee verl has — turn N's prompt tokens are byte-for-byte prior turns' prompt+completion tokens, so the sequence the optimizer trains on matches what was generated. Tests (no Tinker service / model required): - tests/test_tinker_adapter_cumulative.py — token-prompt path samples from tokens; chat path unchanged; non-int prompt falls through. - rllm-model-gateway/tests/unit/test_cumulative_token_mode_local.py — proxy local cumulative non-streaming + streaming ingest + chat translation. Existing cumulative/accumulator suites (28) still pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(gateway): wrap long SSE chunk lines to satisfy ruff E501 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cookbooks): add swe-rl recipe (rllm-swesmith → SWE-bench Verified)
End-to-end SWE-RL cookbook that pairs rLLM's native `rllm-swesmith`
training set with `harbor:swebench-verified` for eval, driving the
in-tree `mini-swe-agent` harness inside per-task sandboxes. Default
model is Qwen/Qwen3.5-9B + LoRA-32, GRPO + async + compact filtering,
64 parallel Daytona sandboxes.
No custom AgentFlow or evaluator: the harness owns the action loop,
each task's `tests/test.sh` is the verifier (pytest for swesmith, the
official SWE-bench harness for Verified), and the gateway captures
trajectories transparently.
Files: prepare_data.py (dataset pull), train.py + train_{tinker,verl}.sh
(recipes), test.py (catalog + harness smoke tests), README.md, pyproject.toml.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cookbooks/swe-rl): use rllm.rollout.* keys; drop stale gateway overrides
The unified trainer config exposes sampling params at
`rllm.rollout.{train,val}.{temperature,top_p}`. The `sampling.*` paths
copied from `examples/harbor_swe/train_harbor.sh` predate the unified
config and break Hydra struct validation. Same template also passed
`rllm.gateway.public_url` / `rllm.gateway.sampling_params_priority`,
neither of which exist in the current schema — drop them.
Verified by resolving the config with --cfg job: no validation errors.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(swe-rl): run on rLLM-native SandboxedAgentFlow path; unblock async training
Switch the cookbook off the remote Harbor runtime onto rLLM's own
SandboxedAgentFlow path (AgentFlowEngine) and fix the bugs that surfaced
once real rollouts ran end-to-end on Modal sandboxes.
train.py / train_*.sh:
- Pass MiniSweAgentHarness as agent_flow so AgentTrainer auto-wires
SandboxTaskHooks + per-task verifiers via AgentFlowEngine (not the
remote_runtime/RemoteAgentFlowEngine path). Sandbox backend selected by
SWE_SANDBOX_BACKEND (default modal).
- Load the val split as "default" (Harbor-pulled name), not "test".
- Async training requires train_batch_size=1 and raise_on_error=false; set
both. Effective batch is mini_batch_size(16) groups x group_size(8).
- Add SWE_VAL_MAX to cap the 500-task SWE-bench-Verified val set.
- Drop stale rllm.remote_runtime.* overrides; doc the sandbox backends.
rllm/data/utils.py:
- task_from_row now roots the Task at the row's task_path and merges
task.toml/Dockerfile metadata, so per-task verifier + image resolution
work on the training path (fixes "No verifier configured").
rllm/gateway/tinker_adapter.py:
- Translate TerminationEvent into an OpenAI-standard 400
context_length_exceeded instead of a 500. litellm maps it to a
non-retryable ContextWindowExceededError, so an over-length in-sandbox
agent stops immediately instead of retrying until the run-timeout
SIGKILL — which was stalling group completion and wedging async steps.
rllm/harnesses/mini_swe_agent.py:
- Retry the in-sandbox uv install (Modal->GitHub egress resets the
connection intermittently and aborted the install under set -e).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(gateway): cumulative token mode for the in-process (Tinker) path
`cumulative_token_mode` (drift-free multi-turn token forwarding: turn 2+ is
rewritten to a pre-tokenized prompt built by renderers.bridge_to_next_turn,
avoiding decode→re-encode drift) previously only worked for the HTTP-proxy
(vLLM/verl) path — the cumulative handlers always routed to an HTTP worker's
/v1/completions, so backends that run in-process via `local_handler` (Tinker)
fell back to re-rendering + re-tokenizing the full conversation every turn.
This wires the same feature through the local_handler path, reusing the
existing backend-agnostic TokenAccumulator + Prime Intellect `renderers`
bridge:
- tinker_adapter: the handler now detects a pre-tokenized `prompt` (list[int])
and samples straight from it via the engine's existing
get_token_output_from_token_input + assemble_model_output, returning a
completions-style body (prompt_token_ids + choices[].token_ids) — the shape
the gateway's cumulative handler already extracts token IDs from. The chat
(messages) path is unchanged.
- proxy: _handle_cumulative_non_streaming routes to local_handler when present
(no HTTP worker); added _handle_cumulative_streaming_local to synthesize an
SSE stream from the single in-process completion (mirrors
_handle_streaming_local), with the same token ingest.
Net effect: Tinker rollouts get the same prefix-extension guarantee verl has —
turn N's prompt tokens are byte-for-byte prior turns' prompt+completion tokens,
so the sequence the optimizer trains on matches what was generated.
Tests (no Tinker service / model required):
- tests/test_tinker_adapter_cumulative.py — token-prompt path samples from
tokens; chat path unchanged; non-int prompt falls through.
- rllm-model-gateway/tests/unit/test_cumulative_token_mode_local.py — proxy
local cumulative non-streaming + streaming ingest + chat translation.
Existing cumulative/accumulator suites (28) still pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(swe-rl): enable cumulative token mode + raise context window
Enables the in-process cumulative-token path (cherry-picked from #658) in the
tinker recipe for testing: rllm.gateway.cumulative_token_mode=true +
renderer_family=qwen3.5. Also raises training.max_length=65536 /
data.max_prompt_length=57344 so long mini-swe-agent trajectories fit, and
save_freq=10.
NOTE: the cumulative feature commit on this branch is a cherry-pick of #658 for
local testing — drop it (rebase onto main) once #658 merges to avoid duplication.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(swe-rl): add synchronous tinker training script
train_tinker_sync.sh: on-policy variant of train_tinker.sh for testing —
drops async_training (synchronous generate→train) and uses a real
data.train_batch_size (default 4; effective batch = train_batch_size x
group_size). Same model/sandbox/context/cumulative settings otherwise.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(swe-rl): disable cumulative token mode (incompatible with thinking model)
Qwen3.5 is a thinking model (<think>...</think> per assistant turn). The normal
re-render path strips prior-turn reasoning from history (mini-swe-agent stores
the think-stripped message.content), but cumulative mode carries forward the RAW
completion tokens (tinker_engine completion_ids=response_tokens, incl. <think>),
which renderers.bridge_to_next_turn concatenates verbatim. The model then sees an
ever-growing stack of its own prior reasoning — out-of-distribution for multi-turn
— degrading into short, submit-early, zero-reward trajectories.
Renderer mismatch was ruled out: the bridge (PrimeIntellect qwen3.5) and the
engine renderer are token-identical; the bridged prompt is well-formed. The
issue is purely reasoning carry-forward. Cumulative mode is fine for
non-thinking models / single-turn; for thinking models, correct multi-turn
rollout requires stripping prior reasoning (i.e. re-tokenization), which is
fundamentally at odds with exact-token reuse.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(swe-rl): switch to Qwen3.5-4B + terminus2 harness
- agent_flow -> Terminus2Harness (was MiniSweAgentHarness) in train.py;
docstring/script prose updated accordingly.
- model.name / MODEL_PATH -> Qwen/Qwen3.5-4B; renderer_family -> qwen3.5;
experiment names -> r2egym-terminus2-qwen3.5-4b[-sync/-verl].
- re-enable rllm.gateway.cumulative_token_mode in the tinker scripts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(swe-rl): bump n_parallel_tasks to 128 (tinker + sync)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add train_fireworks.sh Co-Authored-By: 1stprinciple <ztyharbin@gmail.com> * Support FireworksEngine in gateway Co-Authored-By: 1stprinciple <ztyharbin@gmail.com> * add max_prompt_length limit Co-Authored-By: 1stprinciple <ztyharbin@gmail.com> * use tokenizer_model when exists. Co-Authored-By: 1stprinciple <ztyharbin@gmail.com> * style(gateway): fix ruff formatting (trailing whitespace) Co-Authored-By: 1stprinciple <ztyharbin@gmail.com> * feat(swe-rl): scale Fireworks rollout deployment to 4 replicas Set fireworks_config.rollout_deployment_replica_count=4 in both the async and sync Fireworks scripts (the prior fireworks_config.replica_count key does not exist in the backend config). Co-Authored-By: 1stprinciple <ztyharbin@gmail.com> --------- Co-authored-by: 1stprinciple <ztyharbin@gmail.com>
…ks (#668) * feat(algorithm): add ECHO env-observation loss across verl/tinker/fireworks ECHO (arXiv:2605.24517) augments GRPO with an auxiliary cross-entropy loss on environment-observation tokens — the tool/terminal output the policy conditions on but that GRPO masks out of its loss. Advantages are identical to GRPO; the only change is the extra loss term, so every rollout (including failures) becomes dense supervision at no extra rollout cost. Selectable via the unified trainer on all three backends: rllm.algorithm.adv_estimator=echo # lambda defaults to paper's 0.05 rllm.algorithm.env_loss_coef=<lambda> # override; 0.0 == plain GRPO Shared core (backend-agnostic): - rLLMAdvantageEstimator.ECHO + env_loss_coef on AlgorithmConfig; None resolves to 0.05 for echo, else 0.0 (sentinel distinguishes unset from explicit 0). - echo registered as a GRPO-advantage alias in the estimator registry. Per-backend env-CE term (the only backend-specific part): - verl: added in CustomPPOLoss, reusing the existing forward pass log-probs and GRPO's loss_agg_mode/global-batch normalization. Single pass, exact, free. - tinker: gradient-accumulated cross_entropy pass (weights=lambda on obs tokens), returned separately so policy-pass logprobs stay aligned. Clean because tinker's optim_step applies no extra normalization. - fireworks: same second pass, but normalization is folded into weights/advantages and GradAccNormalization is forced to NONE (its counting normalization spans passes and would otherwise rescale the policy gradient). Defaults to 0.0 everywhere, so existing GRPO runs are unchanged. Adds ECHO config tests and documents the switch in the swe-rl cookbook README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(design): auxiliary-loss framework (AuxiliaryLoss) + ECHO/SDAR clients Design doc for a general token-level auxiliary-loss abstraction so algorithms like ECHO and SDAR are a registered spec + config instead of bespoke per-backend surgery. Covers: named token masks (data layer), the AuxiliaryLoss interface (mask/target/weight/requires), a registry, the aux_losses config list, the two backend executors (in-process for verl; gradient-accumulated cross_entropy pass for tinker/fireworks), ECHO and gap-gated SDAR as reference clients, the verl/managed capability model with graceful degradation, and an incremental migration plan that ports the just-merged ECHO onto the framework first. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(algorithm): generalize ECHO into an AuxiliaryLoss framework Prototype of the auxiliary-loss abstraction from design/auxiliary-losses.md (migration steps 1-3): token-level auxiliary losses become a registered spec + config instead of bespoke per-backend code, and ECHO is ported onto it as the first client. Behavior-preserving for ECHO. Shared core (rllm/trainer/algorithms/aux_loss.py): - AuxiliaryLoss spec (mask / weight / requires), register_aux_loss registry, EnvPredictionLoss, and build_aux_losses() with env_loss_coef / adv_estimator=echo back-compat sugar. New algorithm.aux_losses config list (a stack of {type, coef, ...} specs). Backend executors (the only backend-specific part): - verl: CustomPPOLoss._apply_aux_losses folds each term into the single existing forward/backward pass (replaces _add_env_loss). - tinker: _get_aux_loss_futures submits one gradient-accumulated cross_entropy pass per loss (replaces _build_env_datum/_get_env_loss_futures). - fireworks: _build_aux_datums + the normalization fold / GradAccNormalization.NONE now driven by build_aux_losses() rather than env_loss_coef. - Shared managed datum builder in rllm/trainer/tinker/aux_loss.py (aux_positions + build_aux_ce_datum), used by both tinker and fireworks. Adding a new aux loss is now one registered class + config, no per-backend surgery. SDAR's dynamic-weight + teacher-forward extensions (the `requires` field) are the next milestone and currently raise NotImplementedError. Tests: tests/unified_trainer/test_aux_loss.py covers the registry, config resolution + ECHO sugar, the managed datum builder, and the executor math (incl. empty-observation safety). ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style(tinker): satisfy ruff UP038 in aux_loss isinstance Use `int | float` instead of `(int, float)` in the isinstance call (rllm/trainer/tinker/aux_loss.py:51) to fix the pre-commit ruff failure on PR #668. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(algorithms): rewrite RL algorithms overview + register in nav (#675) The core-concepts/rl-algorithms page was stale (documented an old kl_ctrl/AgentPPOTrainer API, only GRPO/PPO/ReMax) and unlinked from the nav. Rewrite it as an accurate overview of the current algorithm surface: - built-in advantage estimators (grpo, reinforce, reinforce_plus_plus_baseline, prpo, rloo, echo) with exact adv_estimator strings - ECHO + the auxiliary-loss framework (env_loss_coef / aux_losses, per-backend cost, metrics) - how to build custom RL algorithms: custom advantage estimator (@register_rllm_adv_estimator) and custom auxiliary loss (@register_aux_loss) - config quick reference + cross-links to advantage-estimator / configuration / capability-matrix / backends Registers the page under the "Advanced algorithms" nav group. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…#674) End-to-end terminal-agent RL recipe, sibling of cookbooks/swe-rl: - train on a local set of Harbor-format terminal-agent tasks (a .tar.zst you provide via TB_TRAIN_TARBALL), eval on harbor:terminal-bench@<TB_EVAL_VERSION> (default 2.0; flip to 2.1 once published) - terminus2 harness runs inside per-task sandboxes; per-task tests/test.sh is the verifier (writes /logs/verifier/reward.txt) - tinker / tinker-sync / verl / fireworks / fireworks-sync train scripts - README documents the ECHO switch (rllm.algorithm.adv_estimator=echo) The training dataset is not public yet, so the README/scripts describe it generically and take the tarball path from TB_TRAIN_TARBALL. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
#677) * feat(harness): configurable rollout limits — uncapped turns, longer timeout, Modal lifetime headroom Stop cutting sandboxed CLI rollouts short, and make the limits configurable rather than hardcoded: - terminus2: max_turns defaults to None (no artificial cap — Harbor's own default) instead of 50. RLLM_TERMINUS_MAX_TURNS is emitted only when set, and the in-sandbox driver treats an absent value as uncapped. - BaseCliHarness: per-rollout run_timeout default 1800 -> 3600s; configure() now consumes an `agent_timeout` override. - eval CLI: add `--agent-timeout SECONDS` (routed through configure(); also published as RLLM_HARNESS_RUN_TIMEOUT_S so the sandbox lifetime tracks it). - modal_backend: derive the sandbox lifetime as run_timeout + install_timeout + headroom so the container can't be reaped before the agent's own timeout fires — the exit-137 / "Sandbox already shut down" failure. The lifetime clock starts at create, before install + run + verify, so a flat default collided with the run timeout. RLLM_MODAL_SANDBOX_TIMEOUT_S still overrides. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(cookbooks): wire TERMINUS_MAX_TURNS knob + tune terminal-rl fireworks config - terminal-rl & swe-rl train.py: read the TERMINUS_MAX_TURNS env knob (empty/0 = uncapped) and pass it to Terminus2Harness(max_turns=...). - all train_*.sh: default TERMINUS_MAX_TURNS=100 (overridable per run), so the operational cap lives in the scripts rather than the harness. - terminal-rl/train_fireworks.sh: grow the context window to match longer, uncapped rollouts — training.max_length 64k->128k, data.max_prompt_length 56k->120k, enable gateway.cumulative_token_mode, and lower async staleness_threshold to 0.5. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Surface the loaded flow's effective knobs as a "Config" row in the `rllm eval` header (alongside Benchmark / Model / Agent / Evaluator): max turns, temperature, run/install timeouts, max concurrent. Knobs are read off the agent instance via a curated (attr, label, formatter) spec, so a flow that doesn't expose one simply omits that chip and an agent exposing none gets no row — graceful across harnesses (react, CLI agents) and harbor runtimes. Rendered as dim-key / bold-value chips joined by a dim "·"; spaces within a chip are non-breaking so a pair like "run timeout 3600s" never splits across a line wrap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(fireworks): per-trajectory session affinity + prompt-cache metric Fireworks reuses a replica's prompt-prefix KV only when a trajectory's turns route to the same replica (keyed on a session id via request headers). The rollout path set no such header, so multi-turn requests scattered across replicas and re-prefilled the (growing) prompt every turn. (a) Session affinity: - Gateway threads the per-rollout session id into the in-process handler body (rllm_session_id) at all local-handler call sites. - tinker_adapter forwards it only to engines advertising supports_session_affinity (Fireworks), so the tinker path is unchanged. - FireworksEngine sets x-multi-turn-session-id / x-session-affinity per request. Since DeploymentSampler.async_completions_stream has no per-call header hook, _inference_headers is monkeypatched to merge headers stashed in an async-safe ContextVar set around each call. (b) Prompt-cache metric: - _record_prompt_cache accumulates ServerMetrics.cached_prompt_tokens / prompt_tokens, adds a per-call prompt_cache_hit_ratio to the metrics dict, and logs a rolling hit-ratio every 50 completions. Low ratio => affinity not pinning a trajectory to one replica. Both are automatic — no config change. Verify by watching the "Fireworks prompt cache: hit_ratio=..." log climb across a rollout. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(fireworks): rename session headers + drop in-process prompt-cache recording - Rename affinity_headers -> session_headers (the per-request x-multi-turn-session-id / x-session-affinity headers). - Drop the in-process prompt-cache recording (_record_prompt_cache, the rolling counters, and the per-call prompt_cache_hit_ratio metric); read cache behaviour from the Fireworks console instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…n the picker Two CLI UX fixes for the Fireworks eval path. Sampling params: - `rllm eval` now seeds temperature=1.0, top_p=1.0 (mirrors the cookbook training rollout), overridable by --sampling-params / --temperature / --top-p and always shown in the header's gateway-enforced "Sampling" row. - Removed the duplicate temperature from the "Config" panel — it showed the harness's requested value (0.7) while the gateway enforced a different one (e.g. 1.0), so the same param appeared twice with conflicting values. - Terminus2Harness default temperature 0.7 -> 1.0 (and the driver fallback) so nothing lingers at 0.7 when sampling isn't gateway-enforced. Fireworks deployment fetch: - fetch_fireworks_deployed_models only queried the deployedModels endpoint (populated when a custom/fine-tuned model is bound to a deployment), so a dedicated deployment of a base model — which has no deployedModel entry and is addressed for inference by its accounts/<acct>/deployments/<id> name — was invisible in the `rllm setup` picker, even though that's the exact id evals run against. Now also lists READY deployments by their id (e.g. gpdl1exc). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The agentflow_engine tqdm bar only updated the completed count; add a set_postfix with the cumulative mean episode reward so the current score is visible live during a run instead of only per-rollout lines. Falls back to is_correct for error/empty episodes. Display only. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… metrics, prompt-cache affinity
Terminal-bench eval on Modal scored ~30-33% vs official Terminus-2 ~40%, with
flaky "Command failed" warnings. Root cause was infra, not model quality.
The Modal sandbox lifetime was derived from a global env var (a flat 4800s),
while the agent runs for the per-task agent_timeout (600-12000s) and the
verifier then runs in the same box. 14 of 89 terminal-bench@2.0 tasks have
agent+verifier > 4800s, so the box was reaped before the verifier's exec
("Sandbox already shut down") -> the whole episode became a 0-step ERROR
(all agent work discarded). With retry_limit=1, each became a permanent zero.
Changes:
- eval/_resolution: size the Modal sandbox per-task, max(override,
agent_timeout + verifier_timeout + install + slack), so the box always
outlives the agent + verifier it hosts.
- eval/runner: eval retry_limit 1 -> 2, so transient infra failures get one
retry instead of becoming permanent zeros (matches the official harness).
- engine/agentflow_engine: live running accuracy/reward on the progress bar
(acc N/done=% reward=...) so the score is visible as the eval runs.
- sandbox/protocol + modal_backend: classify an exec killed at its own timeout
as SandboxCommandTimeout instead of a generic "Command failed (exit -1)";
cli_harness catches it and logs at INFO ("reached its time budget") — an
agent spending its full per-task budget is expected, the verifier still
scores the partial state.
- gateway proxy: inject x-session-affinity / x-multi-turn-session-id from the
rollout session id on the HTTP path, so eval multi-turn trajectories reuse
Fireworks' per-replica prompt cache. Previously only the in-process training
path set affinity; eval (HTTP proxy path) sent none.
Note: modal_backend.py also carries a pre-existing _CreateRateLimiter (Modal
create-rate pacing) that shares the new `import time`; it is bundled here
because it lives in the same file and is also eval-reliability related.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ask agent_timeout
BaseCliHarness.run() resolved the rollout exec timeout as
`task.metadata.get("agent_timeout", self.run_timeout)`, so a task carrying a
per-task agent_timeout always won and an operator-set
RLLM_HARNESS_RUN_TIMEOUT_S was only a fallback that never fired. Harbor tasks
always carry one — task_from_row merges task.toml's [agent].timeout_sec into
metadata — so e.g. training tb-opus-pass with RLLM_HARNESS_RUN_TIMEOUT_S=1800
silently ran rollouts to the task's 3600s budget. The intended wall-clock cap
was ignored.
Make run_timeout a hard ceiling when an operator sets it (env var or
--agent-timeout): effective timeout = min(per-task agent_timeout, run_timeout).
When it's unset, the task's own agent_timeout governs, so eval still honors each
benchmark's per-task budget (no regression to the terminal-bench parity work).
A run_timeout_is_cap flag (captured from the env at import, flipped on by
configure() for --agent-timeout) distinguishes the two.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…xy resilience (#684) * feat(sandbox/modal): isolate each run's sandboxes by run id Name the Modal App per-run (rllm-sandbox-<run_id>) and stamp each sandbox with a name + rllm_run_id tag, so on a shared account you can list/terminate just one run's (leaked) sandboxes via 'modal app stop rllm-sandbox-<run_id>'. run_id comes from RLLM_RUN_ID (sanitized) or a random per-process id (new rllm.env.rllm_run_id()). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(eval): proxy-death monitor, empty-completion canary, streaming UI uploads proxy.py: capture the litellm proxy's stdout+stderr (was DEVNULL) and add a monitor thread that loudly reports if the proxy dies mid-run — previously a dead proxy was silent and every rollout returned empty with no visible cause. agentflow_engine.py: flag rollouts whose LLM completions are ALL empty (upstream down — dead proxy / no healthy workers / model) and stream each enriched+scored episode to the UI as it finishes. runner.py + tracking.py: invoke on_episode_complete streaming (per-rollout) instead of an end-of-run burst; make the UI drain timeout env-tunable (RLLM_UI_DRAIN_TIMEOUT_S, default 600s). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * build(deps): bump fireworks-ai to 1.2.0a83 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gateway): raise worker health-check timeout 5s -> 20s Under a concurrency spike the single litellm proxy worker is slow to answer /health while busy forwarding requests; a 5s timeout marked it dead (3 strikes) and route() then raised 'No healthy workers available', failing agent turns. 20s tolerates the transient slowness. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(eval): auto-pick free port for eval proxy so concurrent jobs don't collide The auto-started LiteLLM proxy hardcoded port 4000, so two concurrent `rllm eval` jobs collided on it — and the loser's startup probe could see the winner's proxy answering and then clobber its routing via /admin/reload. EvalProxyManager now binds a free port by default (proxy_port=None), with a retry-on-bind-race so concurrent launches each settle on their own port. Add --proxy-port to pin a deterministic port when debugging a single run. The gateway layer was already free-port, so port 4000 was the only collision point. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: signalrush <269811712+signalrush@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
rLLM's native sandbox path (SandboxTaskHooks + rllm/sandbox backends, used when a non-`harbor:` agent runs a harbor-sourced dataset) reimplements task setup/exec instead of calling Harbor, and had drifted from how Harbor's Modal environment actually starts and configures a task. Bring it back into line: - Prebuilt images boot as-is. A task that declares its own [environment].docker_image ships a fully-built image; replaying its Dockerfile RUN steps on top double-applies the build. Default replay_dockerfile off in that case (matches Harbor's should_use_prebuilt_docker_image). SWE-bench-style tasks (no docker_image: base image + RUN extras) keep replay on. - Resource limits actually apply. Harbor task.toml uses size strings (memory = '4G'); rLLM only read memory_mb/storage_mb, so the limit was silently dropped to the Modal default. Normalize the string forms to *_mb. - Real user switching on Modal. exec() now applies `user` via `su <user> -s /bin/bash -c` (matching harbor.environments.modal) instead of ignoring it and running everything as root. - Task env reaches every exec. [environment].env is exported into every command (ModalSandbox.set_env), mirroring Harbor's per-exec Secret, instead of a one-shot `export` that didn't survive a fresh shell. - Keepalive on sandbox create, so the box stays alive regardless of the image's ENTRYPOINT/CMD (matches Harbor). Verifier/reward dirs locked away from agent_user are now chowned to the verifier's user (root unless a distinct verifier_user is set) so the now-su'd verifier can still write reward files. Tests: tests/eval/test_harbor_parity.py + tests/sandbox/test_modal_backend.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ion on the queued log line (#690) * feat(trainer): per-task datum/step/segment breakdown when a task is queued A task is consumed as one GRPO group, which prefix-merges into training rows; previously only batch-level merge metrics were visible. Log per-task at INFO when the group is finalized: number of groups, trajectories, steps (turns), and datums (rows after prefix-merge), plus steps/datum. _segment_count() applies the same prefix-extension check the backend transform uses, so the per-task datum count matches what's trained and the merge ratio is visible per task, not just batch aggregate (steps/datum == 1 is a clean cumulative trajectory; >1 a prefix break). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(trainer): add per-task group reward distribution to the queued log line Extend the per-task "Task ... queued" accounting (cherry-picked from feat/native-renderers) with the GRPO group's reward distribution: avg/min/max over its trajectories. Advantages are driven by within-group reward spread, so a min==max group carries no learning signal — surfacing this per task makes a degenerate group visible at a glance, alongside the steps->datums merge ratio. _traj_reward() mirrors the reward extraction in _log_prompt_group_finished (trajectory-level reward, falling back to the last step's reward) so the per-task average matches the per-episode rewards logged there. Unscored trajectories are excluded; an all-unscored group reports 0.0 without error. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Reproduce Ai2's Tmax recipe (arXiv:2606.23321) in rLLM. Adds: - rllm/data/tmax_builder.py: native builder for the TMax-15K corpus. The Harbor-registry copy their README cites (tmax/TMax-15K-Harbor) is not in the public Harbor registry, so build from Hugging Face instead — join allenai/TMax-15K (test_final_state pytest verifier + description) with allenai/tmax-15k-open-instruct (per-task prebuilt images) by task_id, and materialize Harbor task dirs. Reward = test_final_state.py pytest pass/fail -> /tmp/rllm/reward.json. Same pattern as r2egym/swesmith builders. - rllm/registry/datasets.json: tmax-15k catalog entry (builder-backed). - cookbooks/tmax/: prepare_data.py, train.py, train_verl.sh (faithful full-FT DPPO->GRPO + centered advantages, group 32, 65K, lr 1e-6, no KL, 64 turns), train_fireworks.sh / train_tinker.sh (LoRA variants), test.py, README. Sibling of cookbooks/terminal-rl. Builder join + materialization validated on real HF data; in-container reward execution needs Docker Hub image access. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
#692) * feat(gateway): classify and explain TokenAccumulator resets in the log Cumulative-token mode previously logged a single ambiguous line on reset ("Resetting TokenAccumulator (was at turn N, M messages)") that collapsed four distinct causes and omitted the session id, so it was impossible to tell why a reset happened. A turn-1 reset in particular could be an upstream retry, a rewritten history, an assistant-only delta, or a renderer that can't bridge — all indistinguishable. Introduce a ResetReason taxonomy named by the observable structural relationship between the incoming request and the accumulated snapshot: - duplicate: incoming list identical to the last processed turn (conversation did not advance) — likely upstream retry - prefix_changed: snapshot-covered messages shrank or diverged (history compacted/edited, or session id reused) - empty_delta: clean extension, but only assistant message(s) added - renderer_no_bridge: valid extension the renderer declined A single plan_turn() classifier replaces the split is_cumulative / extract_new_messages logic at the proxy call site. reset() now logs at INFO with the session id, the machine-readable reason, a short plain-language explanation, reason-specific diagnostics (age_s, first_divergent_index, ...), and a reset_count that surfaces reset storms on one session. Switch the snapshot from a whole-list fingerprint to per-message fingerprints so prefix_changed can report the first divergent message and a short preview. Tests: per-reason classification + reset-logging coverage added; existing cumulative-mode unit + integration tests still pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(gateway): add snapshot/incoming token counts to TokenAccumulator reset log The reset line now reports snapshot_tokens (prompt+completion of the last ingested turn = the accumulated history size right before the reset) and, for prefix_changed resets, incoming_tokens (best-effort token count of the incoming, typically compacted, request). Comparing snapshot_tokens against the model's context limit makes token-limit-triggered compaction obvious from the log, and incoming_tokens shows how far the history shrank. snapshot_tokens is free (the accumulator already holds the token IDs); incoming_tokens is computed via renderer.render_ids guarded so diagnostics can never break the reset path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style(gateway): satisfy ruff-format on TokenAccumulator reset log string Join the reset log format string onto one line; ruff-format (pre-commit) rejected the manual two-line split. No behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…hful (#693) * feat(sandbox): make Daytona backend training-ready & Harbor-eval-faithful Bring the Daytona sandbox backend to parity with what e4e91cf (+#684) did for Modal, so TERMINAL_SANDBOX_BACKEND=daytona reproduces Harbor's Daytona eval performance and trains stably at GRPO scale. All Path-B execution semantics now match how Harbor's Daytona environment starts/configures a task. P0 — fidelity blockers (Daytona reward now matches Harbor): - Per-exec user switching: exec honors `user` via `su <user> -s /bin/bash -c` (resolving int UIDs via getent), so agent and verifier run under their declared users and the verifier-dir ownership setup in _resolution actually isolates. Mirrors modal_backend._build_exec_command and harbor's Daytona env. - SandboxCommandTimeout + guaranteed hard-kill: timed commands are wrapped in coreutils `timeout -k 10 <t>` (SIGTERM then SIGKILL) with the SDK timeout as a +60s backstop. Exit 124/137 and SDK-level timeout exceptions map to SandboxCommandTimeout, which cli_harness treats as "expected, score anyway" rather than a failure. A runaway agent can no longer hang the rollout. - set_env: declared [environment].env is re-exported into every exec (Daytona shells are independent), so the verifier reliably observes task env. P1 — training-scale stability: - Create retry + rate-limiting: a _CreateRateLimiter token bucket (RLLM_DAYTONA_SANDBOX_CREATE_RPS/_BURST) plus a _create retry loop with linear backoff on transient/rate-limit/5xx errors; a vanished snapshot / real validation error stays non-transient and cold-falls-back. Prevents GRPO's concurrent-create burst from losing rollouts. - Task-sized lifetime: _resolution sizes Daytona auto_stop_interval from agent+verifier+install+slack (RLLM_DAYTONA_SANDBOX_AUTOSTOP_MIN floor), so a long or stalled rollout isn't reaped mid-run. Also filter build_daytona_snapshot's Resources(**...) to compute-only keys (required for the new lifecycle kwarg; also fixes create_timeout previously leaking into Resources()). Tests: 22 new unit tests covering su/env wrapping, transient classification, the rate limiter, exec timeout->SandboxCommandTimeout (124/137, SDK exception, and the 124-without-timeout edge), and the create retry loop — all against fakes so no SDK/API key is required. Design doc at design/daytona-training-readiness.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(sandbox): unify sandbox lifetime knob into provider-agnostic RLLM_SANDBOX_TIMEOUT_S The sandbox-lifetime override was Modal-specific (RLLM_MODAL_SANDBOX_TIMEOUT_S), and the just-added Daytona path used a separate RLLM_DAYTONA_SANDBOX_AUTOSTOP_MIN. Two names for one concept ("how long may a sandbox live") is wrong now that the same training scripts can target either backend. Introduce a single provider-agnostic knob, RLLM_SANDBOX_TIMEOUT_S (seconds): - rllm/env.py: new sandbox_timeout_override_s() reads it, falling back to the deprecated RLLM_MODAL_SANDBOX_TIMEOUT_S (one-time warning) so existing scripts keep working. - _resolution.py: compute the per-task lifetime floor (agent+verifier+install+ slack) ONCE and raise it to the override, then each backend expresses it in its own unit — Modal's hard `timeout` in seconds, Daytona's idle `auto_stop_interval` in minutes. Dedupes the arithmetic that was copy-pasted across both branches. - modal_backend._default_sandbox_timeout(): read the shared helper. - Drop RLLM_DAYTONA_SANDBOX_AUTOSTOP_MIN (was unreleased, added in this branch). Adopt the new name in the clean tmax scripts (train_tinker.sh, train_verl.sh) and document it in both cookbook READMEs. The provider-specific create-rate knobs (*_CREATE_RPS/_BURST) stay separate — they tune different platform rate limits. Tests: provider-agnostic override floors both backends; legacy alias honored with canonical precedence; per-task sizing unchanged when no override is set. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(sandbox): run Daytona sandboxes ephemeral by default RL rollout and warm-queue sandboxes are strictly single-use (create → run → verify → delete on close()), so default DaytonaSandbox to ephemeral=True with auto_delete_interval=0, matching how harbor's Daytona environment always runs. Two payoffs: - Higher quota: Daytona's ephemeral tier carries a larger per-sandbox limit, so tasks declaring more than the standard 10 GB disk (tmax declares 20 GB) now schedule instead of failing every create with "Disk request 20GB exceeds maximum allowed per sandbox (10GB)". This is why we honor the task's declared storage rather than clamping it — ephemeral lifts the ceiling. - Leak-proofing: an ephemeral sandbox is deleted (not left stopped) when it stops, so a SIGKILL'd run that never reaches close()/atexit no longer strands a stopped box forever — the failure mode behind the ~1.3 TB of leaked stopped sandboxes seen on the shared account. ephemeral is an overridable kwarg (=False falls back to auto_delete_interval=-1, never-delete). Tested via a fake daytona module so no SDK/API key is required. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
#694) * fix(harness): restore TerminationReason on the SandboxedAgentFlow path CLI-harness rollouts (terminus2 / mini-swe-agent) always came back ENV_DONE, so compact filtering couldn't distinguish timeouts/errors/length-truncation and the policy was punished for infrastructure failures (a timed-out or sandbox-reaped rollout shouldn't train as a reward-0 sample). - BaseCliHarness.run() now returns an outcome Episode carrying the reason it observed: TIMEOUT on SandboxCommandTimeout, ERROR (+ details) on exec failure, None on a clean exit; stamps metadata["max_turns"] when the harness caps turns. Stays stateless (returns via value) so the shared-instance parallel rollouts are unaffected. - AgentFlowEngine grows derive_termination_reason() and replaces the blunt ENV_DONE fallback in _finish_episode: last finish_reason=="length" -> MAX_RESPONSE_LENGTH_EXCEEDED, turns at cap -> MAX_TURNS_EXCEEDED, else ENV_DONE. Only runs when the harness left it None, so harness-set TIMEOUT/ERROR win. - Relocate TerminationReason/TerminationEvent from rllm.workflows.workflow to rllm.types (they're used across engine, harnesses, eval, trainer — no longer workflow-specific). workflow.py re-exports them so existing imports keep working; first-party rllm/ import sites updated to the canonical location. MAX_PROMPT_LENGTH_EXCEEDED detection is deferred (TODO in derive_termination_reason). Tests: updated tests/harnesses/test_cli_harness.py (Episode return + TIMEOUT/ERROR /max_turns classification) and added tests/engine/test_termination_derivation.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refine: don't infer MAX_RESPONSE_LENGTH/MAX_TURNS on the CLI-harness path Reading harbor's Terminus-2 showed neither is recoverable from gateway traces: - finish_reason=="length" is not episode-terminal here. On a truncated response the agent appends an error and recursively re-queries the LLM (terminus_2.py _query_llm), so it continues — a trailing "length" is coincidental, not the cause. Combined with base.yaml's mask_max_response_length_exceeded=True, the old heuristic could silently drop otherwise-fine (incl. successful) episodes. - The trace count is LLM *calls*, not turns (retries, length-recovery re-queries and summarization inflate it), so n_steps>=max_turns can't reliably detect the cap; the true turn count lives only inside the sandbox driver. derive_termination_reason now returns ENV_DONE for clean exits; the meaningful signal is the harness-set TIMEOUT/ERROR (the actual fix). Drop the now-unused max_turns metadata stamping. Tests pin "trailing length is not terminal" as a regression guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refine: inline ENV_DONE fallback, drop trivial derive_termination_reason It only ever returned ENV_DONE; inline it in _finish_episode and delete its test. Tighten the harness comments/docstrings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…695) * feat(gateway): rllm tunnel CLI + ngrok backend + daemon auto-pickup Make stable-tunnel setup ergonomic and shareable. Free cloudflared quick tunnels get 429-throttled; ngrok with a reserved domain is stable but previously required a manual `ngrok http` in a side terminal + a hardcoded URL pinned per-script (only valid for one person's account). Adds an `rllm tunnel` CLI that runs a backend as a detached daemon and records its live URL to ~/.rllm/tunnel.json; training auto-discovers it with no per-run config and nothing personal in shared scripts. - gateway/tunnel.py: extract a shared _Tunnel base from CloudflaredTunnel; add NgrokTunnel (ngrok http --url --log json, actionable hints for authtoken/1-session/domain errors); create_tunnel() dispatch; spawn_detached() daemon + state-file helpers; resolve_auto_tunnel() ($RLLM_GATEWAY_TUNNEL > running daemon > cloudflared quick tunnel) - hooks.enable_gateway_tunnel: resolve via resolve_auto_tunnel and warn on quick-tunnel fallback (was a silent cloudflared default) - eval/config.py: load_/save_tunnel_config (per-user backend/domain/port) - cli/tunnel.py + main.py: new `rllm tunnel {setup,up,status,down}` group - cli/train.py: sandbox-routing header shows the resolved tunnel URL - base.yaml: document "ngrok"/"ngrok:<domain>" + daemon auto-pickup - tmax/train_fireworks.sh: drop hardcoded ngrok URL (now portable) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * update training scripts * cookbooks(terminal-rl): gateway port 9090 + drop hardcoded ngrok tunnel train_fireworks.sh now matches tmax: gateway on 9090 (the rllm tunnel daemon default) with no hardcoded rllm.gateway.tunnel, so it auto-discovers the tunnel via `rllm tunnel up`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: ruff format (join implicit string concatenations) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
`rllm view` returned a 400 "Bad run id" for any run whose directory name contains '@' (e.g. terminal-bench@2.0) or '#' (model#deployment). The SPA fetches /api/runs/<id>/index with the id percent-encoded, but the handler validated the still-encoded string against a charset regex excluding '%'/'@'/'#', and never URL-decoded it before the filesystem lookup. Decode the run id + filename with urllib.parse.unquote before validation and lookup (both the /index and /episodes/<file> routes), and permit '@'/'#' in the safe-id/safe-file patterns. Path-traversal is still blocked by the ".."/"/"/"\\" guard and the _under_root check, both of which now run on the decoded value. Co-authored-by: signalrush <269811712+signalrush@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ode bridge Adds rllm.renderers: one bridge-capable renderer interface (prime-rl's protocol) for every backend's models. get_renderer resolves prime-rl native -> TinkerRendererAdapter (wraps tinker_cookbook AND Fireworks-cookbook renderers, e.g. deepseek_v4) -> chat-template fallback. BridgingRendererMixin synthesizes bridge_to_next_turn for any deterministic renderer and is verified byte-identical to prime-rl's hand-tuned qwen3 bridge. - gateway: resolve the cumulative-mode renderer via rllm.renderers (+ renderer_name config). The chat-template fallback is best-effort (runtime prefix-check guards; drift resets, never corrupts) instead of a hard error. - FireworksEngine: opt-in unified renderer via renderer_name/renderer_family (the DeepSeek-V4 path); default keeps the chat-template path so existing Fireworks models do not regress. Tests: 48 pass (renderer + gateway), ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… ChatTemplateParser (#702) * fix(fireworks): use unified renderer for turn-0 and skip ChatTemplateParser when pinned FireworksEngine built ChatTemplateParser unconditionally, whose construction eagerly runs apply_chat_template (verify_equivalence). Some served templates reject the probe messages — e.g. GLM-5.2 raises "'str object' has no attribute 'items'" — crashing engine init before training. The GLM script already pins RENDERER_FAMILY=glm-5, but it was only wired to the gateway; FireworksEngine never received it, so it fell back to chat_parser. Now: - fireworks_backend passes rllm.gateway.renderer_family to FireworksEngine, so the engine renders turn 0 with the same renderer the gateway uses for the cumulative bridge (turns 1+) — keeping them consistent. - FireworksEngine resolves the renderer first; when it pins a native (prime-rl or tinker) renderer it uses it and skips ChatTemplateParser entirely. Default (renderer_family=auto, no renderer_name) is unchanged — chat_parser as before. Test: pinned family uses the renderer and leaves chat_parser None; default still builds chat_parser. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(renderers): auto-detect Fireworks-cookbook models (GLM-5.2) without config The Fireworks cookbook ships renderer *implementations* (glm5, deepseek_v4, ...) but no model->renderer map, and tinker_cookbook's recommender doesn't know them, so nothing auto-routed GLM-5.2 to the cookbook renderer — you had to pin renderer_family=glm-5. - registry: resolve() now owns a Fireworks-cookbook prefix map (GLM-5.x -> glm5, DeepSeek-V4 -> deepseek_v4), consulted after prime-rl's exact-match map; imports training.renderer so cookbook renderers self-register; degrades gracefully if the cookbook isn't installed (never hard-fails). - FireworksEngine: resolve the renderer the same way the gateway does (always), so engine (turn 0) and gateway (turns 1+) pick the same renderer. Adopt it when pinned or when auto-detection lands on a cookbook renderer (e.g. GLM-5.2), and skip ChatTemplateParser; prime-rl models under plain auto keep chat_parser (no regression for working qwen runs). GLM-5.2 now trains with no renderer config: auto -> cookbook glm5 on both engine and gateway. Tests: cookbook prefix mapping + graceful fallback. 14 pass, ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…(both backends) (#705) * fix(fireworks): parse completions via unified renderer in assemble_model_output Skipping ChatTemplateParser when a unified renderer is active left the inherited assemble_model_output asserting chat_parser is set (bypass_render_with_parser=True), crashing on the first response: AssertionError: chat_parser must be set when bypass_render_with_parser=True Override assemble_model_output in FireworksEngine: when a unified renderer is active, parse the completion via renderer.parse_response (ParsedResponse -> content/reasoning/ tool_calls); otherwise delegate to the inherited chat_parser path. Covers both turn 0 (get_model_response) and turns 1+ (the gateway token-prompt path), which both call it. Test: assemble_model_output parses via the renderer with chat_parser=None. 15 pass, ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(fireworks): make bypass_render_with_parser reflect renderer ownership bypass_render_with_parser was hardcoded True even when a unified renderer was active — a lie (True means "render/parse with ChatTemplateParser"), and the root reason assemble_model_output asserted on a None chat_parser. Set it to `self.renderer is None`: False when the renderer owns rendering+parsing, True when ChatTemplateParser does. Flipping the flag alone isn't sufficient — the parent's bypass=False branch is hardcoded to a tinker_cookbook renderer ((Message, term) parse_response), which the unified renderer (ParsedResponse) isn't — so the assemble_model_output override stays. This just stops the attribute from lying (and misleading completer.py etc.). Test: bypass_render_with_parser is False with a renderer, True without. 15 pass, ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(engine): unify renderer handling in shared TinkerEngine for both backends Per review: the unified rllm.renderers handling belonged in the shared TinkerEngine, not a FireworksEngine-only override. Now both backends render+parse through one path. - TinkerEngine: add self.unified_renderer (default None) + a shared _render_prompt_token_input helper. assemble_model_output and the render path branch on the unified renderer first, then ChatTemplateParser, then the tinker_cookbook renderer (incl. VLM image chunks). Base behavior is unchanged (unified_renderer stays None for the Tinker backend), so chat_parser / tinker-renderer / VLM paths are preserved. - FireworksEngine: set self.unified_renderer via rllm.renderers.resolve and reuse the shared render+parse; drop the assemble_model_output override and the inline render. self.renderer stays None (the shared legacy/VLM branch is unreachable here). No external code reads rollout_engine.renderer, so introducing unified_renderer is safe. 15 renderer tests + 55 engine tests pass (the 1 fail / 4 errors are pre-existing env issues: verl not installed, torch.LongTensor). ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(fireworks): honor explicit bypass_render_with_parser=True (escape hatch) FireworksEngine ignored the config's bypass_render_with_parser (it was swallowed by **kwargs) and always resolved a unified renderer. Make it a real parameter: when True, skip the unified renderer and use ChatTemplateParser — an explicit escape hatch, matching TinkerEngine (which already honors bypass=True via its unbuilt unified path). Default (False) keeps auto-resolution. Test: bypass_render_with_parser=True with renderer_family pinned -> unified_renderer None, chat_parser built, bypass True. 16 tests pass, ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…the accumulator (#707) Minimal change vs terminal-rl. In cumulative-token mode an identical resend (upstream timeout/retry of a turn already processed) was classified as a reset: plan_turn returned action="reset" with ResetReason.DUPLICATE, so the proxy dropped the drift-free token state, re-tokenized from text as a fresh turn-0, and forced a spurious segment break for what is a single step. Handle DUPLICATE as a replay instead: regenerate from the same accumulated prompt and overwrite the current turn in place — do NOT reset. - token_accumulator: plan_turn returns action="replay" for DUPLICATE; ingest_turn(..., advance=False) overwrites the current turn without bumping turn_count. - proxy: handle() replays from acc.prev_prompt_ids via _handle_cumulative_turn (replay=True), plumbed through the cumulative handlers as advance=not replay. Logged once per duplicate at INFO (regenerating in place, no reset). A fresh sample (not a cached one) is returned, and each duplicate costs a real generation — so a retry that depends on resampling (e.g. terminus2's context-length summarization fallback) still makes progress, and the generation latency naturally rate-limits any retry loop. Coalescing concurrent in-flight duplicates is intentionally out of scope (rare; added complexity). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…faults (#708) A transient blip on the shared training client around the per-step weight hot-load (eager_parameter_sync_step=1) crashed entire multi-hour runs: the Fireworks SDK's forward_backward / optim_step / save_and_hotload have no retry ("failures propagate so the training loop can crash"), and a step-N forward_backward TimeoutError at step ~17 tore down the run. Wrap the per-step training-client RPCs (forward, forward_backward incl. aux, optim_step, save_and_hotload) in a retry helper that, on a transient error (timeout / connection / channel reset / 5xx markers), optionally reconnects the client to the SAME job and retries with linear backoff. Non-transient errors (bad data) and cancellation propagate immediately. - Reconnect refreshes the ReconnectableClient via the SDK's own _connect path (rlor_mgr.wait_for_existing(job_id) -> client._use_endpoint), re-attaching to the same job so server-side optimizer/gradient state is preserved. Used for forward/forward_backward/optim_step. save_and_hotload retries without reconnect (the WeightSyncer holds the raw client; re-dispatch is idempotent). - Config (fireworks_infra.common): step_max_retries (default 2, 0 disables), step_retry_backoff_s (default 10). Each retry logs at WARNING. CAVEAT (documented in _run_training_op): forward_backward / optim_step mutate server-side state. A retry assumes the failed RPC didn't commit it — true for channel/dispatch failures; a *false* timeout could double-apply for one step. Retries are kept low; this bounded, rare perturbation beats crashing the run. For genuine slowness, raise step_timeout instead. Adds tests/trainer/test_fireworks_policy_trainer_retry.py (transient classification, retry-succeeds, retry-exhausted, non-transient-not-retried, zero-retries-disables, reconnect on/off, reconnect-without-mgr no-op). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…rmat tasks Harbor-format tasks (e.g. honeycomb's AWS/LocalStack tasks) declare an [environment.healthcheck] that boots an in-image service (LocalStack) and COPY files (start_localstack.sh, ready.d seeds) into the image. Two gaps stopped them running on the native sandbox path: 1. No healthcheck handling — rLLM boots the container with `sleep infinity` and never runs the image CMD, so the service never starts. Add a presence-guarded _run_healthcheck step in SandboxTaskHooks.setup (after env materialization, before the agent) that execs the declared command on a bounded retry loop and raises on exhaustion. No-op when no healthcheck is declared (every non-service eval/training task), mirroring harbor's run_healthcheck. 2. COPY dropped on Daytona — the cold path pulls the Dockerfile's FROM base and replays only RUN steps, silently dropping COPY/ENV/WORKDIR. Build the real Dockerfile via daytona's Image.from_dockerfile for Dockerfile-based tasks (replay_dockerfile true) — the same primitive harbor's own Daytona environment uses — in both the cold path and build_daytona_snapshot. env_key now fingerprints the whole build context for these tasks so two tasks sharing FROM+RUN but differing in COPYed data (e.g. different ready.d seeds) don't collide on one snapshot / warm-queue sandbox. Also: a missing verifier reward file is now an explicit error (was a silent reward 0.0) so a broken verifier is distinguishable from an agent that genuinely scored 0, matching harbor's RewardFileNotFoundError. Verified E2E on remote Daytona with terminus2 on a honeycomb AWS/LocalStack task: the real image builds (COPY intact), the healthcheck boots LocalStack + seeds ready.d, and the verifier runs (6/7 tests pass; the miss is the agent, not infra). modal from_dockerfile is a tracked follow-up (untested here; _FROM_DOCKERFILE_BACKENDS). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EzkMPi5tAr3e4T5HBQ14Dt
* feat(terminal-rl): classify infra/grading failures distinctly across eval + train
Wall-clock timeouts and grading-infra failures were collapsed into env_done /
a generic reward 0, so timeouts were mislabeled and broken verifiers trained as
real task failures. Mirror Harbor's granular failure taxonomy end-to-end.
Taxonomy (rllm/types.py)
- Add TerminationReason.{AGENT_SETUP_TIMEOUT, ENV_START_TIMEOUT, VERIFIER_TIMEOUT,
GRADING_ERROR, SANDBOX_ERROR}; keep TIMEOUT = agent wall-clock (reward stays valid).
- Add INFRA_ERROR_REASONS and a canonical exception-class -> reason map
(termination_reason_from_error), keyed on Harbor's class names. trial_helper's
map_termination_reason now delegates to it (single source of truth, both paths).
Agent timeout label (was the core bug)
- terminus2 driver self-limits agent.setup/agent.run with asyncio.wait_for and
raises Harbor's AgentSetupTimeoutError/AgentTimeoutError, then writes an
ExceptionInfo-shaped outcome sentinel and exits 0 (partial state still graded).
- BaseCliHarness.run reads the sentinel (authoritative over the | tee-masked exit
code); falls back to an elapsed-vs-budget backstop when none is present. Exec
gets a grace window so the sentinel write wins the race.
Grading errors (eval + train)
- EvalOutput gains a structured `error`; ShellScriptEvaluator tags verifier
timeout / missing / empty / unparseable reward / tests-dir upload, and
PythonModuleEvaluator tags verifier crashes, with Harbor-aligned names.
- _finish_episode promotes a grading error to the matching infra reason
(VERIFIER_TIMEOUT / GRADING_ERROR), overriding only non-infra outcomes; grading
stays unconditional so a timeout is still scored on partial state.
Filtering + error-rate tracking
- CompactFilteringConfig gains mask_{verifier_timeout,grading_error,sandbox_error,
agent_setup_timeout,env_start_timeout} (base.yaml defaults them True under the
existing master switch); agent TIMEOUT is deliberately not masked.
- Train: buffer records episode/infra_error plus the per-reason series.
- Eval: EvalResult.termination_breakdown + EvalItem.termination_reason; only infra
reasons count as errors (agent timeouts don't).
Tests for the sentinel/backstop classification and the evaluator error tagging.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(harness): classify mid-run sandbox death as SANDBOX_ERROR via is_alive() probe
A dead sandbox and a benign non-zero CLI exit both surface as the same generic
exec exception, so the prior code marked both ERROR. Probe sandbox.is_alive()
in the failure handler: if the box is gone it's infra (SANDBOX_ERROR, reward
untrustworthy); a non-zero exit on a live box stays ERROR. The probe is guarded
(absent/raising is_alive falls back to ERROR), so non-probing backends are
unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(eval): show per-category termination breakdown in the eval summary
The results panel showed only an aggregate "Errors" count. Add a
"Terminations" row listing each termination_reason with its count
(infra-error reasons highlighted), so a flaky endpoint or broken verifier
is visible at a glance instead of hidden. Per-item termination_reason and
the granular error type are already persisted to results.json / rllm view.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(metrics+log): drop "Rollout completed." prefix; add batch/infra_error parity
- agentflow_engine: the per-rollout line was noisy ("[uid] Rollout completed.
Rewards: ..."); drop the prefix, keep rewards/timing/termination.
- unified_trainer: add batch/infra_error (sum of infra-reason rates), mirroring
the buffer/async path's episode/infra_error so the sync trainer reports the
aggregate infra-failure rate too. Per-reason batch/termination_reason/* already
covers every reason via the enum loop.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(terminus2): driver crashed on harbor 0.3.0 — define timeout classes locally + bump harbor to 0.13.2
Regression from the timeout-labelling change: the in-sandbox driver imported
`from harbor.trial.errors import AgentSetupTimeoutError, AgentTimeoutError`, but
the sandbox installs harbor 0.3.0 which has NO `harbor.trial.errors` module (it
was added later; the host venv runs 0.13.2, where I verified the import — wrong
version). So the driver ImportError'd at startup on every task → the agent never
ran → "No traces found" → reward 0, mislabelled ENV_DONE across the whole run.
Fix: define AgentSetupTimeoutError/AgentTimeoutError locally in the driver (only
the class *names* matter — the harness maps them to a TerminationReason via the
sentinel), so the driver no longer depends on that module and works on any harbor
version. Also bump the pinned harbor_version 0.3.0 -> 0.13.2 to match the host and
get a modern Terminus-2 (verified API-compatible: all ctor kwargs + setup/run
signatures + BaseEnvironment abstract methods line up). NOTE: the bump only takes
effect on a snapshot REBUILD (the install marker no-ops a baked snapshot); the
local-class fix is what unblocks the existing 0.3.0 snapshot immediately.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(classify): MODEL_ERROR for downed-upstream rollouts; sandbox setup → SANDBOX_ERROR
- MODEL_ERROR: _finish_episode now detects "no usable model output" — the agent
made no LLM calls (no traces) OR every completion came back empty — and labels
it MODEL_ERROR instead of a clean ENV_DONE with reward 0. This is the dead-proxy
/ unreachable-gateway signature (a *correct* rollout is never reclassified).
- Sandbox provisioning: setup-phase failures are tagged so retry-exhaustion
classifies SANDBOX_ERROR (e.g. DaytonaValidationError) instead of generic ERROR,
and the error_type is recorded in metadata.
- Add TerminationReason.MODEL_ERROR (infra family) + mask_model_error (base.yaml
True); map Daytona provisioning errors to SANDBOX_ERROR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(terminus2): self-heal harbor version in the sandbox instead of a bare ready-marker
The install skipped on any `.terminus2-ready` marker, so a snapshot that baked an
old harbor (0.3.0) kept it forever — the version bump only mattered on a manual
rebuild. Skip now only when harbor is ALREADY at the pinned version; otherwise
upgrade in place (create the venv only if missing, don't clobber). A stale 0.3.0
snapshot self-heals to 0.13.2 on next run. Rebuild the snapshot to avoid paying
the reinstall per cold sandbox.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(daytona): deliver long-exec completions — session polling + TCP keepalive
The one-shot process.exec rides a single HTTP request that stays byte-silent
until the command completes; NAT middleboxes on the client path (WSL2
measured: drop at ~245-250s idle) silently discard the flow, so any command
over ~4 minutes finishes in the sandbox but its response is black-holed —
the call blocks until the SDK read timeout (budget+65s) and the episode is
mislabeled TIMEOUT. Bisect: 240s delivered; 250/260/270/300/315/330s all lost.
Two independent defenses:
- execs with budgets >180s (RLLM_DAYTONA_SESSION_EXEC_THRESHOLD_S) ride a
session command: async submit + adaptive short polls (1s→15s) + log fetch;
no connection ever idles long enough to be dropped. In-sandbox coreutils
timeout stays the deterministic killer (exit 124 → SandboxCommandTimeout).
- TCP keepalive (60s probes) injected into the SDK HTTP pools, best-effort:
keeps L4 flow-table entries alive for every other call. Validated: a 300s
one-shot exec delivers with keepalive, is lost 6/6 without.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cli-harness): trust the sentinel on exec timeout; classify ExecCompletionLost
A SandboxCommandTimeout no longer unconditionally labels the episode
TIMEOUT: the driver's outcome sentinel is consulted first (it survives
transport loss). Clean sentinel → ENV_DONE with an ExecCompletionLost
audit marker in episode metadata (reward is valid; the transport event
stays visible in tracking); typed sentinel → mapped reason; no sentinel →
TIMEOUT as before. Also adds live regression tests (gated on
DAYTONA_API_KEY / RLLM_LIVE_SLOW=1) asserting exec completions are
delivered past the NAT drop line instead of hanging to the read timeout.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(eval): align no-reward-file test with the typed error taxonomy
Upstream terminal-rl added a test asserting the pre-taxonomy RuntimeError;
this branch replaces that raise with the typed RewardFileNotFoundError
EvalOutput (same intent — a missing verdict is an infra error, not a
score of 0 — now engine-classifiable). Update the test accordingly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(sandbox): fix stale max_concurrent default assertion (4 -> 64)
Upstream raised SandboxedAgentFlow.max_concurrent's default to 64 but the
configure() test still asserted 4 — failing on a clean terminal-rl checkout.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style: ruff-format daytona backend + delivery test
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(integration): skip keepalive delivery test on transient Daytona 5xx
A fast server-side error is a platform blip, not the delivery-loss
regression (loss = hanging to the read timeout, still asserted).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: signalrush <269811712+signalrush@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sampling out of FireworksEngine intermittently died on httpx ReadTimeout/ReadError, logged as "Sampling failed permanently after 1 attempts" with an empty message. _completions_with_retry classified transient errors by string-matching str(exc)/class name, but timeouts and transport errors have an empty str(exc) and a class name matching none of the markers, so they were treated as non-transient and gave up on the first attempt (the SDK's own retry can't help: its 300s budget is smaller than the per-attempt http_timeout). Retry fix: - Classify httpx.TimeoutException / TransportError as transient by type, so read timeouts and connection resets (ReadError) are retried instead of failing the whole sampling call. - Include the exception class name in retry / permanent-failure logs, since timeouts render an empty message. Timeout config consolidation (no change to effective defaults): - sample_timeout stays 600 (the prior default). On a streaming SSE call this is httpx's per-read timeout — a cap on inter-token silence, not total generation — so it does not clip long-but-active generations. Made rollout_engine.sample_timeout the single source of truth; fireworks_infra.deployments.rollout.sample_timeout now follows it. - Removed dead training.client_timeout (no consumer in the repo or the training SDK). - Grouped the training-step RPC knobs (step_timeout / step_max_retries / step_retry_backoff_s / weight_sync_timeout) as one timeout + retry policy. - Shared trainer readiness timeout: reference follows policy.timeout_s. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…op-health monitoring (#727) * fix(fireworks): bound sampling-retry hold time to curb duplicate churn FireworksEngine is wired as the gateway's in-process handler, so _completions_with_retry runs inside the agent's HTTP call and holds that connection (byte-silent, no heartbeat on the cumulative path) for the whole retry sequence. The prior 10/20/30/40s backoff could hold ~100s+, past the client/tunnel tolerance — the client then re-sends, surfacing as TokenAccumulator "duplicate: regenerating in place" churn plus wasted regeneration (esp. during the every-step weight sync that resets connections). Keep the transient-error classification, but cap the total retry wall-clock (_RETRY_BUDGET_S=90s) and per-retry backoff (_RETRY_BACKOFF_CAP_S=15s): ride out the common transient reset, then fail fast so a persistent outage is released to the client's own retry instead of stalling the connection. Permanent-failure log now distinguishes "retry budget exhausted" from "permanent" and reports elapsed seconds. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * perf(gateway): take per-request CPU off the single event loop for high concurrency The gateway serves all rollout LLM calls on one asyncio event loop (a thread inside the trainer process). At high concurrency (n_parallel_tasks in the hundreds) that loop can't flush responses / fire heartbeats because it's busy doing per-request CPU work, so connections stall and the agent's retry stack re-sends identical requests -> TokenAccumulator "duplicate" flood + rollout timeouts. Two contained fixes that target the measured hotspots: - Offload tokenizer rendering off the loop: turn-0 render_ids (FireworksEngine.get_model_response) and the cumulative bridge_to_next_turn (proxy handle()) now run via run_in_executor. Tokenizers release the GIL, so these parallelize across worker threads AND the loop stays free to service I/O. turn-0 render on ≤120K-token prompts was the dominant blocker (matches the turn=1-dominant duplicate distribution). - Stop capturing raw_request/raw_response on traces by default (capture_raw_payloads=False). Training reads only token-id/logprob/message fields (rllm.engine.trace_converter), so serializing the full ≤120K-token prompt + response via model_dump was pure per-request CPU on the loop. Does not change training data or trace token/logprob content. Follow-ups (not in this commit): whitespace heartbeat on the cumulative path; run the gateway in its own process to remove GIL contention with the trainer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gateway): extend whitespace heartbeat to the cumulative (turn 1+) path The keep-alive heartbeat only wrapped the turn-0 chat path; cumulative turn-1+ requests were byte-silent for their whole (queued + generating) duration, so a tunnel/NAT idle timer would cut the connection and the agent would re-send — surfacing as TokenAccumulator "duplicate" churn. Extract the heartbeat into a shared `_respond_with_heartbeat(result_coro)` and route both the turn-0 (`_non_streaming_result`) and cumulative (`_cumulative_non_streaming_result`) paths through it. The cumulative result is now produced by a coroutine returning (bytes, status); its accumulator side effects (ingest_turn / update_prefix) run under asyncio.shield, so they still complete exactly once even if the client drops mid-flight. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(gateway): loop-health monitor to diagnose event-loop starvation Adds a diagnostic-only background task that logs, every ~20s: gateway loop health: lag_ms p50/p99/max | thread_cpu=% | inflight cur/max and stamps `inflight` + `loop_lag_ms` onto each TokenAccumulator duplicate line. - lag = event-loop scheduling overshoot (how long the loop couldn't run). - thread_cpu = the loop thread's own CPU utilisation, which disambiguates why: high lag + high thread_cpu -> self-CPU bound (offload / do less) high lag + low thread_cpu -> loop thread starved of the GIL by the trainer -> a separate gateway process (mode=process) helps - inflight = concurrent in-flight generations (tracked around the heartbeat result task, so it's accurate for buffered and heartbeat-streamed responses). Lets the gateway-in-own-process decision be made from data rather than guessed. No behavioural change. Validated: healthy=~1ms/0%, busy-wait=246ms/89%, idle-block=160ms/24% (the two high-lag signatures are cleanly separable). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * perf(gateway): orjson on the hot JSON paths to cut per-core CPU Loop-health data showed the gateway loop thread pegged at ~99% of one core at ~350 concurrent (self-CPU bound, not GIL-starved), with JSON (de)serialization the dominant cost. Add a fastjson helper (orjson with a stdlib fallback + a type-safety net) and swap the per-request hot paths: - middleware: request-body parse + param-injection re-serialize - proxy: request-body parse + non-streaming/cumulative response serialize - token_accumulator: per-message fingerprint (hash orjson bytes directly) Measured on representative payloads: response serialize (16K token_ids + logprobs) 20.1ms -> 1.8ms (11.2x), request parse (~480KB) 2.7x, fingerprint 6x. Round-trip identical; sorted-key fingerprint stable and internally consistent. This raises single-core request capacity (may be enough on its own); the multi-process split remains the path to using all cores if not. Streaming/error/worker JSON paths (cold for the Fireworks non-streaming setup) left on stdlib. orjson added as a declared dependency. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * perf(gateway): orjson httpx bodies + offload trace build off the loop Two more single-process loop-unblockers, from the loop-health finding that the gateway thread is pegged at ~98% of one core with multi-second lag: 1) httpx orjson patch (fireworks_engine): the Fireworks SDK POSTs the full ≤120K-token prompt via client.post(json=payload); httpx serialized it with stdlib json on the event-loop thread (~4ms+/req, the dominant on-loop cost we couldn't reach from our own code). Patch httpx._content.encode_json to use orjson (compact/UTF-8/no-NaN — matches httpx's wire semantics). Guarded, best-effort: probes the (headers, ByteStream) shape against the real function and skips (keeping stdlib) on mismatch, so an httpx upgrade can't break sampling. Measured ~4ms -> sub-ms on a 120K-int prompt. 2) Trace build offloaded (proxy): build_trace_record + model_dump (copying the ≤120K/16K token-id lists) ran inline on the loop before the response returned. Move them into the executor via _persist_trace/_build_trace_data and off the response critical path; only the async store write touches the loop. sync_traces still forces synchronous completion. Both keep the loop responsive (fewer heartbeat misses -> fewer connection cuts / duplicates). They do not lift the GIL 1-core ceiling — multi-process remains the path to using all cores. 236 gateway tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * perf(gateway): orjson for the Fireworks SDK streaming response parse async_completions_stream parses every SSE chunk with stdlib json.loads on the event-loop thread; across many concurrent streams (one chunk per token) that per-chunk parse is a continuous on-loop cost. Swap the SDK sampling module's `json` reference for a shim whose `loads` is orjson and that delegates all other attributes (dumps, JSONDecodeError, …) to stdlib — so nothing else changes. orjson.JSONDecodeError subclasses ValueError, so the SDK's `except (ValueError, TypeError)` still catches malformed chunks. Measured ~3x per chunk (1.9µs -> 0.6µs); this is eliminated work (orjson does the same parse in less CPU), so unlike the trace offload it raises the throughput ceiling. Total impact scales with chunks-per-response (streaming granularity) — the next loop-health thread_cpu reading will show the real drop. Best-effort/idempotent install next to the existing httpx + header patches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(gateway): run the gateway in its own process (num_workers=1) The gateway runs as a thread inside the trainer, sharing one GIL; loop-health showed the gateway loop pegged at ~98% of a core, starving the trainer's Python (weight-sync, episode processing). This adds an opt-in separate-process mode so the gateway gets its own event loop / GIL. The only coupling is that the in-process handler (FireworksEngine) can't cross a process boundary, so the subprocess rebuilds an equivalent one from a serializable recipe — kept backend-agnostic in the gateway package: - FireworksEngine.handler_factory_spec() -> (import_path, config): attaches to the SAME deployment via the sampler's base_url/model (no re-provision); API key from env, never serialized. (renderer_family now retained for this.) - rllm/gateway/worker_handlers.py::build_fireworks_handler(config): rebuilds tokenizer + DeploymentSampler + FireworksEngine + create_tinker_handler. - gateway package: `python -m rllm_model_gateway --handler-factory module:func --handler-config x.json` dynamic-imports the factory and passes the built local_handler to create_app. Package never names a backend. - GatewayManager: rllm.gateway.num_workers (default 0 = trainer thread, unchanged). 1 = single separate process via _start_gateway_subprocess (reuses _start_process spawn/health-poll; longer startup budget for engine build; temp config cleaned up in stop()). >1 raises NotImplementedError (needs the session-sticky front — the next step). Trainer<->gateway comms (weight_version, traces) already go over HTTP, so they work unchanged against a subprocess. Validated: generic factory plumbing end-to-end (dummy factory served a request); spec extraction + spec<->factory key contract; 236 gateway tests pass. The live Fireworks build (real DeploymentSampler attach) is validated on a training run. Expected: frees the trainer from GIL contention; gives the gateway a dedicated core (~98%->~100%, marginal). Does NOT by itself clear high concurrency on the gateway — one core is still one core; that needs num_workers>1 (front router). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gateway): silence per-request uvicorn access logs in subprocess mode The separate-process gateway ran uvicorn at info, so every "POST /v1/chat/completions 200 OK" was logged — a flood at hundreds of concurrent requests. Disable uvicorn access_log unless log_level=debug (thread mode already suppressed these by running uvicorn at 'warning'). Our own INFO logs (loop-health, TokenAccumulator duplicates) are on the rllm_model_gateway logger and are unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(trainer): event-loop health monitor for the trainer loop Mirrors the gateway's loop-health monitor so the trainer's single event loop is observable too — to catch when the *trainer's* data pipeline (on-loop enrich_episode_with_traces, advantage/batch prep) becomes the bottleneck now that the gateway can run in its own process. - rllm/utils/loop_health.py: reusable run_loop_health_monitor(label, gauges=...) logging lag_ms p50/p99/max + thread_cpu% (+ optional gauges) every ~20s. Same discriminator as the gateway: high lag + high thread_cpu = self-CPU bound; high lag + low thread_cpu = GIL-starved. Logger pinned to INFO so it stays visible when the rllm.* hierarchy is raised to WARNING. - AgentFlowEngine: store n_parallel_tasks; add read-only `inflight`/`pending` gauges (derived from the concurrency semaphore) for the monitor. - unified_trainer._fit_fully_async: start the monitor as a background task (label "trainer", gauges=inflight/pending), cancel on shutdown. Diagnostic only. Validated: lag/thread_cpu discriminator + gauge rendering. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Replace the per-sync-window quota in SyncCoordinator with a continuously
evaluated capacity, mirroring AReaL's StalenessManager.get_capacity:
dispatch allowed iff in_flight_groups < (1+staleness)*mini_batch (staleness)
and running_rollouts < n_parallel_tasks (concurrency)
Budget frees on every consume, filter, and rollout-task completion via
_signal_capacity() -- not only at weight sync -- so freed slots refill
immediately instead of the old front-load-then-idle burst within a step.
on_sync_complete now only bumps the version.
New metrics: async/running_rollouts, async/max_in_flight_groups,
async/max_concurrent_rollouts. Adds tests/trainer/test_sync_coordinator.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ity summarization (#725) * feat(harness): terminus2-native — Terminus-2 over native tool-calling with harbor-parity summarization Same terminal interaction model as terminus2 (tmux keystroke batches + screen observations), but the conversation rides native tool-calling rails: execute_commands tool calls, role:tool results, and reasoning_content echoed verbatim on assistant messages so think-mode models (DeepSeek-V4-Pro) retain reasoning across turns. Measured on Fireworks: the user-message path renders echoed reasoning to zero prompt tokens; the tool-calling path renders it fully. Context management is ported verbatim from harbor terminus_2.py so eval deltas stay attributable to the transport alone: - three-subagent summarization dance (summary -> fresh-context questions -> answers), prompt texts byte-identical to harbor's - reactive fallback chain on context-length errors: unwind whole turns from the tail -> full dance -> short summary -> no-LLM handoff -> "Technical difficulties" fabricated reply (episode never crashes) - proactive summarization when estimated free tokens < 8000; tokens estimated from real usage anchors + chars/4 instead of litellm's model map (whose unmapped-model 1M default silently disabled summarization in raw harbor) - context-length errors fast-fail out of the retry loop (a doomed payload retried 4x is just 4 identical failures) Knobs: enable_summarize, proactive_summarization_threshold, context_limit_tokens (env RLLM_TERMINUS_CONTEXT_LIMIT wins so eval runs can pin the model's real limit, e.g. 240000 for v4 on Fireworks). Validated end-to-end on TB2 (v4-pro, effort=max): the dance fired in production and the 207-step episode recovered to a PASS; 100% reasoning persistence across all episodes; blended pass@1 ~66% / pass@3 ~76% vs 54.3 for the classic user-message harness (official in-house: 67.9). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: apply ruff format Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(data): tb-v2 dataset catalog entries + HF/local task builders Adds tb-v2-debug / tb-v2-opus-pass-at-1 / tb-v2-full (and friends) to the dataset registry, backed by new hf_tasks_builder/local_tasks_builder that materialize task archives from HF (mobius-lab/tb-v2-tasks) or local dirs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(gateway): preflight the gateway port before expensive setup The gateway binds 0.0.0.0:<port> from a background thread/subprocess, so a bind conflict surfaced as an opaque 30s health-poll timeout — after minutes of Fireworks infra provisioning. Probe-bind the port at trainer init (before the backend provisions anything) and again in start(), and raise GatewayPortInUseError naming the holder-lookup command and, when the port is the tunnel daemon's upstream, the likely concurrent-run cause. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(fireworks): retry raw ssl.SSLError during sampling The Fireworks SDK's SSE stream can surface raw ssl.SSLError (e.g. bad_record_mac) unwrapped by httpx; the transient classifier missed it, failing the sampling call permanently on attempt 1 — 26% of episodes died to this in a single run. Classify ssl.SSLError as a network error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(terminal-rl): qwen3p5-35b debug launch script + cookbook updates Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(deps): pin fireworks-training-cookbook to the locked commit The unpinned git ref resolved to cookbook HEAD, which now requires fireworks-ai>=1.2.0a86 and conflicts with our ==1.2.0a83 pin, making rllm[fireworks] unresolvable on fresh installs. Pin the commit recorded in uv.lock (the combo running in production). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(lint): apply pre-commit ruff fixes to PR-changed files Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sandbox): demote per-command failure logs to debug A non-zero verifier exit is the expected outcome for an unsolved task; the raised exception already carries the detail and callers pick the severity. Matches the docker backend, which already logged at debug. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(terminal-rl): tune qwen3p5 debug config (context 101k, port 9091, 4 replicas) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: signalrush <269811712+signalrush@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Tianhao Wu <thw@gpu-dp-6wgmd-rj5sb.slurm-compute.slurm.svc.cluster.local>
…rbor-par…" This reverts commit b5eeb93.
…m_workers>1) (#728) * feat(gateway): ConsistentHashPolicy for multi-worker session sharding Deterministic session_id -> worker mapping (stable sha256 over the fixed registered worker set, ordered by url) for num_workers>1: a session's chat turns AND its trace/session-control ops must always land on the same worker (which holds that session's in-memory accumulator/traces/params). Health churn doesn't remap sessions (a dead worker's sessions fail cleanly rather than silently landing on a worker that lacks their state). Sessionless requests fall back to least-loaded. Phase 1 of num_workers>1; not yet wired into a front. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(gateway): thin session-sharding front for multi-worker mode Adds a dedicated thin reverse proxy (rllm_model_gateway.front) that sits in front of N backend gateway workers and routes by session_id: - /sessions/{sid}/... -> the owning worker (ConsistentHashPolicy), forwarding the ORIGINAL path so the worker's own middleware keys its per-session state; responses are streamed (SSE + heartbeat pass through). - POST /sessions (create): sid from the body -> owner. - global ops (weight_version / flush / reload / batch_delete) -> fan out to all workers, aggregating the result. - /health. Run via `python -m rllm_model_gateway --front --worker URL --worker URL ...`. The front never parses chat bodies or holds session state — workers (ordinary num_workers=1 gateways) do all of that; the front stays thin so one core routes for many workers. Validated end-to-end with 2 dummy workers: sticky routing (same sid->same worker), balanced distribution, traces routed to owner, weight_version fan-out reaching both workers, delete routed to owner. Not yet spawned by GatewayManager (next commit). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(gateway): GatewayManager spawns N workers + front for num_workers>1 Generalizes the separate-process gateway to multiple workers: - num_workers==1: one gateway on port (unchanged). - num_workers>1: N worker gateways on port+1..port+N (each a full local_handler gateway via the handler-factory, cumulative ON), behind a thin `--front` session-sharding proxy on `port` (the tunnel target). Each worker is health-polled with extra startup headroom, then the front. Refactors spawning into _gateway_cmd (worker vs front) + _poll_health(port); tracks all processes in _processes and terminates them (front first) in stop(). Trainer<->gateway control plane is unchanged: it hits the front (port), which routes/fans-out — so set_weight_version / aget_traces / adelete_session need no changes. Validated via a manager-driven smoke (2 workers + front): requests shard across both worker processes, sticky routing, weight_version fan-out reaches both, 3 processes spawned. Drops the num_workers>1 guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(gateway): tag per-worker logs with the worker port (num_workers>1) With multiple gateway workers behind the front, their `loop health` and `TokenAccumulator duplicate` INFO lines interleave indistinguishably. Prefix both with a worker label (the worker's port) so a hot/among-N worker is identifiable. Empty label for the single-process case (no change there). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gateway): silence httpx per-request INFO logs in gateway processes httpx logs one INFO line per request; with the multi-worker front forwarding every chat call to a worker (and workers calling Fireworks), that floods the trainer stdout ("INFO:httpx:HTTP Request: POST .../v1/chat/completions 200 OK"). Set the httpx/httpcore loggers to WARNING in the gateway processes unless log_level=debug. Our own INFO logs (loop-health, duplicates) are unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style(gateway): ruff import-sort + format Fixes pre-commit: I001 unsorted-imports in manager.py and ruff-format line collapsing in proxy.py/server.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The async continuous-dispatch generation loop drives rollouts via process_task_with_retry, bypassing AgentFlowEngine.run() and its end-of-step batch_delete. Every completed rollout's gateway session + traces (fat cumulative token-id lists) therefore linger in the gateway's in-memory store for the whole run, growing the per-worker heap unbounded. CPython's cyclic GC then walks that graph on the event-loop thread, stalling request handling (loop-health lag spikes, thread_cpu ~100%, /health timeouts). - unified_trainer._generation_loop: delete each gateway session in a try/finally once the episode is buffered (its traces are already fetched into it), bounding the store to the in-flight set. No-op for gateway-less engines; also frees the session on permanent failure. - gateway server.main(): gc.freeze() the static tokenizer/renderer/sampler and raise the gen-2 threshold. Scoped to the standalone gateway process (main() never runs for the in-trainer thread-mode gateway) so it can't perturb the trainer's GC. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The gateway binds 0.0.0.0:<port> from a background thread/subprocess, so a bind conflict surfaced as an opaque 30s health-poll timeout — after minutes of Fireworks infra provisioning. Probe-bind the port at trainer init (before the backend provisions anything) and again in start(), and raise GatewayPortInUseError naming the holder-lookup command and, when the port is the tunnel daemon's upstream, the likely concurrent-run cause. Riders: typing->collections.abc import lint in worker_handlers.py and loop_health.py (zero behavior change). Co-authored-by: Tianhao Wu <thw@gpu-dp-6wgmd-rj5sb.slurm-compute.slurm.svc.cluster.local> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Harbor's Terminus-2 enables context summarization ("compaction") by
default: when the context fills it spawns summarization subagents that
compress history, which fragments the trajectory the gateway captures
for training.
Plumb an `enable_summarize` knob through the terminus2 harness
(RLLM_TERMINUS_ENABLE_SUMMARIZE -> in-sandbox driver -> Terminus2) and
have the terminal-rl cookbook read TERMINUS_ENABLE_SUMMARIZE. It defaults
to on when train.py is run standalone (Harbor's default); the train_*.sh
scripts set it to 0 to turn compaction off.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…s; add RLLM_HARNESS_VERIFIER_TIMEOUT_S cap (#736) * fix(sandbox): size Modal/Daytona lifetime from effective agent + verifier timeout Two problems made rollouts (esp. SWE-bench Verified) look like they "run forever" or get reaped mid-verify: 1. The agent timeout used for sizing was `task.metadata["agent_timeout"] or RLLM_HARNESS_RUN_TIMEOUT_S`, so a task's large baked-in agent_timeout bypassed the operator cap. Now RLLM_HARNESS_RUN_TIMEOUT_S is a hard cap on the per-task value (matching cli_harness._effective_timeout). 2. The lifetime added install + a flat 600 slack but not the (enforced) verifier_timeout in a way that reliably covers a long verifier — a task whose verifier legitimately runs ~3000s (verifier.timeout_sec) outlived the sandbox. Now: verifier declared -> lifetime = agent + verifier_timeout + 300 verifier absent -> lifetime = agent + 600 raised to the RLLM_SANDBOX_TIMEOUT_S override. So the sandbox is always slightly longer than agent + the enforced verifier budget, with no config needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(sandbox): RLLM_HARNESS_VERIFIER_TIMEOUT_S — operator cap on the verifier timeout Symmetric to RLLM_HARNESS_RUN_TIMEOUT_S (agent). Tasks bake in a per-task verifier.timeout_sec (SWE-bench Verified declares 3000); this env caps it as a hard ceiling, applied to both the enforced verifier timeout (ShellScriptEvaluator) and the sandbox-lifetime sizing via a shared _effective_verifier_timeout(task). Unset = use the task's declared value. Lets an operator shrink an over-declared verifier for fast iteration without editing task.toml. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…reworks (DPPO, CISPO, GSPO) (#713) * feat(algorithm): unified custom-loss abstraction across verl/tinker/fireworks (DPPO) A single per-token loss-term contract — `LossContext -> (per_token_loss, agg_mask, metrics)` — that runs identically in-process on verl and via `forward_backward_custom` on tinker/fireworks (the two managed backends share that mechanism). Subsumes the old policy-vs-aux split: ECHO is now just a term over the observation mask. Core (rllm/trainer/algorithms/loss.py): - @register_loss registry, LossContext, resolve_loss_terms, loss_plugins import hook. - Built-in terms: dppo_tv, dppo_kl, ppo_clip, env_prediction (ECHO). dppo_tv is unit-verified to reproduce verl 0.8's compute_policy_loss_dppo_tv per-token loss. Backends: - verl: register rLLM terms into POLICY_LOSS_REGISTRY on Ray workers (patch hook); never shadows native dppo_tv/dppo_kl. ECHO stays on CustomPPOLoss._apply_aux_losses. - tinker/fireworks: shared forward_backward_custom adapter folds surrogate + ECHO into one pass (no separate aux CE pass); fireworks forces GradAccNormalization.NONE. Public API: @rllm.register_loss / rllm.LossContext (same lazy-export style as @rllm.rollout / @rllm.evaluator). Config: algorithm.losses / loss_plugins / mu_source (mu defaults to inference log-probs — the tmax DPPO choice). Back-compat: native loss_fn (vanilla/ppo/grpo) and aux_losses/env_loss_coef paths are unchanged; the unified path engages only for rLLM-registered losses or an explicit algorithm.losses list. Tests: tests/unified_trainer/test_custom_loss.py (12). See design/unified-custom-loss.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(algorithm): deprecate the standalone aux-loss framework; ECHO is now a single loss term Collapse the two parallel systems into one. ECHO is defined once as the `env_prediction` term (`@register_loss("env_prediction", aux_mask=MASK_OBSERVATION)`); the old `aux_loss` framework (`@register_aux_loss`, `AuxiliaryLoss`, `EnvPredictionLoss`) becomes a thin deprecation shim that emits `DeprecationWarning` and delegates to the unified registry. - loss.py: `register_loss(name, aux_mask=...)` records a static token region used by the managed CE realization; `resolve_additive_terms()` resolves `aux_losses`/`env_loss_coef` into unified `ResolvedTerm`s (with `.mask`). ECHO = `env_prediction` term. - verl: `_apply_aux_losses` now evaluates the term function over a LossContext (single source of truth) instead of hardcoded CE. - tinker/fireworks: aux executors source positions/coef from `ResolvedTerm` (efficient CE realization unchanged); `build_aux_losses` -> `resolve_additive_terms`. - aux_loss.py: deprecation shim; `build_aux_losses` is a non-warning internal alias, `register_aux_loss`/`AuxiliaryLoss`/`get_aux_loss` warn. `EnvPredictionLoss` removed. Back-compat: `aux_losses` / `env_loss_coef` / `adv_estimator=echo` still work (resolve to the `env_prediction` term). Migration: `@register_aux_loss` -> `@rllm.register_loss(name, aux_mask=MASK_OBSERVATION)`, `aux_losses:` -> `losses:`. Tests: test_aux_loss.py rewritten for the deprecation/delegation (33 unified-trainer tests pass). See design/unified-custom-loss.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(algorithm): verl-style single-loss selector; remove the auxiliary-loss framework Per review: drop the `losses` list and the standalone aux-loss system; mirror verl's model of one loss selected by a string. A loss is a single function that returns the COMPLETE scalar objective and aggregates via a backend-injected `ctx.aggregate` — exactly like a verl POLICY_LOSS_REGISTRY function. - loss.py: LossContext gains `aggregate`; losses return `(scalar, metrics)`. `resolve_loss` returns the single selected loss (or None for a backend-native one). Built-ins: ppo_clip, dppo_tv, dppo_kl, ppo_clip_env. dppo_tv still matches verl 0.8's kernel (unit-tested). - ECHO is no longer a framework: it's the `ppo_clip_env` loss, which adds the obs-token CE term inside its own body. `adv_estimator=echo` defaults loss_fn=ppo_clip_env (env_loss_coef 0.05). `algorithm.loss_params` carries loss-specific params (verl-style). - verl: CustomPPOLoss runs the rLLM loss in-process over a LossContext (aggregate=agg_loss with global-batch norm), native kernel otherwise. Removed _apply_aux_losses and the POLICY_LOSS_REGISTRY shim/worker hook. - tinker/fireworks: forward_backward_custom runs the single loss; removed the separate aux CE passes (_get_aux_loss_futures, _build_aux_datums) and aux metric plumbing. Deleted: aux_loss.py, tinker/aux_loss.py, verl/custom_loss.py, design/auxiliary-losses.md, test_aux_loss.py; config `losses`/`aux_losses`. Tests rewritten (19 pass). See design/unified-custom-loss.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(algorithm): add CISPO and GPG built-in losses CISPO (arXiv:2506.13585): clips the importance-sampling weight with a stop-gradient but keeps every token's gradient through log_prob (no token dropped, unlike PPO clip). GPG: clip-free REINFORCE-style policy gradient with group-normalized advantages. Both are per-token functions of (pi, mu, advantages, mask) — direct fits for the LossContext, so they run on all three backends. CISPO is unit-verified against verl 0.8's kernel and against ppo_clip's gradient behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(algorithm): add GSPO + LossContext.seq_reduce for sequence-level losses GSPO (arXiv:2507.18071): a sequence-level length-normalized importance ratio, clipped PPO-style and applied per token, using verl's stop-gradient identity so the value is the detached sequence ratio while the gradient flows per-token through log_prob. Adds one primitive to the abstraction: `LossContext.seq_reduce(values, mask, reduction)` reduces over each sequence and broadcasts back to per-token — a row on verl (B,T), the whole datum on the managed path. `ctx.aggregate` gains a `mode` override so GSPO forces seq-mean-token-mean. gspo is unit-verified against verl 0.8's kernel; seq_reduce and per-token gradient flow are tested. This is all `geo_mean` (GMPO) needs too. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(algorithm): layered loss resolution — entry-point discovery + loss_plugins on workers get_loss/is_custom_loss now fall back to lazy `rllm.losses` entry-point discovery on a registry miss (mirrors rllm.eval.agent_loader/evaluator_loader), so a pip-installed loss package is found with no config and works across process boundaries (incl. verl Ray workers, which call get_loss there). CustomPPOLoss also imports algorithm.loss_plugins on the worker so the explicit-list path registers there too. Explicit inline import still works unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(algorithm): native-first loss routing — prefer a backend's fused kernel over the custom path resolve_loss now takes the backend's native fused-kernel menu and returns None when loss_fn is in it, so the fast native kernel is used instead of the rLLM custom path — even when an rLLM loss of the same name exists. Only losses a backend can't run natively fall back: - verl: CustomPPOLoss runs the in-process rLLM loss only if loss_mode is NOT in verl's POLICY_LOSS_REGISTRY (so native dppo_tv/gspo/cispo on 0.8 win). - tinker: native_losses=TINKER_KNOWN_LOSSES → cispo/ppo/dro use the server string path. - fireworks: native_losses=BUILTIN_LOSSES (grpo/importance_sampling/dapo/dro/gspo/cispo) use the builtin server kernel; only dppo_tv/ppo_clip_env take forward_backward_custom. Avoids the ~3x forward_backward_custom cost whenever a fused kernel exists. Losses routed native take hyperparameters from backend config (eps_clip→clip_ratio), not loss_params. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(algorithm): centralize native-loss detection in native_loss_names(backend) One map of what each backend has a native fused kernel for, derived from the backend's own source of truth (no drift across versions): - verl: POLICY_LOSS_REGISTRY (vanilla, dppo_tv, dppo_kl, gspo, sapo, gpg, clip_cov, kl_cov, geo_mean, cispo, bypass_mode) - tinker: tinker.types.LossFnType (cross_entropy, importance_sampling, ppo, cispo, dro) - fireworks: builtin_losses.BUILTIN_LOSSES (grpo, importance_sampling, dapo, dro, gspo, cispo) Replaces the scattered inline derivations (verl POLICY_LOSS_REGISTRY check, tinker TINKER_KNOWN_LOSSES, fireworks _native_loss_names) with native_loss_names("<backend>"); all three now feed resolve_loss's native-first routing from this one place. Returns ∅ when the backend isn't importable → falls back to the rLLM custom path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(algorithm): rename ppo_clip_env loss → echo Shorter, matches adv_estimator=echo (which selects it). No behavior change — echo is rLLM-only (no backend has a native kernel), so it always runs the custom path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(algorithm): rename gpg loss → reinforce Our gpg loss is exactly -advantages*logπ — the REINFORCE loss (no ratio, no clip); the "GPG" name really refers to the advantage normalization, which lives in the estimator, not this loss. `reinforce` is the accurate, recognizable name (verl calls the same loss `gpg`, Fireworks `reinforce`). verl's native-menu references stay `gpg` (that's verl's name). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(fireworks): skip builtin-loss validation for custom (forward_backward_custom) losses _init_fireworks_infra and resolve_builtin_loss validated algorithm.loss_fn against Fireworks' builtin kernel menu unconditionally at startup, which rejected rLLM custom losses (e.g. dppo_tv) with "Unsupported policy_loss" before they ever reached the client-side forward_backward_custom path. Guard both with resolve_loss(..., native_losses= native_loss_names("fireworks")): a loss that isn't a Fireworks builtin skips the builtin validation/resolution (and self._builtin_loss stays unused, since the custom path returns early). Native/default losses (grpo, cispo, ...) validate as before. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(loss): unify loss_agg_mode across backends; fix managed grad-accum normalization The managed (tinker/fireworks) custom-loss adapter hardcoded one aggregation (token-mean-within-datum, cross-datum mean) and ignored algorithm.loss_agg_mode, while forcing GradAccNormalization.NONE on Fireworks. Under gradient accumulation (mini_batch_size/fwd_bwd_group_size > 1) the per-pass means are SUMMED, not averaged, so the gradient — and the effective LR — is inflated by num_fwd_bwd_passes (16x for the a35b DPPO cookbook). verl was already correct: its injected aggregate = agg_loss(..., batch_num_tokens, global_batch_size, dp_size) folds the GLOBAL normalizer into every micro-batch. Unify on one loss_agg_mode (verl's names/semantics), realized per backend: - loss.py: register_loss(agg_mode=) + RLLM_LOSS_AGG_MODE + ResolvedLoss.agg_mode; resolve_loss picks loss-pin > config > DEFAULT_LOSS_AGG_MODE ("token-mean", matching verl + Fireworks' RL guidance). gspo pins seq-mean-token-mean. - custom_loss.py: build_custom_loss is mode-aware (within-seq token-mean for seq-mean-token-mean, else masked sum; cross-seq sum) and adds server_normalized: * Fireworks (True): return a RAW SUM; server divides across the whole window. * Tinker (False): divide client-side by the pass's global count (correct for its single-pass usage; per-pass under accumulation — documented). - fireworks_policy_trainer.py: custom path builds server_normalized=True and optim_step sets GradAccNormalization from resolved.agg_mode (NUM_LOSS_TOKENS for token-mean, NUM_SEQUENCES for seq-mean-*) instead of NONE — accumulation-invariant. Also restore offpolicy/* diagnostics on the Fireworks custom path (proximal-vs- inference log-probs): the custom branch returned before the builtin path computed them, so dppo_tv runs logged no offpolicy metrics (and DPPO's keep-mask depends on exactly that train/inference gap). Tests: agg_mode resolution + pin, register_loss validation, Fireworks accumulation-invariance (raw sums compose), Tinker client-side per-mode divisor. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * perf(fireworks): drop diagnostic proximal forward on the custom-loss path The offpolicy/* metrics I restored required a proximal forward pass (_compute_proximal_logprobs) on every fwd-bwd pass — mini_batch_size/ fwd_bwd_group_size extra forwards per optimizer step (16x for the a35b DPPO cookbook), a noticeable training-loop slowdown. On the custom (DPPO) path mu = the inference log-probs already on the datums, so pi_old is never recomputed and that forward is purely diagnostic. Remove it; the loss still emits its own diagnostics (e.g. dppo_tv/mask_frac). The builtin GRPO path is unchanged — it needs proximal for its ratio denominator. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gateway): stamp traces with the proxy's fanned-out weight_version (fix async staleness) async/staleness_* read a flat value equal to weight_version (min=mean=max, e.g. 20 at step 21) — every trajectory group was tagged weight_version=0. Root cause: build_trace_record preferred the engine-stamped weight_version in the response body over the proxy's tracked value. In a multi-worker gateway (num_workers>1) each worker subprocess rebuilds its own rollout engine whose self.weight_version is never updated — only proxy.weight_version is (the front fans /admin/weight_version out to every worker). So the engine stamped a stale 0 into every response, clobbering the correct proxy value → all steps 0 → group 0 (_min_weight_version) → staleness = coordinator.weight_version. Fix: prefer the proxy's tracked version (correct on every worker); fall back to the engine-stamped response value only when the proxy isn't tracking one. This matches build_trace_record_from_chunks (streaming), which already used the proxy value only, so the two paths are now consistent. Fixes staleness for the multi-worker subprocess, single subprocess, and in-thread gateway configs. Off-policy metrics were already correct (they compare recomputed proximal vs inference log-probs, not the trace tag), so this is purely the staleness gauge. tests: TestBuildTraceRecordWeightVersion — proxy wins over stale response, base version 0 not clobbered, response fallback preserved when proxy untracked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(gateway): trim the weight_version precedence comment Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * change(algorithm): default loss_agg_mode to seq-mean-token-mean Was token-mean. seq-mean-token-mean weights every sequence (trajectory) equally regardless of length — the GRPO-original behavior and a better default for the variable-length agentic trajectories rLLM targets. Single source of truth is rllm.algorithm.loss_agg_mode (base.yaml). Via the sync_config shared-key mapping it also propagates to verl's actor.loss_agg_mode: a non-null rllm yaml default now wins over verl's token-mean default (previously null let verl's default stand). So the new default applies uniformly to the managed (tinker/fireworks, native + custom-loss) and verl backends. Setting loss_agg_mode=null explicitly still defers to verl's own default. - base.yaml: loss_agg_mode null -> seq-mean-token-mean - AlgorithmConfig: field default + from_config fallback -> seq-mean-token-mean - loss.py DEFAULT_LOSS_AGG_MODE (custom-loss fallback) -> seq-mean-token-mean - test updated Behavior change: runs that didn't set loss_agg_mode and relied on token-mean now get seq-mean-token-mean; pass loss_agg_mode=token-mean to keep the old behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(async): skip fwd/bwd + optim on an all-dropped batch instead of crashing If every trajectory in a batch is dropped as malformed (empty output_logprobs — e.g. failed/overloaded generations that returned no completion), the managed backends called forward_backward([]) and crashed with "ValueError: No data provided to forward_backward", killing the whole run (observed after ~100 steps on a35b when an inference-overload spike made a full fwd-bwd chunk's rollouts all fail). Two guards: - forward_backward_from_trajectory_groups (fireworks + tinker): when transform yields no datums, log a warning and return early (skip forward_backward) with train/num_sequences=0, train/dropped_all_sequences=1. - _training_loop: track trainable sequences across passes; run update_policy + weight sync only when >0. An all-dropped step accumulated no gradient, so stepping the optimizer would apply an undefined/zero-gradient update and bump the LR schedule + trigger a weight sync for nothing. on_training_step_complete is likewise gated so a skipped batch doesn't count toward the sync trigger. Adds async/trained_this_step metric. The transient overload that produced the empty batch is expected; the trainer now rides it out instead of dying. (Root cause of the empty batches — rollout concurrency vs inference replica capacity — is a config concern, separate.) tests: TestAllMalformedBatch — all-empty-logprobs batch -> empty datums + dropped_malformed_sequences count; mixed batch keeps the valid trajectory. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(fireworks): default bypass_mode=True so the GRPO clip isn't inert in async The managed async loop runs one optimizer step per batch (no inner PPO epochs), so a recomputed proximal equals the current policy (pi_old == pi_theta): the clip ratio pi_theta/pi_old is ~1, the clip does nothing, and the proximal forward is wasted. fireworks_backend already ASSUMED bypass_mode defaults True (its validation reads rc.get("bypass_mode", True)), but the gradient path read the AlgorithmConfig field whose default is None -> False, so it silently recomputed proximal and ran an inert clip. Honor True by default in the policy trainer: pi_old := rollout logprobs, making the clip a real pi_theta/pi_rollout trust region against the actual behavior policy (per-token, correct weight version, from the recorded rollout logs) -- the same 2-policy anchor DPPO/SAO use. Also skips the redundant proximal forward. The offpolicy/* diagnostic (real proximal vs rollout) is only computed when bypass=false, since under bypass prox==rollout makes it trivially 1. Scope: Fireworks only. Tinker never recomputes proximal (already uses rollout logprobs directly), so it's unaffected. verl keeps its own default -- verl's bypass=True path requires rollout_log_probs in the batch and asserts otherwise, so a global default flip would be unsafe there; verl users opt in explicitly. Set rollout_correction.bypass_mode=false to restore the 3-policy proximal path (only meaningful with multi-update-per-batch PPO). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: fix ruff (pre-commit) on the custom-loss files Pre-commit (ruff v0.11.4 lint + format) was failing on the feature's files. Auto-fixed: UP037 (drop quotes from annotations under `from __future__ import annotations`), UP035 (Callable from collections.abc), UP007 (X|Y), F401 (unused torch import in seq_reduce), I001 (import sorting). Manual: wrapped an E501 line in loss.py; `# noqa: E402` on the two imports after pytest.importorskip in test_custom_loss.py (skip-if-no-torch pattern). ruff-format applied to loss.py / unified_trainer.py / verl_backend.py. No behavior change; 27 custom-loss/transform tests still pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(dppo): C=inf (drop truncated-IS) to match the DPPO paper dppo_tv/dppo_kl multiplied the score by a *truncated* importance ratio (min(pi/mu, 20)) via the shared _truncated_is helper. But the DPPO paper (arXiv:2602.04879, Eq. 23) sets C=inf for DPPO: the divergence mask IS the trust region, and Sec. 5.4 ("Pitfalls of Truncated Importance Sampling") shows truncation reintroduces the low-prob-token bias DPPO exists to remove. Use the untruncated ratio as a detached weight on the score; _ratio's +/-20 log-ratio clamp stays purely as inf protection. Remove the now-unused _truncated_is helper (cispo/ppo_clip keep their own finite clip on purpose). test_dppo_tv_matches_verl_formula: reference switched to C=inf and a large-ratio (~150) kept token added so a truncation regression is actually caught (was inert before: random inputs never exceeded the cap). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(loss): rename LossContext fields pi/mu/ref -> logp_curr/logp_old/logp_ref `mu` was overloaded — it means the proximal old-policy in one mode and the rollout/behavior policy in another (they only coincide under bypass_mode). Name the fields by role, aligned to verl's own field names, so the distinction is explicit and the bypass identity reads as a statement ("in bypass, logp_old == logp_rollout") rather than a hidden collapse: pi -> logp_curr current policy being optimized (grad) [verl log_prob] mu -> logp_old the ratio denominator (proximal old policy) [verl old_log_probs] ref -> logp_ref KL reference policy [verl ref_log_prob] Pure mechanical rename, no behavior change. Updated the dataclass, all loss bodies + docstrings, the two LossContext construction sites (verl_backend, tinker/custom_loss), and the tests. verl's "ref" mesh / ref_policy_wg is untouched (only the 3 LossContext kwargs changed). 25/25 custom-loss tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(loss): trim overly verbose comments (dppo C=inf, fireworks bypass/offpolicy/all-dropped) Comment-only; no behavior change. Also aligns the fireworks comments with the logp_old naming. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: add "Custom loss functions" page (built-ins + how to register your own) New docs/training/custom-loss.mdx under Advanced algorithms: the built-in policy losses (ppo_clip, dppo_tv/kl, cispo, gspo, reinforce, echo), the LossContext contract + aggregate/seq_reduce reducers, loss_agg_mode, native-first backend routing, and a worked custom-loss example (IcePop) with loss_plugins wiring. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(loss): expose logp_rollout on LossContext (raw inference log-probs) Adds an optional `logp_rollout` field = the behavior policy's recorded sampling log-probs, distinct from `logp_old` (the ratio denominator). They coincide under bypass_mode but differ when a proximal old-policy is recomputed (non-bypass / verl old_log_probs vs rollout_log_probs). Lets a custom loss reference the true sampling distribution independently of the denominator — needed for train/inference-mismatch corrections (TIS/IcePop), which land in a future PR. Populated on every backend: verl `rollout_log_probs` (when present), managed path always the datum's raw `logprobs` (regardless of any `mu_arrays` proximal override). Documented in the custom-loss page; test asserts it's populated and distinct from logp_old under a proximal override. 26/26 tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(config): data-driven estimator→loss default; make echo a normal loss; drop mu_source Three cleanups from review of AlgorithmConfig: 1. Replace the ECHO-specific `if` in __post_init__ with a data-driven `_ESTIMATOR_DEFAULT_LOSS` map ({ECHO: "echo"}); loss_fn defaults from it when unset. Extensible and not special-cased. 2. Drop the dedicated `env_loss_coef` config field. echo now reads its coefficient from loss_params like every other loss reads its params (e.g. dppo_tv's delta), defaulting to 0.05 in the loss body. Removes the field, its from_config read, the auto-default, the resolve_loss injection, and the base.yaml entry. 3. Remove `mu_source`: it was read in exactly one place — a warning that "proximal" isn't supported on the Fireworks custom path — i.e. a functional no-op. The logp_curr/logp_old/logp_rollout naming now documents the semantics. Tests updated to the new design (estimator→loss default; env_loss_coef via loss_params + loss-body default). 34/34 config + custom-loss tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
`icepop` gates each token on the train/inference ratio d = exp(logp_curr -
logp_rollout): zeros the gradient when d leaves [icepop_alpha, icepop_beta]
(default 0.5/5.0), and weights the score by d inside the band (paper's
M(k)=k·1[a<=k<=b]). Masks out-of-range tokens rather than clipping them.
Uses logp_rollout (the true per-token sampling distribution, correct even across
weight versions in a partial rollout) with logp_curr as the "train" side — no
proximal forward, so it sidesteps the multi-weight-version proximal problem and
works on the default bypass path across all backends. Guards on logp_rollout=None
(loud error, no silent fall back to the proximal logp_old).
Emits icepop/ratio_{mean,min,max} over trainable tokens + mask_frac for
monitoring the discrepancy distribution (batch-true on verl; per-sequence then
averaged on the managed per-datum path). Tightened the logp_rollout docstring;
added to the built-in-losses doc table + a guarded worked example. 29/29 tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…egister_loss) Rename register_rllm_adv_estimator -> register_adv_estimator and get_rllm_adv_estimator -> get_adv_estimator, and surface the decorator at the top level as `rllm.register_adv_estimator`, mirroring how `rllm.register_loss` is exposed. This gives custom advantage estimators the same clean, discoverable registration path as custom losses. - rllm/__init__.py: lazy top-level `register_adv_estimator` export - trainer/algorithms/__init__.py: export register_adv_estimator, get_adv_estimator, RLLM_ADV_ESTIMATOR_REGISTRY - trainer/algorithms/advantage.py: rename + keep back-compat aliases (register_rllm_adv_estimator / get_rllm_adv_estimator) so existing custom-estimator imports keep working. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…chat history (#744) Harbor's Chat.chat() drops reasoning_content from assistant history messages unless Terminus2 is constructed with interleaved_thinking=True (Harbor default False), so every request after turn 1 loses all past thinking — bad for reasoning-model distillation, where the captured requests are the SFT context. Plumb the flag through the harness: an interleaved_thinking class attr (env-defaulted from RLLM_TERMINUS_INTERLEAVED_THINKING so `rllm eval` can toggle it without code), a build_env passthrough, and the driver forwarding it to Terminus2(...). Default stays off, matching Harbor. Verified A/B on Fireworks deepseek-v4-pro: with the flag on, every request's history assistant messages carry the exact prior-turn reasoning_content; off, none do. Co-authored-by: Tianhao Wu <44137758+thwu1@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
#745) The interleaved_thinking knob reads its env var as the class-attribute default, but enable_summarize was a plain True — and build_env re-exports the attribute into the sandbox, unconditionally overwriting whatever RLLM_TERMINUS_ENABLE_SUMMARIZE the operator set on the host. Eval runs launched with RLLM_TERMINUS_ENABLE_SUMMARIZE=0 therefore still compacted: long trajectories got summarization handoffs ("You are picking up work from a previous AI agent"), breaking the cumulative prefix structure of the recorded chat histories. Read the env var as the default, mirroring interleaved_thinking, and add a build_env round-trip test for both knobs. Co-authored-by: signalrush <signalrush@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…n fwd/bwd KL to sampler) (#746) Per token, with r = p/q = exp(logp_curr - logp_rollout) against the SAMPLER policy (not a frozen reference): L = -sg[r]*A*logp + bwd_kl_coef*(-log r + r - 1) + fwd_kl_coef*(r*log r - r + 1) Unlike PPO clip (zeros out-of-band gradients) or CISPO/IcePop (clamp/mask the weight), every token always keeps its gradient; the trust region is enforced softly by the KL penalties, using the sample-wise non-negative unbiased estimators under q (k3-style). All knobs are opt-in: fwd_kl_coef / bwd_kl_coef default to 0.0, and ratio_clamp_min / ratio_clamp_max (default None) optionally bound the detached PG weight CISPO-style while the KL terms keep the true r. Co-authored-by: signalrush <signalrush@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…radient KL forms (#747) Per the latest spec, the KL(p, q) regularizers get a kl_approx switch (default False): - kl_approx=False (new default): score-function forms with the exact KL gradient in expectation under q. Backward: -logp (grad -∇logp = ∇KL(q||p); plain CE on the sampled tokens). Forward: sg[r·log r]·logp (grad r·log r·∇logp = ∇KL(p||q), using E_q[r·∇logp] = 0). - kl_approx=True (previous behavior): Schulman's non-negative unbiased value estimators — backward k3 (-log r + r - 1, grad (r-1)·∇logp), forward (r·log r - r + 1, same per-token gradient as the False form). The reinforce_kl/kl_bwd and kl_fwd metrics always report the estimator values, so logged KLs stay comparable across the knob. Co-authored-by: signalrush <signalrush@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ailure budget clock (#748) Three ways a recoverable sampling failure was killing whole rollouts: - 429 was not in the transient markers, so rate-limit responses failed the request permanently instead of backing off (routine at high concurrency). - Raw ssl.SSLError (e.g. bad_record_mac) escapes the SDK's SSE stream unwrapped by httpx, so the isinstance network-error check missed it. - The retry budget was measured from REQUEST START, so a long healthy generation that died mid-stream had already spent its own streaming time and got zero retries — systematically failing the longest completions. The clock now starts at the first failure. Also log response headers on final failure (request ids / rate-limit state for provider-side debugging). Co-authored-by: signalrush <signalrush@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
… logs to debug (#749) Two fixes: - Modal small-file uploads went through _exec_unchecked, so a failed write silently no-op'd: the task environment came up broken and its reward-0 trained as a legitimate failure instead of surfacing as SANDBOX_ERROR. Route the write through exec() so callers (setup hooks / evaluator) classify the raise correctly. - Agents fail shell commands constantly as part of normal exploration; logging every nonzero exit at WARNING floods training logs. Demote the per-command failure logs to DEBUG on modal/daytona/local (the exception path still carries the details). Co-authored-by: signalrush <signalrush@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…rward_backward (#751) The serving stack can (rarely) return a sampled token id past the trainer's embedding — e.g. a padded-lm-head slot drawn under full-distribution sampling (temperature=1.0, top_p=1.0, no top_k). One such id fails the entire forward_backward server-side with "Invalid token id ... Valid range is [0, vocab_size)" and kills the run. Observed in production: id 248090 sampled once in ~30M tokens on qwen3p5-35b-a3b (vocab 248077), crashing a full-param run at step 5; LoRA shapes tolerate it (padded trainer vocab), POLICY_TRAINER full-param shapes do not. Fix: transform_trajectory_groups_to_datums now takes vocab_size and drops any trajectory containing an id >= vocab_size before datum construction — prompt ids are scanned too, since cumulative token mode echoes past sampled turns into later prompts. Mirrors the existing dropped-malformed-sequences idiom. - metrics: batch/dropped_oov_sequences (count) and batch/oov_drop_rate - fireworks: vocab bound from len(AutoTokenizer(model.tokenizer_model)) (the served model path is not HF-resolvable; model config may carry vocab_size null, so the tokenizer is the authority) - tinker: bound from the training client's tokenizer - fail-open: unresolvable tokenizer disables the filter with one warning Co-authored-by: Tianhao Wu <thw@gpu-dp-6wgmd-rj5sb.slurm-compute.slurm.svc.cluster.local> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…<stamp>/<content> (#752) Relaunching with the same project/experiment name previously reused the same artifact directories, so a new run clobbered or interleaved with an earlier run's outputs: - episode logs went to logs/<project>/<experiment>/episodes for every run - the Fireworks SFT backend's default_local_dir is a fixed shared /tmp path (and even an explicit --output collides across relaunches) Every run now gets a per-run UTC stamp (UnifiedTrainer._run_stamp, computed once and shared by all of a run's artifact trees): - episode logs: logs/<project>/<experiment>/<stamp>/episodes - fireworks SFT dir: <default_local_dir>/<experiment>/<stamp>/ Deliberately NOT stamped: tinker-SFT and verl checkpoint dirs — their crash resume reads the stable path (get_last_checkpoint / resume_mode), so a blind stamp would break recovery; those need a resume-aware layout instead. The Fireworks SFT dir is safe to stamp because its resume is server-side (job DCP); the local dir only holds logs/metadata. Co-authored-by: Tianhao Wu <thw@gpu-dp-6wgmd-rj5sb.slurm-compute.slurm.svc.cluster.local> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…l forward) (#753) * feat(trainer): compute off-policy metrics in the custom-loss path (no proximal forward) The managed custom-loss path (forward_backward_custom, shared by tinker/fireworks) never logged offpolicy/* because they compared a recomputed proximal against the rollout — and under bypass_mode (the default) logp_old == logp_rollout, so the comparison was inert and skipped. Instead compare logp_curr (the current-policy forward the loss pass already runs) against logp_rollout. This is the train/inference mismatch we actually care about, it is non-trivial even under bypass_mode, and it costs no extra forward pass. - add shared offpolicy_metrics(curr, rollout, masks) helper in algorithms/loss.py, curated to the genuinely useful set: kl (signed k1), k3_kl, ratio/{mean,max}, chi2_token, training_ppl, rollout_ppl. Drops the noisy/redundant metrics (logprob/prob_abs_diff, log_ppl_* family, ratio/min, prob_pearson_corr, chi2_seq). - wire it into build_custom_loss: accumulate logp_curr.detach()/logp_rollout/mask per datum, merge once after the loop (benefits both fireworks and tinker). - keep the offpolicy/ namespace on the fireworks custom-loss surfacing (no train/ prefix). - collapse the non-bypass builtin _compute_offpolicy_metrics to delegate to the shared helper, so both paths emit the identical curated set (net -86/+80 lines). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(trainer): rename _compute_offpolicy_metrics old_logprobs -> curr_logprobs Pure naming change, no behavior. The builtin non-bypass path feeds prox_logprobs (a forward on the current weights at batch start, so proximal == current) into this arg; naming it old_logprobs made it read as "old vs rollout" when it is really current vs rollout. Rename the param + call-site kwarg to match the shared helper. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(trainer): clarify offpolicy_metrics measures staleness + train/inference mismatch The current-vs-rollout gap is not just train/inference engine mismatch: its dominant source is usually staleness — the current weights have drifted from the sampler snapshot that generated the rollout (which is why the rollout log-probs are off-policy at all). Fix the docstring/comment to name both sources. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(trainer): clarify custom-loss offpolicy is logged even under bypass_mode The custom-loss path computes offpolicy/* from logp_curr vs rollout in the closure, unconditionally — the bypass gate is only on the native builtin path. Also align the wording with the off-policy gap = staleness + train/inference mismatch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: ruff-format offpolicy_metrics (two blank lines before module block) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…path The in-process Tinker/Fireworks handler's token-in path (_token_prompt_completion) dropped per-token routing matrices from its completions-style response: it calls assemble_model_output directly — which, unlike the get_model_response* wrappers, does NOT copy routing_matrices onto the ModelOutput — and omitted the field from the choice entirely. With cumulative_token_mode on (turns 2+ all take this token-in path), the gateway trace stored routing_matrices=None, so router_replay=R3 silently no-opped end-to-end on the Fireworks backend. Read routing matrices off the TokenOutput and stamp them on the choice, as the chat path already does. Adds unit coverage for the token-in path and the proxy->trace carry. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…force_kl (#760) The importance-ratio diagnostics only reported the high tail: offpolicy/ratio logged mean+max, reinforce_kl logged mean+clamp_frac. But the sampler-trainer mismatch (e.g. fp8 rollout vs bf16 trainer) pushes log-ratios in BOTH directions — ratio << 1 marks tokens where the trainer assigns far less probability than the sampler did, whose policy-gradient weight vanishes under IS. Watching only max shows ratio explosion but is blind to gradient starvation. - offpolicy_metrics: add offpolicy/ratio/min (batch min over masked tokens, same tensor as mean/max). - reinforce_kl: add reinforce_kl/ratio_min and ratio_max over trainable tokens, icepop-style masked select (per-sequence on the managed per-datum path, true batch stats on verl — same caveat as icepop). Tests: extremes match hand-computed e^-2/e^1 ratios; masked tokens excluded from the extremes; offpolicy min/max both tails verified. Co-authored-by: Tianhao Wu <thw@gpu-dp-6wgmd-rj5sb.slurm-compute.slurm.svc.cluster.local> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ning distillation (supersedes #739) (#759) * feat(from-eval,sft): model-agnostic reasoning-preserving automerge + CUSTOMIZED masking rllm dataset from-eval read only the final step's conversation and dropped reasoning_content, so it could not distill a reasoning model. It now walks each attempt's steps into self-describing rows — every message carries a 'trainable' flag — deterministically, no flag decides it: - Reasoning is preserved MODEL-AGNOSTICALLY as a structured ThinkingPart, NOT a hardcoded '<think>' string: the model's renderer picks the format at training time (deepseek <think>, qwen, harmony, ...). Content is a uniform list of parts so the parquet column stays one type. Verified: the same curated dataset renders correctly under both the deepseek and qwen renderers, each in its own format. - Automerge walk (message-level analogue of verl.transform._process_trajectory): merges steps that appear identically as history into one row (all turns trained) and splits where they don't. A non-thinking trajectory is one prefix chain -> single row; a reasoning turn seals its segment (its thinking is stripped from history) so the next turn starts a new row -> per-turn rows, the inference- faithful form. Every step is trained; only a step with no assistant turn is skipped. - SFT masking is ALWAYS CUSTOMIZED, driven by each message's 'trainable' flag. Self-describing rows use their flags; flag-less rows (external --train-file) derive a default in the loader (all assistant turns, or the last when tokenize_and_mask_method=stepwise). Removes resolve_train_on_what and the train_on_what enum threading — one masking path. Removes the obsolete last-step-only extraction (_episode_to_messages / _clean_message / _load_messages). No CLI flags added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sft): promote the final Fireworks LoRA so it survives job teardown The SFT loop saved every checkpoint promotable=False and deletes the trainer job on close, so the trained LoRA was a resumable-only DCP blob garbage-collected after the ~30-day retention window — never a servable model. Make the final save promotable=True and promote_latest() it to a permanent account model before infra.close(), mirroring the RL path and the SDK sft recipe. Promotion failure is non-fatal (the promotable row outlives deletion; can be promoted manually). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(from-eval): fully data-driven merge — interleaved histories, tool-call passthrough Minimal follow-ups to the automerge: - Replace the seg_sealed flag with a reasoning-aware prefix match: the episode data alone decides. Non-interleaved runs (history strips thinking) split per-turn exactly as before; interleaved-thinking runs (history RETAINS reasoning) keep merging, with context messages preserving their ThinkingParts — matching what the model actually saw at inference. Net-negative LOC. - Carry tool_calls / tool_call_id / name through the rebuilt messages (the old _clean_message preserved them; native tool-calling harnesses need them) and include tool_calls in the prefix comparison. - Update test_curation.py's content assertions to the parts-list contract (they still asserted the old string form and were failing on this branch). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(sft): red tests capturing PR #739 audit findings F1-F7 + schema/bridge contracts 11 failing tests + 2 intentionally-unresolved imports documenting: - F1 role_colon default crashes on [thinking,text] parts (no renderer auto-detect) - F2 dict-shaped tool_calls crash tinker renderers (need pydantic ToolCall) - F3 parquet round-trip stamps tool_calls/trainable None -> TypeErrors - F4 automerge prefix desync on empty history messages - F5 cross-task dedup collision (assistant-only signature) - F6 empty trainable target + missing merge/split telemetry - F7 verl backend silently accepts structured rows it cannot train plus contracts for rllm.data.sft_schema, rllm.data.sft_bridges, 'rllm dataset import', and 'rllm sft --renderer'. Each test asserts the fixed-state behavior and fails today for the documented defect; subsequent commits flip them green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(sft): explicit SFT row schema + normalized tinker loader boundary New rllm/data/sft_schema.py (pydantic v2): TextPart/ThinkingPart, SFTToolCall (canonical OpenAI dict shape), SFTMessage (trainable, extra=forbid, .to_tinker_message() lowering to real tinker ToolCall objects), SFTRow (row-level metadata preserved), and normalize_* helpers that scrub arrow round-trip artifacts (tool_calls/trainable None-stamping, None cross-keys in parts) and derive trainable flags (all/last) for flag-less rows. tinker_dataset.py renders through the schema: normalize -> to_tinker_message -> CUSTOMIZED; per-row errors surface as SFTConfigError with the dataset row index. Fixes the dict-tool_calls AttributeError and the parquet TypeErrors (red tests test_f2_dict_tool_calls_render, test_f3_roundtrip_*). validate_messages_dataset now schema-checks row 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(from-eval): symmetric prefix filtering, task-aware dedup, walk telemetry - Merge walk now filters BOTH sides of the prefix comparison with one keep-predicate, so empty observations no longer desync the window and spuriously split clean prefix chains (test_empty_history_message_...). - Dedup signature keys on task_id + full conversation (role/text/ thinking/tool_calls), not assistant text alone - distinct tasks sharing short commands no longer collide (test_dedup_does_not_...). - Steps whose last assistant turn is empty (no text, no tool_calls) are skipped instead of trained as empty targets. - CurationStats gains segments_merged/segments_split/ steps_skipped_no_assistant/targets_skipped_empty; degenerate splitting (>=3 targets, 0 merges) logs a warning. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(sft): renderer auto-detection + --renderer flag; explicit verl gate - renderer_name defaults to null (= auto) in tinker/fireworks yamls; build_sft_data resolves via tinker's get_recommended_renderer_name with a family-heuristic fallback (qwen3_5/qwen3/deepseekv3/llama3) for models missing from tinker's map (e.g. Qwen3-0.6B raises KeyError). Explicit names get warn_if_renderer_not_recommended. - Text-only renderers (role_colon/llama3) + reasoning/tool-call rows now fail fast with an actionable SFTConfigError instead of a deep RendererError (fixes the default-CLI-path crash on from-eval output). - rllm sft gains --renderer, wired through spec.overrides data.renderer_name for tinker/fireworks. - VerlSFTBackend.validate_spec rejects structured rows (parts content / trainable flags) with a clear pointer to tinker/fireworks instead of crashing inside torchrun. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(from-eval): validate curated rows through the SFT schema at emission Each automerged row now passes normalize_row().to_record() before being emitted, so malformed provider payloads fail at curation time (logged + counted in CurationStats.rows_invalid) instead of deep inside a training backend. Emitted dict shape is unchanged for well-formed rows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(dataset): source-format bridges + 'rllm dataset import' command rllm/data/sft_bridges.py: BRIDGES registry with two official bridges — 'messages' (plain OpenAI-style rows, --train-on all|last) and 'think-tags' (parses leading <think>...</think> out of assistant content into structured ThinkingParts; --explode default ON emits one row per assistant turn with CoT-stripped inference-faithful history and a single trainable target; --no-explode keeps one all-trainable row per conversation). Row extras map _task/_reward/_group/_model to schema metadata fields. rllm dataset import FILE --name N --format think-tags|messages registers schema-validated rows in the dataset registry. Smoke on tmp/better-rllm-sft/opus_sijun_800.jsonl: 10 conversations -> 155 exploded rows, 0 bridge failures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(sft): SFT distillation guide (schema, bridges, --explode tradeoff, renderers) New docs/guides/sft-distillation.mdx: the eval -> from-eval|import -> sft pipeline, the structured SFT row schema (thinking parts vs baked <think>), automerge stats to watch, the think-tags --explode tradeoff (per-turn CoT coverage vs O(T^2) history duplication), renderer auto-detection and --renderer, backend support matrix (tinker/fireworks; verl gated), a worked Opus-traces example, and an SFTConfigError troubleshooting table. Registered under Guides in docs.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(sft): genericize think-tags bridge; scrub private dataset names from public surfaces The think-tags bridge and its docs referenced a contributor's name ("sijun") and private tmp/ file names (opus_sijun_800.jsonl, etc.). Reframe it generically as "rows whose assistant turns carry a leading <think>...</think> block", a common convention for distilled reasoning traces. Metadata semantics change: drop the private _EXTRA_KEY_MAP that renamed underscore keys (_task -> task_id, _reward -> reward, ...). _row_fields now passes through EVERY non-messages top-level key verbatim as row-level metadata (underscore-prefixed keys included, no renames, no silent drops). SFTRow is extra=allow and to_record() preserves extras, so this needs no schema change. Users who want canonical column names rename keys in their own bridge or preprocess the file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(eval): first-class tinker:// checkpoint models — auto-route to the Tinker OAI provider When the resolved --model starts with tinker:// and no --base-url is given, rllm eval now forces the built-in tinker provider (pinned to the Tinker OAI endpoint) regardless of the configured provider, and even with no config — the sampler-checkpoint path alone determines where it can run. Requires TINKER_API_KEY in the environment (clear actionable error otherwise) and emits one routing line. Every other provider path is unchanged. Live-verified that the Tinker OAI endpoint accepts a tinker://<run>:train:0/sampler_weights/final path as the chat-completion model, and that LiteLLM's proxy round-trips that raw ://-and-slashes model_name cleanly (HTTP 200) — so no proxy alias workaround was needed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(sft): --logger/--ui flags — route SFT metrics through rllm Tracking (wandb/mlflow/ui) Wire rLLM's tracking plugin selection into `rllm sft`. SFTSpec gains a `logger` field; the CLI adds `--logger` (repeatable, curated backends) and `--ui/--no-ui` mirroring `rllm train` (auto-enabled when logged in). Resolution starts from console, adds --logger values, appends ui when enabled, dedupes; None keeps the backend yaml default. tinker/fireworks build_config now honor spec.logger and run tracking finish() in a try/finally (the ui backend tees stdout and holds a session). verl drops ui with a warning (verl's Tracking asserts on it). Docs: new "Logging training progress" section. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sandbox): explicit timeout kwarg overrides sized lifetime in _create_base_sandbox build_modal_snapshot passes its own build_timeout while _sandbox_resource_kwargs also sizes a Modal 'timeout' — the duplicated kwarg made every 'rllm snapshot create --sandbox-backend modal' fail with a TypeError. Caller kwargs now win over the per-task resource defaults. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sandbox): tolerate modal SDKs without the Sandbox.create tags kwarg modal < 1.5 has no 'tags' create kwarg (this venv ships 1.4.1), so every sandbox create — including 'rllm snapshot create' — died with a TypeError. Feature-detect the kwarg; older SDKs attach the run-id tag post-create via set_tags(), best-effort. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(sft): full-parameter fireworks SFT (lora_rank=0) + loud max_length truncation warning - SFTSpec validates lora_rank >= 0; 0 = full-parameter fine-tuning. - Tinker SFT backend rejects lora_rank=0 explicitly (SDK is LoRA-only); fireworks flips supports_full_finetune and, at build_config time, rewrites a '-lora' training shape to its POLICY_TRAINER sibling when lora_rank=0. - Swapping --model to a different Fireworks base model now fails fast unless model.tokenizer_model + fireworks_config.policy_trainer_shape_id move with it (silent wrong-tokenizer/wrong-shape trap). - rllm sft --config <yaml>: deep-merge backend-native knobs; CLI flags win. - conversation_to_datum warns loudly when a row renders past data.max_length (the 2048 default silently truncated long trajectory rows mid-conversation). - Promotion logs/comments cover full-weight artifacts (INFERENCE_BASE -> HF_BASE_MODEL), not just LoRA. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018AEaxFWY5KutdmPPhn6dos * fix(sft): close critic findings on full-param support — guard via merged config, DictConfig overrides, tinker gate via overrides, --config precedence honesty - model-swap guard keys on post-merge cfg.model.name (catches overrides model.name swaps) and accepts DictConfig overrides. - tinker's full-finetune gate sees model.lora_rank set via overrides. - rllm sft --config help states the real precedence (file beats flags, --renderer/--gpus win); a --gpus Click default no longer clobbers a file-set trainer.n_gpus_per_node on verl. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018AEaxFWY5KutdmPPhn6dos * fix(cli): sft summary panel reports resolved n_gpus_per_node, not the --gpus default A --config-set trainer.n_gpus_per_node already governed the torchrun launch, but the pre-launch panel printed the Click --gpus value (default 1), under-reporting the GPU count. Read the resolved config instead. Flagged by verl cross-review of the --config feature. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018AEaxFWY5KutdmPPhn6dos * refactor(sft): strip fix-narration comments, consolidate accreted tests Hygiene back-fix over loop-001's full-param SFT work (91ee357, ef1ec7d, 00c152e). Behavior-preserving: rllm/ changes are comment/docstring only. - Drop fix-narration / restatement comments: cli/sft.py "Panel honesty" + precedence parenthetical; fireworks_backend.py supports_full_finetune justification, _align_shape critic-response graft, "# Normalize" line restatement, per-mode promotion prose (retention-window constraint kept); tinker_backend.py fireworks cross-narration; fireworks.yaml shape-pair backend-behavior restatement. - Consolidate accreted tests via parametrize, preserving case coverage 1:1: test_sft_dispatcher.py 15 fireworks build/validate functions -> test_fireworks_build_config_resolves_shape_and_tokenizer, test_fireworks_provision_doc_parses_sft, test_spec_lora_rank_validation, test_tinker_rejects_full_finetune; test_sft_command.py config trio -> test_sft_config_file_precedence + shared sft_no_train fixture. - De-tautologize the panel test: build_config stub no longer re-implements the overrides->config merge (that merge is covered by test_verl_build_config_maps_spec); it returns a fixed 8-GPU config so the panel-reads-resolved-cfg assertion keeps teeth. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Tianhao Wu <thw@gpu-dp-6wgmd-rj5sb.slurm-compute.slurm.svc.cluster.local> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: signalrush <signalrush@users.noreply.github.com>
Proposal to remove the O(N²) prompt-token duplication in multi-turn session traces by storing per-turn deltas + a parent pointer (a delta chain), reconstructing full prompts on read. Leverages the existing bridge_to_next_turn prefix-extension invariant; both training transforms already reduce full prompts back to deltas. Inspired by verifiers v1's message graph (quadratic → linear trace size). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Removes the O(N²) prompt-token duplication in multi-turn session traces.
Each cumulative-mode turn now stores only its prompt *delta* (new tokens
since the previous turn) plus a parent_trace_id, instead of the full
cumulative prompt; the full prompt is rebuilt on read by walking the
chain, so clients and the trainer see the unchanged TraceRecord shape.
- models: TraceRecord.{prompt_delta_token_ids, parent_trace_id} (defaulted,
backward compatible with existing full-prompt traces).
- token_accumulator: last_trace_id chain pointer; reset() breaks the chain.
- data_process: apply_chain() compresses prompt→delta; build_trace_record
gains trace_id/parent_len/parent_trace_id.
- proxy: _chain_link() computed synchronously in request order before each
ingest; wired into all 5 trace-capture sites (turn-0 chat/stream,
cumulative non-stream/stream, local stream).
- token_chain: reconstruct_prompt_ids() rebuilds full prompts by parent
pointer (order-independent; leaves unresolvable partial chains untouched).
- server: reconstruct on read in the session/trace endpoints.
- tests: test_token_chain.py (13 tests); full gateway suite green (256).
Phase 2 (carry deltas end-to-end into the training transforms) to follow.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jeffreysijuntan
changed the title
jeffreysijuntan
force-pushed
the
terminal-rl
branch
2 times, most recently
from
July 27, 2026 01:30
5b55970 to
332bb74
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Removes the O(N²) prompt-token duplication in multi-turn gateway session traces (each turn re-stores the full cumulative prompt). Inspired by verifiers v1's message graph — quadratic → linear trace size.
Contains the RFC (
design/gateway-dag-token-storage.md) and the Phase 1 implementation.Approach
In cumulative_token_mode the renderer bridge already guarantees
prompt_ids[k] == prompt_ids[k-1] + completion_ids[k-1] + delta[k]byte-for-byte. So each turn stores only its delta + aparent_trace_id; the full prompt is rebuilt on read by walking the chain. Clients and the trainer see the unchangedTraceRecordshape (transparent), so zero training-side changes.Phase 1 (this PR — implemented)
models:TraceRecord.{prompt_delta_token_ids, parent_trace_id}(defaulted; backward compatible with existing full-prompt traces).token_accumulator:last_trace_idchain pointer;reset()breaks the chain (segment boundary).data_process:apply_chain()compresses prompt→delta;build_trace_recordgainstrace_id/parent_len/parent_trace_id.proxy:_chain_link()computed synchronously in request order before each ingest (async store can then land in any order); wired into all 5 trace-capture sites.token_chain:reconstruct_prompt_ids()rebuilds full prompts by parent pointer (order-independent; leaves unresolvable partialsince/limitchains untouched rather than fabricating).server: reconstruct on read in the session/trace endpoints.Edge cases handled: turn-0 / post-reset / duplicate-replay → chain roots (stored full); segment breaks; non-cumulative + legacy traces unchanged.
Tests:
tests/unit/test_token_chain.py(13 new); full gateway unit suite green (256 passed). Includes a storage-is-linear assertion and byte-exact reconstruction round-trips.Phase 2 (follow-up)
Carry deltas end-to-end into the verl/tinker transforms (they already slice full prompts back into deltas), removing the on-the-wire O(N²) and the redundant prefix-slicing. Gated + parity-tested.
🤖 Generated with Claude Code