feat(terminus2): observability - record observed decision turns

[giuliolovisotto]

Terminus2 now records canonical turns at Harbor's actual main-agent decision boundary, before terminal commands execute. Each turn retains the selected response's exact input, answer, reasoning, observed response time, and model-server/response-ID reference. A decision can issue several commands; retries and summarization requests do not become additional task turns. Completed decisions survive parser errors, tool failures, and cancellation.

The producer supplies execution identity so collection can attach captured calls to their invocation, and records known summarization boundaries separately. Rejected responses, synthetic fallback messages, and missing or reused response IDs produce explicit gaps instead of guessed decision evidence. Step counts represent completed nonempty terminal-command batch attempts; resolution remains unset.

Seven existing turn-content and owned-call checks can evaluate this evidence. Full turn-call accounting remains unobserved because the underlying HTTP transport can retry without notifying the adapter. A narrow turn_model_call_scope_incomplete health gate prevents false token-reconciliation findings and false policy-call completeness without suppressing those seven checks. This does not establish full rollout health or task success.

Validation:

• 315 tests passed across the Terminus agent, observability, collection, and health suites. Tests run Harbor's real loop, parser, retry/fallback and summarization logic, then exercise run serialization, capture merge, saved projection, and health checks.
• Coverage includes multiple commands per decision, HTTP 500/429 and timeout retries, all-fail, full/short compaction, synthetic fallback, parser/tool failures, cancellation, missing/reused IDs, concurrent executions, and disabled capture.
• pre-commit run --all-files passed; Fern npm run check passed (0 errors, 1 warning).
• A final-source TB2.1 smoke with a real managed model, sandbox and verifier produced two accepted decision turns, two exactly owned captured calls, and a completed invocation within an 8-turn/360-second bound. Both saved turns exactly matched their referenced request inputs and response answers; step counts were 0 and 1, and resolution remained unset. The model credential stayed local behind a temporary loopback SSH proxy.
• The verifier ran successfully and returned reward 0 because the generated regex failed the task assertions. Saved health reported no findings and retained the explicit token-accounting limitation; this is observability validation, not a successful benchmark result. An earlier bounded smoke also preserved a rejected response and cancelled compaction without inventing a turn.

Additional crafted smokes on the same source used the real Terminus/Harbor loop, HTTP model adapter, OpenSandbox, verifier, saved capture projection, and health checks. The two proactive-compaction cases used a live managed model; the six fault/command cases used explicitly scripted model responses. Independent checks compared saved turn inputs and answers with exact captured calls and checked actual terminal feedback.

Case	Observed evidence
Proactive compaction and continuation	Two valid decisions, five owned calls, one completed compaction with three helper calls, and a subsequent decision using the handoff context. Step counts 0, 1.
Cancellation during compaction	One valid command decision retained, compaction aborted, invocation incomplete, no fabricated second turn. Deliberate response delay triggered the deadline.
HTTP retry recovery	Injected 429 then 500 after server-side capture: four owned calls, two turns; withheld responses were not selected as decisions.
Malformed answer recovery	Three turns, steps 0, 0, 1; malformed answer retained, parser feedback observed, then real command execution.
Rejected completion and reactive compaction	Seven owned calls, three turns and three compaction helper calls. The truncated response remained captured but was excluded from turns.
Multiple commands	Two commands in one decision; the batch counter advanced once.
Terminal failure	Real missing command returned 127; the next decision received the error.
Shell timeout	Real shell timeout returned 124; the next decision received it. This does not exercise Harbor's internal caught TimeoutError.

Collection limitations found by the cancellation smoke remain open:

• The delayed model call completed about 49 seconds after the initial saved snapshot, so that snapshot contained only the first call. Ordinary collection does not automatically refresh late captures.
• A straightforward after-drain merge/projection added the second call but retained the existing canonical invocation's one-call ownership list. Health emitted no specific finding (the overall verdict remained unobserved). Explicitly rebuilding the derived invocation from updated observation evidence restored ownership; this was a separately labeled manual reconstruction, not normal collection behavior.

The corrected smokes validate observability, not benchmark task success; verifier rewards were 0. Initial empty-history fixture attempts were retained as diagnostics and excluded from the successful compaction claims. No production code changed during this additional verification.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-glovisotto-terminus-observability.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

• nemo/gym/evaluation/rollout-health
• nemo/gym/reference/trajectory-capabilities

[giuliolovisotto]

/claude review

[claude]

SHIP — no reliability concerns.

Reviewed the scoring path first. The rollout_health.py change is the only edit that touches health computation, and it is conservative and backward-compatible:

• turn_call_scope_complete = trajectory_observed and not _trajectory_has_gap(trajectory, "turn_model_call_scope_incomplete"). When the gap is absent (every existing producer today), it is True, so not turn_call_scope_complete is False and neither the rollout_token_count_mismatch gate nor policy_calls_observed change behavior. No risk of retroactively suppressing findings for current producers.
• When a producer does emit the gap (the new Terminus2 adapter), the effect is to mark rollout_token_count_mismatch unobserved and drop policy_calls_observed (which in turn makes task_no_successful_model_calls unobserved). That is the correct direction: a producer that cannot establish full turn→call scope should not have token reconciliation or successful-call claims computed against it, rather than emitting a potentially-wrong finding. The new parametrized test test_turn_call_scope_gap_only_gates_completeness_checks pins exactly this (completeness checks gate off, agent_turn_hollow/model_call_failed still evaluate).

The Terminus2 observability adapter is honest about its own limits — it self-reports transport_retry_visibility_unavailable and turn_model_call_scope_incomplete so downstream health refuses the reconciliation checks it can't back. Turn/decision binding uses object-identity matching (harbor_response is response) and refuses reused/missing response IDs, so no guessed associations reach the trajectory. Compaction outcome tracking correctly resets the single-slot state in finally, and the proactive-summarization flag is now reset in a finally (fixes a real leak on cancellation, covered by test_proactive_summarization_flag_resets_on_cancellation). Tests exercise the real Harbor run loop end-to-end through the saved-projection health check across 16 failure modes — this meets the "real rollout behavior, not just green units" bar.

No new HTTP paths; model calls still route through the Gym client with an added x-session-id header. Observations are gated behind observability_enabled via rollout_id_from_run() → _capture_correlation_enabled(), verified by test_disabled_observability_preserves_output_and_headers.

Two NOTES (author's call, non-blocking):

• NeMoGymLLM.call() deepcopys the full request input on every model call when observations are enabled. Cost is O(prompt size × calls) and only paid under observability, but for long trajectories at scale it's non-trivial memory/CPU. Acceptable as the price of an exact pre-mutation snapshot.
• _execute's except asyncio.CancelledError (app.py:494) catches and does not re-raise. This swallows external cancellation and lets the coroutine finish normally. It preserves the prior bare-except: behavior (not a regression), but if cooperative cancellation/shutdown matters here, consider re-raising after recording the invocation status.