feat(orchestrator): Surface waiting background work

[mwolson]

Summary

• Project finite provider background work after the root turn settles.
• Keep provider event ingestion alive until the thread-scoped background roster
  clears.
• Present the normalized orchestrator-v2 roster as an idle runtime so the
  Waiting UI from feat: "waiting" status for threads with open background tasks (i.e. pr monitors) #4415 can consume it without a second sidebar implementation.

Relationship to #4415

#4415 owns the frontend
Waiting direction on main: an idle provider session renders as Waiting in the
web and mobile thread lists. This PR supplies the orchestrator-v2 counterpart.
It projects the provider-thread background roster, keeps late roster clears
observable, and presents a nonempty post-settlement roster as runtime idle.

This PR intentionally does not carry its former sidebar or mobile status
precedence changes. The GitHub base remains t3code/codex-turn-mapping so the
diff contains only the orchestrator-v2 delta. Using #4415's branch as the
literal GitHub base would include the entire CTM stack in this PR. The intended
integration order is #4415 into main, CTM absorbs main, then this focused
delta completes Waiting for orchestrator v2.

Problem and Fix

Problem and Why it Happened	Fix
A completed root turn can still own finite background work, but orchestrator v2 had no normalized post-settlement signal.	Derive one pending list from provider-thread rosters and nonterminal command, dynamic-tool, and subagent items.
Closing provider event ingestion as soon as the root settled could miss the final roster clear.	Keep ingestion open while that provider thread reports finite background work, then persist the late empty roster.
A session-wide pending probe could couple unrelated provider threads on a shared runtime.	Use a thread-scoped probe for root-run ingestion while retaining the session-wide probe for provider idle release.
CTM runtime status can remain transiently running or checkpoint waiting after the authoritative post-settlement roster arrives.	Normalize a nonempty roster to runtime idle in client presentation while preserving the original status in latestRun.
Existing stored provider threads and shells omit the new roster.	Decode pendingBackgroundTasks with an empty-array default.

Defensive Fixes

Problem and Why it Happened	Fix
Claude roster snapshots and completion edges can arrive in either order.	Track the authoritative roster separately from wake eligibility and replay classification.
Replacing a Claude query kills the old process before it can clear its roster.	Clear process-scoped state while preserving only explicitly classified buffered Bash notifications.
A Grok subagent can outlive the root turn through adapter carryover.	Include live carryover subagents in the pending-work probe.
Persisted background state can survive a restart without a provider capable of completing it.	Clear stale rosters and terminalize stale background-capable items during recovery.

Validation

• Focused tests pass for Claude and ACP adapters, run execution, provider
  recovery, contracts, shared pending-work derivation, client presentation, and
  the web timeline.
• Affected package typechecks pass for contracts, shared, client-runtime,
  server, and web.
• Targeted vp check passes for every changed file, and git diff --check is
  clean.
• The final branch is exactly one commit on live
  t3code/codex-turn-mapping.
• The client presentation regressions cover stale running and checkpoint
  waiting shell states without restoring direct sidebar roster precedence.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: a7c4ec12-315e-444c-a41c-c3a18653b967

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[macroscopeapp]

🟡 Medium

t3code/apps/server/src/orchestration-v2/ProviderRuntimeRecoveryService.ts

Line 304 in e46af0a

const recoveredNonterminalRunIds = new Set(runs.map((run) => run.id));

isBackgroundCapableTurnItemType causes replayable waiting runs to have their background-capable turn items (command_execution, dynamic_tool, subagent) cancelled, even though those runs were intentionally skipped from terminalization because a pending/running checkpoint effect still exists. When a waiting run has a replayable checkpoint, reconcileProjection omits it from runs, so its ID is absent from recoveredNonterminalRunIds. The later stale-item loop then matches that run's items via isBackgroundCapableTurnItemType + isNonterminalTurnItemStatus and emits turn-item.updated cancellation events, destroying items that belong to a still-recoverable run and checkpoint. The stale-item loop needs to also exclude turn items whose runId belongs to a replayable waiting run, not just those already in recoveredNonterminalRunIds.

🤖 Copy this AI Prompt to have your agent fix this:

In file @apps/server/src/orchestration-v2/ProviderRuntimeRecoveryService.ts around line 304:

`isBackgroundCapableTurnItemType` causes replayable `waiting` runs to have their background-capable turn items (command_execution, dynamic_tool, subagent) cancelled, even though those runs were intentionally skipped from terminalization because a pending/running checkpoint effect still exists. When a `waiting` run has a replayable checkpoint, `reconcileProjection` omits it from `runs`, so its ID is absent from `recoveredNonterminalRunIds`. The later stale-item loop then matches that run's items via `isBackgroundCapableTurnItemType` + `isNonterminalTurnItemStatus` and emits `turn-item.updated` cancellation events, destroying items that belong to a still-recoverable run and checkpoint. The stale-item loop needs to also exclude turn items whose `runId` belongs to a replayable `waiting` run, not just those already in `recoveredNonterminalRunIds`.

[macroscopeapp]

🟡 Medium orchestration-v2/RunExecutionService.ts:738

After the root terminal is observed, writeIfRunCurrent is stripped from all provider_thread.updated events for the root thread — including events from a superseded attempt whose subscription stays open for background work. When that stale subscription receives a late provider_thread.updated and persists the old providerThread snapshot, it overwrites the new attempt's thread state (for example reverting status or roster cleared by the new attempt). The gate should use a settled-run-safe conditional write or field-specific merge rather than bypassing attempt ownership entirely based on rootTerminalAlreadySeen.

🤖 Copy this AI Prompt to have your agent fix this:

In file @apps/server/src/orchestration-v2/RunExecutionService.ts around line 738:

After the root terminal is observed, `writeIfRunCurrent` is stripped from all `provider_thread.updated` events for the root thread — including events from a superseded attempt whose subscription stays open for background work. When that stale subscription receives a late `provider_thread.updated` and persists the old `providerThread` snapshot, it overwrites the new attempt's thread state (for example reverting status or roster cleared by the new attempt). The gate should use a settled-run-safe conditional write or field-specific merge rather than bypassing attempt ownership entirely based on `rootTerminalAlreadySeen`.

[macroscopeapp]

🟡 Medium orchestration-v2/ProviderRuntimeRecoveryService.ts:319

A stale subagent turn item on an already-settled run gets only a turn-item.updated event, leaving the matching projection.subagents entry in pending/running/waiting indefinitely while its turn item becomes cancelled. The existing nonterminal-run cancellation loop above emits both subagent.updated and turn-item.updated, so the new settled-run path produces contradictory state — downstream code reading the subagent entity will treat a dead subagent as active. The settled-run loop should also emit a subagent.updated event (status cancelled) for matching subagent entries.

🤖 Copy this AI Prompt to have your agent fix this:

In file @apps/server/src/orchestration-v2/ProviderRuntimeRecoveryService.ts around line 319:

A stale `subagent` turn item on an already-settled run gets only a `turn-item.updated` event, leaving the matching `projection.subagents` entry in `pending`/`running`/`waiting` indefinitely while its turn item becomes `cancelled`. The existing nonterminal-run cancellation loop above emits both `subagent.updated` and `turn-item.updated`, so the new settled-run path produces contradictory state — downstream code reading the subagent entity will treat a dead subagent as active. The settled-run loop should also emit a `subagent.updated` event (status `cancelled`) for matching subagent entries.