feat(executors): orphaned-position lifecycle for LP executors — flag, listing, DB-aware stop, resolve - #217
feat(executors): orphaned-position lifecycle for LP executors — flag, listing, DB-aware stop, resolve#217fengtality wants to merge 1 commit into
Conversation
… listing, DB-aware stop, resolve
API-side of the gateway#678 retry-ownership work (canonical design:
docs/retry-architecture.md in the companion gateway PR). Executors created
via the API run in-process with no controller, so the API owns the
"react to a stranded position" role:
- Stop on a terminal executor returns already_terminated with close_type,
position_address, orphaned_position, and hold_reason instead of the 404
dead end #678 hit (terminal executors are popped from memory within one
tick, so "not in memory" almost always means "already terminated").
404 is reserved for ids the DB has never seen.
- Completion flags stranded exposure in the persisted final state:
an involuntary hold (POSITION_HOLD with hold_reason set — an LP close
that exhausted its retries) or a legacy FAILED-with-position gets
orphaned_position: true and an error-level log. Voluntary holds never
match (a successful close clears position_address first).
- GET /executors/positions/orphaned lists recovery candidates
(SQL-filtered to lp_executor; involuntary holds, FAILED-with-position,
and SYSTEM_CLEANUP restarts flagged needs_onchain_reconciliation).
- POST /executors/{id}/resolve-orphan marks a candidate recovered after
the position is closed externally, silencing listings and warnings.
- bots/controllers lp_rebalancer mirror: halt + skip accounting for
executors that ended with a live position (re-creating one would mint a
second position on top of the stranded one).
Validated live on mainnet: forced close-failure cascade terminated as the
involuntary hold, surfaced in the orphan listing with hold_reason,
re-stop returned already_terminated, and resolve-orphan cleared it after
a direct gateway close recovered all funds + rent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HahKfEY9rvKnZijrzUAFSq
Greptile SummaryThe PR adds database-backed discovery and resolution of orphaned LP positions, richer behavior when stopping terminal executors, and controller-side protection against stacking exposure. The controller protection is not synchronized with persisted orphan state, so it fails in both directions across recovery and restart.
Confidence Score: 3/5The PR should not merge until the LP controller reconciles its halt with persisted orphan state across both resolution and process restart. The new in-memory marker permanently suppresses actions after an orphan is resolved, yet resets on restart while the orphan remains unresolved, allowing the same lifecycle mechanism to cause either indefinite inactivity or duplicate LP exposure. Files Needing Attention: bots/controllers/generic/lp_rebalancer/lp_rebalancer.py, services/executor_service.py
|
| Filename | Overview |
|---|---|
| bots/controllers/generic/lp_rebalancer/lp_rebalancer.py | Adds orphan-aware accounting and a controller halt, but the halt is neither cleared after resolution nor restored after restart. |
| services/executor_service.py | Adds DB-aware terminal stopping, orphan persistence/listing, and resolution; resolution currently has no integration with a running LP controller. |
| database/repositories/executor_repository.py | Adds a bounded, SQL-filtered close-type query suitable for retrieving LP orphan candidates. |
| routers/executors.py | Exposes orphan listing and resolution endpoints with service-level error handling. |
| models/executors.py | Extends stop responses and defines the orphan listing response schema. |
| models/init.py | Exports the newly added orphan response models. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[LP executor terminates with live position] --> B[Persist orphan record]
A --> C[Set controller memory marker]
C --> D[Controller returns no actions]
B --> E[Operator closes position externally]
E --> F[Resolve-orphan updates database]
F -. no controller reconciliation .-> D
B --> G[Process restart]
G --> H[Marker initialized to None]
H --> I[Controller may create another LP position]
Reviews (1): Last reviewed commit: "feat(executors): orphaned-position lifec..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| if self._orphaned_position_address: | ||
| self.logger().debug( | ||
| f"Halted: position {self._orphaned_position_address} from a FAILED executor is " | ||
| "still open on-chain and requires manual recovery" | ||
| ) | ||
| return actions |
There was a problem hiding this comment.
When an operator closes or adopts the stranded position and calls resolve-orphan, only the database record is updated; _orphaned_position_address remains set, so every subsequent action cycle returns no actions and the rebalancer stays halted until restart.
There was a problem hiding this comment.
This is the intended operator flow rather than a missed reconciliation: resolve_orphan writes to the hummingbot-api database, while the lp_rebalancer controller runs in a separate bot process with no channel to that record — restarting the controller is the acknowledgment step after recovery (and in the ExecutorService topology there is no controller at all; controller_id is only a label). Auto-clearing the in-memory halt from inside the controller would defeat the guard it exists to provide. The recovery runbook now states this explicitly: hummingbot/condor@d8012bd adds "restart the controller (or its bot) after resolving" to guides/lp_executor.md.
| # Set when a FAILED LP executor still reports a live on-chain position; the | ||
| # controller halts new position creation until it is recovered manually | ||
| self._orphaned_position_address: Optional[str] = None |
There was a problem hiding this comment.
When the process restarts with an unresolved orphan in the database, the controller initializes _orphaned_position_address to None and never restores it, allowing the rebalancer to open another LP position on top of the stranded position.
There was a problem hiding this comment.
Known and documented as an accepted residual — see docs/retry-architecture.md §9 (hummingbot/gateway#679): the halt is deliberately process-local because the controller runs in the bot process and has no persisted state to rebuild it from after a restart. The durable guard for the restart case is this PR's DB side: the orphan is persisted in the executor's final state (hold_reason / orphaned_position), listed by GET /executors/positions/orphaned, and surfaced to agents as warnings (hummingbot/condor#204), so it survives restarts even though the in-memory latch does not.
1 participant
Summary
API-side of the gateway#678 retry-ownership work. Canonical design:
docs/retry-architecture.md(in the gateway PR), esp. §5: executors created via the API run in-process with no controller, so the API owns the "react to a stranded position" role.already_terminatedwithclose_type,position_address,orphaned_position, andhold_reasoninstead of the 404 dead end #678 hit (terminal executors are popped from memory within one tick, so "not in memory" almost always means "already terminated"). 404 is reserved for ids the DB has never seen.POSITION_HOLDwithhold_reason— an LP close that exhausted its retries) or a legacyFAILED-with-position getsorphaned_position: truepersisted plus an error-level log. Voluntary holds never match (a successful close clearsposition_addressfirst).GET /executors/positions/orphaned: recovery candidates, SQL-filtered tolp_executor— involuntary holds,FAILED-with-position, andSYSTEM_CLEANUPrestarts (flaggedneeds_onchain_reconciliation).POST /executors/{id}/resolve-orphan: mark recovered after the position is closed externally; silences listings and agent warnings.bots/controllerslp_rebalancer mirror: halt + skip accounting for executors that ended with a live position.Companion PRs
docs/retry-architecture.md(in the gateway PR)Validation
Validated live on mainnet: a forced close-failure cascade terminated as the involuntary hold, surfaced in the orphan listing with
hold_reason, re-stop returnedalready_terminated, and resolve-orphan cleared it after a direct gateway close recovered all funds + rent.🤖 Generated with Claude Code
https://claude.ai/code/session_01HahKfEY9rvKnZijrzUAFSq