Skip to content

Prevent durable PR waits from exhausting execution operation limits #334

Description

@JWThewes

Problem

PR-per-unit executions currently poll pull request state with durable timers. A long review wait can therefore consume the durable execution's operation budget even though no application work is happening.

The observed execution reached Lambda's 3,000-operation limit and failed with ExecutionLimitExceeded, while its process row remained RUNNING. During that wait:

  • the provider had already merged the reviewed comparison-feature head, but local state still showed the unit and PR as ready;
  • the completed views construction stages and draft PR were still valid;
  • the watchdog would not notice the terminal durable execution until the configured local expiry;
  • the existing repair path would reset completed stages and replay successful construction.

Expected behavior

  • A PR review wait persists one callback and does not grow durable operations while provider state is unchanged.
  • Provider reconciliation wakes that callback only for merge, closure, reviewed head/target movement, or queued authenticated feedback.
  • Concurrent scheduler and feedback wakeups deliver a callback at most once.
  • Terminal durable executions are detected while their local process rows are still active.
  • Repair reconciles remote PR state, validates merged reviewed-head ancestry, preserves completed stages and artifacts, and resumes an existing draft PR through the current orchestrator alias.
  • GitHub, GitLab, multi-repository, and stale-run ownership cases behave consistently.

Proposed implementation

  • Add a sparse DynamoDB maintenance index for active executions and parked unit PR waits.
  • Replace durable timer polling with persisted callback ownership and an EventBridge-driven provider reconciler.
  • Extend the durable watchdog to inspect every indexed active execution.
  • Make lane repair provider-aware and state-preserving.
  • Add focused regression coverage for long waits, transitions, feedback wakeups, duplicate delivery, early durable failure, and recovery.

Dependency

This work uses the project-level source-control service and authentication model introduced by #333.

Acceptance criteria

  • Long unchanged PR waits create no repeated durable operations.
  • A merged reviewed head is marked merged only after ancestry validation.
  • Completed construction stages retain their state, attempts, and artifacts during repair.
  • An existing draft PR resumes without creating a replacement or rerunning construction.
  • The repaired execution advances after that PR merges.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions