Summary
skopos does not model retries or backoff. Rate limits are survivable today via on_status: {429: empty_events}, but the drain simply yields nothing for that page rather than waiting and retrying — so heavily throttled APIs lose data or stall.
Varieties observed
| Variety |
Vendors / notes |
HTTP 429 + Retry-After |
X/Twitter API v2 (monthly cap), generic rate-limited REST |
| Secondary / abuse rate limits |
GitHub secondary limits |
| Transient 5xx |
general resilience for long drains |
Implementation options
- A. Document-level
retry block. retry: {max_attempts, base_delay, max_delay, jitter, on: [429, 5xx], respect_retry_after: true}, applied around the request execution in client/http.go. Pro: one self-contained, isolated runtime addition; smallest blast radius of the buildable gaps. Con: interacts with pagination/progress — a retried page must not double-advance state.
- B. Per-request override of the document default (some endpoints are stricter). Pro: precise. Con: more surface; only worth it if a real spec needs it — defer until then.
- C. Extend
on_status with a retry action (alongside skip/fail/empty_events/invalidate_cache). Pro: reuses existing dispatch. Con: backoff/Retry-After state doesn't fit the current per-status one-shot model cleanly.
Recommendation: A — a document-level retry block honouring Retry-After, exponential backoff with jitter, applied before pagination advances. Add B only when a spec demands it.
Acceptance criteria
Summary
skopos does not model retries or backoff. Rate limits are survivable today via
on_status: {429: empty_events}, but the drain simply yields nothing for that page rather than waiting and retrying — so heavily throttled APIs lose data or stall.Varieties observed
Retry-AfterImplementation options
retryblock.retry: {max_attempts, base_delay, max_delay, jitter, on: [429, 5xx], respect_retry_after: true}, applied around the request execution inclient/http.go. Pro: one self-contained, isolated runtime addition; smallest blast radius of the buildable gaps. Con: interacts with pagination/progress — a retried page must not double-advance state.on_statuswith aretryaction (alongsideskip/fail/empty_events/invalidate_cache). Pro: reuses existing dispatch. Con: backoff/Retry-Afterstate doesn't fit the current per-status one-shot model cleanly.Recommendation: A — a document-level
retryblock honouringRetry-After, exponential backoff with jitter, applied before pagination advances. Add B only when a spec demands it.Acceptance criteria
retryblock parses + validates; retries fire on the configured statuses with exponential backoff + jitter.Retry-After(seconds and HTTP-date forms) is honoured when present.cmd/skopos/testdata/where the test server returns 429 then 200 and the drain recovers.docs/schema.md/docs/runtime.mdupdated;make schema-docregenerated.