Skip to content

[Feature] Retry / backoff / Retry-After honouring #56

Description

@P1llus

Summary

skopos does not model retries or backoff. Rate limits are survivable today via on_status: {429: empty_events}, but the drain simply yields nothing for that page rather than waiting and retrying — so heavily throttled APIs lose data or stall.

Varieties observed

Variety Vendors / notes
HTTP 429 + Retry-After X/Twitter API v2 (monthly cap), generic rate-limited REST
Secondary / abuse rate limits GitHub secondary limits
Transient 5xx general resilience for long drains

Implementation options

  • A. Document-level retry block. retry: {max_attempts, base_delay, max_delay, jitter, on: [429, 5xx], respect_retry_after: true}, applied around the request execution in client/http.go. Pro: one self-contained, isolated runtime addition; smallest blast radius of the buildable gaps. Con: interacts with pagination/progress — a retried page must not double-advance state.
  • B. Per-request override of the document default (some endpoints are stricter). Pro: precise. Con: more surface; only worth it if a real spec needs it — defer until then.
  • C. Extend on_status with a retry action (alongside skip/fail/empty_events/invalidate_cache). Pro: reuses existing dispatch. Con: backoff/Retry-After state doesn't fit the current per-status one-shot model cleanly.

Recommendation: A — a document-level retry block honouring Retry-After, exponential backoff with jitter, applied before pagination advances. Add B only when a spec demands it.

Acceptance criteria

  • retry block parses + validates; retries fire on the configured statuses with exponential backoff + jitter.
  • Retry-After (seconds and HTTP-date forms) is honoured when present.
  • A retried page does not double-advance pagination/progress state.
  • Golden under cmd/skopos/testdata/ where the test server returns 429 then 200 and the drain recovers.
  • docs/schema.md / docs/runtime.md updated; make schema-doc regenerated.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions