[coordinator] retry transient snowflake errors

[ayubun]

we should retry transient snowflake generation failures with backoff (i.e. ratelimited, 5xx), while failing permanent errors immediately (i.e. bad request)

Summary by CodeRabbit

• Bug Fixes
  • Improved ID generation reliability by automatically retrying transient service and network failures.
  • Added handling for rate-limit responses and server errors.
  • Prevented unnecessary retries for permanent client errors.
  • Ensured requests stop retrying after a bounded number of attempts.

[coderabbitai]

📝 Walkthrough

Walkthrough

Changes

Snowflake ID generation now validates HTTP statuses, classifies retryable failures, applies bounded exponential backoff, and includes tests for transient errors, transport disconnects, permanent client errors, and retry exhaustion.

Snowflake ID retry handling

Layer / File(s)	Summary
Retry policy and execution osprey_coordinator/src/snowflake_client.rs	Adds retry limits, exponential delays, retryable error classification, and a bounded retry loop around ID generation.
Response validation and retry tests osprey_coordinator/src/snowflake_client.rs	Validates HTTP responses before JSON parsing and adds async in-process HTTP tests for retry and termination behavior.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant SnowflakeClient
  participant generate_id_once
  participant SnowflakeEndpoint
  SnowflakeClient->>generate_id_once: Request ID
  generate_id_once->>SnowflakeEndpoint: Send HTTP request
  SnowflakeEndpoint-->>generate_id_once: Return response or disconnect
  generate_id_once-->>SnowflakeClient: Return ID or error
  SnowflakeClient->>SnowflakeClient: Classify error and apply backoff

Loading

Suggested reviewers: alphonsoc

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: retrying transient Snowflake errors in the coordinator.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch ayu/snowflake-clients

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@osprey_coordinator/src/snowflake_client.rs`:
- Around line 36-38: Update retry_delay to implement exponential backoff: for
successive attempts, produce 50/100/200 ms using RETRY_BASE_DELAY_MILLIS, rather
than multiplying by (attempt + 1) squared. Preserve the existing Duration return
type and attempt-based retry sequencing.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 3c8265e9-9f45-48e6-b17b-5a0cce96d45d

📥 Commits

Reviewing files that changed from the base of the PR and between 699ff64 and 42d6187.

📒 Files selected for processing (1)

• osprey_coordinator/src/snowflake_client.rs