fix: add retry logic for GitHub API calls in scale-down Lambda

[shivdesh]

Summary

Add exponential backoff retry for transient GitHub API failures (5xx, 429) when de-registering runners during scale-down operations.

Problem

When the scale-down Lambda attempts to de-register a runner from GitHub, transient API failures (e.g., 502 Server Error) cause the operation to fail. The current code catches the error, logs it, but still terminates the EC2 instance. This leaves stale/offline runner entries in the GitHub org settings.

Solution

• Add withRetry() helper with configurable max retries (3) and exponential backoff delays (1s, 2s, 4s)
• Wrap deleteSelfHostedRunnerFromOrg/Repo calls with retry logic
• Only terminate EC2 instance if GitHub de-registration succeeds
• If de-registration fails after all retries, leave the instance running so the next scale-down cycle can retry

Changes

• lambdas/functions/control-plane/src/scale-runners/scale-down.ts:
  • Added RETRY_CONFIG, sleep(), isRetryableError(), and withRetry() helper functions
  • Added deleteGitHubRunner() wrapper that uses retry logic
  • Modified removeRunner() to only terminate EC2 if all GitHub de-registrations succeed

Testing

• Tested locally with simulated 502 errors
• Retry logic handles 5xx errors and 429 rate limiting

[sekhar-isovalent]

Could you upstream this change ?

[sekhar-isovalent]

Lets try and research open issues upstream and fix that if found and we can sync the change to our module.

[brilong]

FYI, in the spirit of upstreaming changes, my upstream PR for nested virtualization is almost approved. Working on one more tweak.

[brilong]

Also, upstream recently released v7.4.1. We might want to compare and integrate their changes into our fork until we can get our upstream tweaks included.

[shivdesh]

upstream change: github-aws-runners#5061
@sekhar-isovalent @brilong