Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Backoff and Retry Mechanism for Image Pull #1409

Merged
merged 7 commits into from
Feb 19, 2025

Conversation

honeyankit
Copy link
Contributor

@honeyankit honeyankit commented Feb 19, 2025

This pull request introduces a retry mechanism for fetching Docker images, which includes changes to the ImageService and corresponding test files. The main goal is to improve the robustness of image fetching by implementing retries with exponential backoff when Dependabot hit the rate limit (429 error) while downloading docker images from GitHub package registry.

This PR is created to address the recent spikes in the rate limit issues.

Enhancements to image fetching:

  • src/image-service.ts: Added constants MAX_RETRIES and INITIAL_DELAY_MS, and a sleep function to facilitate retry logic. Modified fetchImage to fetchImageWithRetry, incorporating retry attempts with exponential backoff and handling of specific error messages such as "429 Too Many Requests". [1] [2]

Updates to test files to use the new retry mechanism:

@honeyankit honeyankit self-assigned this Feb 19, 2025

if (
error instanceof Error &&
(error.message.includes('429') ||
Copy link
Contributor Author

@honeyankit honeyankit Feb 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Docker API typically returns a 429 Too Many Requests error for abuse rate limits. Just to be sure, I added a condition to check for either 429 or Too Many Requests. This way, even if one of these terms is missing, the error can still be detected, allowing for a retry.

@honeyankit honeyankit marked this pull request as ready for review February 19, 2025 03:21
@honeyankit honeyankit requested a review from a team as a code owner February 19, 2025 03:21
@honeyankit
Copy link
Contributor Author

honeyankit commented Feb 19, 2025

Steps to make changes to this PR in Codespace:

  1. Make changes to the PR
  2. Execute the below command in the Codespace terminal after any change to the PR

2.1

npm run lint-check
npm run format-check -- --write
npm run test

2.2

nvm install;nvm use;npm clean-install;npm ci;npm run package

Note: If you do execute step 2.2 and commit the code then CI will fail with the below error:

Run script/check-diff
Detected uncommitted changes after build:
diff --git a/dist/main/index.js b/dist/main/index.js
index c09ccea..8f50b37 1006[4](https://github.com/github/dependabot-action/actions/runs/7720200685/job/21044694134?pr=1156#step:7:5)4
Binary files a/dist/main/index.js and b/dist/main/index.js differ
diff --git a/dist/main/index.js.map b/dist/main/index.js.map
index cc44481..e[5](https://github.com/github/dependabot-action/actions/runs/7720200685/job/21044694134?pr=1156#step:7:6)0840f 100[6](https://github.com/github/dependabot-action/actions/runs/7720200685/job/21044694134?pr=1156#step:7:7)44
Binary files a/dist/main/index.js.map and b/dist/main/index.js.map differ
  1. Commit and push the code changes
  2. Once PR is approved simply merge the PR. This will make the PR deploy in the production

@@ -2,6 +2,12 @@ import * as core from '@actions/core'
import Docker from 'dockerode'
import {Readable} from 'stream'

const MAX_RETRIES = 5 // Maximum number of retries
const INITIAL_DELAY_MS = 2000 // Initial delay in milliseconds for backoff
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@abdulapopoola I have added the delay of 2 seconds

@markhallen markhallen merged commit c26fdc1 into main Feb 19, 2025
9 checks passed
@markhallen markhallen deleted the honeyankit/add-retry-for-image-update branch February 19, 2025 10:24
@Nishnha
Copy link
Member

Nishnha commented Feb 20, 2025

@honeyankit and I considered adding retries to

try {
const image = await docker.getImage(imageName).inspect()
if (!force) {
core.info(`Resolved ${imageName} to existing ${image.RepoDigests}`)
return
} // else fallthrough to pull
} catch (e: unknown) {
. We originally thought this code checks whether the image exists on the remote server, but it actually checks for the existence of the image on the local server, so we do not need to add retries.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants