✨ Add --skip-errors option to continue scans on check/probe failures#5117
Open
jimjag wants to merge 1 commit into
Open
✨ Add --skip-errors option to continue scans on check/probe failures#5117jimjag wants to merge 1 commit into
jimjag wants to merge 1 commit into
Conversation
Previously, if a single check or probe failed with a runtime error, the entire scan aborted and no results were reported. This is especially disruptive when scanning many repositories with --repos or --org, where one transient API failure fails the whole run. Add a --skip-errors option so that when one or more checks or probes fail at runtime, Scorecard continues running the remaining checks/probes and reports results for everything that succeeded. Failures are logged to stderr and the command exits with a zero status. Changes: - Add SkipErrors option and --skip-errors CLI flag. - Thread skipErrors through scorecard.Run into the probe and raw-data collection paths so a failure skips only the affected item instead of aborting the run. - Do not return the "one or more checks failed" error (non-zero exit) when --skip-errors is set. - Add unit tests for the flag and the skip-vs-abort behavior, and document the option in the README. Signed-off-by: Jim Jagielski <jimjag@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What kind of change does this PR introduce?
Feature.
What is the current behavior?
If a single check or probe fails with a runtime error (for example a transient GitHub API failure such as a
404onListWorkflowRunsByFileName), the entire scan aborts and no results are reported. This is especially disruptive when scanning many repositories with--reposor--org, where one transient failure fails the whole run.Example:
What is the new behavior (if this is a feature change)?
Adds a
--skip-errorsoption. When set, if one or more checks or probes fail at runtime, Scorecard continues running the remaining checks/probes and reports results for everything that succeeded. Failures are logged to stderr and the command exits with a zero status.Adds the
SkipErrorsoption and--skip-errorsCLI flag.Threads
skipErrorsthroughscorecard.Runinto the probe and raw-data collection paths, so a failure skips only the affected item instead of aborting the run.Suppresses the "one or more checks failed" error (non-zero exit) when
--skip-errorsis set.Tests for the changes have been added (for bug fixes/features)
Which issue(s) this PR fixes
NONE
Special notes for your reviewer
The public
scorecard.RunAPI is unchanged; a newWithSkipErrorsoption was added and defaults tofalse, so existing callers (serve, cron, attestor, scdiff) keep their current behavior. The default CLI behavior is also unchanged unless--skip-errorsis passed.Does this PR introduce a user-facing change?
Yes — a new optional
--skip-errorsCLI flag. Default behavior is unchanged.