feat: add Copa patching and integrate Renovate - #4756
Conversation
Signed-off-by: Abhishek Sheth <absheth@microsoft.com>
Signed-off-by: Abhishek Sheth <absheth@microsoft.com>
Codecov Report✅ All modified and coverable lines are covered by tests.
Additional details and impacted files@@ Coverage Diff @@
## master #4756 +/- ##
==========================================
- Coverage 54.49% 47.86% -6.63%
==========================================
Files 134 290 +156
Lines 12329 22908 +10579
==========================================
+ Hits 6719 10966 +4247
- Misses 5116 10981 +5865
- Partials 494 961 +467
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Adds automated vulnerability remediation through Copa-patched images and Renovate dependency management.
Changes:
- Adds Copa scanning, patching, attestation, and publishing workflow.
- Replaces Dependabot with Renovate.
- Documents patched images and updates release guidance.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/copa-patch.yaml |
Adds multi-architecture image patching workflow. |
.github/workflows/renovate.yaml |
Runs self-hosted Renovate weekly. |
.github/renovate.json5 |
Configures dependency and security updates. |
.github/dependabot.yml |
Removes Dependabot configuration. |
.github/workflows/release-pr.yaml |
Updates release checklist and action pins. |
docs/RELEASE.md |
Replaces Dependabot release guidance. |
website/docs/security.md |
Documents vulnerability-patched images. |
Suppressed comments (2)
.github/workflows/release-pr.yaml:30
- This regresses
setup-gofrom v7.0.0 to v6.4.0 even though the repository already uses the v7 pin across its Go workflows (for example,.github/workflows/release.yaml:45). Keep the existing v7.0.0 SHA rather than introducing an unrelated action downgrade.
uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0
.github/workflows/copa-patch.yaml:119
- This second Copa job also pins
harden-runnerv2.19.4 instead of the repository's current v2.21.0 SHA used by existing workflows such as.github/workflows/release.yaml:35and.github/workflows/scan-vulns.yaml:35. Align this pin with the current version.
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
|
||
| # Vulnerability-Patched Images | ||
|
|
||
| In addition to the regular release images, Gatekeeper publishes automatically **patched** variants of recent releases that pick up fixed versions of Go dependencies and the Go standard library **without waiting for the next release**. They are produced on a weekly schedule using [Project Copacetic (Copa)](https://github.com/project-copacetic/copacetic) and published to the same repositories as the regular `gatekeeper` and `gator` images: |
| # Serialize every run on a ref (weekly schedule + any manual dispatch) so two runs can't race | ||
| # on the same vX.Y.Z-N revision number or -patched promotion. | ||
| concurrency: | ||
| group: copa-patch-${{ github.ref }} |
| staging="${STAGING_INPUT:-ghcr.io/${{ github.repository_owner }}/copa-staging}" | ||
| intermediate_repo="${staging}/${IMAGE}" |
| if [[ "${total}" -eq 0 ]]; then | ||
| echo "has_fixes=false" >> "$GITHUB_OUTPUT" | ||
| echo "Existing ${RELEASE}-patched already fixes all current CVEs — skipping (idempotent)." | tee -a "$GITHUB_STEP_SUMMARY" | ||
| exit 0 | ||
| fi |
| if [[ "${MODE}" != "dry-run" ]]; then | ||
| last=$(crane ls "${PRIMARY_REPO}" 2>/dev/null | grep -E "^${RELEASE}-[0-9]+$" | sed "s/^${RELEASE}-//" | sort -n | tail -1 || true) | ||
| rev=$(( ${last:-0} + 1 )) |
| - name: Re-wrap with attestations and publish to canonical | ||
| id: rewrap | ||
| if: steps.gate.outputs.has_fixes == 'true' && steps.cfg.outputs.mode == 'canonical' |
| vulnerabilityAlerts: { | ||
| enabled: true, | ||
| vulnerabilityFixStrategy: "lowest", // take the earliest fixed version | ||
| labels: ["security"], // `backport` is added per release branch below |
| steps: | ||
| - name: Harden Runner | ||
| uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0 | ||
| uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4 |
| issues: write | ||
| steps: | ||
| - name: Harden Runner | ||
| uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4 |
| releases: ${{ steps.resolve.outputs.matrix }} | ||
| steps: | ||
| - name: Harden Runner | ||
| uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4 |
Signed-off-by: Abhishek Sheth <absheth@microsoft.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.
Suppressed comments (6)
.github/workflows/release-pr.yaml:30
- This downgrades actions/setup-go from v7.0.0 to v6.4.0 even though the release workflow still needs the same Go setup behavior. Keep the existing v7.0.0 SHA instead of regressing an unrelated action dependency.
uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0
.github/workflows/copa-patch.yaml:426
- This gate compares only aggregate vulnerability counts. A dependency/toolchain upgrade can remove multiple findings while introducing a different fixable CVE, and the lower total would still be published. Compare the per-architecture vulnerability identities/packages before and after and reject any newly introduced finding in addition to requiring a lower count.
if [[ "${total}" -ge "${BEFORE}" ]]; then
echo "::error::patch did not reduce fixable CVEs (before=${BEFORE}, after=${total})"
exit 1
fi
.github/workflows/copa-patch.yaml:466
- The canonical publish path only creates and promotes GHCR tags; this workflow has no Docker Hub login or push. That contradicts the PR description and #4480, which promise both immutable and floating tags on Docker Hub and GHCR. Add the corresponding Docker Hub publication and verification path, or explicitly revise the stated scope before merging.
ghcr_ref="${SOURCE_REGISTRY}/${IMAGE}:${RELEASE}-${REV}"
website/docs/security.md:87
- This describes patched images as automatically produced by a recurring workflow, but the workflow's only schedule is commented out, so after merge no recurring run can publish these images. Either enable the schedule or document that patched images are not yet automatically available.
In addition to the regular release images, Gatekeeper publishes automatically **patched** variants of recent releases that pick up fixed versions of Go dependencies and the Go standard library **without waiting for the next release**. They are produced by a recurring [Project Copacetic (Copa)](https://github.com/project-copacetic/copacetic) workflow and published to **GHCR** (the regular release images are additionally published to Docker Hub):
.github/workflows/release-pr.yaml:25
- This changes the existing release workflow from actions/checkout v7.0.1 back to v7.0.0. The downgrade is unrelated to the Renovate migration and discards the newer patch release already used across the repository; retain the existing v7.0.1 SHA.
This issue also appears on line 30 of the same file.
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0
.github/workflows/copa-patch.yaml:74
- The manual input is accepted without enforcing the stable
vX.Y.Zformat assumed by the image tags and the revision regex below. A value such aslatestor a prerelease can therefore publish officiallatest-N/prerelease patched tags outside the documented scheme. ValidateRELEASE_TAGagainst the stable-tag regex before adding it to the matrix, as the release workflow already does for its version input.
if [[ -n "${RELEASE_TAG}" ]]; then
matrix=$(jq -cn --arg t "${RELEASE_TAG}" '[$t]')
JaydipGabani
left a comment
JaydipGabani
left a comment
There was a problem hiding this comment.
Thanks for working on this. Some of the feedback can be follow up prs, I will leave the judgement to you on what needs to be follow up vs fixed now.
|
|
||
| - name: Set up Go | ||
| uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 | ||
| uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0 |
There was a problem hiding this comment.
why modify this to older version?
| # Weekly canonical run — gated OFF for the initial merge. | ||
| # schedule: | ||
| # - cron: "17 8 * * 1" | ||
| workflow_dispatch: |
There was a problem hiding this comment.
The schedule is disabled here, but website/docs/security.md describes patched images as being published automatically by a recurring workflow. Could we either enable the schedule before merging or update the documentation to clearly state that Copa patching is currently manual/disabled? As written, users may expect patched tags to be produced automatically even though no scheduled run will occur.
There was a problem hiding this comment.
is there a way to not delete and run both for sometime? I am afraid of missing out on fixes from dependabot.
| run: | | ||
| set -euo pipefail | ||
| if [[ "${MODE}" != "dry-run" ]]; then | ||
| last=$(crane ls "${PRIMARY_REPO}" 2>/dev/null | grep -E "^${RELEASE}-[0-9]+$" | sed "s/^${RELEASE}-//" | sort -n | tail -1 || true) |
There was a problem hiding this comment.
This lookup fails open. Because the entire pipeline ends with || true, registry, authentication, and network errors are indistinguishable from a repository with no matching tags. A transient failure would therefore select revision 1.
Please preserve failures from crane ls and fail the job unless the registry query succeeds. Only the expected “no matching revision tags” result should produce an empty last value.
Why this matters: A registry outage could make the workflow choose an already-used revision.
| if [[ "${MODE}" != "dry-run" ]]; then | ||
| last=$(crane ls "${PRIMARY_REPO}" 2>/dev/null | grep -E "^${RELEASE}-[0-9]+$" | sed "s/^${RELEASE}-//" | sort -n | tail -1 || true) | ||
| rev=$(( ${last:-0} + 1 )) | ||
| if crane digest "${PRIMARY_REPO}:${RELEASE}-${rev}" >/dev/null 2>&1; then |
There was a problem hiding this comment.
A nonzero crane digest result does not necessarily mean this tag is absent; it may also indicate an authentication, network, rate-limit, or registry failure. If the registry recovers before the later push, this workflow could overwrite an existing tag that is documented as immutable.
Please distinguish an explicit not-found response from operational failures and proceed only for the former. All other errors should fail the workflow closed.
Why this matters: The existing check does not reliably enforce the immutable-tag guarantee.
| docker run --rm --pull always --platform "$p" "${PATCHED_REF}" --help >/dev/null 2>&1 | ||
| ec=$? | ||
| set -e | ||
| if [[ "${ec}" != "0" && "${ec}" != "2" ]]; then |
There was a problem hiding this comment.
This accepts exit code 2 while discarding all output, but 2 can also be produced by a Go panic or another abnormal failure. A crashing patched binary could therefore pass the smoke test.
The successful fork runs reportedly returned 0, so requiring 0 would be the safest check. If exit code 2 must remain supported, please capture stdout/stderr and accept it only when expected help or usage output is present, while rejecting panic output.
| echo "CVE totals across ${PATCH_ARCHES}: before=${BEFORE} after=${total} | stdlib before=${BEFORE_STDLIB} after=${stdlib_total}" | ||
| echo "| **all** | **${BEFORE} → ${total}** | **${BEFORE_STDLIB} → ${stdlib_total}** |" >> "$GITHUB_STEP_SUMMARY" | ||
| # Surface fixable library CVEs Copa couldn't reach. | ||
| library_total=$(( total - stdlib_total )) |
There was a problem hiding this comment.
Could we derive and validate residual findings directly rather than calculating library residuals as total - stdlib_total? If Trivy output changes, contains duplicate findings, or categorizes a result unexpectedly, this arithmetic could produce a misleading summary while the publication decision still proceeds.
It would be safer to normalize the report once and derive totals, subsets, severities, and residual package details from that same finding set.
| n=$(docker buildx imagetools inspect "${GHCR_REF}" --raw \ | ||
| | jq '[.manifests[] | select(.annotations["vnd.docker.reference.type"] == "attestation-manifest")] | length') | ||
| echo "Attestation manifests on ${GHCR_REF}: ${n}" | ||
| if [[ "${n}" -lt 1 ]]; then echo "::error::no attestations on ${GHCR_REF}"; exit 1; fi |
There was a problem hiding this comment.
This check succeeds when the image index contains only one attestation manifest, even though three platforms are being published. Could we verify that every expected image manifest has the required SBOM and provenance attestations, rather than checking only that at least one attestation exists?
Ideally, map each platform manifest digest to its attestation manifests and fail if any published platform is missing either expected attestation type.
|
|
||
| # Vulnerability-Patched Images | ||
|
|
||
| In addition to the regular release images, Gatekeeper publishes automatically **patched** variants of recent releases that pick up fixed versions of Go dependencies and the Go standard library **without waiting for the next release**. They are produced by a recurring [Project Copacetic (Copa)](https://github.com/project-copacetic/copacetic) workflow and published to **GHCR** (the regular release images are additionally published to Docker Hub): |
There was a problem hiding this comment.
The workflow’s weekly schedule is commented out, so patched images will not currently be published automatically. Manual runs also default to a no-write dry run.
Could we either enable the schedule before merging or update this section to clearly describe the feature as manual/disabled during its initial rollout? As written, users may expect recurring -patched images that are not actually being produced.
| | `vX.Y.Z-R` | An **immutable** patch revision (`R` = `1`, `2`, …). Each run that fixes newly-available CVEs publishes the next number. | | ||
| | `vX.Y.Z-patched` | A **floating** tag that always points at the newest `vX.Y.Z-R`. Use this to track the latest patched build of that release. | | ||
|
|
||
| All published architectures `linux/amd64`, `linux/arm64`, and `linux/arm/v7` are rebuilt and patched. |
There was a problem hiding this comment.
Could we soften or qualify the statement that all architectures “are rebuilt and patched”? The workflow allows Copa to ignore individual patching errors, and an architecture may pass through unchanged or retain fixable vulnerabilities.
Something like “all supported architectures are included in the patched image and are scanned for applicable fixes” would better match the workflow’s best-effort behavior described below.
What this PR does / why we need it:
Adds recurring Copa patching and integrates renovate for vulnerability remediation in published images and the source dependencies.
Which issue(s) this PR fixes (optional, using
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when the PR gets merged):Fixes #4480
Special notes for your reviewer:
Closes #4480.
Copa image patching -
.github/workflows/copa-patch.yamlA scheduled workflow that scans released
gatekeeperandgatorimages (latest stable + previous minor) for fixable Go library/stdlib CVEs, patches them with Copacetic, re-scans to confirm the fix, and republishes them as immutablevX.Y.Z-R+ floatingvX.Y.Z-patchedmulti-arch tags on GHCR carrying the same SBOM + SLSA provenance attestations the release images already ship. OriginalvX.Y.Ztags are never modified.scheduleis commented out, so nothing runs on merge; manual runs default to a no-write dry-run.website/docs/security.md.Renovate -
.github/renovate.json5,.github/workflows/renovate.yamlReplaces
.github/dependabot.ymlwith Renovate, preserving today's behavior and adding release branch security coverage:release-3.xbranches (vulnerabilityAlerts+ OSV, resolved to the lowest fixed version).go mod tidyafter gomod changes, and a Dependency Dashboard for visibility.Renovate is driven via the Dependency Dashboard + PR checkboxes (rebase/retry) rather than
@dependabotcomments.Testing
amd64/arm64/arm/v7for both images and published the-R/-patchedtags.