Add recurring Copa patching workflow to produce vulnerability-patched container images #4480

Summary

Project Copacetic (Copa) has merged Go binary patching support (copacetic#1388), shipping in Copa v0.14.0. This is directly relevant to Gatekeeper — our production images (gatekeeper, gator) are built on gcr.io/distroless/static-debian12 which contains no OS packages to patch. The only vulnerability surface is the compiled Go binaries and their dependencies/stdlib.

This issue proposes adding a recurring GitHub Actions workflow that uses Copa to produce patched variants of released images, addressing Go dependency and stdlib CVEs without requiring a full Gatekeeper release.

Motivation

• Gatekeeper images use distroless — traditional OS-level patching tools are ineffective. Copa's new Go patching capability is uniquely suited to this use case.
• We already run Trivy + govulncheck on every push (scan-vulns.yaml), but scanning only detects vulnerabilities — it doesn't remediate them between releases.
• Users in regulated environments or with strict CVE remediation SLAs often can't wait for the next Gatekeeper release to pick up Go dependency or stdlib fixes.
• Copa patches images without a full rebuild, producing a thin patch layer on top of the existing release image and preserving layer integrity.
• Copa's Go patching can fix both Go module/dependency CVEs and stdlib CVEs (by rebuilding with a newer patch-level Go compiler, e.g., go1.26.0 → go1.26.3).

Prerequisites

• Copa v0.14.0 must be released — Go patching (copacetic#1388) is merged to main but not yet in a release. This workflow should be implemented once v0.14.0 ships.
• VCS commit metadata in binaries — Copa's binary rebuild path requires Go binaries to have embedded VCS commit hashes (go version -m). This is the default for Go 1.18+ when .git is present in the build context. We should verify our Dockerfile doesn't strip this (it shouldn't — we use standard go build).

Proposed Design

Workflow: .github/workflows/copa-patch.yaml

Trigger: Weekly schedule (e.g., Mondays) + workflow_dispatch for manual runs.

Scope: Patch the latest stable + previous minor release images (e.g., v3.22.x and v3.21.x) for:

• ✅ gatekeeper — distroless + Go binary → Go patching
• ✅ gator — distroless + Go binary → Go patching
• ⏭️ gatekeeper-crds — scratch base + kubectl, no Go binary → skip

Steps (per image, per release):

1. Determine target release tags (latest stable + previous minor)
2. Pull release image from GHCR
3. Trivy scan → JSON report (--ignore-unfixed --pkg-types library)
4. If zero fixable Go vulnerabilities → skip, no patch needed
5. Copa patch:
     COPA_EXPERIMENTAL=1 copa patch \
       -i ghcr.io/open-policy-agent/gatekeeper:v3.22.0 \
       -r scan.json \
       --pkg-types library \
       --toolchain-patch-level patch \
       --library-patch-level patch \
       --ignore-errors \
       -t v3.22.0-$REVISION
6. Trivy re-scan patched image → verify CVE reduction, log delta
7. Push patched image to DockerHub + GHCR

Tagging Scheme

• -R increments monotonically (-1, -2, -3, ...).
• -patched is re-tagged on each successful run to point to the latest -R.
• If a scan finds zero fixable CVEs, no new -R is produced and -patched stays unchanged.

Registry Push Targets

Consistent with existing release workflow:

• DockerHub: openpolicyagent/gatekeeper:v3.22.0-1, openpolicyagent/gatekeeper:v3.22.0-patched
• GHCR: ghcr.io/open-policy-agent/gatekeeper:v3.22.0-1, ghcr.io/open-policy-agent/gatekeeper:v3.22.0-patched

Same for gator.

Workflow Skeleton

name: Copa Patch

on:
  schedule:
    - cron: '17 8 * * 1'  # Weekly, Mondays 08:17 UTC
  workflow_dispatch:
    inputs:
      release_tag:
        description: 'Release tag to patch (e.g., v3.22.0). Defaults to latest stable + previous minor.'
        required: false

jobs:
  resolve-releases:
    runs-on: ubuntu-latest
    outputs:
      releases: ${{ steps.resolve.outputs.matrix }}
    steps:
      - name: Determine target releases
        id: resolve
        run: |
          # Resolve latest stable + previous minor via gh release list
          # Output as JSON matrix: ["v3.22.0", "v3.21.1"]

  patch:
    needs: resolve-releases
    runs-on: ubuntu-latest
    strategy:
      matrix:
        image: [gatekeeper, gator]
        release: ${{ fromJson(needs.resolve-releases.outputs.releases) }}
      fail-fast: false
    steps:
      - name: Determine patch revision
        # Query GHCR for existing -R tags, compute next revision number

      - name: Pull release image
        run: docker pull ghcr.io/open-policy-agent/${{ matrix.image }}:${{ matrix.release }}

      - name: Scan with Trivy
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: 'ghcr.io/open-policy-agent/${{ matrix.image }}:${{ matrix.release }}'
          format: json
          output: scan.json
          ignore-unfixed: true
          pkg-types: library

      - name: Check for fixable vulnerabilities
        id: check
        run: |
          # Parse scan.json, exit early if 0 fixable CVEs
          # Set output: has_fixes=true/false

      - name: Setup BuildKit
        if: steps.check.outputs.has_fixes == 'true'
        run: docker buildx create --name copa-buildkit --use --bootstrap

      - name: Install Copa
        if: steps.check.outputs.has_fixes == 'true'
        # Install Copa >= v0.14.0

      - name: Patch image
        if: steps.check.outputs.has_fixes == 'true'
        run: |
          COPA_EXPERIMENTAL=1 copa patch \
            -i ghcr.io/open-policy-agent/${{ matrix.image }}:${{ matrix.release }} \
            -r scan.json \
            --pkg-types library \
            --toolchain-patch-level patch \
            --library-patch-level patch \
            --ignore-errors \
            -t ${{ matrix.release }}-${{ env.REVISION }}

      - name: Verify patched image
        if: steps.check.outputs.has_fixes == 'true'
        # Re-scan with Trivy, log before/after CVE counts

      - name: Login to GHCR + DockerHub
        if: steps.check.outputs.has_fixes == 'true'
        # docker login for both registries

      - name: Push patched images
        if: steps.check.outputs.has_fixes == 'true'
        run: |
          TAG="${{ matrix.release }}"
          REV="${{ env.REVISION }}"
          IMAGE="${{ matrix.image }}"

          # Push immutable revision tag
          docker push ghcr.io/open-policy-agent/${IMAGE}:${TAG}-${REV}
          docker push openpolicyagent/${IMAGE}:${TAG}-${REV}

          # Update floating -patched tag
          docker tag ghcr.io/open-policy-agent/${IMAGE}:${TAG}-${REV} \
                     ghcr.io/open-policy-agent/${IMAGE}:${TAG}-patched
          docker tag openpolicyagent/${IMAGE}:${TAG}-${REV} \
                     openpolicyagent/${IMAGE}:${TAG}-patched
          docker push ghcr.io/open-policy-agent/${IMAGE}:${TAG}-patched
          docker push openpolicyagent/${IMAGE}:${TAG}-patched

Considerations

Copa Go Patching Mechanics

• Gatekeeper uses distroless → no Go toolchain in runtime image → Copa uses the binary rebuild path (extracts VCS metadata, clones source, applies go get updates, rebuilds binary).
• --toolchain-patch-level patch handles stdlib CVEs by rebuilding with a newer Go patch release.
• --library-patch-level patch handles Go module CVEs via conservative semver-compatible updates.
• --ignore-errors ensures a single module failure doesn't block other patches.

Multi-Architecture

• Gatekeeper builds for linux/amd64, linux/arm64, linux/arm/v7. Copa patching needs to run per-platform and the patched manifest list should cover all architectures.

When This Blocks

• This workflow is blocked on Copa v0.14.0 being released. Track copacetic releases for availability.

References

• Project Copacetic (Copa)
• Copa Go patching PR — copacetic#1388
• Copa Go patching request — copacetic#1386
• Copa app-level patching docs
• Gatekeeper release workflow
• Gatekeeper vulnerability scanning workflow