fix(workflow-executor): apply Debian security updates to the Docker image

[Scra3]

Why

The Validate Dockerfile build job has been failing on every build since 12 Sep on two HIGH CVEs in libpcre2-8-0:

Total: 2 (HIGH: 2, CRITICAL: 0)
libpcre2-8-0  CVE-2026-86145  HIGH  fixed  10.42-1 → 10.42-1+deb12u1
              CVE-2026-89161

This is not branch-specific. main and every open PR fail the same step, and a commit that passed on 11 Sep fails today with no code change: the advisories landed in between.

The image builds and the smoke test passes. Only the Trivy gate (step 8, Scan OS packages (blocking)) fails.

Why not just bump the digest

That was my first instinct, and I checked it before writing this: the newest node:22-bookworm-slim still ships the vulnerable 10.42-1. Upstream has not rebuilt.

newest node:22-bookworm-slim → libpcre2-8-0 10.42-1
pinned digest 813a7480…      → libpcre2-8-0 10.42-1

The digest pin is the right call for reproducibility, but it also guarantees Debian security updates never reach the image on their own. Applying them in the base stage addresses the class of failure rather than this instance of it, so the next advisory does not need a digest hunt.

What

• The runtime stage runs apt-get update && apt-get upgrade -y --with-new-pkgs and drops the index afterwards. In runtime rather than base so a refresh rebuilds COPY layers, not yarn install and lerna run build.
• APT_EPOCH build arg, set by CI to github.run_id-run_attempt. It sits in the layer's cache key, so every run refetches the Debian index: a rebuild of a released tag, a re-run of a failed job and both arches of one release all resolve against the current archive. The gated build and the push share the value, so the push is a cache hit and ships the image the scan saw.
• apt-get update -o APT::Update::Error-Mode=any: an unreachable mirror fails the build instead of exiting zero on a half patched layer.

Verified locally, with the CI's own scan settings

--scanners vuln --pkg-types os --severity CRITICAL,HIGH --ignore-unfixed:

	Before	After
Trivy HIGH/CRITICAL	2	0
libpcre2-8-0	10.42-1	10.42-1+deb12u1
docker/smoke-test.sh	pass	pass
Image size	~410 MB	~435 MB

Also checked the cache key: a second build with the same APT_EPOCH shows the apt layer as CACHED, a different value re-runs it.

Trade off, stated plainly

The image is reproducible from the commit plus the build date, not from the commit alone. That is the point: the digest pin had frozen the package set and nobody noticed until CI went red. What stays fixed is the base digest, the lockfile-driven dependency tree, the SBOM on every push and the blocking scan before publish. The README no longer claims byte reproducibility.

Unblocks the Docker job on main and on every open PR, including #1897.

🤖 Generated with Claude Code

Note

Apply Debian security updates to workflow-executor Docker image

• Adds an apt-get update && apt-get upgrade -y --with-new-pkgs step in the runtime stage of Dockerfile so published images install current Debian security and package updates rather than relying solely on the pinned base image.
• Introduces an APT_SECURITY_EPOCH build arg composed of the current UTC ISO week plus a serial read from apt-refresh; the epoch busts the Docker build cache to force APT layer refresh on week rollover or serial bump.
• The epoch is computed in both the validate and publish jobs in docker-publish.yml and passed through the gated and push-by-digest builds so the published image matches the one that was validated.
• Risk: base package contents now vary over time, so reproducibility of runtime layers depends on archive availability and the epoch value; bump the serial in apt-refresh to force a refresh.

Changes since #1899 opened

• Replaced APT_SECURITY_EPOCH build argument with APT_EPOCH across the docker-publish workflow and workflow-executor Dockerfile, computing the epoch value from github.run_id and github.run_attempt instead of a weekly security epoch [d3f93d2]
• Added -o APT::Update::Error-Mode=any flag to the apt-get update command in the workflow-executor Dockerfile runtime stage [d3f93d2]
• Deleted the apt-refresh file from the workflow-executor docker directory [d3f93d2]
• Updated the documented image size from approximately 410 MB to approximately 435 MB in the workflow-executor docker README and clarified that the dependency tree is fully pinned [d3f93d2]

^{Macroscope summarized 2c4b08f.}

[macroscopeapp]

🟡 Medium workflow-executor/Dockerfile:19

This apt-get update/unversioned apt-get upgrade makes identical builds from the same base digest produce different runtime images as Debian publishes updates. Pin the repository to a dated snapshot or record and install explicit package versions to preserve reproducibility.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @packages/workflow-executor/Dockerfile around line 19:

This `apt-get update`/unversioned `apt-get upgrade` makes identical builds from the same base digest produce different runtime images as Debian publishes updates. Pin the repository to a dated snapshot or record and install explicit package versions to preserve reproducibility.

[Scra3]

Right, and it was the most serious thing wrong with the first version — thanks. Fixed in b2d34e5, though not quite the way you suggested.

The diagnosis needs one correction. A stale layer cannot put a vulnerable image in the registry: the publish job scans the loaded image with a blocking Trivy gate and only pushes afterwards (docker-publish.yml, Scan OS packages (blocking) → Push by digest). So the real failure mode is not a vulnerable release, it is a CI that goes red and stays red through re-runs, because every retry restores the same cached layer. Worth stating precisely, since it changes who is harmed and how they find out.

On the fix: I did not take the rotated-ARG shape as written, because "deliberately rotated" means a human has to remember. Nobody rotates it, and the person staring at a red Trivy gate has no way to discover the knob exists — so it reads as solved while behaving exactly like today. Instead CI computes the value:

- name: Compute apt security epoch
  id: apt
  run: echo "epoch=$(date -u +%G-W%V)" >> "$GITHUB_OUTPUT"

passed as build-args: APT_SECURITY_EPOCH=${{ steps.apt.outputs.epoch }} to the validate build, the gated build and the push. Same value in the gated build and the push on purpose: a different one there would rebuild the apt layer and publish an image the scan never saw.

Also moved the layer from base to runtime. Identical shipped packages, since runtime is FROM base, but a buster in base invalidates yarn install and lerna run build beneath it. In runtime there is nothing below but COPY lines, so the weekly refresh costs ~30s instead of a cold monorepo rebuild. That was the part of your suggestion I could not take as-is.

Verified locally: the layer shows CACHED on an unchanged epoch and re-runs on a changed one, libpcre2-8-0 lands at 10.42-1+deb12u1, Trivy reports 0, and docker/smoke-test.sh passes.

[macroscopeapp]

The weekly epoch prevents a permanently stale cached layer, but it does not resolve the reproducibility concern: the same pinned base and Dockerfile can still produce different package versions across weeks as Debian changes. Pinning a snapshot or explicit versions is still needed for reproducible output. Would you like me to prepare that change?

[Scra3]

You are right that reproducibility is not restored, and I am declining the fix rather than disputing the diagnosis — so let me be explicit about the trade instead of leaving it implicit.

Reproducible output and automatic patching are in direct opposition here. Pinning a snapshot or explicit versions gives back byte-identical builds, and in exchange the image is frozen at that snapshot until a human bumps it. That is precisely the failure this PR exists to fix: the digest pin already froze the package set, nobody noticed for months, and CI went red on two HIGH CVEs that Debian had fixed. Snapshot pinning reproduces that shape with a different knob.

There is a coherent design where both hold — pin a snapshot date and have the weekly job bump the date — but it is materially more machinery than the problem in front of us, and it still ships whatever that week's snapshot contains. It is not more reproducible in any sense that matters to an operator, only more auditable.

So the choice is deliberate: the image is reproducible from the commit plus the build date, not from the commit alone. What we keep is the pinned base digest, a lockfile-driven dependency tree, an SBOM on every push, and a blocking scan before publish, so a build is auditable after the fact even when it is not bit-identical.

The Dockerfile now states this in the comment rather than claiming otherwise — the first version of this PR asserted it "keeps the pin reproducible", which was simply false and is the kind of comment worth catching.

If the team later wants determinism back, the honest form is targeted --only-upgrade of named packages, one line per advisory, which trades the automation away openly. Worth a separate discussion, not this PR.

[qltysh]

Coverage Impact

This PR will not change total coverage.

🚦 See full report on Qlty Cloud »

🛟 Help

• Diff Coverage: Coverage for added or modified lines of code (excludes deleted files). Learn more.

• Total Coverage: Coverage for the whole repository, calculated as the sum of all File Coverage. Learn more.

• File Coverage: Covered Lines divided by Covered Lines plus Missed Lines. (Excludes non-executable lines including blank lines and comments.)

  • Indirect Changes: Changes to File Coverage for files that were not modified in this PR. Learn more.

[hercemer42]

Spec: no Linear ticket is linked anywhere on this PR, so the functional check was limited to the PR description.

Applies to: packages/workflow-executor/docker/README.md
Claude Opus 5 (claude-opus-5[1m]): Should fix: The docker README still promises a reproducible 410 MB image that this change no longer builds.

Code

docker/README.md:5 claims roughly 410 MB and reproducible, while the PR measures 435 MB and Dockerfile:63 drops reproducibility.

A way out: scope the claim to the dependency install and document the apt-refresh lever there.

Reproduction

1. Open docker/README.md to size the runtime image.
2. It reports 410 MB and calls the build reproducible.
3. Pull the published image and measure 435 MB with OS packages resolved at build time.

Applies to: the PR as a whole
Claude Opus 5 (claude-opus-5[1m]): Preferential: The reproducibility trade off lives only in Dockerfile comments, so the next reader relitigates it.

Code

Dockerfile:63 is the only record of the trade off, and no ADR under docs/adr/** covers it.

A way out: run /adr on the decision.

[Scra3]

README: fixed in d3f93d2. It now says ~435 MB with a fully pinned dependency tree, which is what the lockfile still guarantees.

ADR: skipped. The decision is one RUN line and trivially reversible, so it fails the hard-to-inverse gate.

[hercemer42]

Claude Opus 5 (claude-opus-5[1m]): Closed: the README now reports 435 MB and scopes the pinning claim to the dependency tree.

How it was checked

docker/README.md:5 no longer calls the build reproducible, which the runtime stage stopped being.

The apt-refresh half of the finding is moot because d3f93d2 deleted the file.

No check pins the claim, and nothing in the repo has ever guarded it.

Claude Opus 5 (claude-opus-5[1m]): Closed: the decision shrank to one RUN line in d3f93d2, which fails the hard to reverse gate.

How it was checked

The predicate was a decision costly to invert, and reverting is now deleting four lines.

Dockerfile:63 keeps the reason in the file, which is what the ADR would have carried.