docs: update flow + diagrams + developer guide

.github/workflows/race.yml

@@ -0,0 +1,37 @@
+name: Race Detector
+
+on:
+  workflow_dispatch: {}
+  pull_request:
+    branches:
+      - main
+
+jobs:
+  race:
+    name: Run tests with race detector
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        go-version: [1.20.x]
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Set up Go
+        uses: actions/setup-go@v4
+        with:
+          go-version: ${{ matrix.go-version }}
+
+      - name: Install build tools (for cgo / race detector)
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y build-essential
+
+      - name: Ensure CGO enabled
+        run: echo "CGO_ENABLED=1" >> $GITHUB_ENV
+
+      - name: Run tests with race detector
+        run: |
+          go test -race ./... -v

CHANGELOG.md

@@ -0,0 +1,14 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## [Unreleased]
+
+- Add `--registry-ca-validate` flag: when supplied with `--registry-ca`, Watchtower can validate the provided CA bundle on startup and fail fast on misconfiguration. Prefer using this over `--insecure-registry` in production.
+
+- Security: registry TLS verification is now secure-by-default for internal HEAD/token requests; `--insecure-registry` is opt-in for testing.
+- Registry CA support: add `--registry-ca` to provide a PEM bundle merged into system roots, and `--registry-ca-validate` to fail-fast on invalid bundles.
+- Registry token caching: in-memory, concurrent-safe token cache added for registry auth tokens (honors `expires_in`), with deterministic and concurrency unit tests.
+- Testability: refactored registry transport construction and exposed test helpers; added an injectable `now` variable for deterministic time-dependent tests.
+- Docs: added detailed update flow docs, diagrams, and a developer guide (`docs/update-flow*.md`, PlantUML, and rendered SVG).
+- CI: added a GitHub Actions workflow to run `go test -race ./...` with CGO enabled; recommended containerized `-race` run steps added to the developer guide.

README.md

@@ -33,6 +33,21 @@ $ docker run --detach \
 
 Watchtower is intended to be used in homelabs, media centers, local dev environments, and similar. We do **not** recommend using Watchtower in a commercial or production environment. If that is you, you should be looking into using Kubernetes. If that feels like too big a step for you, please look into solutions like [MicroK8s](https://microk8s.io/) and [k3s](https://k3s.io/) that take away a lot of the toil of running a Kubernetes cluster. 
 
+### Using a custom registry CA (private registries)
+
+If you run Watchtower against a private registry that uses a custom TLS certificate, provide the CA bundle and enable validation at startup so Watchtower fails fast on misconfiguration:
+
+```
+$ docker run --detach \
+  --name watchtower \
+  --volume /var/run/docker.sock:/var/run/docker.sock \
+  containrrr/watchtower \
+  --registry-ca /etc/ssl/certs/my-registry-ca.pem \
+  --registry-ca-validate=true
+```
+
+Prefer providing a CA bundle and enabling `--registry-ca-validate` over disabling TLS verification with `--insecure-registry` in production environments.
+
 ## Documentation
 The full documentation is available at https://containrrr.dev/watchtower.
 

cmd/root.go

@@ -20,6 +20,7 @@ import (
 	"github.com/containrrr/watchtower/pkg/container"
 	"github.com/containrrr/watchtower/pkg/filters"
 	"github.com/containrrr/watchtower/pkg/metrics"
+	"github.com/containrrr/watchtower/pkg/registry"
 	"github.com/containrrr/watchtower/pkg/notifications"
 	t "github.com/containrrr/watchtower/pkg/types"
 	"github.com/robfig/cron"
@@ -118,6 +119,30 @@ func PreRun(cmd *cobra.Command, _ []string) {
 	removeVolumes, _ := f.GetBool("remove-volumes")
 	warnOnHeadPullFailed, _ := f.GetString("warn-on-head-failure")
 
+	// Configure TLS verification for registry HEAD/token requests. Default is secure (verify certs).
+	insecureRegistry, _ := f.GetBool("insecure-registry")
+	registry.InsecureSkipVerify = insecureRegistry
+	if insecureRegistry {
+		log.Warn("TLS certificate verification for registry requests is disabled (insecure). This should only be used for testing.)")
+	}
+
+	registryCABundle, _ := f.GetString("registry-ca")
+	if registryCABundle != "" {
+		registry.RegistryCABundle = registryCABundle
+		log.Debugf("Using registry CA bundle: %s", registryCABundle)
+	}
+
+	// Optionally validate CA bundle at startup
+	validateCABundle, _ := f.GetBool("registry-ca-validate")
+	if validateCABundle && registry.RegistryCABundle != "" {
+		if pool := registry.GetRegistryCertPool(); pool == nil {
+			log.Fatalf("Failed to validate registry CA bundle at %s", registry.RegistryCABundle)
+		}
+		log.Info("Registry CA bundle validated successfully")
+	} else if validateCABundle && registry.RegistryCABundle == "" {
+		log.Fatalf("--registry-ca-validate was set but no --registry-ca was provided")
+	}
+
 	if monitorOnly && noPull {
 		log.Warn("Using `WATCHTOWER_NO_PULL` and `WATCHTOWER_MONITOR_ONLY` simultaneously might lead to no action being taken at all. If this is intentional, you may safely ignore this message.")
 	}

docs/SUMMARY_CHECKPOINT.md

@@ -0,0 +1,29 @@
+# Summary Checkpoint
+
+This file marks a checkpoint for summarizing repository changes.
+
+All future requests that ask to "summarise all the changes thus far" should consider
+only changes made after this checkpoint was created.
+
+Checkpoint timestamp (UTC): 2025-11-13T12:00:00Z
+
+Notes:
+- Purpose: act as a stable anchor so that subsequent "summarise all the changes thus far"
+  requests will include only modifications after this point.
+- Location: `docs/SUMMARY_CHECKPOINT.md`
+
+Recent delta (since previous checkpoint):
+
+- Added CLI flags and wiring: `--registry-ca` and `--registry-ca-validate` (startup validation).
+- Implemented secure-by-default registry transport behavior and support for a custom CA bundle.
+- Introduced an in-memory bearer token cache (honors `expires_in`) and refactored time usage
+  to allow deterministic tests via an injectable `now` function.
+- Added deterministic unit tests for the token cache (`pkg/registry/auth/auth_cache_test.go`).
+- Added quickstart documentation snippets to `README.md`, `docs/index.md`, and
+  `docs/private-registries.md` showing `--registry-ca` + `--registry-ca-validate`.
+- Created `CHANGELOG.md` with an Unreleased entry for the new `--registry-ca-validate` flag.
+- Ran package tests locally: `pkg/registry/auth` and `pkg/registry/digest` — tests passed
+  (some integration tests were skipped due to missing credentials).
+
+If you want the next checkpoint after more changes (e.g., mapping the update call chain,
+documenting data shapes, or adding concurrency tests), request another summary break.

docs/arguments.md

@@ -460,8 +460,34 @@ Alias for:
 		--notification-report
 		--notification-template porcelain.VERSION.summary-no-log
 
+
             Argument: --porcelain, -P
 Environment Variable: WATCHTOWER_PORCELAIN
      Possible values: v1
              Default: -
 ```
+
+## Registry TLS options
+
+Options to configure TLS verification when Watchtower talks to image registries.
+
+```text
+            Argument: --insecure-registry
+Environment Variable: WATCHTOWER_INSECURE_REGISTRY
+                Type: Boolean
+             Default: false
+```
+
+```text
+            Argument: --registry-ca
+Environment Variable: WATCHTOWER_REGISTRY_CA
+                Type: String (path to PEM bundle inside container)
+             Default: -
+```
+
+```text
+            Argument: --registry-ca-validate
+Environment Variable: WATCHTOWER_REGISTRY_CA_VALIDATE
+                Type: Boolean
+             Default: false
+```

docs/developer-guide.md

@@ -0,0 +1,78 @@
+<!-- Developer guide: local dev and test commands -->
+# Developer Guide — Running tests & race detector
+
+This short guide covers how to run unit tests locally and how to run the race-enabled test suite in a Linux container (recommended for Windows hosts).
+
+## Prerequisites
+
+- Go toolchain (version compatible with project go.mod). To run `go test` locally, ensure `go` is in your PATH.
+- Docker (for running a Linux container to execute `-race` with CGO enabled)
+- Optional: GitHub CLI `gh` to open PRs from the command line.
+
+## Run unit tests locally
+
+From the repository root:
+
+PowerShell
+
+```powershell
+go test ./... -v
+```
+
+If you only want to run a package tests, run:
+
+```powershell
+go test ./pkg/registry/auth -v
+```
+
+## Run race detector (recommended via container on Windows)
+
+The Go race detector requires cgo and a C toolchain. On Linux runners this is usually available; on Windows it's simplest to run tests inside a Linux container.
+
+Example (PowerShell):
+
+```powershell
+docker run --rm -v "${PWD}:/work" -w /work -e CGO_ENABLED=1 golang:1.20 bash -lc "apt-get update && apt-get install -y build-essential ; /usr/local/go/bin/go test -race ./... -v"
+```
+
+Notes:
+- The command mounts the current working directory into the container and installs `build-essential` to provide a C toolchain so `-race` works.
+- If you prefer a faster run, run `go test -run TestName ./pkg/yourpkg -race`.
+
+## Render PlantUML diagrams (local)
+
+To render PlantUML into SVG using Docker (no Java/PlantUML install required):
+
+```powershell
+docker run --rm -v "${PWD}:/work" -w /work plantuml/plantuml -tsvg docs/diagrams/update-flow.puml
+```
+
+Move the generated SVG into the docs assets folder:
+
+```powershell
+mkdir docs/assets/images -Force
+Move-Item docs/diagrams/update-flow.svg docs/assets/images/update-flow.svg -Force
+```
+
+## Create a branch and PR (example)
+
+Example git commands:
+
+```powershell
+git checkout -b docs/update-flow
+git add docs/update-flow.md docs/diagrams/update-flow.puml docs/developer-guide.md docs/assets/images/update-flow.svg
+git commit -m "docs: add update flow docs, diagrams and developer guide"
+git push -u origin docs/update-flow
+```
+
+If you have the GitHub CLI installed you can open a PR with:
+
+```powershell
+gh pr create --title "docs: update flow + diagrams" --body "Adds update flow documentation, a PlantUML diagram and developer guide." --base main
+```
+
+If `gh` is not installed you can open a PR via GitHub web UI after pushing the branch.
+
+---
+
+If you'd like, I can push the branch and attempt to open the PR for you now.

docs/diagrams/update-flow.puml

@@ -0,0 +1,46 @@
+@startuml
+title Watchtower Update Flow
+actor User as CLI
+participant "cmd (root)" as CMD
+participant "internal/actions.Update" as ACT
+participant "container.Client" as CLIENT
+participant "pkg/registry/digest" as DIG
+participant "pkg/registry/auth" as AUTH
+participant "pkg/registry" as REG
+database "Docker Engine" as DOCKER
+
+CLI -> CMD: trigger runUpdatesWithNotifications()
+CMD -> ACT: Update(client, UpdateParams)
+ACT -> CLIENT: ListContainers(filter)
+loop per container
+    ACT -> CLIENT: IsContainerStale(container, params)
+    CLIENT -> CLIENT: PullImage (maybe)
+    CLIENT -> DIG: CompareDigest(container, registryAuth)
+    DIG -> AUTH: GetToken(challenge)
+    AUTH -> AUTH: getCachedToken / storeToken
+    DIG -> REG: newTransport() (uses --insecure-registry / --registry-ca)
+    DIG -> DOCKER: HEAD manifest with token
+    alt digest matches
+        CLIENT --> ACT: no pull needed
+    else
+        CLIENT -> DOCKER: ImagePull(image)
+    end
+    CLIENT --> ACT: HasNewImage -> stale/newestImage
+end
+ACT -> ACT: SortByDependencies
+ACT -> CLIENT: StopContainer / StartContainer (with lifecycle hooks)
+ACT -> CLIENT: RemoveImageByID (cleanup)
+ACT --> CMD: progress.Report()
+
+note right of AUTH
+  Tokens are cached by auth URL (realm+service+scope)
+  ExpiresIn (seconds) sets TTL when provided
+end note
+
+note left of REG
+  TLS is secure-by-default
+  `--registry-ca` provides PEM bundle
+  `--registry-ca-validate` fails startup on invalid bundle
+end note
+
+@enduml

docs/index.md

@@ -63,3 +63,17 @@ the following command:
         volumes:
           - /var/run/docker.sock:/var/run/docker.sock
     ```
+
+Quick note: if your registry uses a custom TLS certificate, mount the CA bundle and enable startup validation so Watchtower fails fast on misconfiguration:
+
+```bash
+docker run --detach \
+  --name watchtower \
+  --volume /var/run/docker.sock:/var/run/docker.sock \
+  --volume /etc/ssl/private-certs:/certs \
+  containrrr/watchtower \
+  --registry-ca /certs/my-registry-ca.pem \
+  --registry-ca-validate=true
+```
++
+Prefer this over `--insecure-registry` for production.

docs/private-registries.md

@@ -205,3 +205,45 @@ A few additional notes:
 
 4.  An alternative to adding the various variables is to create a ~/.aws/config and ~/.aws/credentials files and 
     place the settings there, then mount the ~/.aws directory to / in the container.
+
+## Token caching and required scopes
+
+Watchtower attempts to minimize calls to registry auth endpoints by caching short-lived bearer tokens when available.
+
+- Token cache: When Watchtower requests a bearer token from a registry auth endpoint, it will cache the token in-memory keyed by the auth realm + service + scope. If the token response includes an `expires_in` field, Watchtower will honor it and refresh the token only after expiry. This reduces load and rate-limit pressure on registry auth servers.
+
+- Required scope: Watchtower requests tokens with the following scope format: `repository:<image-path>:pull`. This is sufficient for read-only operations required by Watchtower (HEAD or pull). For registries enforcing fine-grained scopes, ensure the provided credentials can request tokens with `pull` scope for the repositories you want to monitor.
+
+- Credential sources: Watchtower supports these sources (in priority order):
+  1. Environment variables: `REPO_USER` and `REPO_PASS`.
+  2. Docker config file (`DOCKER_CONFIG` path or default location, typically `/root/.docker/config.json` when running in container) including support for credential helpers and native stores.
+
+When possible, prefer using short-lived tokens or credential helpers and avoid embedding long-lived plaintext credentials in environment variables.
+
+### Providing a custom CA bundle
+
+For private registries using certificates signed by an internal CA, prefer providing a PEM encoded CA bundle to disable verification bypassing. Use the `--registry-ca` flag or the `WATCHTOWER_REGISTRY_CA` environment variable to point to a file inside the container with one or more PEM encoded certificates. Watchtower will merge the provided bundle with system roots and validate registry certificates accordingly.
+
+Example (docker run):
+
+```bash
+docker run -v /etc/ssl/private-certs:/certs -e WATCHTOWER_REGISTRY_CA=/certs/my-registry-ca.pem containrrr/watchtower
+```
+
+This is the recommended approach instead of `--insecure-registry` for production deployments.
+
+#### Quick example: validate CA at startup
+
+If you want Watchtower to fail fast when the provided CA bundle is invalid or missing, mount the CA into the container and enable validation:
+
+```bash
+docker run --detach \
+  --name watchtower \
+  --volume /var/run/docker.sock:/var/run/docker.sock \
+  --volume /etc/ssl/private-certs:/certs \
+  containrrr/watchtower \
+  --registry-ca /certs/my-registry-ca.pem \
+  --registry-ca-validate=true
+```
++
+This makes misconfiguration explicit during startup and is recommended for unattended deployments.