test(tier4): Tor-down + cosign-verify + clock/ENOSPC + per-service uid fault-injection

[VijitSingh97]

Summary

Adds the four real-hardware (tier-4) coverage gaps a coverage audit found — all harness code, runs at the v1.8 e2e/release gate on real hardware (gouda), not PR CI. Cannot be run locally; verified by shellcheck/shfmt/bash -n plus manual tracing (see below).

1. mini-stack: Tor/SOCKS-down scenario — assert no clearnet fallback and safe degradation #563 Tor-container-down / SOCKS unreachable (TOP PRIVACY PRIORITY, was uncovered at every tier) — fault_tor_down() in tests/integration/run.sh:1196 (called from run_fault_injection()): stops the tor container, asserts (a) zero clearnet egress leak via assert_egress_posture (reuses tests/integration/benchmarks/bench-verify-egress.sh's /proc/net/tcp proof — the same one the steady-state battery runs) and (b) pithead doctor exits non-zero and does not print "All checks passed." while tor is down, then restarts tor and re-asserts both the egress proof and pithead status.

2. cosign verify path (Sign release images with cosign and verify them in pithead upgrade #376/Post-publish release smoke test: real cosign verify + real dashboard upgrade against the published bundle (#376/#59) #459) — verify_release_images_direct() in scripts/release-smoke.sh:176: exercises pithead's own verify_release_images() function directly (not a reimplementation) in the extracted, published bundle dir. Positive case (needs a signed release): the real function against the real pinned digests + committed cosign.pub must pass. Negative case (security-critical, runs regardless of signing): a digest tampered in a copy of the bundle's docker-compose.yml must make the function abort fail-closed. Called from main at scripts/release-smoke.sh:306.

3. clock-drift/NTP + ENOSPC verdicts (doctor: check the egress firewall is actually installed, stratum :3333 is listening, and the dashboard answers #383) — fault_clock_drift() (tests/integration/run.sh:1235) PATH-shadows the timedatectl binary (the same wrapper trick fault_firewall_rollback uses for sudo) so doctor's real NTP check classifies a real unsynced report — without touching the box's actual clock (mining is time-sensitive). fault_disk_enospc() (tests/integration/run.sh:1254) bind-mounts a 1MiB tmpfs over the dashboard data dir and fills it solid, forcing a real kernel ENOSPC (distinct from the existing fault_db_readonly's EACCES) and asserting db_healthy:false, then unmounts and asserts recovery.

4. Per-service runtime non-root uid (Run containers as a non-root user (drop root inside the container) #255/Security follow-ups: dashboard host-networking, Tari gRPC allow-list, assert_safe_dir #91) — tests/integration/run.sh:615 (inside assert_running_state's local-mode block, so it runs on every matrix scenario and at --check): docker exec <svc> id -u for all 9 services against an audited expected uid — tor=100, monerod/p2pool/xmrig-proxy/dashboard/tari=1000, and caddy/docker-proxy/docker-control=0 (root, mitigated by cap_drop: ALL + isolation, not uid — verified empirically against the actual pulled images, see below).

docs/testing-strategy.md's "Known gaps" section is updated to mark all four as covered.

Verification (cannot run against real hardware locally)

• bash -n, shellcheck --severity=warning, shfmt -i 4 -d — clean on both changed scripts, using the exact make lint-sh file set.
• tests/integration/selftest.sh — 132/132 still pass (nothing broken in the sourced pure helpers).
• make lint-docs-voice / markdownlint — clean on the doc edit.
• Traced end-to-end against synthetic fixtures (pulled the real caddy:2.11.4, tecnativa/docker-socket-proxy:v0.4.2, and ubuntu:24.04/alpine:3.24 images to empirically confirm each service's actual runtime uid before hardcoding the expected table — caddy and the two Docker socket proxies genuinely run as root by design, so the check pins that audited posture rather than asserting something false).
• Sourced pithead in a scratch dir and called verify_release_images() directly through the exact cd ... && bash -c 'source ./pithead; verify_release_images' invocation the harness uses, with a fake failing cosign, to confirm the non-zero exit propagates correctly for the negative case.
• Sourced pithead and called clock_sync_status() under the PATH-shadowed timedatectl, and ran the full ./pithead doctor invocation under the shadow, to confirm the exact "NOT NTP-synchronized" string the assertion checks for.
• Validated the docker-compose.yml digest-tamper sed against a synthetic file end-to-end.
• Passed a ponytail-review pass (dropped a dead pre-existence guard and simplified the tamper-digest fixture).

What needs the real box to actually execute

• Everything above is unexercised against real containers/Tor/timedatectl-on-a-real-host/a real signed release bundle — this is why the task scopes it to run at the v1.8 e2e gate on gouda, not PR CI.
• Item 1's doctor-loud-failure assertion is expected to currently FAIL against today's doctor: check_egress_firewall_installed and check_tor_clearnet_egress both SKIP (dr_info, not dr_fail/dr_warn) when the tor container isn't running, so nothing else in doctor currently names a tor-only outage — a healthy-otherwise box can print "All checks passed." with tor stopped. The harness is written to prove that gap, not paper over it; closing it needs a small doctor product fix (an explicit tor-down verdict), which is out of scope for this test-only PR and should be filed as a follow-up.
• Item 2's positive case only runs when the target release is actually cosign-signed (opt-in per Sign release images with cosign and verify them in pithead upgrade #376); on an unsigned release it warns and skips, which is documented as expected in the code.
• Item 4's expected-uid table (caddy/docker-proxy/docker-control = root) reflects the current, intentional posture — if those images are ever hardened to non-root, the table needs a one-line update alongside that change.

Does not merge on its own — awaiting the v1.8 cut's real-hardware run on gouda.

🤖 Generated with Claude Code