test(tier3): fake monerod + failover scenarios in the mini-stack (#616)

Wires a fake monerod (control endpoint to toggle down/busy/synced) into
docker-compose.fake.yml and sets LOCAL_MONERO_HOST=fake-monerod so the
dashboard actually probes it (else monerod is treated remote and never
checked). Adds CI-runnable scenarios: monerod down→reject→readmit
(#31/#564), busy/mid-reorg reject, double-outage both-must-recover, and
Tari-optional-keeps-mining (#562, its own compose cycle since it's a
boot-time flag). Fixes the stale header comment (said monerod, tested Tari).

Closes #562
Closes #564

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Author: VijitSingh97

docs/testing-strategy.md

@@ -64,10 +64,11 @@ The deploy-time axes — each changes a real runtime path. Full table and assert
 | Situation | Trigger | Tier |
 |---|---|---|
 | monerod down → **reject workers** (stop `xmrig-proxy`) | unreachable ≥ `NODE_DOWN_AFTER_SEC` | 1 ✅ · 3 ▶ · 4 ▶ |
-| Tari down + required → reject; Tari down + non-blocking → **ignore** | `tari_down ∧ TARI_REQUIRED?` | 1 ✅ |
+| monerod busy / mid-reorg (HTTP 200, `status≠OK`) → **reject workers** | RPC answers but distrusted | 1 ✅ · 3 ▶ |
+| Tari down + required → reject; Tari down + non-blocking → **ignore** | `tari_down ∧ TARI_REQUIRED?` | 1 ✅ · 3 ▶ |
 | Recovery hysteresis — readmit only after stable `NODE_RECOVERY_AFTER_SEC` | reachable again | 1 ✅ |
 | Transient blip / never-reachable → **no** false reject | debounce / `ever_up` | 1 ✅ |
-| Double outage; readmit only when **both** healthy | both down → both up | 1 ✅ (added) |
+| Double outage; readmit only when **both** healthy | both down → both up | 1 ✅ (added) · 3 ▶ |
 | #35 latch × #31 failover coexist after release | down post-release | 1 ✅ (added) · 3 ▶ |
 | Stop/start fails → retry next cycle (idempotent) | docker error | 1 ✅ |
 
@@ -259,16 +260,22 @@ tier 3/4:
   root-owned file under the dashboard data dir and asserts the pool-flip `apply` (which runs
   `ensure_directories` → `ensure_owner`) chowns it to uid 1000 — the #255 "scan contents, not just
   the dir" regression. Runs at the release gate only (needs root to create a foreign-uid inode).
-- **Real-container monerod failover in PR CI.** The primary-node reject/readmit cycle only runs on
-  the manual tier-4 box (`--fault-injection`); the mini-stack (tier 3) breaks Tari, not monerod.
-- **Non-blocking-Tari "ignore" path with real containers.** Unit-tested only; the mini-stack proves
-  Tari-down-while-required (reject) but never Tari-down-while-optional (keep mining). This is the
-  path that silently kills yield if it regresses to a reject.
-- **monerod busy / mid-reorg failover.** The contract test proves the client reads a busy node as
-  unreachable; no mini-stack or fault-injection scenario asserts the dashboard actually rejects
-  workers on a busy-but-alive node (a real reorg state, distinct from a clean stop).
-- **Double outage, both-must-recover.** Unit-tested (monerod ∧ Tari down → readmit only when both
-  healthy); never driven with real containers, so the recovery ordering is unproven end-to-end.
+- **Real-container monerod failover in PR CI.** ✅ Now tier-3 scenarios 6/7 in the mini-stack: the
+  compose env had a fake `monerod` container wired at the network level (`MONERO_RPC_URL`) but the
+  dashboard's `LOCAL_MONERO_HOST` default didn't match it, so `MONERO_NODE_HOST != LOCAL_MONERO_HOST`
+  put the dashboard on the "remote" code path, which never probes reachability — a monerod outage
+  was a silent no-op end-to-end. Setting `LOCAL_MONERO_HOST` to the fake's hostname fixed the wiring;
+  scenarios 6/7 down/readmit the real `itest-xmrig-proxy` container against it. The tier-4
+  `--fault-injection` box run still covers the real binary/real kernel leg.
+- **Non-blocking-Tari "ignore" path with real containers.** ✅ Mini-stack scenario 11: recreates the
+  stack with `dashboard.tari_required=false` (baked in at container boot, so it needs its own
+  compose cycle) and asserts `itest-xmrig-proxy` stays running through a Tari outage — the path that
+  silently kills yield if it regresses to a reject.
+- **monerod busy / mid-reorg failover.** ✅ Mini-stack scenario 8: the fake's `busy` mode (HTTP 200,
+  `status≠OK`) drives the same reject/readmit cycle as a clean outage.
+- **Double outage, both-must-recover.** ✅ Mini-stack scenario 9: both monerod and Tari down →
+  rejected; recovering only Tari leaves it rejected; recovering monerod too readmits — the recovery
+  ordering proven end-to-end, not just at the unit level.
 - **Partial-start / stop-failure idempotency.** The control loop's "container fails to start/stop →
   retry next cycle" is unit-only; no tier-3/4 scenario injects a docker start/stop error.
 - **`pithead doctor` on a real box.** ✅ The `--check` phase now runs `doctor` and asserts exit 0

tests/integration/mini-stack/docker-compose.fake.yml

@@ -44,6 +44,12 @@ services:
       MONERO_NODE_USERNAME: ""
       MONERO_NODE_PASSWORD: ""
       MONERO_NODE_HOST: "fake-monerod"
+      # Must equal MONERO_NODE_HOST so the dashboard takes the "local" RPC-driven path
+      # (mining_dashboard.collector.logs.get_monero_sync_status) instead of the "remote" path,
+      # which never probes reachability and makes monerod-down/#31/#564 a silent no-op — the
+      # fake-monerod container below was wired at the network level but never actually read for
+      # node-health without this.
+      LOCAL_MONERO_HOST: "fake-monerod"
       MONERO_PRUNE: "true"
       TARI_GRPC_ADDRESS: "fake-tari:18142"
       DOCKER_PROXY_URL: "tcp://docker-proxy:2375"
@@ -52,7 +58,9 @@ services:
       # deployment's p2pool/xmrig-proxy on the same host.
       SYNC_GATE_CONTAINERS: "itest-p2pool,itest-xmrig-proxy"
       REJECT_WORKERS_CONTAINER: "itest-xmrig-proxy"
-      TARI_REQUIRED: "true"
+      # Overridable (scenario 11, #562) since it's baked in at container boot — a live toggle
+      # isn't possible, so that scenario recreates the stack with TARI_REQUIRED=false.
+      TARI_REQUIRED: "${TARI_REQUIRED:-true}"
       XVB_ENABLED: "false"
       XVB_POOL_URL: ""
       XVB_DONOR_ID: ""

tests/integration/mini-stack/run-mini-stack.sh

@@ -6,12 +6,19 @@
 # test-mini-stack`.
 #
 # Scenarios:
-#   1. boot syncing            → dashboard HOLDS itest-p2pool + itest-xmrig-proxy (#35)
-#   2. both chains synced      → dashboard RELEASES them
-#   3. monerod down            → dashboard REJECTS workers (stops itest-xmrig-proxy) (#31)
-#   4. monerod back            → dashboard READMITS workers
-#   +. healthchecks (#79)      → REAL loop fires a liveness heartbeat to the fake-hc receiver
-#                                (pure dead-man's switch; node-health alerting is Telegram #121)
+#    1. boot syncing              → dashboard HOLDS itest-p2pool + itest-xmrig-proxy (#35)
+#    2. monerod synced, Tari not  → still HELD (the gate needs both required chains)
+#    3. both chains synced        → dashboard RELEASES them
+#    +. healthchecks (#79)        → REAL loop fires a liveness heartbeat to the fake-hc receiver
+#                                   (pure dead-man's switch; node-health alerting is Telegram #121)
+#    4. Tari down (required)      → dashboard REJECTS workers (stops itest-xmrig-proxy) (#31)
+#    5. Tari back                 → dashboard READMITS workers
+#    6. monerod down              → dashboard REJECTS workers (#31/#564)
+#    7. monerod back              → dashboard READMITS workers (#564)
+#    8. monerod busy/mid-reorg    → dashboard REJECTS, then READMITS on recovery
+#    9. monerod + Tari both down  → REJECTS; recovering only one does NOT readmit; both does
+#   10. dashboard restart         → the one-way sync latch survives (#35 persistence)
+#   11. Tari OPTIONAL, Tari down  → dashboard keeps mining, workers stay accepted (#562)
 #
 set -uo pipefail
 
@@ -191,10 +198,6 @@ log "scenario 3b: the dashboard fires a real healthchecks heartbeat"
 wait_hc "healthchecks: real loop fired a liveness heartbeat" '^/ph$' 40
 
 # 4. Tari down while required → reject workers (stop the proxy); itest-p2pool keeps running. (#31)
-#    NOTE: monerod-down failover is deliberately NOT simulated here — the dashboard's monerod
-#    down-path falls back to log-scraping a real `monerod` container, which this fake stack has
-#    no equivalent of. That path is covered on real hardware by the tier-4 --fault-injection
-#    phase. Tari has no such fallback, so its reject/readmit exercises the failover cleanly.
 log "scenario 4: rejects workers when required Tari is down"
 set_tari down
 assert_state "rejected on Tari outage: itest-xmrig-proxy stopped" itest-xmrig-proxy exited 90
@@ -209,9 +212,46 @@ log "scenario 5: readmits workers when Tari recovers"
 set_tari synced
 assert_state "readmitted after Tari recovery: itest-xmrig-proxy running" itest-xmrig-proxy running 90
 
-# 6. Dashboard restart after release → the one-way latch is persisted, so the miner is NOT
-#    re-held: both containers stay running across the restart. (#35 persistence)
-log "scenario 6: a dashboard restart does not re-hold a released miner"
+# 6. monerod down → reject workers (stop the proxy); itest-p2pool keeps running. (#31/#564)
+#    Needs LOCAL_MONERO_HOST == MONERO_NODE_HOST in the compose env — otherwise the dashboard
+#    treats monerod as "remote" and never probes it for reachability at all (see that env var's
+#    comment in docker-compose.fake.yml).
+log "scenario 6: rejects workers when monerod is down"
+set_monerod down
+assert_state "rejected on monerod outage: itest-xmrig-proxy stopped" itest-xmrig-proxy exited 90
+if [ "$(cstate itest-p2pool)" = "running" ]; then
+    c_ok "monerod-outage rejection leaves itest-p2pool running (only the proxy fails over)"
+else
+    c_bad "monerod-outage rejection leaves itest-p2pool running" "itest-p2pool is '$(cstate itest-p2pool)'"
+fi
+
+# 7. monerod recovers → readmit (after the recovery-hysteresis window). (#564)
+log "scenario 7: readmits workers when monerod recovers"
+set_monerod synced
+assert_state "readmitted after monerod recovery: itest-xmrig-proxy running" itest-xmrig-proxy running 90
+
+# 8. monerod busy (HTTP 200 but status != OK, e.g. mid-reorg) — the client must distrust the
+#    heights and treat the node as unreachable, not synced, the same as a clean outage.
+log "scenario 8: rejects workers when monerod reports busy/mid-reorg"
+set_monerod busy
+assert_state "rejected on monerod busy: itest-xmrig-proxy stopped" itest-xmrig-proxy exited 90
+set_monerod synced
+assert_state "readmitted after monerod busy clears: itest-xmrig-proxy running" itest-xmrig-proxy running 90
+
+# 9. Double outage — monerod AND Tari both down → rejected; recovering only Tari must NOT
+#    readmit (monerod is still down); recovering monerod too readmits both.
+log "scenario 9: double outage — readmits only once BOTH nodes recover"
+set_monerod down
+set_tari down
+assert_state "rejected on double outage: itest-xmrig-proxy stopped" itest-xmrig-proxy exited 90
+set_tari synced
+assert_stays "still rejected with only Tari recovered" itest-xmrig-proxy exited 8
+set_monerod synced
+assert_state "readmitted once both nodes recovered: itest-xmrig-proxy running" itest-xmrig-proxy running 90
+
+# 10. Dashboard restart after release → the one-way latch is persisted, so the miner is NOT
+#     re-held: both containers stay running across the restart. (#35 persistence)
+log "scenario 10: a dashboard restart does not re-hold a released miner"
 compose restart dashboard >/dev/null 2>&1
 for _ in $(seq 1 30); do
     compose exec -T dashboard python3 -c \
@@ -221,6 +261,30 @@ done
 assert_stays "itest-p2pool stays up across restart" itest-p2pool running 6
 assert_stays "itest-xmrig-proxy stays up across restart" itest-xmrig-proxy running 6
 
+# 11. Tari OPTIONAL (dashboard.tari_required=false) → a Tari outage must NOT reject workers;
+#     Monero mining keeps going. (#562 — the silent-yield-loss path: unit-tested at
+#     test_data_service.py::test_tari_down_ignored_when_non_blocking; this drives the same
+#     decision through real containers.) TARI_REQUIRED is baked into the dashboard container at
+#     boot, so this needs its own compose cycle rather than a live toggle.
+log "scenario 11: Tari-optional — a Tari outage does not reject workers (#562)"
+compose down -v --remove-orphans >/dev/null 2>&1 || true
+TARI_REQUIRED=false compose up -d >/dev/null 2>&1
+api_up=0
+for _ in $(seq 1 30); do
+    if compose exec -T dashboard python3 -c \
+        "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/api/state', timeout=3)" >/dev/null 2>&1; then
+        api_up=1
+        break
+    fi
+    sleep 2
+done
+[ "$api_up" = 1 ] && c_ok "Tari-optional stack: dashboard API is up" || c_bad "Tari-optional stack: dashboard API is up" "no /api/state after ~60s"
+# Tari is non-blocking, so monerod alone gates the sync-hold; release it to reach steady state.
+set_monerod synced
+assert_state "Tari-optional: released itest-xmrig-proxy running" itest-xmrig-proxy running 90
+set_tari down
+assert_stays "Tari-optional: itest-xmrig-proxy keeps mining through a Tari outage" itest-xmrig-proxy running 8
+
 echo ""
 log "mini-stack: $PASS passed, $FAIL failed"
 [ "$FAIL" -eq 0 ]