Fleetlock janitor sidecar fails to release stale reboot leases

[letan-assistant]

Summary

The janitor sidecar in files/fleetlock.yaml (added in e952c15, deployed 2026-03-08) is supposed to delete the fleetlock-default lease when its holderIdentity no longer matches any current node's machine ID. It does not work.

Impact

On 2026-06-06 the cluster's OS auto-updates were found to be fully stalled. The fleetlock-default lease (namespace default) had been held for ~112 days by machine ID 010bf3597d08415bad40906d032083a8, which matches no current node — it is a ghost from an original node destroyed in a prior make deploy (deploys recreate cloud instances).

Because the ghost holder never releases the lock, every node loops forever in upgrade-os.sh on POST /v1/pre-reboot getting a non-200, so staged rpm-ostree upgrades never reboot. Agent nodes were stuck on FCOS 43.20260202 for ~4 months.

The janitor was added specifically to prevent this, but failed to do its job.

Evidence the janitor is broken

• Pod running 90 days, janitor container: 0 restarts.
• Janitor logs are completely empty (it should log "Lease holder ... not found among current nodes, deleting lease").
• kubectl invoked inside the janitor container hangs (in-pod kubectl exec into control-plane pods also hangs).
• The lease survived the entire 90 days and had to be deleted manually.

Likely contributing factors in the script

• No --request-timeout on the kubectl calls → a hung get-lease wedges the while loop forever on the first iteration.
• 2>/dev/null on the get-lease swallows any error, so failures are invisible in logs.
• 1800s sleep makes iteration slow to observe.

Proposed fix (not yet applied)

• Add --request-timeout=30s to all kubectl calls.
• Drop 2>/dev/null so failures surface in logs.
• Possibly shorten the sleep / add explicit logging per iteration.
• Investigate why in-pod kubectl hangs on the control-plane node (API reachability from pod network, kubelet exec stream) — the fleetlock Go client on the same pod works fine, so this is specific to the kubectl path.

Workaround

kubectl --kubeconfig elsa.yaml delete lease fleetlock-default -n default

Safe and self-healing — fleetlock recreates the lease on the next real reboot request. After deletion a waiting node acquired the lock within 15s and proceeded to reboot.

[letan-assistant]

Second bug found during the 2026-06-06 rollout: post-reboot lock release is unreliable

After the stale lease was deleted, elsa-agent-1 acquired the lock, rebooted, and successfully upgraded 43.20260202 → 43.20260217 (Ready, uncordoned). But it never released the lock. The fleetlock server logs show only elsa-agent-0 polling pre-reboot (blocked), and zero steady-state requests from agent-1 — i.e. fleetlock-release.sh never got a release request through to the server after the upgrade reboot. agent-1 held the lock for 15+ minutes until manually released, blocking agent-0 the whole time.

Why this matters

This is the same fleet-wide stall as the original ghost lease, but the janitor sidecar (this issue's main subject) would not catch it either — agent-1 is a valid, current node, so the janitor's "holder not among current nodes" check never fires. The orphaned-lock-by-a-live-node case needs a different safety net than the janitor.

Suspected causes (fleetlock-release.service in agent.bu.mustache / control_plane.bu.mustache)

• Type=oneshot + RemainAfterExit=yes, runs once at boot (WantedBy=multi-user.target, After=k3s-agent.service). If the curl to the fleetlock ClusterIP (10.43.0.100:8080) doesn't succeed during the early-boot window — kube-proxy / CNI / the fleetlock pod not yet reachable from the freshly-booted node — the script burns its 84×5s=7min of retries and exits without ever releasing. No automatic re-run afterward.
• No server-side TTL/expiry on the lease, so a missed release is permanent until manual intervention.

Proposed mitigations (to discuss, defer until cluster is up to date)

• Make release self-healing rather than one-shot-at-boot: e.g. a small systemd timer that periodically reconciles "if this node holds the lock AND this node is up/Ready/steady, release it," instead of a single best-effort attempt.
• And/or harden fleetlock-release.service with Restart=on-failure (drop RemainAfterExit) so a failed release retries.
• Consider a lease TTL / auto-expiry as a backstop for both failure modes.

Workaround (same as the ghost-lease case)

kubectl --kubeconfig elsa.yaml delete lease fleetlock-default -n default — safe once the holding node has finished upgrading and is Ready/uncordoned.

[letan-assistant]

Actual root cause found (corrects the earlier comments) + fix

Investigation continued. The earlier theories (degraded control-plane networking, fleetlock 'matches no Kubernetes Nodes', early-boot release races) were largely a red herring caused by flaky kubectl exec into control-plane pods masking the real issue. The actual reason the janitor never worked for 90 days:

kubectl in the alpine/k8s image does not use in-cluster credentials on its own — it falls back to http://localhost:8080 and gets connection-refused. The original script piped that through 2>/dev/null, so holder came back empty every loop, the janitor concluded 'no stale lease', and slept — silently, forever. (fleetlock's main container works because the Go client auto-loads in-cluster config; kubectl does not.)

Proof: running the janitor logic in a pod with the fleetlock SA logged:

WARN cannot read lease: ... Get "http://localhost:8080/api?timeout=20s": dial tcp [::1]:8080: connect: connection refused

Fix (committed to files/fleetlock.yaml)

• Point kubectl at the API explicitly with the SA token: --server=https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT --certificate-authority=<sa>/ca.crt --token=$(cat <sa>/token). Token re-read each call (rotation-safe).
• --request-timeout=20s on every call so a hung call can't wedge the loop.
• Log every iteration so the janitor is visibly alive, never indistinguishable from a hang.
• Drop 2>/dev/null so failures surface.

Validated end-to-end on a healthy agent node with a synthetic ghost lease: the janitor detected the unmatched holder and deleted the lease within one poll. (Can't validate on the current control-plane node because kubectl exec there is flaky, unrelated to this fix.)

Scope decision

Kept the janitor minimal: ghost-holder detection only (holder matches no current node). The 'orphaned lock held by a live, Ready+schedulable node' case (what stalled agent-0 today via fleetlock failing to release) is intentionally not covered, on the assumption that a clean redeploy restores reliable fleetlock releases. Residual risk: if post-redeploy fleetlock still fails to release a lock held by a live node, this minimal janitor won't clean it — we'd then add the orphaned-holder logic or fix/upgrade fleetlock.

Ships via make deploy (recreates the control-plane; user_data is ForceNew on vultr_instance), not a manual kubectl apply.

[letan-assistant]

Status update (2026-06-08) — auth fix shipped; fleetlock healthy; janitor still wedges (new cause isolated)

Shipped

• The in-cluster auth fix (point kubectl at https://$KUBERNETES_SERVICE_HOST:$PORT with the SA token instead of falling back to localhost:8080) landed in commit f4063c2 and was deployed in the full cluster redeploy on 2026-06-08. All nodes are now on FCOS 44.20260510.
• Related: the deeper reason OS auto-update was dead — Zincati left enabled, blocking rpm-ostree upgrade — was fixed separately by masking zincati.service (commit 6a9d618). That's distinct from this janitor issue but was the headline 'cluster doesn't auto-update' cause.

Good news — the original fleetlock release bug is gone on the fresh cluster

Post-redeploy, the fleetlock (Go) container processed steady-state reboot-lock releases from all three nodes' new machine-ids (18:07–18:09) with no matches no Kubernetes Nodes errors. That error was almost certainly a stale-informer artifact of the 90-day-old pod; a fresh fleetlock matches nodes correctly. So coordinated releases work again.

Still open — the janitor is wedged, and it's NOT the auth bug or connectivity

On the freshly-redeployed control-plane, the janitor logs started (poll=300s api=https://10.43.0.1:443) and then goes completely silent — 0 heartbeats, 0 warnings for 25+ min despite a 5-min poll. So it's stuck on its first API call.

Key isolation: fleetlock's Go client reaches the API fine from the same pod/node (it just released three locks), so this is not control-plane pod→ClusterIP connectivity. The difference is that kubectl does full API discovery (and self-throttles — earlier logs showed client-side throttling ... Waited before sending request delay=5s and context deadline exceeded on /api), while the Go client makes targeted calls. So kubectl discovery over the control-plane pod's ClusterIP path is slow/throttled enough to wedge the janitor, even though raw connectivity is fine.

Note this reproduced across two fresh control-plane instances (the 06-06 recreate and the 06-08 full redeploy), so it's a property of kubectl-on-the-control-plane-node, not a one-off degraded node.

The janitor's logic is correct — validated on an agent node (where kubectl works), it deletes a synthetic ghost lease within one poll. It's specifically the control-plane placement + kubectl discovery that breaks it.

Recommended next step

Make the janitor not depend on kubectl discovery. Options, roughly in order of preference:

• Rewrite its calls to hit the API REST endpoints directly via curl + the SA token with a hard --max-time (no discovery, bounded, logs every iteration) — earlier deferred as 'over-engineering for a degraded node', but now justified since a clean control-plane reproduces it.
• Or use kubectl get --raw <path> (skips discovery) + a persisted --cache-dir.
• Or move the janitor off the control-plane node (it only needs API access + RBAC, not control-plane locality).

Leaving this issue open: the auth bug is fixed and fleetlock is healthy, but the janitor safety-net is still not reliably functional in production.