fix(alerting): KeycloakRealmBackupStale fires NoData spuriously (#559) (#28)

VERDICT: conversion artifact, NOT a real stale backup. The keycloak-dr-backup
CronJob is healthy (every 6h, last runs Complete, latest S3 aws-sso realm
export ~80m old, metric last sample ~6m ago).

Root cause: keycloak_backup_* is PUSHED once per 6h run, so it's present only
within VM's ~5m instant-staleness window — empty for ~5h55m of every cycle. The
VMRule's `(time()-max(...) > 26h) or absent_over_time([26h])` returns a series
only when firing; vmalert treats an empty result as OK, but the verbatim
Grafana port turns the healthy empty result into NoData -> no_data_state=Alerting
-> a spurious ticket/page every inter-push gap.

Fix: query A now = `time() - max_over_time(keycloak_backup_last_success_timestamp_seconds{realm="aws-sso"}[26h])`
(the real age, tolerant of the sparse push), threshold C `> 93600` (26h), and
NoData (no sample in 26h at all = backup genuinely stopped) keeps
no_data_state=Alerting. This reproduces both original clauses without the
spurious fire. relative_time_range.from raised to 94320 to cover the 26h lookback.

The other two converted rules (cert-expiry, tailscale) query continuously-scraped
metrics, so they don't hit this and are unchanged.

Refs thunderbird/platform-infrastructure#559, #367.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: aatchison

terraform/alerting-converted-vmrules.tf

@@ -51,17 +51,27 @@ resource "grafana_rule_group" "keycloak_dr_backup" {
       runbook_url = "https://github.com/thunderbird/platform-infrastructure/blob/main/docs/keycloak-staff-sso.md"
     }
 
+    # A = seconds since the last successful aws-sso backup, computed with a 26h
+    # range (max_over_time) so it tolerates the SPARSE push cadence: the backup
+    # pushes keycloak_backup_* once per 6h run, so a bare instant selector is
+    # empty for ~5h55m of every cycle. The original VMRule's
+    # `(time()-max(...) > 26h) or absent_over_time([26h])` returns a series only
+    # when firing; ported verbatim into a Grafana instant query that empty result
+    # became NoData -> no_data_state=Alerting -> a spurious page every inter-push
+    # gap. max_over_time([26h]) instead always yields the real age while a sample
+    # exists in 26h, and yields NO data only when the backup has been absent for
+    # >26h (the genuine "CronJob stopped" case) -> NoData=Alerting then fires.
     data {
       ref_id         = "A"
       datasource_uid = local.victoriametrics_ds_uid
       relative_time_range {
-        from = 600
+        from = 94320
         to   = 0
       }
       model = jsonencode({
         refId         = "A"
         datasource    = { type = "prometheus", uid = local.victoriametrics_ds_uid }
-        expr          = "(time() - max(keycloak_backup_last_success_timestamp_seconds{realm=\"aws-sso\"}) > 26 * 3600) or absent_over_time(keycloak_backup_success{realm=\"aws-sso\"}[26h])"
+        expr          = "time() - max_over_time(keycloak_backup_last_success_timestamp_seconds{realm=\"aws-sso\"}[26h])"
         instant       = true
         range         = false
         intervalMs    = 1000
@@ -97,7 +107,7 @@ resource "grafana_rule_group" "keycloak_dr_backup" {
         datasource = { type = "__expr__", uid = "__expr__" }
         conditions = [{
           type      = "query"
-          evaluator = { type = "gt", params = [0] }
+          evaluator = { type = "gt", params = [93600] }
           operator  = { type = "and" }
           query     = { params = ["C"] }
           reducer   = { type = "last", params = [] }