Repository navigation
fix(alerting): KeycloakRealmBackupStale fires NoData spuriously (sparse-push conversion artifact) (#559) - #28
Merged
Conversation
|
Ran Plan for project: Show OutputTerraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
~ update in-place
- destroy
Terraform will perform the following actions:
# grafana_rule_group.catalog_argocd will be destroyed
# (because grafana_rule_group.catalog_argocd is not in configuration)
- resource "grafana_rule_group" "catalog_argocd" {
- disable_provenance = true -> null
- folder_uid = "efhbgu7k4a2o0b" -> null
- id = "0:efhbgu7k4a2o0b:catalog-argocd-paging" -> null
- interval_seconds = 60 -> null
- name = "catalog-argocd-paging" -> null
- org_id = "1" -> null
- rule {
- annotations = {
- "description" = "An ArgoCD app has been health_status=Degraded for 15m — a tracked resource failed its runtime health check (crash-loop, failed readiness, stuck PVC) even if Git-synced. Sustained >15m usually means a real managed service (Traefik/Keycloak/vmauth) is broken. Matches only Degraded (excludes transient Progressing/Missing)."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "An ArgoCD application is Degraded on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "15m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "argocd"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "ArgoCDApplicationDegraded" -> null
- no_data_state = "OK" -> null
- uid = "afq15flidozr4f" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "sum(argocd_app_info{cluster=\"mzla-eks-shared01\",health_status=\"Degraded\"})"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 900 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 900 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 0,
]
- type = "gt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 900 -> null
- to = 0 -> null
}
}
}
- rule {
- annotations = {
- "description" = "The application-controller (reconcile/self-heal) or repo-server (manifest rendering) is fully down on shared01 — the fleet's GitOps loop stalls, drift goes uncorrected, no app can sync/self-heal across all clusters. Selector uses the live `container` label (no app_kubernetes_io_component here)."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "ArgoCD application-controller or repo-server is down on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "10m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "argocd"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "ArgoCDControllerOrRepoServerDown" -> null
- no_data_state = "Alerting" -> null
- uid = "ffq15flig6w3kd" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "min(max by (container) (up{cluster=\"mzla-eks-shared01\",namespace=\"argocd\",container=~\"application-controller|repo-server\"}))"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 1,
]
- type = "lt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
}
# grafana_rule_group.catalog_eso will be destroyed
# (because grafana_rule_group.catalog_eso is not in configuration)
- resource "grafana_rule_group" "catalog_eso" {
- disable_provenance = true -> null
- folder_uid = "bfhbgu9i6fjswa" -> null
- id = "0:bfhbgu9i6fjswa:catalog-eso-paging" -> null
- interval_seconds = 60 -> null
- name = "catalog-eso-paging" -> null
- org_id = "1" -> null
- rule {
- annotations = {
- "description" = "The ESO controller is down fleet-wide on shared01 (namespace external-secrets) — no ExternalSecret reconciles. Existing Secrets keep serving, but any rotation / new ExternalSecret / token refresh (vmauth bearer tokens, Keycloak/Tailscale OAuth) silently stops. for=10m tolerates a rollout blip (60s scrape interval)."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "External Secrets Operator is down on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "10m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "external-secrets"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "ESOControllerDown" -> null
- no_data_state = "Alerting" -> null
- uid = "bfq15fla70yyoc" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "sum(up{cluster=\"mzla-eks-shared01\",namespace=\"external-secrets\"} == 1)"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 1,
]
- type = "lt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
}
# grafana_rule_group.catalog_keycloak will be destroyed
# (because grafana_rule_group.catalog_keycloak is not in configuration)
- resource "grafana_rule_group" "catalog_keycloak" {
- disable_provenance = true -> null
- folder_uid = "dfhbgu8xprhtsb" -> null
- id = "0:dfhbgu8xprhtsb:catalog-keycloak-paging" -> null
- interval_seconds = 60 -> null
- name = "catalog-keycloak-paging" -> null
- org_id = "1" -> null
- rule {
- annotations = {
- "description" = "No Keycloak pod is up in namespace keycloak on shared01 — total staff-SSO outage, including AWS Identity Center SAML federation (aws-sso realm). The 3-replica HA StatefulSet tolerates losing 1-2 pods, so this pages only on full-fleet loss. for=2m for fast paging without flapping on rolling restarts."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/blob/main/docs/keycloak-staff-sso.md"
- "summary" = "All Keycloak replicas are down on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "2m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "keycloak"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "KeycloakAllReplicasDown" -> null
- no_data_state = "Alerting" -> null
- uid = "efq15flf9bk74a" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "sum(up{cluster=\"mzla-eks-shared01\",namespace=\"keycloak\"} == 1)"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 1,
]
- type = "lt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
- rule {
- annotations = {
- "description" = ">5% 5xx responses from Keycloak on shared01 for 10m = serving errors (DB failover, cache split-brain, OOM) even though pods are up — missed by the pod-down rule. http_server_requests_seconds_count carries a `status` label; the denominator is guarded against divide-by-zero by no_data_state=OK."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/blob/main/docs/keycloak-staff-sso.md"
- "summary" = "Keycloak 5xx rate > 5% on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "OK" -> null
- for = "10m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "keycloak"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "KeycloakHigh5xxRate" -> null
- no_data_state = "OK" -> null
- uid = "bfq15flfebcw0e" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "sum(rate(http_server_requests_seconds_count{cluster=\"mzla-eks-shared01\",namespace=\"keycloak\",status=~\"5..\"}[5m])) / sum(rate(http_server_requests_seconds_count{cluster=\"mzla-eks-shared01\",namespace=\"keycloak\"}[5m]))"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 0.05,
]
- type = "gt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
Warning: Output length greater than max comment size. Continued in next comment. |
|
Continued plan output from previous comment. Show Output - uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
}
# grafana_rule_group.catalog_kubernetes will be destroyed
# (because grafana_rule_group.catalog_kubernetes is not in configuration)
- resource "grafana_rule_group" "catalog_kubernetes" {
- disable_provenance = true -> null
- folder_uid = "dfhbgu7yr6nlsc" -> null
- id = "0:dfhbgu7yr6nlsc:catalog-kubernetes-paging" -> null
- interval_seconds = 60 -> null
- name = "catalog-kubernetes-paging" -> null
- org_id = "1" -> null
- rule {
- annotations = {
- "description" = "At least one mzla-eks-shared01 node is NotReady and not cordoned (kube_node_spec_unschedulable==0) for 10m. Workloads are evicting and cluster capacity is shrinking. Chart ships this at warning/15m and routes nowhere; this Grafana copy pages. Check node health, kubelet, and EC2 instance status."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "A shared01 node has been NotReady for 10m"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "10m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "kubernetes"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "KubeNodeNotReady (shared01)" -> null
- no_data_state = "OK" -> null
- uid = "ffq15flbkyxvka" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "count(kube_node_status_condition{cluster=\"mzla-eks-shared01\",condition=\"Ready\",status=\"true\"} == 0 and on(node) kube_node_spec_unschedulable{cluster=\"mzla-eks-shared01\"} == 0)"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 0,
]
- type = "gt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
- rule {
- annotations = {
- "description" = "node count minus live kubelet targets > 0 on mzla-eks-shared01: at least one node's kubelet is down/unscraped, so its pods are unmanaged and the control plane has lost ground truth for that node. Highest-signal node-plane outage. Chart classifies KubeletDown critical/15m but it never reaches PagerDuty."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "A shared01 node has no live kubelet scrape target for 10m"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "10m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "kubernetes"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "KubeletDown (shared01)" -> null
- no_data_state = "OK" -> null
- uid = "afq15flcwf0g0d" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "count(kube_node_info{cluster=\"mzla-eks-shared01\"}) - count(up{cluster=\"mzla-eks-shared01\",job=\"kubelet\",metrics_path=\"/metrics\"} == 1)"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 0,
]
- type = "gt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
- rule {
- annotations = {
- "description" = "A PVC on mzla-eks-shared01 has less than 5% free space and is minutes from write failures. These back stateful core infra (vmstorage 100Gi, VictoriaLogs, Keycloak, Teleport). A full volume = metrics/log ingestion loss or auth-plane corruption. Only the critical <5% variant is promoted to page; the chart's 15%-prediction warning stays on the chart path. Expand the PVC."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "A shared01 PersistentVolume is under 5% free"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "5m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "kubernetes"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "KubePersistentVolumeFillingUpCritical (shared01)" -> null
- no_data_state = "OK" -> null
- uid = "cfq15flcwf0g1a" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "count((kubelet_volume_stats_available_bytes{cluster=\"mzla-eks-shared01\"} / kubelet_volume_stats_capacity_bytes{cluster=\"mzla-eks-shared01\"}) < 0.05 and kubelet_volume_stats_used_bytes{cluster=\"mzla-eks-shared01\"} > 0)"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 0,
]
- type = "gt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
- rule {
- annotations = {
- "description" = "A container in a user-facing core namespace (monitoring/keycloak/teleport/argocd/external-secrets/kube-system/traefik) has been CrashLoopBackOff for 15m on mzla-eks-shared01. Namespace-scoped so a crashlooping Keycloak/Traefik/ArgoCD/ESO/VM pod pages while general app workloads stay on the chart's warning path. for=15m survives a rolling deploy."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "A core-namespace pod is CrashLoopBackOff for 15m on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "15m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "kubernetes"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "KubePodCrashLoopingCoreNamespaces (shared01)" -> null
- no_data_state = "OK" -> null
- uid = "bfq15flcywwsgb" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "count(max_over_time(kube_pod_container_status_waiting_reason{cluster=\"mzla-eks-shared01\",reason=\"CrashLoopBackOff\",namespace=~\"monitoring|keycloak|teleport|argocd|external-secrets|kube-system|traefik\"}[5m]) >= 1)"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 0,
]
- type = "gt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
}
# grafana_rule_group.catalog_teleport will be destroyed
# (because grafana_rule_group.catalog_teleport is not in configuration)
- resource "grafana_rule_group" "catalog_teleport" {
- disable_provenance = true -> null
- folder_uid = "afhbgu9xft7uob" -> null
- id = "0:afhbgu9xft7uob:catalog-teleport-paging" -> null
- interval_seconds = 60 -> null
- name = "catalog-teleport-paging" -> null
- org_id = "1" -> null
- rule {
- annotations = {
- "description" = "The Teleport process is down/unscrapeable on shared01 (standalone single-replica). ALL SSO-mediated kube and SSH access to the fleet is gone — break-glass-adjacent blast radius, hence a Teleport-specific page even though chart KubePodNotReady also fires. no_data_state=Alerting catches a vanished target."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "Teleport is down on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "5m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "teleport"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "TeleportDown" -> null
- no_data_state = "Alerting" -> null
- uid = "bfq15flgx94hsb" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "max(up{cluster=\"mzla-eks-shared01\",namespace=\"teleport\"})"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 1,
]
- type = "lt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
- rule {
- annotations = {
- "description" = "Teleport's own health verdict (process_state) is sustained degraded (>=2) for 10m: up enough to scrape but not serving correctly (backend unreachable, cert problems). More precise than up==0. for=10m avoids paging on transient recovering/starting during a restart."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "Teleport process_state is degraded on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "10m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "teleport"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "TeleportProcessDegraded" -> null
- no_data_state = "OK" -> null
- uid = "efq15flgx94hte" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "max(process_state{cluster=\"mzla-eks-shared01\",namespace=\"teleport\"})"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 1,
]
- type = "gt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
}
# grafana_rule_group.catalog_traefik will be destroyed
# (because grafana_rule_group.catalog_traefik is not in configuration)
- resource "grafana_rule_group" "catalog_traefik" {
- disable_provenance = true -> null
- folder_uid = "efhbguai6gv7ka" -> null
- id = "0:efhbguai6gv7ka:catalog-traefik-paging" -> null
- interval_seconds = 60 -> null
- name = "catalog-traefik-paging" -> null
- org_id = "1" -> null
- rule {
- annotations = {
- "description" = "No Traefik pod is reporting up{namespace=\"traefik\"} for 5m on mzla-eks-shared01. Traefik is the single NLB ingress for shared01 (victoriametrics.infra.thunderbird.net, sso.infra.thunderbird.net) — all externally-routed services are unreachable. no_data_state=Alerting so a vanished scrape target also fires."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "Traefik ingress is down on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "5m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "traefik"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "TraefikDown (shared01)" -> null
- no_data_state = "Alerting" -> null
- uid = "ffq15fldqdslcb" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "sum(up{cluster=\"mzla-eks-shared01\",namespace=\"traefik\"})"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 1,
]
- type = "lt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
- rule {
- annotations = {
- "description" = "Sustained >5% 5xx at the web/websecure entrypoints on mzla-eks-shared01 for 10m = Traefik or all backends failing user-facing requests. Entrypoint-level (not per-service) so a single noisy backend won't trip it. no_data_state=OK so no traffic does not fire."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "Traefik entrypoint 5xx rate > 5% on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "OK" -> null
- for = "10m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "traefik"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "TraefikHighEntrypointErrorRate (shared01)" -> null
- no_data_state = "OK" -> null
- uid = "cfq15flg0sg00d" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "sum(rate(traefik_entrypoint_requests_total{cluster=\"mzla-eks-shared01\",code=~\"5..\",entrypoint=~\"web|websecure\"}[5m])) / sum(rate(traefik_entrypoint_requests_total{cluster=\"mzla-eks-shared01\",entrypoint=~\"web|websecure\"}[5m]))"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 0.05,
]
- type = "gt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
}
# grafana_rule_group.catalog_victoriametrics will be destroyed
# (because grafana_rule_group.catalog_victoriametrics is not in configuration)
- resource "grafana_rule_group" "catalog_victoriametrics" {
- disable_provenance = true -> null
- folder_uid = "ffhbgu8hreubka" -> null
- id = "0:ffhbgu8hreubka:catalog-victoriametrics-paging" -> null
- interval_seconds = 60 -> null
- name = "catalog-victoriametrics-paging" -> null
- org_id = "1" -> null
- rule {
- annotations = {
- "description" = "No vminsert replica is up. vminsert is the sole ingestion entrypoint for ALL clusters remote-writing to shared01 — all replicas down = fleet-wide observability blackout. Gated on total-down to avoid paging on a rolling restart."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "All vminsert replicas are down on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "3m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "victoriametrics"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "VMClusterInsertPathDown" -> null
- no_data_state = "Alerting" -> null
- uid = "bfq15fleaczcwb" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "count(up{job=~\".*vminsert.*\"} == 1)"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 1,
]
- type = "lt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
- rule {
- annotations = {
- "description" = "No vmselect replica is up. vmselect serves every query — Grafana, vmalert evaluation, the external read path. All down = dashboards and alert eval go blind. D7 CAVEAT: this rule queries the VM datasource, so if vmselect is the eval backend and it is fully down this rule may not fire reliably; true external deadman is tracked in the self-monitoring follow-up."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "All vmselect replicas are down on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "3m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "victoriametrics"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "VMClusterSelectPathDown" -> null
- no_data_state = "Alerting" -> null
- uid = "dfq15flecuvpcb" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "count(up{job=~\".*vmselect.*\"} == 1)"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 1,
]
- type = "lt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
- rule {
- annotations = {
- "description" = "3 vmstorage replicas, replicationFactor=2. Losing one is tolerated; losing two means writes cannot satisfy RF and queries may return partial results — active data-loss risk. Single-node-down is intentionally NOT paged."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "Fewer than 2 vmstorage replicas up on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "5m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "victoriametrics"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "VMStorageInsufficientReplicas" -> null
- no_data_state = "Alerting" -> null
- uid = "ffq15flecuvpda" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "count(up{job=~\".*vmstorage.*\"} == 1)"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 2,
]
- type = "lt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
- rule {
- annotations = {
- "description" = "vmstorage flips read-only when disk fills, stalling fleet-wide ingestion. 85% used (data / (data + free)) sustained for 30m so it only fires on a real fill, leaving runway to expand the PVC."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "vmstorage disk > 85% full on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "30m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "victoriametrics"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "VMStorageDiskRunsOutOfSpace" -> null
- no_data_state = "OK" -> null
- uid = "cfq15flefcs1sa" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "sum(vm_data_size_bytes{job=~\".*vmstorage.*\"}) / (sum(vm_free_disk_space_bytes{job=~\".*vmstorage.*\"}) + sum(vm_data_size_bytes{job=~\".*vmstorage.*\"}))"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 1800 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 1800 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 0.85,
]
- type = "gt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 1800 -> null
- to = 0 -> null
}
}
}
- rule {
- annotations = {
- "description" = "A cluster's vmagent is DROPPING samples (queue full / persistently rejected) — that data is gone forever and the cluster silently disappears from observability with no backfill. The firing series' `cluster` label identifies which. for=10m rides out a brief queue blip during a shared01 restart."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "A vmagent is dropping remote-write samples"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "10m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "victoriametrics"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "VMAgentRemoteWriteDroppingData" -> null
- no_data_state = "OK" -> null
- uid = "bfq15flefcs1td" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "sum(increase(vmagent_remotewrite_packets_dropped_total[5m])) by (cluster)"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 0,
]
- type = "gt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
}
# grafana_rule_group.keycloak_dr_backup will be updated in-place
~ resource "grafana_rule_group" "keycloak_dr_backup" {
id = "0:dfhbgu8xprhtsb:keycloak-dr-backup"
name = "keycloak-dr-backup"
# (4 unchanged attributes hidden)
~ rule {
name = "KeycloakRealmBackupStale"
# (10 unchanged attributes hidden)
~ data {
~ model = jsonencode(
~ {
~ expr = "(time() - max(keycloak_backup_last_success_timestamp_seconds{realm=\"aws-sso\"}) > 26 * 3600) or absent_over_time(keycloak_backup_success{realm=\"aws-sso\"}[26h])" -> "time() - max_over_time(keycloak_backup_last_success_timestamp_seconds{realm=\"aws-sso\"}[26h])"
# (4 unchanged attributes hidden)
}
)
# (3 unchanged attributes hidden)
~ relative_time_range {
~ from = 600 -> 94320
# (1 unchanged attribute hidden)
}
}
~ data {
~ model = jsonencode(
~ {
~ conditions = [
~ {
~ evaluator = {
~ params = [
~ 0 -> 93600,
]
# (1 unchanged attribute hidden)
}
# (4 unchanged attributes hidden)
},
]
# (4 unchanged attributes hidden)
}
)
# (3 unchanged attributes hidden)
# (1 unchanged block hidden)
}
# (1 unchanged block hidden)
}
}
Plan: 0 to add, 1 to change, 7 to destroy.
Plan: 0 to add, 1 to change, 7 to destroy.
|
Contributor
Author
|
atlantis unlock |
|
All Atlantis locks for this PR have been unlocked and plans discarded |
VERDICT: conversion artifact, NOT a real stale backup. The keycloak-dr-backup
CronJob is healthy (every 6h, last runs Complete, latest S3 aws-sso realm
export ~80m old, metric last sample ~6m ago).
Root cause: keycloak_backup_* is PUSHED once per 6h run, so it's present only
within VM's ~5m instant-staleness window — empty for ~5h55m of every cycle. The
VMRule's `(time()-max(...) > 26h) or absent_over_time([26h])` returns a series
only when firing; vmalert treats an empty result as OK, but the verbatim
Grafana port turns the healthy empty result into NoData -> no_data_state=Alerting
-> a spurious ticket/page every inter-push gap.
Fix: query A now = `time() - max_over_time(keycloak_backup_last_success_timestamp_seconds{realm="aws-sso"}[26h])`
(the real age, tolerant of the sparse push), threshold C `> 93600` (26h), and
NoData (no sample in 26h at all = backup genuinely stopped) keeps
no_data_state=Alerting. This reproduces both original clauses without the
spurious fire. relative_time_range.from raised to 94320 to cover the 26h lookback.
The other two converted rules (cert-expiry, tailscale) query continuously-scraped
metrics, so they don't hit this and are unchanged.
Refs thunderbird/platform-infrastructure#559, #367.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
aatchison
force-pushed
the
fix/559-keycloak-backup-stale-nodata
branch
from
June 23, 2026 19:55
a4db4b4 to
f8199cc
Compare
|
Ran Plan for project: Show OutputTerraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# grafana_rule_group.keycloak_dr_backup will be updated in-place
~ resource "grafana_rule_group" "keycloak_dr_backup" {
id = "0:dfhbgu8xprhtsb:keycloak-dr-backup"
name = "keycloak-dr-backup"
# (4 unchanged attributes hidden)
~ rule {
name = "KeycloakRealmBackupStale"
# (10 unchanged attributes hidden)
~ data {
~ model = jsonencode(
~ {
~ expr = "(time() - max(keycloak_backup_last_success_timestamp_seconds{realm=\"aws-sso\"}) > 26 * 3600) or absent_over_time(keycloak_backup_success{realm=\"aws-sso\"}[26h])" -> "time() - max_over_time(keycloak_backup_last_success_timestamp_seconds{realm=\"aws-sso\"}[26h])"
# (4 unchanged attributes hidden)
}
)
# (3 unchanged attributes hidden)
~ relative_time_range {
~ from = 600 -> 94320
# (1 unchanged attribute hidden)
}
}
~ data {
~ model = jsonencode(
~ {
~ conditions = [
~ {
~ evaluator = {
~ params = [
~ 0 -> 93600,
]
# (1 unchanged attribute hidden)
}
# (4 unchanged attributes hidden)
},
]
# (4 unchanged attributes hidden)
}
)
# (3 unchanged attributes hidden)
# (1 unchanged block hidden)
}
# (1 unchanged block hidden)
}
}
Plan: 0 to add, 1 to change, 0 to destroy.
Plan: 0 to add, 1 to change, 0 to destroy.
|
mzeier
approved these changes
Jun 23, 2026
Contributor
Author
|
atlantis apply |
|
Ran Apply for project: ╷
│ Warning: Deprecated Parameter
│
│ The parameter "dynamodb_table" is deprecated. Use parameter "use_lockfile"
│ instead.
╵
grafana_rule_group.keycloak_dr_backup: Modifying... [id=0:dfhbgu8xprhtsb:keycloak-dr-backup]
grafana_rule_group.keycloak_dr_backup: Modifications complete after 1s [id=1:dfhbgu8xprhtsb:keycloak-dr-backup]
Apply complete! Resources: 0 added, 1 changed, 0 destroyed. |
|
Locks and plans deleted for the projects and workspaces modified in this pull request:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Verdict: conversion artifact, not a real stale backup
KeycloakRealmBackupStale(groupkeycloak-dr-backup) fires withgrafana_state_reason=NoData/[no value], but the DR backup is healthy:keycloak-dr-backupruns every 6h, not suspended; last 3 jobs Complete (latest ~80m ago). Only Failed jobs are 18d old (the VM-outage era).s3://mzla-keycloak-dr-use1/realm-exports/aws-sso/...export ~80 min old (inside 26h).keycloak_backup_last_success_timestamp_seconds{realm="aws-sso"}last sample ~6 min ago.Root cause
keycloak_backup_*is pushed once per 6h run, so a bare instant selector only sees it ~5 min per 6h cycle. The VMRule's(time()-max(...) > 26h) or absent_over_time([26h])returns a series only when firing; vmalert reads empty as OK, but the verbatim Grafana port turns the healthy empty result into NoData →no_data_state=Alerting→ a page every inter-push gap.Fix
time() - max_over_time(keycloak_backup_last_success_timestamp_seconds{realm="aws-sso"}[26h])(real age, tolerant of the sparse push).> 93600(26h).Alerting— now means "no sample in 26h at all" = genuineabsent_over_timecase.relative_time_range.from→94320to cover the 26h lookback.Other two converted rules (cert-expiry, tailscale) query continuously-scraped metrics → unaffected, unchanged.
No
atlantis apply/ no merge — for review. Refs thunderbird/platform-infrastructure#559 , thunderbird/platform-infrastructure#367 .🤖 Generated with Claude Code