Skip to content

Commit 76d0346

Browse files
aatchisonclaude
andauthored
fix(alerting): name degraded ArgoCD app + repair 3 NoData-broken rules (#29)
* feat(alerting): name the degraded app in ArgoCDApplicationDegraded page The alert collapsed `argocd_app_info{...health_status="Degraded"}` to a single scalar via `sum(...)`, so a page could only say "an app is Degraded" with no way to tell which one. - Query: `sum(...)` -> `sum by (name) (...)` so the rule fires one alert instance per degraded app, carrying the `name` label. - Annotations: summary/description template `{{ $labels.name }}`. - Notification policy: add a route (above the generic severity=page route) matching alertname=ArgoCDApplicationDegraded, grouping by ["alertname","name"] -> one PagerDuty incident per degraded app, each auto-resolving independently. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fmt: move comment out of jsonencode alignment block terraform fmt -check flagged the inline comment splitting the model{} '=' alignment group; relocate it above the data{} block. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(alerting): repair 3 NoData-broken rules (tailscale ops, ESO, euc1 cert) Investigation of the NoData/spurious alerts surfaced three genuinely broken rules (the kube/keycloak count-based rules are healthy-by-design — empty = nothing wrong — and were left unchanged). All three new expressions were validated against live VictoriaMetrics (each returns one series = 0 healthy). - TailscaleOperatorDown: was firing permanently. `sum(up{namespace="tailscale", job=~".*tailscale-operator.*"})` matches nothing — the operator isn't scraped, so there is no `up` series on any cluster; the empty result became NoData -> no_data_state=Alerting. Rewrite to `1 - max(kube_deployment_status_replicas_available{cluster="mzla-eks-shared01", namespace="tailscale", deployment="operator"})` (continuously scraped; 0 when healthy, 1 when down, NoData only if the Deployment is deleted -> Alerting). - TailscaleOAuthSecretSyncErrors: silently blind. Metric name was wrong (`external_secrets_sync_calls_error_total` doesn't exist; real name is `externalsecret_sync_calls_error`) and ESO labels the target ns as `exported_namespace`, not `namespace`. Repoint to the continuously-present `externalsecret_status_condition{exported_namespace="tailscale", name="operator-oauth", condition="Ready", status="False"}` (Ready=False signal). - EUC1CertNotReady: could never fire (logic bug) AND showed NoData when healthy. `{condition="True"} == 0` emits value 0 when not-ready, which never crosses the gt-0 threshold. Switch to `{condition="False"}` (1 when not Ready, 0 when Ready) and no_data_state NoData -> OK. - EUC1CertExpiringIn14Days: healthy state showed NoData; no_data_state -> OK (expr already returns a value within the 14-day window). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent f8defc5 commit 76d0346

3 files changed

Lines changed: 53 additions & 12 deletions

File tree

‎terraform/alerting-catalog-paging.tf‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1203,11 +1203,14 @@ resource "grafana_rule_group" "catalog_argocd" {
12031203
service = "argocd"
12041204
}
12051205
annotations = {
1206-
summary = "An ArgoCD application is Degraded on shared01"
1207-
description = "An ArgoCD app has been health_status=Degraded for 15m — a tracked resource failed its runtime health check (crash-loop, failed readiness, stuck PVC) even if Git-synced. Sustained >15m usually means a real managed service (Traefik/Keycloak/vmauth) is broken. Matches only Degraded (excludes transient Progressing/Missing)."
1206+
summary = "ArgoCD application {{ $labels.name }} is Degraded on shared01"
1207+
description = "ArgoCD application {{ $labels.name }} has been health_status=Degraded for 15m — a tracked resource failed its runtime health check (crash-loop, failed readiness, stuck PVC) even if Git-synced. Sustained >15m usually means a real managed service (Traefik/Keycloak/vmauth) is broken. Matches only Degraded (excludes transient Progressing/Missing)."
12081208
runbook_url = "https://github.com/thunderbird/platform-infrastructure/issues/80"
12091209
}
12101210

1211+
# `sum by (name)` keeps one series per ArgoCD app so the alert fires as a
1212+
# separate instance per degraded app (label `name`), surfaced in the page
1213+
# via {{ $labels.name }} and the per-app notification route in alerting.tf.
12111214
data {
12121215
ref_id = "A"
12131216
datasource_uid = var.prometheus_datasource_uid
@@ -1218,7 +1221,7 @@ resource "grafana_rule_group" "catalog_argocd" {
12181221
model = jsonencode({
12191222
refId = "A"
12201223
datasource = { type = "prometheus", uid = var.prometheus_datasource_uid }
1221-
expr = "sum(argocd_app_info{cluster=\"mzla-eks-shared01\",health_status=\"Degraded\"})"
1224+
expr = "sum by (name) (argocd_app_info{cluster=\"mzla-eks-shared01\",health_status=\"Degraded\"})"
12221225
instant = true
12231226
intervalMs = 1000
12241227
maxDataPoints = 43200

‎terraform/alerting-converted-vmrules.tf‎

Lines changed: 29 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -140,10 +140,18 @@ resource "grafana_rule_group" "tailscale" {
140140
service = "tailscale"
141141
}
142142
annotations = {
143-
summary = "Tailscale operator pod down >5m on shared01"
144-
description = "The Tailscale operator is reporting fewer than 1 up target in the tailscale namespace on shared01. Tailnet admin access (operator-managed proxies and ingress) may be impaired. Check the tailscale-operator deployment in the tailscale namespace."
143+
summary = "Tailscale operator down >5m on shared01"
144+
description = "The tailscale-operator Deployment on shared01 has <1 available replica. Tailnet admin access (operator-managed proxies and ingress) may be impaired. Check: kubectl -n tailscale get deploy operator; kubectl -n tailscale describe deploy operator."
145145
}
146146

147+
# A = 1 when the operator Deployment has 0 available replicas, 0 when healthy.
148+
# The original `sum(up{namespace="tailscale", job=~".*tailscale-operator.*"})`
149+
# was a conversion artifact: the operator is NOT scraped (no `up` series exists
150+
# for the tailscale namespace on any cluster), so the verbatim port returned an
151+
# empty result -> NoData -> no_data_state=Alerting -> a permanent spurious fire.
152+
# kube_deployment_status_replicas_available is continuously scraped by
153+
# kube-state-metrics and yields a real value while the Deployment exists; it
154+
# goes NoData only if the Deployment is deleted (a genuine outage) -> Alerting.
147155
data {
148156
ref_id = "A"
149157
datasource_uid = local.victoriametrics_ds_uid
@@ -154,7 +162,7 @@ resource "grafana_rule_group" "tailscale" {
154162
model = jsonencode({
155163
refId = "A"
156164
datasource = { type = "prometheus", uid = local.victoriametrics_ds_uid }
157-
expr = "sum(up{namespace=\"tailscale\", job=~\".*tailscale-operator.*\"}) < 1"
165+
expr = "1 - max(kube_deployment_status_replicas_available{cluster=\"mzla-eks-shared01\", namespace=\"tailscale\", deployment=\"operator\"})"
158166
instant = true
159167
range = false
160168
intervalMs = 1000
@@ -282,10 +290,17 @@ resource "grafana_rule_group" "tailscale" {
282290
service = "tailscale"
283291
}
284292
annotations = {
285-
summary = "operator-oauth ExternalSecret returning sync errors"
286-
description = "The operator-oauth ExternalSecret in the tailscale namespace is returning sync errors (external_secrets_sync_calls_error_total rate > 0 over 15m). If the OAuth client secret goes stale the Tailscale operator loses tailnet auth. Check the ExternalSecret and its SecretStore in the tailscale namespace."
293+
summary = "operator-oauth ExternalSecret not Ready in tailscale namespace"
294+
description = "The operator-oauth ExternalSecret (tailscale namespace) has Ready=False, so ESO is not reconciling the OAuth client secret. If the secret goes stale the Tailscale operator loses tailnet auth. Check: kubectl -n tailscale describe externalsecret operator-oauth and its SecretStore."
287295
}
288296

297+
# A = 1 when the operator-oauth ExternalSecret is Ready=False, 0 when Ready.
298+
# The original `rate(external_secrets_sync_calls_error_total{namespace="tailscale"...})`
299+
# never matched anything: the metric is `externalsecret_sync_calls_error` (no
300+
# `external_secrets`/`_total` form exists), and ESO labels the target namespace
301+
# as `exported_namespace`, not `namespace` (which is ESO's own ns). The rule was
302+
# silently blind (no_data_state=OK). externalsecret_status_condition is
303+
# continuously scraped and present whether Ready or not.
289304
data {
290305
ref_id = "A"
291306
datasource_uid = local.victoriametrics_ds_uid
@@ -296,7 +311,7 @@ resource "grafana_rule_group" "tailscale" {
296311
model = jsonencode({
297312
refId = "A"
298313
datasource = { type = "prometheus", uid = local.victoriametrics_ds_uid }
299-
expr = "rate(external_secrets_sync_calls_error_total{namespace=\"tailscale\", name=\"operator-oauth\"}[15m]) > 0"
314+
expr = "externalsecret_status_condition{exported_namespace=\"tailscale\", name=\"operator-oauth\", condition=\"Ready\", status=\"False\"}"
300315
instant = true
301316
range = false
302317
intervalMs = 1000
@@ -358,7 +373,7 @@ resource "grafana_rule_group" "euc1_cert_expiry" {
358373
name = "EUC1CertExpiringIn14Days"
359374
condition = "C"
360375
for = "1h"
361-
no_data_state = "NoData"
376+
no_data_state = "OK"
362377
exec_err_state = "Error"
363378
labels = {
364379
severity = "ticket"
@@ -431,7 +446,7 @@ resource "grafana_rule_group" "euc1_cert_expiry" {
431446
name = "EUC1CertNotReady"
432447
condition = "C"
433448
for = "15m"
434-
no_data_state = "NoData"
449+
no_data_state = "OK"
435450
exec_err_state = "Error"
436451
labels = {
437452
severity = "ticket"
@@ -444,6 +459,11 @@ resource "grafana_rule_group" "euc1_cert_expiry" {
444459
runbook_url = "https://github.com/thunderbird/platform-infrastructure/blob/main/docs/keycloak-staff-sso.md"
445460
}
446461

462+
# condition="False" series = 1 when the cert's Ready condition is False, 0 when
463+
# Ready. The previous `{condition="True"} == 0` was a real logic bug: not-ready
464+
# made it emit value 0, which never crosses the `gt 0` threshold (and ready made
465+
# it empty), so the alert could never fire. This form emits 1 (fires) when not
466+
# Ready and 0 (Normal) when Ready.
447467
data {
448468
ref_id = "A"
449469
datasource_uid = local.victoriametrics_ds_uid
@@ -454,7 +474,7 @@ resource "grafana_rule_group" "euc1_cert_expiry" {
454474
model = jsonencode({
455475
refId = "A"
456476
datasource = { type = "prometheus", uid = local.victoriametrics_ds_uid }
457-
expr = "certmanager_certificate_ready_status{namespace=\"keycloak\", name=\"keycloak-tls\", condition=\"True\"} == 0"
477+
expr = "certmanager_certificate_ready_status{namespace=\"keycloak\", name=\"keycloak-tls\", condition=\"False\"}"
458478
instant = true
459479
range = false
460480
intervalMs = 1000

‎terraform/alerting.tf‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,24 @@ resource "grafana_notification_policy" "root" {
7676
group_interval = "5m"
7777
repeat_interval = "4h"
7878

79+
# Per-app route for ArgoCDApplicationDegraded. MUST stay ABOVE the generic
80+
# paging route below: routes are evaluated in order and this alert carries
81+
# severity=page, so the generic route would otherwise group all degraded apps
82+
# into one incident. Grouping by `name` (the per-app label from `sum by (name)`)
83+
# yields one PagerDuty incident per degraded app, each auto-resolving on its
84+
# own, with the app name in the title via {{ $labels.name }}.
85+
policy {
86+
contact_point = grafana_contact_point.pagerduty_platform_infra.name
87+
group_by = ["alertname", "name"]
88+
continue = false
89+
90+
matcher {
91+
label = "alertname"
92+
match = "="
93+
value = "ArgoCDApplicationDegraded"
94+
}
95+
}
96+
7997
policy {
8098
contact_point = grafana_contact_point.pagerduty_platform_infra.name
8199
group_by = ["alertname"]

0 commit comments

Comments
 (0)