Skip to content

Commit e61da7c

Browse files
aatchisonclaude
andauthored
fix(alerting): de-page thundermail-ticket-spike-monitor (ArgoCD Degraded) (#30)
Exclude name="thundermail-ticket-spike-monitor" from the ArgoCDApplicationDegraded paging query. It is a non-critical CronJob (Zendesk ticket-count poller) whose transient run failures leave a Failed Job in history, holding the ArgoCD app Degraded for up to ~45m (failedJobsHistoryLimit:3 x 15m cadence) -> exceeds the alert's for:15m -> pages platform on-call for a low-stakes monitor. This fully suppresses the page (and alert) for that one app; all other shared01 apps still page per-app as before. Accepted because the monitor is non-critical; the durable fix (make the monitor resilient / not retain failed Jobs) is tracked in thunderbird/platform-infrastructure#611. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 76d0346 commit e61da7c

1 file changed

Lines changed: 5 additions & 1 deletion

File tree

‎terraform/alerting-catalog-paging.tf‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1211,6 +1211,10 @@ resource "grafana_rule_group" "catalog_argocd" {
12111211
# `sum by (name)` keeps one series per ArgoCD app so the alert fires as a
12121212
# separate instance per degraded app (label `name`), surfaced in the page
12131213
# via {{ $labels.name }} and the per-app notification route in alerting.tf.
1214+
# thundermail-ticket-spike-monitor is excluded: it is a non-critical CronJob
1215+
# (Zendesk count poller) whose transient run failures hold the ArgoCD app
1216+
# Degraded for up to ~45m and page platform on-call. De-paged per
1217+
# thunderbird/platform-infrastructure#611 (durable fix = monitor resilience).
12141218
data {
12151219
ref_id = "A"
12161220
datasource_uid = var.prometheus_datasource_uid
@@ -1221,7 +1225,7 @@ resource "grafana_rule_group" "catalog_argocd" {
12211225
model = jsonencode({
12221226
refId = "A"
12231227
datasource = { type = "prometheus", uid = var.prometheus_datasource_uid }
1224-
expr = "sum by (name) (argocd_app_info{cluster=\"mzla-eks-shared01\",health_status=\"Degraded\"})"
1228+
expr = "sum by (name) (argocd_app_info{cluster=\"mzla-eks-shared01\",health_status=\"Degraded\",name!=\"thundermail-ticket-spike-monitor\"})"
12251229
instant = true
12261230
intervalMs = 1000
12271231
maxDataPoints = 43200

0 commit comments

Comments
 (0)