Repository navigation
Commit 76d0346
fix(alerting): name degraded ArgoCD app + repair 3 NoData-broken rules (#29)
* feat(alerting): name the degraded app in ArgoCDApplicationDegraded page
The alert collapsed `argocd_app_info{...health_status="Degraded"}` to a
single scalar via `sum(...)`, so a page could only say "an app is
Degraded" with no way to tell which one.
- Query: `sum(...)` -> `sum by (name) (...)` so the rule fires one alert
instance per degraded app, carrying the `name` label.
- Annotations: summary/description template `{{ $labels.name }}`.
- Notification policy: add a route (above the generic severity=page route)
matching alertname=ArgoCDApplicationDegraded, grouping by
["alertname","name"] -> one PagerDuty incident per degraded app, each
auto-resolving independently.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fmt: move comment out of jsonencode alignment block
terraform fmt -check flagged the inline comment splitting the model{}
'=' alignment group; relocate it above the data{} block.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(alerting): repair 3 NoData-broken rules (tailscale ops, ESO, euc1 cert)
Investigation of the NoData/spurious alerts surfaced three genuinely broken
rules (the kube/keycloak count-based rules are healthy-by-design — empty =
nothing wrong — and were left unchanged). All three new expressions were
validated against live VictoriaMetrics (each returns one series = 0 healthy).
- TailscaleOperatorDown: was firing permanently. `sum(up{namespace="tailscale",
job=~".*tailscale-operator.*"})` matches nothing — the operator isn't scraped,
so there is no `up` series on any cluster; the empty result became
NoData -> no_data_state=Alerting. Rewrite to
`1 - max(kube_deployment_status_replicas_available{cluster="mzla-eks-shared01",
namespace="tailscale", deployment="operator"})` (continuously scraped; 0 when
healthy, 1 when down, NoData only if the Deployment is deleted -> Alerting).
- TailscaleOAuthSecretSyncErrors: silently blind. Metric name was wrong
(`external_secrets_sync_calls_error_total` doesn't exist; real name is
`externalsecret_sync_calls_error`) and ESO labels the target ns as
`exported_namespace`, not `namespace`. Repoint to the continuously-present
`externalsecret_status_condition{exported_namespace="tailscale",
name="operator-oauth", condition="Ready", status="False"}` (Ready=False signal).
- EUC1CertNotReady: could never fire (logic bug) AND showed NoData when healthy.
`{condition="True"} == 0` emits value 0 when not-ready, which never crosses the
gt-0 threshold. Switch to `{condition="False"}` (1 when not Ready, 0 when Ready)
and no_data_state NoData -> OK.
- EUC1CertExpiringIn14Days: healthy state showed NoData; no_data_state -> OK
(expr already returns a value within the 14-day window).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>1 parent f8defc5 commit 76d0346
3 files changed
Lines changed: 53 additions & 12 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1203 | 1203 | | |
1204 | 1204 | | |
1205 | 1205 | | |
1206 | | - | |
1207 | | - | |
| 1206 | + | |
| 1207 | + | |
1208 | 1208 | | |
1209 | 1209 | | |
1210 | 1210 | | |
| 1211 | + | |
| 1212 | + | |
| 1213 | + | |
1211 | 1214 | | |
1212 | 1215 | | |
1213 | 1216 | | |
| |||
1218 | 1221 | | |
1219 | 1222 | | |
1220 | 1223 | | |
1221 | | - | |
| 1224 | + | |
1222 | 1225 | | |
1223 | 1226 | | |
1224 | 1227 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
140 | 140 | | |
141 | 141 | | |
142 | 142 | | |
143 | | - | |
144 | | - | |
| 143 | + | |
| 144 | + | |
145 | 145 | | |
146 | 146 | | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
147 | 155 | | |
148 | 156 | | |
149 | 157 | | |
| |||
154 | 162 | | |
155 | 163 | | |
156 | 164 | | |
157 | | - | |
| 165 | + | |
158 | 166 | | |
159 | 167 | | |
160 | 168 | | |
| |||
282 | 290 | | |
283 | 291 | | |
284 | 292 | | |
285 | | - | |
286 | | - | |
| 293 | + | |
| 294 | + | |
287 | 295 | | |
288 | 296 | | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
289 | 304 | | |
290 | 305 | | |
291 | 306 | | |
| |||
296 | 311 | | |
297 | 312 | | |
298 | 313 | | |
299 | | - | |
| 314 | + | |
300 | 315 | | |
301 | 316 | | |
302 | 317 | | |
| |||
358 | 373 | | |
359 | 374 | | |
360 | 375 | | |
361 | | - | |
| 376 | + | |
362 | 377 | | |
363 | 378 | | |
364 | 379 | | |
| |||
431 | 446 | | |
432 | 447 | | |
433 | 448 | | |
434 | | - | |
| 449 | + | |
435 | 450 | | |
436 | 451 | | |
437 | 452 | | |
| |||
444 | 459 | | |
445 | 460 | | |
446 | 461 | | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
447 | 467 | | |
448 | 468 | | |
449 | 469 | | |
| |||
454 | 474 | | |
455 | 475 | | |
456 | 476 | | |
457 | | - | |
| 477 | + | |
458 | 478 | | |
459 | 479 | | |
460 | 480 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
76 | 76 | | |
77 | 77 | | |
78 | 78 | | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
79 | 97 | | |
80 | 98 | | |
81 | 99 | | |
| |||
0 commit comments