Repository navigation
fix(alerting): name degraded ArgoCD app + repair 3 NoData-broken rules - #29
Merged
Merged
Conversation
The alert collapsed `argocd_app_info{...health_status="Degraded"}` to a
single scalar via `sum(...)`, so a page could only say "an app is
Degraded" with no way to tell which one.
- Query: `sum(...)` -> `sum by (name) (...)` so the rule fires one alert
instance per degraded app, carrying the `name` label.
- Annotations: summary/description template `{{ $labels.name }}`.
- Notification policy: add a route (above the generic severity=page route)
matching alertname=ArgoCDApplicationDegraded, grouping by
["alertname","name"] -> one PagerDuty incident per degraded app, each
auto-resolving independently.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
terraform fmt -check flagged the inline comment splitting the model{}
'=' alignment group; relocate it above the data{} block.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Ran Plan for project: Show OutputTerraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# grafana_notification_policy.root will be updated in-place
~ resource "grafana_notification_policy" "root" {
id = "0:policy"
# (7 unchanged attributes hidden)
~ policy {
~ group_by = [
"alertname",
+ "name",
]
# (7 unchanged attributes hidden)
- matcher {
- label = "severity" -> null
- match = "=~" -> null
- value = "page|critical" -> null
}
+ matcher {
+ label = "alertname"
+ match = "="
+ value = "ArgoCDApplicationDegraded"
}
}
~ policy {
~ contact_point = "pagerduty-platform-infra-low" -> "pagerduty-platform-infra"
# (7 unchanged attributes hidden)
- matcher {
- label = "severity" -> null
- match = "=" -> null
- value = "warning" -> null
}
+ matcher {
+ label = "severity"
+ match = "=~"
+ value = "page|critical"
}
}
~ policy {
# (8 unchanged attributes hidden)
- matcher {
- label = "severity" -> null
- match = "=" -> null
- value = "ticket" -> null
}
+ matcher {
+ label = "severity"
+ match = "="
+ value = "warning"
}
}
+ policy {
+ contact_point = "pagerduty-platform-infra-low"
+ continue = false
+ group_by = [
+ "alertname",
]
+ matcher {
+ label = "severity"
+ match = "="
+ value = "ticket"
}
}
}
# grafana_rule_group.catalog_argocd will be updated in-place
~ resource "grafana_rule_group" "catalog_argocd" {
id = "0:efhbgu7k4a2o0b:catalog-argocd-paging"
name = "catalog-argocd-paging"
# (4 unchanged attributes hidden)
~ rule {
~ annotations = {
~ "description" = "An ArgoCD app has been health_status=Degraded for 15m — a tracked resource failed its runtime health check (crash-loop, failed readiness, stuck PVC) even if Git-synced. Sustained >15m usually means a real managed service (Traefik/Keycloak/vmauth) is broken. Matches only Degraded (excludes transient Progressing/Missing)." -> "ArgoCD application {{ $labels.name }} has been health_status=Degraded for 15m — a tracked resource failed its runtime health check (crash-loop, failed readiness, stuck PVC) even if Git-synced. Sustained >15m usually means a real managed service (Traefik/Keycloak/vmauth) is broken. Matches only Degraded (excludes transient Progressing/Missing)."
~ "summary" = "An ArgoCD application is Degraded on shared01" -> "ArgoCD application {{ $labels.name }} is Degraded on shared01"
# (1 unchanged element hidden)
}
name = "ArgoCDApplicationDegraded"
# (9 unchanged attributes hidden)
~ data {
~ model = jsonencode(
~ {
~ expr = "sum(argocd_app_info{cluster=\"mzla-eks-shared01\",health_status=\"Degraded\"})" -> "sum by (name) (argocd_app_info{cluster=\"mzla-eks-shared01\",health_status=\"Degraded\"})"
# (3 unchanged attributes hidden)
}
)
# (3 unchanged attributes hidden)
# (1 unchanged block hidden)
}
# (2 unchanged blocks hidden)
}
# (1 unchanged block hidden)
}
Plan: 0 to add, 2 to change, 0 to destroy.
Plan: 0 to add, 2 to change, 0 to destroy.
|
1 similar comment
|
Ran Plan for project: Show OutputTerraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# grafana_notification_policy.root will be updated in-place
~ resource "grafana_notification_policy" "root" {
id = "0:policy"
# (7 unchanged attributes hidden)
~ policy {
~ group_by = [
"alertname",
+ "name",
]
# (7 unchanged attributes hidden)
- matcher {
- label = "severity" -> null
- match = "=~" -> null
- value = "page|critical" -> null
}
+ matcher {
+ label = "alertname"
+ match = "="
+ value = "ArgoCDApplicationDegraded"
}
}
~ policy {
~ contact_point = "pagerduty-platform-infra-low" -> "pagerduty-platform-infra"
# (7 unchanged attributes hidden)
- matcher {
- label = "severity" -> null
- match = "=" -> null
- value = "warning" -> null
}
+ matcher {
+ label = "severity"
+ match = "=~"
+ value = "page|critical"
}
}
~ policy {
# (8 unchanged attributes hidden)
- matcher {
- label = "severity" -> null
- match = "=" -> null
- value = "ticket" -> null
}
+ matcher {
+ label = "severity"
+ match = "="
+ value = "warning"
}
}
+ policy {
+ contact_point = "pagerduty-platform-infra-low"
+ continue = false
+ group_by = [
+ "alertname",
]
+ matcher {
+ label = "severity"
+ match = "="
+ value = "ticket"
}
}
}
# grafana_rule_group.catalog_argocd will be updated in-place
~ resource "grafana_rule_group" "catalog_argocd" {
id = "0:efhbgu7k4a2o0b:catalog-argocd-paging"
name = "catalog-argocd-paging"
# (4 unchanged attributes hidden)
~ rule {
~ annotations = {
~ "description" = "An ArgoCD app has been health_status=Degraded for 15m — a tracked resource failed its runtime health check (crash-loop, failed readiness, stuck PVC) even if Git-synced. Sustained >15m usually means a real managed service (Traefik/Keycloak/vmauth) is broken. Matches only Degraded (excludes transient Progressing/Missing)." -> "ArgoCD application {{ $labels.name }} has been health_status=Degraded for 15m — a tracked resource failed its runtime health check (crash-loop, failed readiness, stuck PVC) even if Git-synced. Sustained >15m usually means a real managed service (Traefik/Keycloak/vmauth) is broken. Matches only Degraded (excludes transient Progressing/Missing)."
~ "summary" = "An ArgoCD application is Degraded on shared01" -> "ArgoCD application {{ $labels.name }} is Degraded on shared01"
# (1 unchanged element hidden)
}
name = "ArgoCDApplicationDegraded"
# (9 unchanged attributes hidden)
~ data {
~ model = jsonencode(
~ {
~ expr = "sum(argocd_app_info{cluster=\"mzla-eks-shared01\",health_status=\"Degraded\"})" -> "sum by (name) (argocd_app_info{cluster=\"mzla-eks-shared01\",health_status=\"Degraded\"})"
# (3 unchanged attributes hidden)
}
)
# (3 unchanged attributes hidden)
# (1 unchanged block hidden)
}
# (2 unchanged blocks hidden)
}
# (1 unchanged block hidden)
}
Plan: 0 to add, 2 to change, 0 to destroy.
Plan: 0 to add, 2 to change, 0 to destroy.
|
mzeier
approved these changes
Jun 29, 2026
… cert)
Investigation of the NoData/spurious alerts surfaced three genuinely broken
rules (the kube/keycloak count-based rules are healthy-by-design — empty =
nothing wrong — and were left unchanged). All three new expressions were
validated against live VictoriaMetrics (each returns one series = 0 healthy).
- TailscaleOperatorDown: was firing permanently. `sum(up{namespace="tailscale",
job=~".*tailscale-operator.*"})` matches nothing — the operator isn't scraped,
so there is no `up` series on any cluster; the empty result became
NoData -> no_data_state=Alerting. Rewrite to
`1 - max(kube_deployment_status_replicas_available{cluster="mzla-eks-shared01",
namespace="tailscale", deployment="operator"})` (continuously scraped; 0 when
healthy, 1 when down, NoData only if the Deployment is deleted -> Alerting).
- TailscaleOAuthSecretSyncErrors: silently blind. Metric name was wrong
(`external_secrets_sync_calls_error_total` doesn't exist; real name is
`externalsecret_sync_calls_error`) and ESO labels the target ns as
`exported_namespace`, not `namespace`. Repoint to the continuously-present
`externalsecret_status_condition{exported_namespace="tailscale",
name="operator-oauth", condition="Ready", status="False"}` (Ready=False signal).
- EUC1CertNotReady: could never fire (logic bug) AND showed NoData when healthy.
`{condition="True"} == 0` emits value 0 when not-ready, which never crosses the
gt-0 threshold. Switch to `{condition="False"}` (1 when not Ready, 0 when Ready)
and no_data_state NoData -> OK.
- EUC1CertExpiringIn14Days: healthy state showed NoData; no_data_state -> OK
(expr already returns a value within the 14-day window).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
aatchison
changed the title
|
Ran Plan for project: Show OutputTerraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# grafana_notification_policy.root will be updated in-place
~ resource "grafana_notification_policy" "root" {
id = "0:policy"
# (7 unchanged attributes hidden)
~ policy {
~ group_by = [
"alertname",
+ "name",
]
# (7 unchanged attributes hidden)
- matcher {
- label = "severity" -> null
- match = "=~" -> null
- value = "page|critical" -> null
}
+ matcher {
+ label = "alertname"
+ match = "="
+ value = "ArgoCDApplicationDegraded"
}
}
~ policy {
~ contact_point = "pagerduty-platform-infra-low" -> "pagerduty-platform-infra"
# (7 unchanged attributes hidden)
- matcher {
- label = "severity" -> null
- match = "=" -> null
- value = "warning" -> null
}
+ matcher {
+ label = "severity"
+ match = "=~"
+ value = "page|critical"
}
}
~ policy {
# (8 unchanged attributes hidden)
- matcher {
- label = "severity" -> null
- match = "=" -> null
- value = "ticket" -> null
}
+ matcher {
+ label = "severity"
+ match = "="
+ value = "warning"
}
}
+ policy {
+ contact_point = "pagerduty-platform-infra-low"
+ continue = false
+ group_by = [
+ "alertname",
]
+ matcher {
+ label = "severity"
+ match = "="
+ value = "ticket"
}
}
}
# grafana_rule_group.catalog_argocd will be updated in-place
~ resource "grafana_rule_group" "catalog_argocd" {
id = "0:efhbgu7k4a2o0b:catalog-argocd-paging"
name = "catalog-argocd-paging"
# (4 unchanged attributes hidden)
~ rule {
~ annotations = {
~ "description" = "An ArgoCD app has been health_status=Degraded for 15m — a tracked resource failed its runtime health check (crash-loop, failed readiness, stuck PVC) even if Git-synced. Sustained >15m usually means a real managed service (Traefik/Keycloak/vmauth) is broken. Matches only Degraded (excludes transient Progressing/Missing)." -> "ArgoCD application {{ $labels.name }} has been health_status=Degraded for 15m — a tracked resource failed its runtime health check (crash-loop, failed readiness, stuck PVC) even if Git-synced. Sustained >15m usually means a real managed service (Traefik/Keycloak/vmauth) is broken. Matches only Degraded (excludes transient Progressing/Missing)."
~ "summary" = "An ArgoCD application is Degraded on shared01" -> "ArgoCD application {{ $labels.name }} is Degraded on shared01"
# (1 unchanged element hidden)
}
name = "ArgoCDApplicationDegraded"
# (9 unchanged attributes hidden)
~ data {
~ model = jsonencode(
~ {
~ expr = "sum(argocd_app_info{cluster=\"mzla-eks-shared01\",health_status=\"Degraded\"})" -> "sum by (name) (argocd_app_info{cluster=\"mzla-eks-shared01\",health_status=\"Degraded\"})"
# (3 unchanged attributes hidden)
}
)
# (3 unchanged attributes hidden)
# (1 unchanged block hidden)
}
# (2 unchanged blocks hidden)
}
# (1 unchanged block hidden)
}
# grafana_rule_group.euc1_cert_expiry will be updated in-place
~ resource "grafana_rule_group" "euc1_cert_expiry" {
id = "0:bfhbgu9i6fjswa:cert-expiry-euc1"
name = "cert-expiry-euc1"
# (4 unchanged attributes hidden)
~ rule {
name = "EUC1CertExpiringIn14Days"
~ no_data_state = "NoData" -> "OK"
# (9 unchanged attributes hidden)
# (3 unchanged blocks hidden)
}
~ rule {
name = "EUC1CertNotReady"
~ no_data_state = "NoData" -> "OK"
# (9 unchanged attributes hidden)
~ data {
~ model = jsonencode(
~ {
~ expr = "certmanager_certificate_ready_status{namespace=\"keycloak\", name=\"keycloak-tls\", condition=\"True\"} == 0" -> "certmanager_certificate_ready_status{namespace=\"keycloak\", name=\"keycloak-tls\", condition=\"False\"}"
# (4 unchanged attributes hidden)
}
)
# (3 unchanged attributes hidden)
# (1 unchanged block hidden)
}
# (2 unchanged blocks hidden)
}
}
# grafana_rule_group.tailscale will be updated in-place
~ resource "grafana_rule_group" "tailscale" {
id = "0:bfhbgu9i6fjswa:tailscale-operator"
name = "tailscale-operator"
# (4 unchanged attributes hidden)
~ rule {
~ annotations = {
~ "description" = "The Tailscale operator is reporting fewer than 1 up target in the tailscale namespace on shared01. Tailnet admin access (operator-managed proxies and ingress) may be impaired. Check the tailscale-operator deployment in the tailscale namespace." -> "The tailscale-operator Deployment on shared01 has <1 available replica. Tailnet admin access (operator-managed proxies and ingress) may be impaired. Check: kubectl -n tailscale get deploy operator; kubectl -n tailscale describe deploy operator."
~ "summary" = "Tailscale operator pod down >5m on shared01" -> "Tailscale operator down >5m on shared01"
}
name = "TailscaleOperatorDown"
# (9 unchanged attributes hidden)
~ data {
~ model = jsonencode(
~ {
~ expr = "sum(up{namespace=\"tailscale\", job=~\".*tailscale-operator.*\"}) < 1" -> "1 - max(kube_deployment_status_replicas_available{cluster=\"mzla-eks-shared01\", namespace=\"tailscale\", deployment=\"operator\"})"
# (4 unchanged attributes hidden)
}
)
# (3 unchanged attributes hidden)
# (1 unchanged block hidden)
}
# (2 unchanged blocks hidden)
}
~ rule {
~ annotations = {
~ "description" = "The operator-oauth ExternalSecret in the tailscale namespace is returning sync errors (external_secrets_sync_calls_error_total rate > 0 over 15m). If the OAuth client secret goes stale the Tailscale operator loses tailnet auth. Check the ExternalSecret and its SecretStore in the tailscale namespace." -> "The operator-oauth ExternalSecret (tailscale namespace) has Ready=False, so ESO is not reconciling the OAuth client secret. If the secret goes stale the Tailscale operator loses tailnet auth. Check: kubectl -n tailscale describe externalsecret operator-oauth and its SecretStore."
~ "summary" = "operator-oauth ExternalSecret returning sync errors" -> "operator-oauth ExternalSecret not Ready in tailscale namespace"
}
name = "TailscaleOAuthSecretSyncErrors"
# (9 unchanged attributes hidden)
~ data {
~ model = jsonencode(
~ {
~ expr = "rate(external_secrets_sync_calls_error_total{namespace=\"tailscale\", name=\"operator-oauth\"}[15m]) > 0" -> "externalsecret_status_condition{exported_namespace=\"tailscale\", name=\"operator-oauth\", condition=\"Ready\", status=\"False\"}"
# (4 unchanged attributes hidden)
}
)
# (3 unchanged attributes hidden)
# (1 unchanged block hidden)
}
# (2 unchanged blocks hidden)
}
# (1 unchanged block hidden)
}
Plan: 0 to add, 4 to change, 0 to destroy.
Plan: 0 to add, 4 to change, 0 to destroy.
|
Contributor
Author
|
atlantis apply |
|
Ran Apply for project: Show Output╷
│ Warning: Deprecated Parameter
│
│ The parameter "dynamodb_table" is deprecated. Use parameter "use_lockfile"
│ instead.
╵
grafana_rule_group.euc1_cert_expiry: Modifying... [id=0:bfhbgu9i6fjswa:cert-expiry-euc1]
grafana_rule_group.catalog_argocd: Modifying... [id=0:efhbgu7k4a2o0b:catalog-argocd-paging]
grafana_rule_group.tailscale: Modifying... [id=0:bfhbgu9i6fjswa:tailscale-operator]
grafana_notification_policy.root: Modifying... [id=0:policy]
grafana_notification_policy.root: Modifications complete after 1s [id=0:policy]
grafana_rule_group.tailscale: Modifications complete after 1s [id=1:bfhbgu9i6fjswa:tailscale-operator]
grafana_rule_group.catalog_argocd: Modifications complete after 1s [id=1:efhbgu7k4a2o0b:catalog-argocd-paging]
grafana_rule_group.euc1_cert_expiry: Modifications complete after 1s [id=1:bfhbgu9i6fjswa:cert-expiry-euc1]
Apply complete! Resources: 0 added, 4 changed, 0 destroyed. |
|
Locks and plans deleted for the projects and workspaces modified in this pull request:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two related alerting changes, both driven by reviewing the live firing/NoData alert state on shared01.
1. Name the degraded app in
ArgoCDApplicationDegradedThe page collapsed
argocd_app_info{...health_status="Degraded"}to a single scalar viasum(...), so it could only say "an app is Degraded" — never which one (confirmed live: the alert instance carried no app label).alerting-catalog-paging.tf— querysum(...)→sum by (name) (...)(one alert instance per degraded app);summary/descriptiontemplate{{ $labels.name }}.alerting.tf— new notification-policy route above the genericseverity=pageroute, matchingalertname=ArgoCDApplicationDegradedand grouping by["alertname","name"]→ one PagerDuty incident per degraded app, each auto-resolving independently.2. Repair 3 genuinely-broken rules (found while auditing NoData state)
Most "NoData" alerts are healthy-by-design (
count(<bad-condition>)is empty when nothing is wrong — verified the base metrics exist, so they still fire on real conditions). Three were actually broken. All three replacement expressions were validated against live VictoriaMetrics (each returns one series =0while healthy):TailscaleOperatorDown— was firing permanently.sum(up{namespace="tailscale", job=~".*tailscale-operator.*"})matches nothing (the operator isn't scraped; noupseries on any cluster) → empty → NoData →no_data_state=Alerting. Rewritten to1 - max(kube_deployment_status_replicas_available{cluster="mzla-eks-shared01", namespace="tailscale", deployment="operator"})(continuously scraped; NoData only if the Deployment is deleted → still Alerting).TailscaleOAuthSecretSyncErrors— silently blind. Wrong metric name (external_secrets_sync_calls_error_totaldoesn't exist) and ESO labels the target ns asexported_namespace, notnamespace. Repointed toexternalsecret_status_condition{exported_namespace="tailscale", name="operator-oauth", condition="Ready", status="False"}.EUC1CertNotReady— could never fire (logic bug):{condition="True"} == 0emits value0when not-ready, which never crosses thegt 0threshold. Switched to{condition="False"}(1 when not Ready, 0 when Ready);no_data_stateNoData → OK.EUC1CertExpiringIn14Days—no_data_stateNoData → OK so the healthy state reads Normal (expr already returns a value within the 14-day window).Verification
Atlantis
planshould show only in-placegrafana_rule_group/grafana_notification_policyupdates — 0 add / 0 destroy.Notes / follow-up
terraform/alerting*.tf.🤖 Generated with Claude Code