Repository navigation
fix(alerting): concise PagerDuty/Slack summary (drop default.message dump) - #27
Merged
Merged
Conversation
mzeier
approved these changes
Jun 23, 2026
Contributor
Author
|
atlantis plan -p grafana |
Contributor
Author
|
atlantis plan -p grafana |
|
Ran Plan for project: Show OutputTerraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
~ update in-place
- destroy
Terraform will perform the following actions:
# grafana_contact_point.pagerduty_platform_infra will be updated in-place
~ resource "grafana_contact_point" "pagerduty_platform_infra" {
id = "0:pagerduty-platform-infra"
name = "pagerduty-platform-infra"
# (2 unchanged attributes hidden)
- pagerduty {
# At least one attribute in this block is (or was) sensitive,
# so its contents will not be displayed.
}
+ pagerduty {
# At least one attribute in this block is (or was) sensitive,
# so its contents will not be displayed.
}
}
# grafana_contact_point.pagerduty_platform_infra_low will be updated in-place
~ resource "grafana_contact_point" "pagerduty_platform_infra_low" {
id = "0:pagerduty-platform-infra-low"
name = "pagerduty-platform-infra-low"
# (2 unchanged attributes hidden)
- pagerduty {
# At least one attribute in this block is (or was) sensitive,
# so its contents will not be displayed.
}
+ pagerduty {
# At least one attribute in this block is (or was) sensitive,
# so its contents will not be displayed.
}
}
# grafana_rule_group.catalog_argocd will be destroyed
# (because grafana_rule_group.catalog_argocd is not in configuration)
- resource "grafana_rule_group" "catalog_argocd" {
- disable_provenance = true -> null
- folder_uid = "efhbgu7k4a2o0b" -> null
- id = "0:efhbgu7k4a2o0b:catalog-argocd-paging" -> null
- interval_seconds = 60 -> null
- name = "catalog-argocd-paging" -> null
- org_id = "1" -> null
- rule {
- annotations = {
- "description" = "An ArgoCD app has been health_status=Degraded for 15m — a tracked resource failed its runtime health check (crash-loop, failed readiness, stuck PVC) even if Git-synced. Sustained >15m usually means a real managed service (Traefik/Keycloak/vmauth) is broken. Matches only Degraded (excludes transient Progressing/Missing)."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "An ArgoCD application is Degraded on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "15m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "argocd"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "ArgoCDApplicationDegraded" -> null
- no_data_state = "OK" -> null
- uid = "afq15flidozr4f" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "sum(argocd_app_info{cluster=\"mzla-eks-shared01\",health_status=\"Degraded\"})"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 900 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 900 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 0,
]
- type = "gt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 900 -> null
- to = 0 -> null
}
}
}
- rule {
- annotations = {
- "description" = "The application-controller (reconcile/self-heal) or repo-server (manifest rendering) is fully down on shared01 — the fleet's GitOps loop stalls, drift goes uncorrected, no app can sync/self-heal across all clusters. Selector uses the live `container` label (no app_kubernetes_io_component here)."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "ArgoCD application-controller or repo-server is down on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "10m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "argocd"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "ArgoCDControllerOrRepoServerDown" -> null
- no_data_state = "Alerting" -> null
- uid = "ffq15flig6w3kd" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "min(max by (container) (up{cluster=\"mzla-eks-shared01\",namespace=\"argocd\",container=~\"application-controller|repo-server\"}))"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 1,
]
- type = "lt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
}
# grafana_rule_group.catalog_eso will be destroyed
# (because grafana_rule_group.catalog_eso is not in configuration)
- resource "grafana_rule_group" "catalog_eso" {
- disable_provenance = true -> null
- folder_uid = "bfhbgu9i6fjswa" -> null
- id = "0:bfhbgu9i6fjswa:catalog-eso-paging" -> null
- interval_seconds = 60 -> null
- name = "catalog-eso-paging" -> null
- org_id = "1" -> null
- rule {
- annotations = {
- "description" = "The ESO controller is down fleet-wide on shared01 (namespace external-secrets) — no ExternalSecret reconciles. Existing Secrets keep serving, but any rotation / new ExternalSecret / token refresh (vmauth bearer tokens, Keycloak/Tailscale OAuth) silently stops. for=10m tolerates a rollout blip (60s scrape interval)."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "External Secrets Operator is down on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "10m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "external-secrets"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "ESOControllerDown" -> null
- no_data_state = "Alerting" -> null
- uid = "bfq15fla70yyoc" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "sum(up{cluster=\"mzla-eks-shared01\",namespace=\"external-secrets\"} == 1)"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 1,
]
- type = "lt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
}
# grafana_rule_group.catalog_keycloak will be destroyed
# (because grafana_rule_group.catalog_keycloak is not in configuration)
- resource "grafana_rule_group" "catalog_keycloak" {
- disable_provenance = true -> null
- folder_uid = "dfhbgu8xprhtsb" -> null
- id = "0:dfhbgu8xprhtsb:catalog-keycloak-paging" -> null
- interval_seconds = 60 -> null
- name = "catalog-keycloak-paging" -> null
- org_id = "1" -> null
- rule {
- annotations = {
- "description" = "No Keycloak pod is up in namespace keycloak on shared01 — total staff-SSO outage, including AWS Identity Center SAML federation (aws-sso realm). The 3-replica HA StatefulSet tolerates losing 1-2 pods, so this pages only on full-fleet loss. for=2m for fast paging without flapping on rolling restarts."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/blob/main/docs/keycloak-staff-sso.md"
- "summary" = "All Keycloak replicas are down on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "2m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "keycloak"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "KeycloakAllReplicasDown" -> null
- no_data_state = "Alerting" -> null
- uid = "efq15flf9bk74a" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "sum(up{cluster=\"mzla-eks-shared01\",namespace=\"keycloak\"} == 1)"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 1,
]
- type = "lt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
- rule {
- annotations = {
- "description" = ">5% 5xx responses from Keycloak on shared01 for 10m = serving errors (DB failover, cache split-brain, OOM) even though pods are up — missed by the pod-down rule. http_server_requests_seconds_count carries a `status` label; the denominator is guarded against divide-by-zero by no_data_state=OK."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/blob/main/docs/keycloak-staff-sso.md"
- "summary" = "Keycloak 5xx rate > 5% on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "OK" -> null
- for = "10m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "keycloak"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "KeycloakHigh5xxRate" -> null
- no_data_state = "OK" -> null
- uid = "bfq15flfebcw0e" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "sum(rate(http_server_requests_seconds_count{cluster=\"mzla-eks-shared01\",namespace=\"keycloak\",status=~\"5..\"}[5m])) / sum(rate(http_server_requests_seconds_count{cluster=\"mzla-eks-shared01\",namespace=\"keycloak\"}[5m]))"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
Warning: Output length greater than max comment size. Continued in next comment. |
|
Continued plan output from previous comment. Show Output }
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 0.05,
]
- type = "gt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
}
# grafana_rule_group.catalog_kubernetes will be destroyed
# (because grafana_rule_group.catalog_kubernetes is not in configuration)
- resource "grafana_rule_group" "catalog_kubernetes" {
- disable_provenance = true -> null
- folder_uid = "dfhbgu7yr6nlsc" -> null
- id = "0:dfhbgu7yr6nlsc:catalog-kubernetes-paging" -> null
- interval_seconds = 60 -> null
- name = "catalog-kubernetes-paging" -> null
- org_id = "1" -> null
- rule {
- annotations = {
- "description" = "At least one mzla-eks-shared01 node is NotReady and not cordoned (kube_node_spec_unschedulable==0) for 10m. Workloads are evicting and cluster capacity is shrinking. Chart ships this at warning/15m and routes nowhere; this Grafana copy pages. Check node health, kubelet, and EC2 instance status."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "A shared01 node has been NotReady for 10m"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "10m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "kubernetes"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "KubeNodeNotReady (shared01)" -> null
- no_data_state = "OK" -> null
- uid = "ffq15flbkyxvka" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "count(kube_node_status_condition{cluster=\"mzla-eks-shared01\",condition=\"Ready\",status=\"true\"} == 0 and on(node) kube_node_spec_unschedulable{cluster=\"mzla-eks-shared01\"} == 0)"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 0,
]
- type = "gt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
- rule {
- annotations = {
- "description" = "node count minus live kubelet targets > 0 on mzla-eks-shared01: at least one node's kubelet is down/unscraped, so its pods are unmanaged and the control plane has lost ground truth for that node. Highest-signal node-plane outage. Chart classifies KubeletDown critical/15m but it never reaches PagerDuty."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "A shared01 node has no live kubelet scrape target for 10m"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "10m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "kubernetes"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "KubeletDown (shared01)" -> null
- no_data_state = "OK" -> null
- uid = "afq15flcwf0g0d" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "count(kube_node_info{cluster=\"mzla-eks-shared01\"}) - count(up{cluster=\"mzla-eks-shared01\",job=\"kubelet\",metrics_path=\"/metrics\"} == 1)"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 0,
]
- type = "gt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
- rule {
- annotations = {
- "description" = "A PVC on mzla-eks-shared01 has less than 5% free space and is minutes from write failures. These back stateful core infra (vmstorage 100Gi, VictoriaLogs, Keycloak, Teleport). A full volume = metrics/log ingestion loss or auth-plane corruption. Only the critical <5% variant is promoted to page; the chart's 15%-prediction warning stays on the chart path. Expand the PVC."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "A shared01 PersistentVolume is under 5% free"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "5m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "kubernetes"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "KubePersistentVolumeFillingUpCritical (shared01)" -> null
- no_data_state = "OK" -> null
- uid = "cfq15flcwf0g1a" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "count((kubelet_volume_stats_available_bytes{cluster=\"mzla-eks-shared01\"} / kubelet_volume_stats_capacity_bytes{cluster=\"mzla-eks-shared01\"}) < 0.05 and kubelet_volume_stats_used_bytes{cluster=\"mzla-eks-shared01\"} > 0)"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 0,
]
- type = "gt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
- rule {
- annotations = {
- "description" = "A container in a user-facing core namespace (monitoring/keycloak/teleport/argocd/external-secrets/kube-system/traefik) has been CrashLoopBackOff for 15m on mzla-eks-shared01. Namespace-scoped so a crashlooping Keycloak/Traefik/ArgoCD/ESO/VM pod pages while general app workloads stay on the chart's warning path. for=15m survives a rolling deploy."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "A core-namespace pod is CrashLoopBackOff for 15m on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "15m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "kubernetes"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "KubePodCrashLoopingCoreNamespaces (shared01)" -> null
- no_data_state = "OK" -> null
- uid = "bfq15flcywwsgb" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "count(max_over_time(kube_pod_container_status_waiting_reason{cluster=\"mzla-eks-shared01\",reason=\"CrashLoopBackOff\",namespace=~\"monitoring|keycloak|teleport|argocd|external-secrets|kube-system|traefik\"}[5m]) >= 1)"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 0,
]
- type = "gt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
}
# grafana_rule_group.catalog_teleport will be destroyed
# (because grafana_rule_group.catalog_teleport is not in configuration)
- resource "grafana_rule_group" "catalog_teleport" {
- disable_provenance = true -> null
- folder_uid = "afhbgu9xft7uob" -> null
- id = "0:afhbgu9xft7uob:catalog-teleport-paging" -> null
- interval_seconds = 60 -> null
- name = "catalog-teleport-paging" -> null
- org_id = "1" -> null
- rule {
- annotations = {
- "description" = "The Teleport process is down/unscrapeable on shared01 (standalone single-replica). ALL SSO-mediated kube and SSH access to the fleet is gone — break-glass-adjacent blast radius, hence a Teleport-specific page even though chart KubePodNotReady also fires. no_data_state=Alerting catches a vanished target."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "Teleport is down on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "5m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "teleport"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "TeleportDown" -> null
- no_data_state = "Alerting" -> null
- uid = "bfq15flgx94hsb" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "max(up{cluster=\"mzla-eks-shared01\",namespace=\"teleport\"})"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 1,
]
- type = "lt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
- rule {
- annotations = {
- "description" = "Teleport's own health verdict (process_state) is sustained degraded (>=2) for 10m: up enough to scrape but not serving correctly (backend unreachable, cert problems). More precise than up==0. for=10m avoids paging on transient recovering/starting during a restart."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "Teleport process_state is degraded on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "10m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "teleport"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "TeleportProcessDegraded" -> null
- no_data_state = "OK" -> null
- uid = "efq15flgx94hte" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "max(process_state{cluster=\"mzla-eks-shared01\",namespace=\"teleport\"})"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 1,
]
- type = "gt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
}
# grafana_rule_group.catalog_traefik will be destroyed
# (because grafana_rule_group.catalog_traefik is not in configuration)
- resource "grafana_rule_group" "catalog_traefik" {
- disable_provenance = true -> null
- folder_uid = "efhbguai6gv7ka" -> null
- id = "0:efhbguai6gv7ka:catalog-traefik-paging" -> null
- interval_seconds = 60 -> null
- name = "catalog-traefik-paging" -> null
- org_id = "1" -> null
- rule {
- annotations = {
- "description" = "No Traefik pod is reporting up{namespace=\"traefik\"} for 5m on mzla-eks-shared01. Traefik is the single NLB ingress for shared01 (victoriametrics.infra.thunderbird.net, sso.infra.thunderbird.net) — all externally-routed services are unreachable. no_data_state=Alerting so a vanished scrape target also fires."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "Traefik ingress is down on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "5m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "traefik"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "TraefikDown (shared01)" -> null
- no_data_state = "Alerting" -> null
- uid = "ffq15fldqdslcb" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "sum(up{cluster=\"mzla-eks-shared01\",namespace=\"traefik\"})"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 1,
]
- type = "lt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
- rule {
- annotations = {
- "description" = "Sustained >5% 5xx at the web/websecure entrypoints on mzla-eks-shared01 for 10m = Traefik or all backends failing user-facing requests. Entrypoint-level (not per-service) so a single noisy backend won't trip it. no_data_state=OK so no traffic does not fire."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "Traefik entrypoint 5xx rate > 5% on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "OK" -> null
- for = "10m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "traefik"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "TraefikHighEntrypointErrorRate (shared01)" -> null
- no_data_state = "OK" -> null
- uid = "cfq15flg0sg00d" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "sum(rate(traefik_entrypoint_requests_total{cluster=\"mzla-eks-shared01\",code=~\"5..\",entrypoint=~\"web|websecure\"}[5m])) / sum(rate(traefik_entrypoint_requests_total{cluster=\"mzla-eks-shared01\",entrypoint=~\"web|websecure\"}[5m]))"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 0.05,
]
- type = "gt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
}
# grafana_rule_group.catalog_victoriametrics will be destroyed
# (because grafana_rule_group.catalog_victoriametrics is not in configuration)
- resource "grafana_rule_group" "catalog_victoriametrics" {
- disable_provenance = true -> null
- folder_uid = "ffhbgu8hreubka" -> null
- id = "0:ffhbgu8hreubka:catalog-victoriametrics-paging" -> null
- interval_seconds = 60 -> null
- name = "catalog-victoriametrics-paging" -> null
- org_id = "1" -> null
- rule {
- annotations = {
- "description" = "No vminsert replica is up. vminsert is the sole ingestion entrypoint for ALL clusters remote-writing to shared01 — all replicas down = fleet-wide observability blackout. Gated on total-down to avoid paging on a rolling restart."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "All vminsert replicas are down on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "3m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "victoriametrics"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "VMClusterInsertPathDown" -> null
- no_data_state = "Alerting" -> null
- uid = "bfq15fleaczcwb" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "count(up{job=~\".*vminsert.*\"} == 1)"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 1,
]
- type = "lt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
- rule {
- annotations = {
- "description" = "No vmselect replica is up. vmselect serves every query — Grafana, vmalert evaluation, the external read path. All down = dashboards and alert eval go blind. D7 CAVEAT: this rule queries the VM datasource, so if vmselect is the eval backend and it is fully down this rule may not fire reliably; true external deadman is tracked in the self-monitoring follow-up."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "All vmselect replicas are down on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "3m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "victoriametrics"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "VMClusterSelectPathDown" -> null
- no_data_state = "Alerting" -> null
- uid = "dfq15flecuvpcb" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "count(up{job=~\".*vmselect.*\"} == 1)"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 1,
]
- type = "lt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
- rule {
- annotations = {
- "description" = "3 vmstorage replicas, replicationFactor=2. Losing one is tolerated; losing two means writes cannot satisfy RF and queries may return partial results — active data-loss risk. Single-node-down is intentionally NOT paged."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "Fewer than 2 vmstorage replicas up on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "5m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "victoriametrics"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "VMStorageInsufficientReplicas" -> null
- no_data_state = "Alerting" -> null
- uid = "ffq15flecuvpda" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "count(up{job=~\".*vmstorage.*\"} == 1)"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 2,
]
- type = "lt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
- rule {
- annotations = {
- "description" = "vmstorage flips read-only when disk fills, stalling fleet-wide ingestion. 85% used (data / (data + free)) sustained for 30m so it only fires on a real fill, leaving runway to expand the PVC."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "vmstorage disk > 85% full on shared01"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "30m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "victoriametrics"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "VMStorageDiskRunsOutOfSpace" -> null
- no_data_state = "OK" -> null
- uid = "cfq15flefcs1sa" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "sum(vm_data_size_bytes{job=~\".*vmstorage.*\"}) / (sum(vm_free_disk_space_bytes{job=~\".*vmstorage.*\"}) + sum(vm_data_size_bytes{job=~\".*vmstorage.*\"}))"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 1800 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 1800 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 0.85,
]
- type = "gt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 1800 -> null
- to = 0 -> null
}
}
}
- rule {
- annotations = {
- "description" = "A cluster's vmagent is DROPPING samples (queue full / persistently rejected) — that data is gone forever and the cluster silently disappears from observability with no backfill. The firing series' `cluster` label identifies which. for=10m rides out a brief queue blip during a shared01 restart."
- "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/issues/80"
- "summary" = "A vmagent is dropping remote-write samples"
} -> null
- condition = "C" -> null
- exec_err_state = "Error" -> null
- for = "10m0s" -> null
- is_paused = false -> null
- labels = {
- "cluster" = "mzla-eks-shared01"
- "service" = "victoriametrics"
- "severity" = "page"
} -> null
- missing_series_evals_to_resolve = 0 -> null
- name = "VMAgentRemoteWriteDroppingData" -> null
- no_data_state = "OK" -> null
- uid = "bfq15flefcs1td" -> null
# (1 unchanged attribute hidden)
- data {
- datasource_uid = "P4169E866C3094E38" -> null
- model = jsonencode(
{
- datasource = {
- type = "prometheus"
- uid = "P4169E866C3094E38"
}
- expr = "sum(increase(vmagent_remotewrite_packets_dropped_total[5m])) by (cluster)"
- instant = true
- refId = "A"
}
) -> null
- ref_id = "A" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "A"
- reducer = "last"
- refId = "B"
- type = "reduce"
}
) -> null
- ref_id = "B" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
- data {
- datasource_uid = "__expr__" -> null
- model = jsonencode(
{
- conditions = [
- {
- evaluator = {
- params = [
- 0,
]
- type = "gt"
}
- operator = {
- type = "and"
}
- query = {
- params = [
- "C",
]
}
- reducer = {
- params = []
- type = "last"
}
- type = "query"
},
]
- datasource = {
- type = "__expr__"
- uid = "__expr__"
}
- expression = "B"
- refId = "C"
- type = "threshold"
}
) -> null
- ref_id = "C" -> null
# (1 unchanged attribute hidden)
- relative_time_range {
- from = 600 -> null
- to = 0 -> null
}
}
}
}
Plan: 0 to add, 2 to change, 7 to destroy.
Plan: 0 to add, 2 to change, 7 to destroy.
|
…dump) (#559) Rebuilt on current main. summary on both contact points -> one-liner [severity] alertname: summary; full description + runbook stay on the PD incident. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
aatchison
force-pushed
the
fix/concise-pagerduty-summary
branch
from
June 23, 2026 19:42
4ce96fb to
d82dbb3
Compare
|
Ran Plan for project: Show OutputTerraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# grafana_contact_point.pagerduty_platform_infra will be updated in-place
~ resource "grafana_contact_point" "pagerduty_platform_infra" {
id = "0:pagerduty-platform-infra"
name = "pagerduty-platform-infra"
# (2 unchanged attributes hidden)
- pagerduty {
# At least one attribute in this block is (or was) sensitive,
# so its contents will not be displayed.
}
+ pagerduty {
# At least one attribute in this block is (or was) sensitive,
# so its contents will not be displayed.
}
}
# grafana_contact_point.pagerduty_platform_infra_low will be updated in-place
~ resource "grafana_contact_point" "pagerduty_platform_infra_low" {
id = "0:pagerduty-platform-infra-low"
name = "pagerduty-platform-infra-low"
# (2 unchanged attributes hidden)
- pagerduty {
# At least one attribute in this block is (or was) sensitive,
# so its contents will not be displayed.
}
+ pagerduty {
# At least one attribute in this block is (or was) sensitive,
# so its contents will not be displayed.
}
}
Plan: 0 to add, 2 to change, 0 to destroy.
Plan: 0 to add, 2 to change, 0 to destroy.
|
Contributor
Author
|
atlantis plan -p grafana |
|
Ran Plan for project: Show OutputTerraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# grafana_contact_point.pagerduty_platform_infra will be updated in-place
~ resource "grafana_contact_point" "pagerduty_platform_infra" {
id = "0:pagerduty-platform-infra"
name = "pagerduty-platform-infra"
# (2 unchanged attributes hidden)
- pagerduty {
# At least one attribute in this block is (or was) sensitive,
# so its contents will not be displayed.
}
+ pagerduty {
# At least one attribute in this block is (or was) sensitive,
# so its contents will not be displayed.
}
}
# grafana_contact_point.pagerduty_platform_infra_low will be updated in-place
~ resource "grafana_contact_point" "pagerduty_platform_infra_low" {
id = "0:pagerduty-platform-infra-low"
name = "pagerduty-platform-infra-low"
# (2 unchanged attributes hidden)
- pagerduty {
# At least one attribute in this block is (or was) sensitive,
# so its contents will not be displayed.
}
+ pagerduty {
# At least one attribute in this block is (or was) sensitive,
# so its contents will not be displayed.
}
}
Plan: 0 to add, 2 to change, 0 to destroy.
Plan: 0 to add, 2 to change, 0 to destroy.
|
Contributor
Author
|
atlantis apply -p grafana |
|
Ran Apply for project: ╷
│ Warning: Deprecated Parameter
│
│ The parameter "dynamodb_table" is deprecated. Use parameter "use_lockfile"
│ instead.
╵
grafana_contact_point.pagerduty_platform_infra: Modifying... [id=0:pagerduty-platform-infra]
grafana_contact_point.pagerduty_platform_infra_low: Modifying... [id=0:pagerduty-platform-infra-low]
grafana_contact_point.pagerduty_platform_infra_low: Modifications complete after 0s [id=0:pagerduty-platform-infra-low]
grafana_contact_point.pagerduty_platform_infra: Modifications complete after 1s [id=0:pagerduty-platform-infra]
Apply complete! Resources: 0 added, 2 changed, 0 destroyed. |
|
Locks and plans deleted for the projects and workspaces modified in this pull request:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The PagerDuty contact points used
summary = "{{ template \"default.message\" . }}", which dumps Value/Labels/Annotations/Source/Silence into #mzla-pages — very noisy (e.g. the KeycloakRealmBackupStale ticket that just fired).Replaces it on both
pagerduty-platform-infraandpagerduty-platform-infra-lowwith a one-line title:[severity] alertname: summary. The full description + runbook_url remain on the PagerDuty incident (one click away) — just not spammed into Slack.No routing/secret/policy changes. Refs thunderbird/platform-infrastructure#559, #80.