Repository navigation
feat(alerting): Grafana deadman heartbeat -> AWS watcher (#560) - #25
Merged
Merged
Conversation
mzeier
approved these changes
Jun 23, 2026
aatchison
force-pushed
the
feat/560-deadman-heartbeat-20260623-101530
branch
from
August 18, 2026 16:42
53e74d7 to
de0fb4c
Compare
Contributor
Author
|
atlantis plan |
1 similar comment
Contributor
Author
|
atlantis plan |
|
Ran Plan for dir: Plan Error |
Contributor
Author
|
atlantis plan |
|
Ran Plan for dir: Plan Error |
|
Ran Plan for dir: Plan Error Show Output |
aatchison
force-pushed
the
feat/560-deadman-heartbeat-20260623-101530
branch
from
August 18, 2026 16:48
de0fb4c to
5e6930d
Compare
Contributor
Author
|
atlantis plan |
|
Ran Plan for dir: Plan Error |
|
Ran Plan for dir: Plan Error Show Output |
Adds the Grafana side of the alerting deadman/heartbeat: - terraform/deadman.tf: a grafana_rule_group `deadman-heartbeat` whose query A is `vector(1)` executed AGAINST the VictoriaMetrics datasource by UID (not a pure in-process __expr__ node) so a stopped heartbeat proves ruler + datasource + delivery (plan B1). Labels deadman=true, severity=heartbeat (deliberately NOT page|critical). Plus a webhook contact point `deadman-ingest` that POSTs each heartbeat to the off-cluster AWS watcher's ingest URL with a bearer token; URL+token read from Secrets Manager mzla/shared-services/grafana-terraform-deadman-url (named under the grafana-terraform* prefix so the Atlantis task role reads it with NO IAM change — plan B2). - terraform/alerting.tf: a nested deadman=true route added to the existing grafana_notification_policy.root singleton (sibling of the page|critical and warning/ticket routes — NOT re-created; existing routes preserved). Tight timers (group_wait 0s, group/repeat 1m). The notification policy root is NOT otherwise modified. terraform fmt clean. The AWS-native watcher is in platform-infrastructure pulumi/environments/grafana-deadman/ (separate PR). Nothing applied (no atlantis apply). Sequence: build+populate the watcher ingest secret first, then apply this. Refs thunderbird/platform-infrastructure#560 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
aatchison
force-pushed
the
feat/560-deadman-heartbeat-20260623-101530
branch
from
August 19, 2026 05:31
5e6930d to
9f85d06
Compare
Contributor
Author
|
atlantis plan |
Contributor
Author
|
atlantis plan |
Contributor
Author
|
atlantis plan |
|
Ran Plan for project: Show OutputTerraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+ create
~ update in-place
Terraform will perform the following actions:
# grafana_contact_point.deadman_ingest will be created
+ resource "grafana_contact_point" "deadman_ingest" {
+ disable_provenance = true
+ id = (known after apply)
+ name = "deadman-ingest"
+ webhook {
# At least one attribute in this block is (or was) sensitive,
# so its contents will not be displayed.
}
}
# grafana_notification_policy.root will be updated in-place
~ resource "grafana_notification_policy" "root" {
id = "0:policy"
# (7 unchanged attributes hidden)
+ policy {
+ contact_point = "deadman-ingest"
+ continue = false
+ group_by = [
+ "alertname",
]
+ group_interval = "1m"
+ group_wait = "0s"
+ repeat_interval = "1m"
+ matcher {
+ label = "deadman"
+ match = "="
+ value = "true"
}
}
# (4 unchanged blocks hidden)
}
# grafana_rule_group.deadman_heartbeat will be created
+ resource "grafana_rule_group" "deadman_heartbeat" {
+ disable_provenance = true
+ folder_uid = "bfhbgu9i6fjswa"
+ id = (known after apply)
+ interval_seconds = 30
+ name = "deadman-heartbeat"
+ rule {
+ annotations = {
+ "description" = "Emits a heartbeat to the AWS-native deadman watcher every cycle. If this stops, the watcher pages on-call. Do NOT pause or delete this rule (issue 560). Runbook: docs/observability.md '## Alerting & Paging' -> Deadman."
+ "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/blob/main/docs/observability.md#deadman--self-monitoring"
+ "summary" = "Grafana alerting deadman heartbeat (always firing while healthy)"
}
+ condition = "C"
+ exec_err_state = "Error"
+ is_paused = false
+ labels = {
+ "deadman" = "true"
+ "severity" = "heartbeat"
}
+ name = "GrafanaAlertingDeadmanHeartbeat"
+ no_data_state = "OK"
+ uid = (known after apply)
+ data {
+ datasource_uid = "P4169E866C3094E38"
+ model = jsonencode(
{
+ datasource = {
+ type = "prometheus"
+ uid = "P4169E866C3094E38"
}
+ expr = "vector(1)"
+ instant = true
+ refId = "A"
}
)
+ ref_id = "A"
+ relative_time_range {
+ from = 600
+ to = 0
}
}
+ data {
+ datasource_uid = "__expr__"
+ model = jsonencode(
{
+ datasource = {
+ type = "__expr__"
+ uid = "__expr__"
}
+ expression = "A"
+ reducer = "last"
+ refId = "B"
+ type = "reduce"
}
)
+ ref_id = "B"
+ relative_time_range {
+ from = 600
+ to = 0
}
}
+ data {
+ datasource_uid = "__expr__"
+ model = jsonencode(
{
+ conditions = [
+ {
+ evaluator = {
+ params = [
+ 0,
]
+ type = "gt"
}
+ operator = {
+ type = "and"
}
+ query = {
+ params = [
+ "C",
]
}
+ reducer = {
+ params = []
+ type = "last"
}
+ type = "query"
},
]
+ datasource = {
+ type = "__expr__"
+ uid = "__expr__"
}
+ expression = "B"
+ refId = "C"
+ type = "threshold"
}
)
+ ref_id = "C"
+ relative_time_range {
+ from = 600
+ to = 0
}
}
}
}
Plan: 2 to add, 1 to change, 0 to destroy.
Plan: 2 to add, 1 to change, 0 to destroy.
|
Contributor
Author
|
atlantis apply |
|
Ran Apply for project: Show Output╷
│ Warning: Deprecated Parameter
│
│ The parameter "dynamodb_table" is deprecated. Use parameter "use_lockfile"
│ instead.
╵
grafana_rule_group.deadman_heartbeat: Creating...
grafana_contact_point.deadman_ingest: Creating...
grafana_contact_point.deadman_ingest: Creation complete after 0s [id=0:deadman-ingest]
grafana_notification_policy.root: Modifying... [id=0:policy]
grafana_rule_group.deadman_heartbeat: Creation complete after 0s [id=0:bfhbgu9i6fjswa:deadman-heartbeat]
grafana_notification_policy.root: Modifications complete after 0s [id=0:policy]
Apply complete! Resources: 2 added, 1 changed, 0 destroyed. |
|
Locks and plans deleted for the projects and workspaces modified in this pull request:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Grafana side of the alerting deadman/heartbeat (issue https://github.com/thunderbird/platform-infrastructure/issues/560). A heartbeat that stops is what pages: while the alerting stack is healthy this rule fires continuously and POSTs to an off-cluster AWS watcher; if Grafana, the ruler, the VictoriaMetrics datasource, or shared01 egress is down, the heartbeat stops and the watcher pages.
The AWS-native watcher half (Lambda + EventBridge Scheduler + DynamoDB + PagerDuty) is a separate PR: https://github.com/thunderbird/platform-infrastructure/pull/572
Refs https://github.com/thunderbird/platform-infrastructure/issues/560
Changes
terraform/deadman.tf(new):grafana_rule_group.deadman_heartbeat— query A isvector(1)executed against the VictoriaMetrics datasource by UID (var.prometheus_datasource_uid), reduced (B) and thresholded> 0(C). Not a pure in-process__expr__node — a datasource-traversing query is required (plan B1) so a stopped heartbeat proves ruler + datasource + delivery. Labelsdeadman=true,severity=heartbeat(deliberately NOTpage|critical, so it never hits the real PD route). Reuses thecore_servicesfolder.grafana_contact_point.deadman_ingest— webhook POSTing each heartbeat to the watcher ingest URL with a bearer token. URL + token read from Secrets Managermzla/shared-services/grafana-terraform-deadman-url, named under thegrafana-terraform*prefix so the Atlantis ECS task role reads it with no IAM change (plan B2).terraform/alerting.tf: a nesteddeadman=trueroute added to the existinggrafana_notification_policy.rootsingleton — a sibling of thepage|criticalandwarning/ticketroutes. The root tree and existing routes are unchanged (plan B3: the singleton is not re-created; the notification policy root is otherwise untouched). Tight timers (group_wait=0s,group/repeat=1m).terraform fmtclean.Atlantis plan expectations
When this plans: 1 sensitive data-source read; create
grafana_contact_point.deadman_ingest+grafana_rule_group.deadman_heartbeat; the notification policy shows the imported baseline plus the newdeadman=trueroute with 0 destroys of the existingpage|critical/warning/ticketroutes.Sequencing / not applied
Nothing is applied — no
atlantis apply. The heartbeat begins firing on apply, so build the watcher (PR #572) and populatemzla/shared-services/grafana-terraform-deadman-urlbefore applying this, or accept benign delivery failures until the ingest URL exists. Recommended order: watcher up → secret populated → apply this.Human step
Create the dedicated PagerDuty Grafana Deadman - Platform Infrastructure service + Events API v2 routing key (used by the watcher, not by this repo). Documented in the watcher PR /
docs/observability.md.🤖 Generated with Claude Code