[aatchison]

What

Excludes name="thundermail-ticket-spike-monitor" from the ArgoCDApplicationDegraded paging query so it no longer pages platform on-call.

- sum by (name) (argocd_app_info{cluster="mzla-eks-shared01",health_status="Degraded"})
+ sum by (name) (argocd_app_info{cluster="mzla-eks-shared01",health_status="Degraded",name!="thundermail-ticket-spike-monitor"})

Why

It's a non-critical CronJob (Zendesk ticket-count poller). A single transient run failure leaves a Failed Job in history, holding the ArgoCD app Degraded for up to ~45m (failedJobsHistoryLimit: 3 × 15-min cadence), which exceeds the alert's for: 15m and pages on-call for a low-stakes monitor.

Scope / trade-off

This fully suppresses the page and the alert for this one app; every other shared01 app still pages per-app as before. Accepted because the monitor is non-critical. The durable fix (make the monitor resilient / stop retaining failed Jobs) is tracked in thunderbird/platform-infrastructure#611.

Verification

Atlantis plan should show a single in-place update to grafana_rule_group.catalog_argocd (0 add / 0 destroy).

🤖 Generated with Claude Code

[mzla-atlantis]

Ran Plan for project: grafana dir: terraform workspace: default

Show Output

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
~ update in-place

Terraform will perform the following actions:

  # grafana_rule_group.catalog_argocd will be updated in-place
~ resource "grafana_rule_group" "catalog_argocd" {
        id                 = "1:efhbgu7k4a2o0b:catalog-argocd-paging"
        name               = "catalog-argocd-paging"
        # (4 unchanged attributes hidden)

      ~ rule {
            name                            = "ArgoCDApplicationDegraded"
            # (10 unchanged attributes hidden)

          ~ data {
              ~ model          = jsonencode(
                  ~ {
                      ~ expr       = "sum by (name) (argocd_app_info{cluster=\"mzla-eks-shared01\",health_status=\"Degraded\"})" -> "sum by (name) (argocd_app_info{cluster=\"mzla-eks-shared01\",health_status=\"Degraded\",name!=\"thundermail-ticket-spike-monitor\"})"
                        # (3 unchanged attributes hidden)
                    }
                )
                # (3 unchanged attributes hidden)

                # (1 unchanged block hidden)
            }

            # (2 unchanged blocks hidden)
        }

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

• ▶️ To apply this plan, comment:

  atlantis apply -p grafana

• 🚮 To delete this plan and lock, click here
• 🔁 To plan this project again, comment:

  atlantis plan -p grafana

Plan: 0 to add, 1 to change, 0 to destroy.

---

• ⏩ To apply all unapplied plans from this Pull Request, comment:

  atlantis apply

• 🚮 To delete all plans and locks from this Pull Request, comment:

  atlantis unlock

[aatchison]

atlantis apply

[mzla-atlantis]

Ran Apply for project: grafana dir: terraform workspace: default

╷
│ Warning: Deprecated Parameter
│ 
│ The parameter "dynamodb_table" is deprecated. Use parameter "use_lockfile"
│ instead.
╵
grafana_rule_group.catalog_argocd: Modifying... [id=1:efhbgu7k4a2o0b:catalog-argocd-paging]
grafana_rule_group.catalog_argocd: Modifications complete after 1s [id=1:efhbgu7k4a2o0b:catalog-argocd-paging]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

[mzla-atlantis]

Locks and plans deleted for the projects and workspaces modified in this pull request:

• dir: terraform workspace: default