Repository navigation
feat(alerting): Cloudflare Access gate probe alerts, paused (platform-infrastructure#1200) - #53
Merged
Merged
Conversation
…-infrastructure#1200) Three rules against the blackbox probe metric job=cloudflare-access-gate: CloudflareAccessGateOpen (200, page), CloudflareAccessGateProbeBroken (non-200 failure, warning) and CloudflareAccessGateProbeAbsent (no series, warning). All ship is_paused = true: until the gate is armed (platform-infrastructure#1205) a 200 is the correct reading.
|
Ran Plan for project: Show OutputTerraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# grafana_rule_group.cloudflare_access_gate will be created
+ resource "grafana_rule_group" "cloudflare_access_gate" {
+ disable_provenance = true
+ folder_uid = "bfhbgu9i6fjswa"
+ id = (known after apply)
+ interval_seconds = 60
+ name = "cloudflare-access-gate"
+ rule {
+ annotations = {
+ "description" = "The gate probe got HTTP 200 from {{ $labels.instance }} for 5 minutes instead of a 302 to thunderbird.cloudflareaccess.com: the host is answering without a Google login. Likely cause: the gated CloudflareAccessApplication CR (argocd/cloudflare-access/applications/) was pruned or deleted, or the operator lost its token and tore the application down. Check: kubectl -n cloudflare-zero-trust-system get cloudflareaccessapplication on mzla-eks-shared01 (every CR needs a non-empty ID and Available=True), the cloudflare-access-applications ArgoCD app, and Zero Trust > Access > Applications. Restore the CR via git, not by hand: selfHeal will revert a hand-made app. Runbook: docs/cloudflare-access.md."
+ "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/blob/main/docs/observability.md#synthetic-probes-blackbox-exporter"
+ "summary" = "Cloudflare Access gate is OPEN on {{ $labels.instance }}"
}
+ condition = "C"
+ exec_err_state = "Error"
+ for = "5m"
+ is_paused = true
+ labels = {
+ "cluster" = "mzla-eks-shared01"
+ "service" = "cloudflare-access"
+ "severity" = "page"
}
+ name = "CloudflareAccessGateOpen"
+ no_data_state = "OK"
+ uid = (known after apply)
+ data {
+ datasource_uid = "P4169E866C3094E38"
+ model = jsonencode(
{
+ datasource = {
+ type = "prometheus"
+ uid = "P4169E866C3094E38"
}
+ expr = "count by (instance) (probe_http_status_code{job=\"cloudflare-access-gate\"} == 200)"
+ instant = true
+ refId = "A"
}
)
+ ref_id = "A"
+ relative_time_range {
+ from = 600
+ to = 0
}
}
+ data {
+ datasource_uid = "__expr__"
+ model = jsonencode(
{
+ datasource = {
+ type = "__expr__"
+ uid = "__expr__"
}
+ expression = "A"
+ reducer = "last"
+ refId = "B"
+ type = "reduce"
}
)
+ ref_id = "B"
+ relative_time_range {
+ from = 600
+ to = 0
}
}
+ data {
+ datasource_uid = "__expr__"
+ model = jsonencode(
{
+ conditions = [
+ {
+ evaluator = {
+ params = [
+ 0,
]
+ type = "gt"
}
+ operator = {
+ type = "and"
}
+ query = {
+ params = [
+ "C",
]
}
+ reducer = {
+ params = []
+ type = "last"
}
+ type = "query"
},
]
+ datasource = {
+ type = "__expr__"
+ uid = "__expr__"
}
+ expression = "B"
+ refId = "C"
+ type = "threshold"
}
)
+ ref_id = "C"
+ relative_time_range {
+ from = 600
+ to = 0
}
}
}
+ rule {
+ annotations = {
+ "description" = "The gate probe for {{ $labels.instance }} has failed for 15 minutes with something other than HTTP 200, so the gate is currently UNMONITORED (this is not a fail-open). Check probe_http_status_code{job=\"cloudflare-access-gate\"}: 403 means Super Bot Fight Mode challenged it, i.e. the WAF skip rule for /api/health is gone or no longer matches mzla/shared-services/metabase-gate-probe (header_value); 0 means timeout, DNS or TLS failure from shared01; 302 with probe_success 0 means the redirect no longer points at thunderbird.cloudflareaccess.com; 5xx means Metabase or the tunnel is down. Runbook: docs/observability.md Synthetic probes."
+ "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/blob/main/docs/observability.md#synthetic-probes-blackbox-exporter"
+ "summary" = "Cloudflare Access gate probe is broken for {{ $labels.instance }}"
}
+ condition = "C"
+ exec_err_state = "Error"
+ for = "15m"
+ is_paused = true
+ labels = {
+ "cluster" = "mzla-eks-shared01"
+ "service" = "cloudflare-access"
+ "severity" = "warning"
}
+ name = "CloudflareAccessGateProbeBroken"
+ no_data_state = "OK"
+ uid = (known after apply)
+ data {
+ datasource_uid = "P4169E866C3094E38"
+ model = jsonencode(
{
+ datasource = {
+ type = "prometheus"
+ uid = "P4169E866C3094E38"
}
+ expr = "count by (instance) (probe_success{job=\"cloudflare-access-gate\"} == 0 and probe_http_status_code{job=\"cloudflare-access-gate\"} != 200)"
+ instant = true
+ refId = "A"
}
)
+ ref_id = "A"
+ relative_time_range {
+ from = 600
+ to = 0
}
}
+ data {
+ datasource_uid = "__expr__"
+ model = jsonencode(
{
+ datasource = {
+ type = "__expr__"
+ uid = "__expr__"
}
+ expression = "A"
+ reducer = "last"
+ refId = "B"
+ type = "reduce"
}
)
+ ref_id = "B"
+ relative_time_range {
+ from = 600
+ to = 0
}
}
+ data {
+ datasource_uid = "__expr__"
+ model = jsonencode(
{
+ conditions = [
+ {
+ evaluator = {
+ params = [
+ 0,
]
+ type = "gt"
}
+ operator = {
+ type = "and"
}
+ query = {
+ params = [
+ "C",
]
}
+ reducer = {
+ params = []
+ type = "last"
}
+ type = "query"
},
]
+ datasource = {
+ type = "__expr__"
+ uid = "__expr__"
}
+ expression = "B"
+ refId = "C"
+ type = "threshold"
}
)
+ ref_id = "C"
+ relative_time_range {
+ from = 600
+ to = 0
}
}
}
+ rule {
+ annotations = {
+ "description" = "No probe_http_status_code{job=\"cloudflare-access-gate\"} series for 15 minutes: the gate is UNMONITORED. Likely causes on mzla-eks-shared01: the blackbox-exporter pod is down or stuck in ContainerCreating (the blackbox-exporter-config ExternalSecret failed to sync from mzla/shared-services/metabase-gate-probe), the VMProbe cloudflare-access-gate in namespace monitoring was deleted, or VMAgent is not scraping it. Check the blackbox-exporter and observability ArgoCD apps. Runbook: docs/observability.md Synthetic probes."
+ "runbook_url" = "https://github.com/thunderbird/platform-infrastructure/blob/main/docs/observability.md#synthetic-probes-blackbox-exporter"
+ "summary" = "Cloudflare Access gate probe has no data"
}
+ condition = "C"
+ exec_err_state = "Error"
+ for = "15m"
+ is_paused = true
+ labels = {
+ "cluster" = "mzla-eks-shared01"
+ "service" = "cloudflare-access"
+ "severity" = "warning"
}
+ name = "CloudflareAccessGateProbeAbsent"
+ no_data_state = "OK"
+ uid = (known after apply)
+ data {
+ datasource_uid = "P4169E866C3094E38"
+ model = jsonencode(
{
+ datasource = {
+ type = "prometheus"
+ uid = "P4169E866C3094E38"
}
+ expr = "absent(probe_http_status_code{job=\"cloudflare-access-gate\"})"
+ instant = true
+ refId = "A"
}
)
+ ref_id = "A"
+ relative_time_range {
+ from = 600
+ to = 0
}
}
+ data {
+ datasource_uid = "__expr__"
+ model = jsonencode(
{
+ datasource = {
+ type = "__expr__"
+ uid = "__expr__"
}
+ expression = "A"
+ reducer = "last"
+ refId = "B"
+ type = "reduce"
}
)
+ ref_id = "B"
+ relative_time_range {
+ from = 600
+ to = 0
}
}
+ data {
+ datasource_uid = "__expr__"
+ model = jsonencode(
{
+ conditions = [
+ {
+ evaluator = {
+ params = [
+ 0,
]
+ type = "gt"
}
+ operator = {
+ type = "and"
}
+ query = {
+ params = [
+ "C",
]
}
+ reducer = {
+ params = []
+ type = "last"
}
+ type = "query"
},
]
+ datasource = {
+ type = "__expr__"
+ uid = "__expr__"
}
+ expression = "B"
+ refId = "C"
+ type = "threshold"
}
)
+ ref_id = "C"
+ relative_time_range {
+ from = 600
+ to = 0
}
}
}
}
Plan: 1 to add, 0 to change, 0 to destroy.
Plan: 1 to add, 0 to change, 0 to destroy.
|
|
Locks and plans deleted for the projects and workspaces modified in this pull request:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The alert half of thunderbird/platform-infrastructure#1200. It reads the metric from the blackbox probe added in thunderbird/platform-infrastructure#1261 (
job="cloudflare-access-gate").New file
terraform/alerting-cloudflare-access.tfadds onegrafana_rule_groupin Core Services, against the VictoriaMetrics datasource:forCloudflareAccessGateOpencount by (instance) (probe_http_status_code{job="cloudflare-access-gate"} == 200)pagepagerduty-platform-infraCloudflareAccessGateProbeBrokencount by (instance) (probe_success{...} == 0 and probe_http_status_code{...} != 200)warningpagerduty-platform-infra-low(Slack)CloudflareAccessGateProbeAbsentabsent(probe_http_status_code{job="cloudflare-access-gate"})warningpagerduty-platform-infra-low(Slack)The rules read status codes rather than
probe_successalone, becauseprobe_successis 0 both when the gate is open and when the probe is broken. A 403 SBFM challenge or a timeout lands in ProbeBroken and never pages. All three useno_data_state = "OK", because an empty result is the healthy state for each, andexec_err_state = "Error", as the other paging rules do. The notification policy is unchanged; existing severity routes handle all three.Ships paused
Every rule has
is_paused = true. Until thunderbird/platform-infrastructure#1205 arms the gate,200is the correct reading, andCloudflareAccessGateOpenwould page nonstop. So this can merge and apply ahead of #1205 and the probe, and it is inert until unpaused.Unpausing happens in follow-up PRs:
probe_http_status_codehas read302steadily.Verification
terraform fmtandterraform validatepass locally (v1.15.8).grafana_rule_group.cloudflare_access_gateto add.Refs thunderbird/platform-infrastructure#1200