alertmanager-webhook-ntfy

What is this?

Alertmanager does not have a built-in receiver for ntfy. Thus, I have decided to create my own receiver that exposes a webhook for Alertmanager to distribute alerts to. This bridge receives incoming webhooks and routes them to specific ntfy topics based on a customizable user-defined YAML file. The configuration file can define multiple rules to route alerts to different topics, to change their ntfy priority, and more.

This bridge works well and has been tested with the kube-prometheus-stack.

Features

• YAML Configuration: Kubernetes engineers will love the YAML configuration that feels like home.. because it's YAML.
• Dynamic Routing: Alerts can be dynamically routed to different ntfy topics such as k8s-core or k8s-workloads based on alert names, labels, severities, or other things while using wildcards!
• Overriding Rules: Rules are evaluated top to bottom, meaning the lowest rule is applied on top of all. If you match alertname=KubeAPI* to apply a ntfy priority of 4, but you later match a label severity=critical to apply a priority of 5, the priority of 5 wins.
• Template Rendering: Prettify ntfy notifications by creating your own alert message and inserting Alertmanager payload variables. (For example: Alert Name: {commonLabels.alertname} will replace {..} with the payload value when rendered.)

Configuration

See config.yaml for an example config file.

Alertmanager Configuration

Alertmanager needs to be configured to send a webhook payload to alertmanager-webhook-ntfy.

webhook_configs:
  - url: 'http://<ip>:8080/alertmanager'
  send_resolved: true

More information in the [Deployment](## Deployment) section.

Deployment Instructions

This application was designed to be easily deployed in a Kubernetes cluster.

The recommended method of deploying is by using Kustomize, but you can deploy the container however you want.

kustomization.yaml

kustomization.yaml:

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: monitoring

resources:
  - deployment.yaml
  - svc.yaml
  - sealedsecret.yaml
  - alertmanagerconfig.yaml

configMapGenerator:
    - name: routing-config
      files:
        - config.yaml

deployment.yaml

deployment.yaml:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: alertmanager-webhook-ntfy-deployment
  labels:
    app: alertmanager-webhook-ntfy-deployment
spec:
  revisionHistoryLimit: 1
  replicas: 1
  selector:
    matchLabels:
      app: alertmanager-webhook-ntfy-pod
  template:
    metadata:
      labels:
        app: alertmanager-webhook-ntfy-pod
    spec:
      containers:
        - name: alertmanager-webhook-ntfy
          image: ghcr.io/cmdr-johnalex/alertmanager-webhook-ntfy:latest
          envFrom:
            - secretRef:
                name: alertmanager-webhook-ntfy-env-secrets
          ports:
            - containerPort: 8080
              name: http-webhook
          volumeMounts:
            - mountPath: /usr/src/app/config.yaml
              subPath: config.yaml
              name: config-volume
      volumes:
        - name: config-volume
          configMap:
            name: routing-config

svc.yaml

svc.yaml:

---
apiVersion: v1
kind: Service
metadata:
  name: alertmanager-webhook-ntfy-svc
spec:
  type: ClusterIP
  selector:
    app: alertmanager-webhook-ntfy-pod
  ports:
    - name: http-webhook-out
      protocol: TCP
      port: 80
      targetPort:  http-webhook

sealedsecret.yaml

sealedsecret.yaml:

Assuming you have SealedSecrets set up for your cluster, you can make one here that defines NTFY_TOKEN or NTFY_USERNAME and NTFY_PASSWORD depending on your authentication method.

If you do not have SealedSecrets set up, you can create a regular Kubernetes Secret with the variables defined there and apply that to your cluster.

If you do not wish to use Secrets or SealedSecrets at all, you can modify your deployment to accept env vars directly instead of envFrom.

          envFrom:
            - secretRef:
                name: alertmanager-webhook-ntfy-env-secrets

becomes

          env:
            - name: NTFY_TOKEN
              value: "tk_XXXX"
            - name: NTFY_USERNAME
              value: "ntfyadmin"
            - name: NTFY_PASSWORD
              value: "password"

However, this is NOT recommended!

alertmanagerconfig.yaml

alertmanagerconfig.yaml:

---
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig

metadata:
  name: alertmanager-webhook-ntfy
  namespace: kube-system

spec:
  route:
    receiver: alertmanager-webhook-ntfy

    groupBy:
      - alertname
      - namespace

    groupWait: 10s
    groupInterval: 5m
    repeatInterval: 12m

  receivers:
    - name: alertmanager-webhook-ntfy
      webhookConfigs:
        - url: http://alertmanager-webhook-ntfy-svc.monitoring.svc.cluster.local:80/alertmanager
          sendResolved: true

---
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig

metadata:
  name: alertmanager-webhook-ntfy
  namespace: monitoring

spec:
  route:
    receiver: alertmanager-webhook-ntfy

    groupBy:
      - alertname
      - namespace

    groupWait: 10s
    groupInterval: 5m
    repeatInterval: 12m

  receivers:
    - name: alertmanager-webhook-ntfy
      webhookConfigs:
        - url: http://alertmanager-webhook-ntfy-svc.monitoring.svc.cluster.local:80/alertmanager
          sendResolved: true

Notice that the file defines two AlertmanagerConfig with one in the kube-system namespace, and the other in the monitoring namespace?

Due to how kube-prometheus-stack is set up by default, you need an AlertmanagerConfig in each namespace where PrometheusRules are defined.

Thank you

If you are considering using this in your own setup, thank you! I hope this project is of use to you.