The Problem

I had just merged a nice batch of disk-usage alerts: PVCs filling up, node filesystem usage, and LINSTOR pool stats. Prometheus rules looked good, alerts showed up as firing in the Prometheus UI, job done.

Then I went looking for where those alerts actually went.

receivers:
- name: "null"
route:
  receiver: "null"
  routes:
  - matchers:
    - alertname = "Watchdog"
    receiver: "null"

That’s the default Alertmanager config kube-prometheus-stack renders if you don’t give it one. Every alert routed to a receiver that does nothing. That includes Watchdog, the one alert whose entire purpose is to prove the pipeline works. The cluster had been dutifully detecting problems for eight months and telling absolutely no one.

Monitoring without alerting is just a dashboard you forget to look at.

What I Wanted

Three properties, in order of importance:

1. Alerts reach my phone, at home or on cellular, with priority that means something (critical wakes me up, warning doesn’t).
2. Silence is never ambiguous. If I hear nothing, it should mean nothing is wrong, not that the cluster, Alertmanager, or my internet is down.
3. Alert content stays mine. Alert text is full of hostnames, PVC names, and error messages. I’d rather not hand that to a third-party push service in the clear.

Why ntfy

ntfy is a single Go binary with a SQLite cache that turns HTTP POSTs into phone push notifications. It idles at a few dozen megabytes of RAM, has Android and iOS apps, and anything that can curl can use it. Beyond Alertmanager, it’s also where backup jobs, Renovate, and n8n workflows can report.

The public instance at ntfy.sh works fine, but on the free tier anyone who guesses a topic name can read it, and there’s a daily message cap. An alert storm is exactly when you’d hit it. Self-hosting fixes both: auth-default-access: deny-all, one admin user for me, and a publish-only token for Alertmanager.

Kustomize, Not Helm

The ntfy docs offer a Kustomize example and point to a community Helm chart. There’s no official chart, and the app is five manifests (Deployment, PVC, Service, ConfigMap, HTTPRoute), so pulling in a third-party chart buys nothing. It drops straight into my existing pattern: a directory under apps/ntfy/envs/metal/ that the ArgoCD ApplicationSet discovers automatically.

The upstream example needed rework anyway: its image pin was more than a major version behind, it hardcodes local-path storage (which pins the pod to one node), and it uses Ingress where I use Gateway API.

A few details worth copying:

spec:
  replicas: 1 # SQLite cache/auth DB is single-writer
  strategy:
    type: Recreate
  template:
    spec:
      containers:
        - name: ntfy
          image: binwiederhier/ntfy:v2.31.1
          args: ["serve", "--config", "/etc/ntfy/server.yml"]
          envFrom:
            - secretRef:
                name: ntfy-auth  # NTFY_AUTH_USERS / NTFY_AUTH_TOKENS
          securityContext:
            readOnlyRootFilesystem: true

Recent ntfy versions can provision users, ACLs, and tokens declaratively with auth-users, auth-access, auth-tokens in the config, or the matching NTFY_AUTH_* environment variables. That’s exactly what GitOps wants: bcrypt hashes and the token go in a Sealed Secret, the ACLs go in plain server.yml, and there’s no kubectl exec ntfy user add state living only on a PVC.

auth-file: /var/cache/ntfy/user.db
auth-default-access: deny-all
auth-access:
  - "alertmanager:homelab-*:wo"
upstream-base-url: https://ntfy.sh   # iOS instant push relay

That last line is the one concession to iOS: Apple only delivers push through APNs, so a self-hosted server forwards a poll request (topic hash and message ID, no content) to ntfy.sh, which wakes the phone, which then fetches the real message from my server.

Wiring Alertmanager: The Namespace Matcher Trap

The natural move with the Prometheus Operator is an AlertmanagerConfig custom resource. My Alertmanager has alertmanagerConfigSelector: {}, so it picks them up from any namespace.

Here’s the catch that isn’t obvious until you read the generated config: the operator injects a namespace="<the CR's namespace>" matcher into every selector-discovered AlertmanagerConfig. That’s a sensible multi-tenant default, since one team’s config shouldn’t hijack another team’s alerts. But it means an AlertmanagerConfig in the prometheus-operator namespace only ever sees alerts from the prometheus-operator namespace.

The fix is a field on the Alertmanager CR itself:

apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
  name: prometheus-operator-kube-p-alertmanager
spec:
  alertmanagerConfiguration:
    name: homelab

The referenced AlertmanagerConfig becomes the global config, with no namespace scoping. It replaces the chart’s default secret entirely, so remember to carry over the inhibit rules (critical suppresses warning, and so on). Otherwise you silently lose them.

The routing ended up as:

Alertmanager 0.26’s webhook receiver can’t set custom headers, but ntfy reads almost everything from query parameters too, and it ships a built-in Alertmanager template that turns the webhook JSON into a readable notification:

webhookConfigs:
  - url: https://ntfy.example.com/homelab-critical?template=alertmanager&priority=5&tags=rotating_light
    sendResolved: true
    httpConfig:
      authorization:
        type: Bearer
        credentials:
          name: alertmanager-receivers
          key: ntfy-token

The Dead Man’s Switch

Self-hosted alerting has an obvious hole: if the cluster dies, it takes the alerting with it. Silence then means “everything is fine” and “everything is on fire”, which violates property #2.

The standard answer is a dead man’s switch: an alert that is always firing, sent to something outside your infrastructure that complains when it stops hearing from you. kube-prometheus-stack already ships the always-firing half: Watchdog.

For the outside half I used healthchecks.io. Alertmanager posts Watchdog to a ping URL; if the pings stop for longer than the period plus grace, healthchecks.io sends a notification through its native ntfy integration pointed at public ntfy.sh on a long random topic, plus email.

graph LR
    P[Prometheus] --> AM[Alertmanager]
    AM -->|alerts| NT[self-hosted ntfy]
    AM -->|Watchdog ping| HC[healthchecks.io]
    AM -->|delivery failing| NS[ntfy.sh]
    AM -->|Watchdog heartbeat| NT
    HC -->|pings stopped| NS
    HC -->|pings stopped| EM[email]
    NT --> PH[phone]
    NS --> PH

The public topic only ever carries “your homelab stopped checking in”, so the privacy trade-off is fine. That one path catches a dead cluster, a broken Alertmanager, and a home internet outage.

The other gap is the self-hosted ntfy itself: if only ntfy is down, Alertmanager keeps pinging healthchecks.io happily while every real alert bounces. The plan was for the upstream AlertmanagerFailedToSendAlerts rule to cover that: Alertmanager notices its own delivery failures and reports them to ntfy.sh instead. Test 3 had other ideas.

The Repeat-Interval Quirk

I set the Watchdog route to groupInterval: 1m and repeatInterval: 1m, expecting a ping every minute. healthchecks.io showed one every two.

Alertmanager only evaluates repeats on group-interval ticks, and at each tick the time since the last send comes out a few milliseconds under the repeat interval, so it skips a tick. Rather than tuning intervals, I set the healthchecks.io period to 2 minutes. With a 10-minute grace window, a 1-minute ping buys nothing anyway.

The Split-DNS Twist

My original plan had a neat trick: point Alertmanager at ntfy’s public URL. If Cloudflare, the public gateway, or my WAN ingress broke, delivery would fail exactly as it would for my phone on cellular, and the fallback route would fire.

Then I remembered how my DNS works. My internal BIND9 is authoritative for the zone, and the split-horizon setup only publishes internal records for HTTPRoutes on the private gateway. The first version of ntfy had only a public route, so it didn’t resolve on my LAN at all, for phones on Wi-Fi or for Alertmanager.

Adding a private HTTPRoute (one-line diff: cilium-gateway-public → cilium-gateway) fixed resolution, but quietly killed the trick: inside the cluster the name now resolves to the private gateway, and Alertmanager never touches the public path.

So the public path needed its own check, from genuinely outside. healthchecks.io only receives pings, so I wrote a tiny Cloudflare Worker on a cron trigger:

async function probe() {
  try {
    const r = await fetch("https://ntfy.example.com/v1/health", {
      cf: { cacheTtl: 0 },
      signal: AbortSignal.timeout(10000),
    });
    return r.ok && (await r.json()).healthy === true;
  } catch (e) {
    return false;
  }
}

export default {
  async scheduled(event, env, ctx) {
    await fetch(env.HC_URL + ((await probe()) ? "" : "/fail"));
  },
};

Every five minutes it fetches the health endpoint through Cloudflare’s edge, the same way my phone would on cellular, and reports into a second healthchecks.io check. A failed probe hits /fail for an immediate alert; a dead Worker (or a Cloudflare outage) simply stops pinging, and the grace window catches that too. It runs comfortably inside the Workers free tier: 288 runs a day against a 100,000 limit.

Testing It

An untested alerting pipeline is a hope, not a pipeline. Three tests:

1. Happy path: POST a synthetic alert to Alertmanager’s API with a five-minute endsAt. A warning arrived on my phone, followed five minutes later by the resolved notification.
2. Dead man’s switch: scale Alertmanager to zero and wait for healthchecks.io to complain. The last ping landed at 21:38; at 21:50, right on the 2-minute period plus 10-minute grace, “homelab-deadman is DOWN” showed up on my phone through ntfy.sh, with the email right behind it.
3. ntfy outage: scale ntfy to zero, fire a test alert, and wait for AlertmanagerFailedToSendAlerts on ntfy.sh. It never came.

The bonus result came first: 27 seconds after ntfy went down, the Cloudflare Worker’s probe got a 503 and healthchecks.io emailed me that ntfy-public was down. I hadn’t even planned to test that path.

Alertmanager was doing its part too. The logs showed Notify attempt failed, will retry later ... unexpected status code 503. But the alert sat in pending, reset, went pending again, and reset again, every five minutes.

The upstream rule computes a failure rate over a 5-minute window, split by reason:

rate(alertmanager_notifications_failed_total[5m])
  / ignoring (reason) group_left
rate(alertmanager_notifications_total[5m]) > 0.01
for: 5m

On a busy cluster that’s fine. On mine, exactly one alert was trying to reach ntfy, and Alertmanager only gives up and counts a failure once per retry cycle, which is the 5-minute group interval. So each reason saw about one failure per window, and alternated between serverError and other. The rate dropped back to zero right around when for: 5m would have been satisfied.

The bigger problem was hiding behind that one: even if it worked, it would only notice ntfy being down while some other alert happened to be sending. On a quiet night ntfy could be dead for days and I’d find out when a real alert went missing.

The fix was two small changes. Watchdog now also posts to an ntfy topic nobody subscribes to, every two minutes (the 1m intervals below land every two, same quirk as before), with ?cache=no&firebase=no so nothing is stored or pushed:

- receiver: ntfy-heartbeat
  matchers:
    - name: alertname
      value: Watchdog
      matchType: "="
  groupInterval: 1m
  repeatInterval: 1m

That gives Alertmanager steady traffic to fail on. And a rule that doesn’t care about reason, routed to the ntfy.sh fallback:

- alert: AlertmanagerWebhookFailing
  expr: |
    sum without (reason) (
      increase(alertmanager_notifications_failed_total{integration="webhook"}[10m])
    ) > 0
  for: 2m
  labels:
    severity: critical

For the record, I left the upstream rule running through all of this. Twenty minutes of steady 503s, and AlertmanagerFailedToSendAlerts never once left pending.

Retest with ntfy at zero again, and this time nothing else firing. By 22:27 every failure in the 10-minute window came from the heartbeat (8.4 of them), and the alert held on that alone. The push that landed on ntfy.sh had a passenger: the upstream AlertmanagerFailedToSendAlerts had finally fired too, 25 minutes after it started flapping. With a failure every couple of minutes it got the steady rate it needed, so the heartbeat fixed both rules.

Where that leaves things:

Lessons

• Check where your alerts go, not just whether they fire. The kube-prometheus-stack default is a polite /dev/null.
• alertmanagerConfiguration beats selector-discovered AlertmanagerConfigs for a single-tenant cluster. No surprise namespace matchers, but carry the inhibit rules over.
• Your monitoring can’t be its own dead man’s switch. Put the “I stopped hearing from you” check somewhere that shares no failure modes with the cluster.
• Split DNS means two paths. If internal and external clients resolve the same name differently, test both, from where each client actually sits.
• Run the failure tests. Each one I ran surfaced something: an unresolvable hostname, a timing quirk, and an alert that could never fire.

Outstanding Work

1. Wire up the other senders. K8up backup results, Renovate, and n8n should report to ntfy instead of logging into the void
2. Fix the alert links. Alertmanager’s externalUrl is cluster-internal, so “view in Alertmanager” from a notification goes nowhere on a phone
3. Upgrade kube-prometheus-stack. Alertmanager 0.26 is old enough that custom webhook headers aren’t an option

References

• https://docs.ntfy.sh/
• https://prometheus-operator.dev/docs/developer/alerting/
• https://healthchecks.io/docs/