Who Watches the Watcher: Closing the Loop on Alerting

thomas@tnepeters.com (Thomas Peters) — Fri, 16 Oct 2026 00:00:00 +0000

The Problem

I had just merged a nice batch of disk-usage alerts: PVCs filling up, node filesystem usage, and LINSTOR pool stats. Prometheus rules looked good, alerts showed up as firing in the Prometheus UI, job done.

Then I went looking for where those alerts actually went.

receivers:
- name: "null"
route:
 receiver: "null"
 routes:
 - matchers:
 - alertname = "Watchdog"
 receiver: "null"

That’s the default Alertmanager config kube-prometheus-stack renders if you don’t give it one. Every alert routed to a receiver that does nothing. That includes Watchdog, the one alert whose entire purpose is to prove the pipeline works. The cluster had been dutifully detecting problems for eight months and telling absolutely no one.

Monitoring on Thomas Peters

Who Watches the Watcher: Closing the Loop on Alerting

The Problem