Checkmk out, Prometheus in: a homelab monitoring migration in three slices

For a long time my homelab had two monitoring systems. Checkmk, on two virtual machines, watched the hosts, the websites and the services. kube-prometheus-stack, inside the Kubernetes cluster, watched the cluster. Two dashboards, two places for alerts, two answers to “is it down?”.

This weekend I moved everything into the Prometheus stack and shut Checkmk down. This post is the plan, the parts I’d reuse, and the honest list of things that broke on the way. If you’re thinking about the same move, the last section is the one to read twice.

The stack, in one paragraph

kube-prometheus-stack is a Helm chart that installs Prometheus, its operator, Alertmanager and node-exporter on a Kubernetes cluster. The operator is the part that matters for this story: instead of editing one big Prometheus config file, you describe what to scrape and what to alert on as Kubernetes resources, called ScrapeConfig, Probe and PrometheusRule. Those are YAML files in Git. Argo CD applies them. That is what made a migration feel like a series of pull requests instead of a weekend of clicking. The cluster itself is described in five nodes, five jobs.

Three slices, three days of specs

I didn’t try to do it in one go. Each slice got a short design document, an implementation plan, and a rollout, with Checkmk left running until the new stack covered the same ground.

Then the Checkmk VMs were shut down and removed from the target list. Cleaning their agent role out of Ansible is a fourth slice, still to do.

Slice 1: the inventory is the source of truth

The trick I’m most pleased with is small. Ansible already knows every host I have. So instead of typing a target list into Prometheus, an Ansible role renders it from the inventory into the cluster repository:

# Generated by awx-homelab (roles/prometheus_targets) from inventory.yml.
# Do not edit by hand.
apiVersion: monitoring.coreos.com/v1alpha1
kind: ScrapeConfig
metadata:
  name: node-exporter-external
  namespace: monitoring
spec:
  staticConfigs:
    - labels:
        site: home
      targets:
        - dns00.koper.dev:9100
        - bunker00.koper.dev:9100
    - labels:
        site: proxmox
      targets:
        - pve-anton.koper.dev:9100
        - pve-fw.koper.dev:9100

Add a host to the inventory, run the monitoring play, commit the generated file, and Prometheus scrapes it. Retire a host and it disappears the same way. The site label comes from the inventory too, and it’s what most of the alert rules group on later.

The node_exporter role itself is convergent: it checks the installed version, only downloads on a mismatch, runs the service as its own user, and opens the port only to the cluster’s subnet. Grafana was already running on a VM from a hand-written compose file; the role adopted it, kept its dashboards and pointed its datasources at the in-cluster Prometheus and Alertmanager.

Slice 2: websites, and a surprise about IPv6

Checkmk had 52 website checks. They became Probe resources for the blackbox exporter, grouped by which reverse proxy serves them, with labels for hosting and certificate type. The first audit failed 20 of 52, and the reasons were interesting rather than bugs:

• Sites behind BunkerWeb resolve, from inside the cluster, to the cluster’s own public address. IPv4 hairpins and fails. IPv6 works. So those probes got their own module that prefers IPv6.
• Sites behind the older proxy use six-day certificates. A “certificate expires in less than 14 days” rule fires on those every day of their life. The certificate rules were split by lifetime.

After regrouping, all 48 remaining sites probed green. Four were dropped pending a decision, because they answered 403 from the cluster or not at all.

Slice 3: the services Checkmk was still watching

Proxmox and its backup server get scraped over their APIs by two small exporters running in the cluster, with read-only tokens. Disk health comes from smartctl_exporter on the five physical nodes, rolled out by a new Ansible role through AWX. PostgreSQL, Valkey, HAProxy and CrowdSec already had exporters; they just needed a port opened and a scrape job. Each job got its own alert file, with thresholds picked to stay quiet on a healthy day.

Alerting: mail by default, phone when it matters

Alertmanager routes by severity. Everything goes to mail. Critical alerts also go to Pushover on my phone. And one alert is supposed to fire forever:

route:
  group_by: ["alertname", "severity", "instance"]
  receiver: mail
  routes:
    # Dead man's switch: Watchdog always fires. One silent Pushover
    # message a day proves Prometheus, Alertmanager and delivery are alive.
    - matchers: ['alertname = "Watchdog"']
      receiver: heartbeat
      repeat_interval: 24h
    - matchers: ['severity = "critical"']
      receiver: critical
      repeat_interval: 1h
inhibit_rules:
  # A critical alert on a host silences the warnings on the same host.
  - source_matchers: ['severity = "critical"']
    target_matchers: ['severity = "warning"']
    equal: ["instance"]

The heartbeat is the part people skip. A monitoring system that dies quietly is worse than none, because you stop looking. A silent daily notification that stops arriving is the cheapest fix for that.

There’s also a small read-only wall board for a Raspberry Pi with a touchscreen, showing open alerts by severity and a Silenced tab. It came out of the same weekend and lives in its own repository.

What went wrong

This is the list I wish I’d had beforehand. In the order it happened.

The Proxmox hosts refused the automation key. Every Ubuntu host took the node_exporter role on the first run. The five Proxmox nodes use root and rejected the key, and their own firewall needed the port opened by hand. 36 of 41 hosts on day one, and a “virtual machine is down” alert for every pve-* host until the rest followed. I silenced those and finished them through AWX.

One host had its own ideas about /opt. The DNS server’s install had made /opt unreadable to other users, so node_exporter’s service user couldn’t traverse it. The role now sets that permission generically.

The first blackbox audit sent 20 real notifications to my phone. See the IPv6 story above. The plan had a careful canary step to prove the alert path. It was never needed: the audit proved it thoroughly, and a two-hour silence covered the noise while the stale series expired.

A generic “target down” rule paged for every new exporter. Adding six new scrape jobs meant six new ways for “virtual machine is down” to fire on things that aren’t virtual machines. The rule is now scoped to host exporters, and a separate catch-all ScrapeTargetDown covers the rest at a lower severity.

Small container things. The Proxmox exporter’s web server needs a writable /tmp for its worker heartbeat, which a read-only root filesystem doesn’t give it. The backup server’s API token started with too few permissions to read anything. The backup server also had no sudo binary, so Ansible couldn’t manage it until it did.

systemd hardening versus PostgreSQL’s client library. The postgres_exporter runs under a service user with ProtectHome on. The PostgreSQL client library looks for certificates in the user’s home directory, which under that setting doesn’t exist as far as the process is concerned. It failed silently until the service user’s home was moved outside /home. The same rollout found that a database the inventory called a replica was in fact a live primary. Monitoring finds things.

Growing the Prometheus volume turned into growing four disks. Thirty days of retention wanted a bigger volume. Longhorn refused the resize, because the storage nodes were below its minimum free space. Behind that was a Talos detail: the Longhorn partitions had been created by an older mechanism that Talos never regrows, so the disks had been half unused since the cluster was built. Growing the partitions in place on all four workers, with a one-shot privileged pod per node, fixed both problems at once. The volume is 50 GB now.

One alert was real. The HAProxy “backend down” rule fired on backends that had no servers at all, which was a bug in the rule. After the fix it kept firing on one backend, because that service really was down. Not everything that fires during a migration is noise.

What I’d tell you before you start

• Slice it, and keep the old system running until each slice is covered. Three small specs with rollout notes beat one big weekend, and the notes are where this post came from.
• Generate target lists from whatever already knows your hosts. For me that’s the Ansible inventory. It removes a whole category of “forgot to add it to monitoring”.
• Labels first, rules second. site and hosting labels made the alert rules short and made regrouping the probes a ten-minute job.
• Expect the first audit to be loud, and plan the silence in advance. Then read every alert, because one of them is real.
• Add the heartbeat on day one.

What’s next

The Checkmk agent role and its firewall rules still have to come out of Ansible, and a Grafana dashboard pass is waiting. The reverse proxy in front of the websites is also getting a migration of its own, which will get its own post. As always, the hardware all of this runs on is on the Homelab page.