~ koper.dev

Every request gets frisked: BunkerWeb in front of my homelab

· by Robin Koper · 7 min read · #Bunkerweb #Waf #Modsecurity #Kubernetes #Security #Homelab

Anything in my homelab that you can reach from the internet, you reach through one door. This blog, the password manager, GitLab, the photo library, a few websites I host for other people. Around thirty sites, one door.

That door is BunkerWeb. This post is about what it does to every request before it’s allowed in, how I add a new site, and the handful of outages that taught me how to run it properly.

What BunkerWeb is

BunkerWeb is nginx with a security department attached. It’s a reverse proxy, so it takes the request from the internet and passes it to the real application inside the lab. On the way, it checks the request against a web application firewall, reputation lists, rate limits and a stack of security headers, and it handles the Let’s Encrypt certificates for every site.

A WAF, or web application firewall, looks at the content of HTTP requests rather than just ports and addresses. It’s the thing that notices a login form being fed SQL, or a scanner asking for /.env. BunkerWeb uses ModSecurity with the OWASP Core Rule Set for that, which is the standard open source rule set.

In my setup it runs inside the Kubernetes cluster as a DaemonSet, one instance per worker node, behind a single virtual IP. The cluster is the one from five nodes, five jobs.

One file per site

The part I like most is how a site gets added. BunkerWeb watches Kubernetes Ingress resources, and every setting is an annotation. So a public site is one YAML file in my cluster repository, and Argo CD applies it. This is a trimmed version of the one for this blog:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: koper-dev
  annotations:
    bunkerweb.io/SECURITY_MODE: "block"
    bunkerweb.io/USE_REVERSE_PROXY: "yes"
    bunkerweb.io/REVERSE_PROXY_HOST: "http://koper-dev.koper-dev.svc.cluster.local:80"
    # A static site: nothing takes a body, nothing needs POST.
    bunkerweb.io/ALLOWED_METHODS: "GET|HEAD"
    bunkerweb.io/MAX_CLIENT_SIZE: "1m"
    bunkerweb.io/USE_MODSECURITY: "yes"
    bunkerweb.io/USE_DNSBL: "yes"
spec:
  ingressClassName: bunkerweb
  rules:
    - host: koper.dev
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: koper-dev
                port:
                  number: 80

Commit it, point DNS at the virtual IP, and a minute later there’s a certificate and a protected site. Removing a site is deleting the file.

What every site gets for free

Most of the protection isn’t in the site files at all. It’s in one global settings file, so a new site starts safe and only loosens what it has to.

  • HTTPS everywhere. Automatic Let’s Encrypt, HTTP redirected to HTTPS, HTTP/2 and HTTP/3. Certificates for sites that no longer exist are cleaned up automatically, so failed renewals don’t eat into Let’s Encrypt’s limits.
  • Only three methods by default. GET, POST and HEAD. Apps with an API opt into PUT, DELETE and friends in their own file. Doing it the other way round, a restrictive list per site and a permissive default, is how you end up with an app API that quietly accepts DELETE from anywhere.
  • Reputation lists. Known Tor exit nodes, bad user agents and two community lists of aggressive scanners are refused at the door. A few countries I have no visitors from are blocked outright.
  • Bad behaviour. An address that collects fifteen 401, 403 or 404 answers within twenty seconds is banned for a day. Scanners hit that in about a second.
  • Rate limits. 35 requests per second per client by default, raised for the few apps that legitimately need more.
  • Security headers. A default Content Security Policy, cross-origin isolation headers, and headers that leak versions (Server, X-Powered-By) removed.
  • The boring files. security.txt and robots.txt are served for every site, so each app doesn’t have to remember them.

Detect first, block later

Turning a WAF straight to block on a real application is a good way to break it. The Core Rule Set is strict, and real apps do strange things: GitLab sends Git pack files, a password manager posts encrypted blobs, a photo app uploads ten gigabytes.

So every new site starts in detect mode. ModSecurity logs what it would have blocked, and lets the request through. After a while I read the log, and there are only two kinds of entries:

  1. Scanners. Requests for /.env, /.git/config, /wp-login.php on a site that has never run WordPress. These are exactly what the WAF is for.
  2. False positives. Real traffic that happens to look suspicious.

The false positives get an exclusion, scoped as tightly as possible: one rule, one path, one reason. The best example came from GitLab. The rule set flags .git/ in a URL, which is sensible, except that Argo CD pulls my cluster repository from GitLab over exactly such URLs. Block mode without an exception would have cut Argo CD off from its own source of truth, and with it every future config change, including the one to fix it. The exclusion drops that rule only for Git’s own endpoints:

SecRule REQUEST_URI "@rx /(info/refs|git-upload-pack|git-receive-pack)$" \
  "id:1001,phase:1,pass,nolog,ctl:ruleRemoveById=930130"

When the log is only scanners, the site flips to block. The password manager needed 90 minutes of real use from every client, web, browser extension and phone, to show 238 hits that were all scanners. Its admin page is also only reachable from my own addresses, regardless of what the WAF thinks.

AI crawlers

One setting I thought about longer than expected. The bad user agent list blocks every AI crawler by default. I let the ones through that respect robots.txt, and lean on the rate limits and a crawl delay for the rest. The few that are known to ignore both stay blocked. I want this blog to be findable, including by the tools people now use to find things.

What running it taught me

BunkerWeb is easy to start and needs some respect to run. Every item below was an outage or a near miss first.

The rate limit and the ban fed each other. A 429 is BunkerWeb’s own “slow down” answer. The bad behaviour module counted 429s as bad behaviour, so a client that briefly went too fast got rate-limited, collected fifteen 429s, and was banned for a day. My own Coder workspace did exactly that. Now only real client errors count.

worker_processes auto read the wrong number. Nginx sizes its workers to the CPU count it sees, which is the node’s 24 cores, not the pod’s share. Each worker loads the full rule set, and a reload briefly doubles everything. That’s how you get an out-of-memory kill at night. Eight workers and an honest memory limit fixed it.

It breaks in the night. Backups run around one in the morning and certificate renewals around five. Under that disk load the default liveness probe, one second, killed perfectly healthy pods. The probes are more patient now.

One broken exclusion stops every reload. BunkerWeb checks the whole configuration before applying it. A single typo in one site’s ModSecurity file blocks config changes for all thirty sites. The running instances keep serving the old config, which makes it look like your change was ignored. Always check the generated config, not just the file in Git.

Git isn’t the only source of truth. Settings changed in BunkerWeb’s web interface land in its database, and the database wins. When a setting refuses to take, that’s the first place I look now.

IPv6 needed a patch. The Kubernetes controller in the version I run recognised only IPv4 addresses, and tried to write the IPv6 virtual IP into the wrong field of every Ingress. Kubernetes rejected all of them. The fix is a corrected copy of that one Python file, mounted over the original from a ConfigMap. Every image upgrade, I check whether it’s still needed. I made a merge request to the Bunkerweb project and this has been merged.

Moving sites over is the real work. This weekend the last sites moved from an older proxy onto BunkerWeb. Each one had its own quirks: an app that serves its own error pages, cookie flags that got lost, a registry that speaks HTTPS to its backend, and security headers that break a console running in an iframe. Every one of them started in detect mode, and that’s the only reason none of them broke.

Would I recommend it

For a homelab with more than a couple of public services, yes. It turns “I should really harden that” into defaults that apply to everything, and it keeps the whole edge in one Git folder. Budget some time for the rule set and for memory, start every site in detect, and read the logs before you flip to block.

How I reach all of this myself, from outside the house, is in The laptop is just a screen, and how it’s all monitored is in Checkmk out, Prometheus in.