CV

Homelab

PersonalJan 2022 – ongoingSolo

My own Debian server running Docker stacks behind a reverse proxy, VPN-only access, infrastructure as code, and the monitoring stack behind this site's Lab page.

Context & objective

I wanted to host my own media, automation and development services at home, reach them from anywhere by name instead of by IP, and expose as little as possible to the internet. The homelab became my playground for doing things properly.

What I built

  • The server: Debian on a Ryzen 7 3700X (8 cores / 16 threads), 64 GB of RAM, an RTX 3070, about 8 TB of storage and a UPS.
  • Services in Docker: media automation, Node-RED, PostgreSQL, a Homepage dashboard and Uptime Kuma; Plex runs natively on the host. Every stack is a plain docker compose file in its own folder, with secrets kept in separate env files, so any of them comes back up with one command.
  • The network: an edge Caddy with automatic HTTPS (Cloudflare DNS challenge) for *.manziltchu.fr. Only Plex and this site’s API are public, both through Caddy on port 443; everything else answers only on my LAN or through WireGuard. AdGuard Home is the DNS, with rewrites so local names resolve at home, and IPv6 runs on a delegated /60 prefix.
  • Hardening: an nftables host firewall that drops everything not explicitly allowed (all new inbound IPv6 included), services with no published ports, reached only through Caddy by container name, databases and MQTT internal to their stack, and SSH keys-only with fail2ban. I wrote it up as a before → after case study.
  • Game streaming from the server to my phone over 4G/5G (Sunshine/Moonlight, NVENC), through WireGuard with a tuned MTU.
  • A Transmission scheduler, a small Python daemon under systemd that caps active downloads at 3, starts the smallest first and retries stuck ones, so the disks never get flooded with writes.
  • Monitoring: Prometheus with node_exporter, cAdvisor and blackbox_exporter, private Grafana dashboards, and my own exporter reading the Freebox OS API (bandwidth) and Moonraker (3D printer).
  • A small public API in Node.js with no dependencies, publishing a safe, cached subset of those metrics. It feeds the Lab page.

Challenges

  • Storage stalls. Writes kept freezing. One 4 TB drive showed about 2,700 UDMA CRC errors, which turned out to be a faulty SATA cable; another kept renegotiating its link through power management; and both drives are SMR, which stalls under heavy writes. A new cable, the max_performance link power policy and the download scheduler fixed it.
  • Moving twice. In a student residence with no inbound ports, I rebuilt access as outbound-only: a Cloudflare tunnel for Plex and Tailscale/Headscale for remote access, with my router in client mode. Now on fibre again, the original design is back.
  • Split DNS. With a local wildcard rewrite, some hostnames have to bypass it, or the house ends up talking to itself instead of the real site.
  • A new box changed the rules. After the move, the new ISP box gave the server a public IPv6 address. The setup had been designed behind IPv4 NAT, where a Docker -p 8080:8080 stays on the LAN; but Docker binds [::] too, so on the new network about 40 services became reachable from the internet over IPv6. I caught it while bringing the homelab back up, confirmed it from my phone on mobile data, and redesigned for a network where NAT no longer protects anything: a default-deny nftables firewall (loaded at boot, after Docker), every service moved behind the proxy, and the Portainer GUI removed. Published ports went from about 40 to just the proxy on 443, and the services I could reach from mobile data before now time out.
  • The tunnel that connected, then nothing. After the hardening, my WireGuard clients could connect but no page loaded. Docker had auto-assigned a subnet to a new network that overlapped the VPN clients’ range, so their packets arrived and the replies went into the Docker bridge. tcpdump and ip route get showed it, and pinning that network’s subnet in Compose fixed it. Those clients had also lost DNS and Plex, because my new firewall trusted the LAN but not the VPN range, so that needed its own rule.
  • Exposing an API safely. Only allowlisted containers and hosts are published, under public names, and the API reaches Docker through a read-only socket proxy.

What I learned

The biggest lessons came from things going wrong, not from the setups that worked the first time.

  • docker -p is not “LAN only”. NAT was hiding the exposure. Now a host firewall is the backstop for the next mistake.
  • Read the disks before blaming the software. SMART data can point to a hardware fault, and the difference between SMR and CMR drives matters.
  • Keep the infrastructure as code. With every stack in a Compose file, what runs on the server is written down.