A self-hosted Kubernetes platform, run like a production estate

Problem

Home lab clusters usually stop at “it’s up.” Mine needed to run apps other people actually depend on (a gym’s booking system, a licence-plate alert pipeline), which meant it had to behave like a platform, not a pet project: recoverable state, no snowflake changes, no single node that takes the whole thing down.

Approach

Four Proxmox VE nodes host a seven-VM k3s cluster (plus one tainted bare-metal worker), managed through Rancher and versioned entirely in one GitOps repo: nothing gets applied by hand. kube-vip runs the control plane and two service-mode floating VIPs so ingress and the admission webhook survive a node loss. Storage is split by workload: NFS for shared/bulk data, iSCSI-backed ZFS zvols via democratic-csi for anything that needs real block storage and can’t tolerate multi-attach. etcd snapshots are encrypted with age and shipped to NAS on a schedule, so a full cluster loss is a restore, not a rebuild. Every namespace gets its own security context (65534:65534, matching the NFS export’s squash mapping) rather than trusting image defaults.

Outcome

Fifteen-plus production workloads run on this cluster today (bookings, alerting pipelines, dashboards, bots, internal tools), all deployed the same way, through the same repo, with the same backup guarantees. The platform work paid for itself the first time a node failed mid-afternoon and nothing user-facing even blipped.