← All projects

Automation design, alerting philosophy

Automated fleet ops for 13 hosts, quiet unless something's wrong

13 hosts patched nightly, alert channel silent on every healthy night

Problem

Thirteen hosts (four hypervisors, a NAS, a media server, the whole k3s fleet) is too many to patch by hand without it becoming a part-time job, and too easy to over-alert on until every notification gets ignored on principle.

Approach

A nightly job walks the fleet, applies routine updates, and where something looks off, runs an automated investigation pass before ever paging a human: most anomalies get root-caused and fixed inline. Safe dependency bumps apply themselves; anything host-level or reboot-class gets flagged for a deliberate maintenance window instead of a blind auto-reboot. The guiding rule was “quiet when healthy”: a notification only fires when there’s a real decision for a human to make, which is what makes the ones that do fire worth reading.

Outcome

Thirteen hosts stay patched with no manual round-trip most nights, and the alert channel earns its notifications instead of spending them on noise nobody reads.