Automation design, alerting philosophy
Automated fleet ops for 13 hosts, quiet unless something's wrong
13 hosts patched nightly, alert channel silent on every healthy night
- CronJob
- Telegram
- LLM-backed investigation
- dependency auto-apply
Problem
Thirteen hosts (four hypervisors, a NAS, a media server, the whole k3s fleet) is too many to patch by hand without it becoming a part-time job, and too easy to over-alert on until every notification gets ignored on principle.
Approach
A nightly job walks the fleet, applies routine updates, and where something looks off, runs an automated investigation pass before ever paging a human: most anomalies get root-caused and fixed inline. Safe dependency bumps apply themselves; anything host-level or reboot-class gets flagged for a deliberate maintenance window instead of a blind auto-reboot. The guiding rule was “quiet when healthy”: a notification only fires when there’s a real decision for a human to make, which is what makes the ones that do fire worth reading.
Outcome
Thirteen hosts stay patched with no manual round-trip most nights, and the alert channel earns its notifications instead of spending them on noise nobody reads.