Field notes on keeping small Linux servers boring and reliable.
Once a quarter I pick one machine, pretend it burned down, and rebuild it from the latest archive on a scratch VM. The first time it took a whole evening and exposed two missing config files. Now it takes forty minutes and a checklist.
The rule I settled on: three copies, two different providers, and one restore drill written into the calendar.
A modest swap file with vm.swappiness=10 turned sudden out-of-memory kills into slow afternoons that I can actually notice and fix. Pair it with per-service memory limits so one hungry process cannot starve the rest.
Disk above 85%, a certificate expiring within two weeks, and a service that has restarted more than three times in an hour. Everything else went into a weekly digest, and I started reading alerts again.
Seven days of access logs answers nearly every question I have ever asked of them. Keeping more mostly kept things I would rather not be responsible for.