Sysadmin Emergency Runbook: High CPU, Memory & Disk I/O Triage
Sysadmin Emergency Runbook: Triage One-Liners
Keep this runbook open when responding to high-severity telemetry alerts.
1. High CPU / Load Average Spike
bash
# Top 10 CPU-consuming processes with memory and user
ps aux --sort=-%cpu | head -n 11
# Interactive thread-level CPU monitor
htop -d 10
# Check load vs CPU core count
uptime && nproc
2. Memory & OOM Killer Inspection
bash
# High memory consumers
ps aux --sort=-%mem | head -n 11
# Check for OOM killer invocations
dmesg -T | grep -i -E "oom|out of memory|killed process"
# Real-time memory split
free -h -w
3. Storage / Disk Full (100% inode or block)
bash
# Find largest directories starting from root
du -ahx / 2>/dev/null | sort -rh | head -n 20
# Check inode saturation (frequently caused by session files or mail queues)
df -i
# Delete old systemd journal logs keeping only 3 days
sudo journalctl --vacuum-time=3d
4. Network Connections & Port Listening
bash
# Active listening sockets with process PID
sudo ss -tulpn
# Active established connections by remote IP count
netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n