Sysadmin Emergency Runbook: High CPU, Memory & Disk I/O Triage

Sysadmin Emergency Runbook: Triage One-Liners

Keep this runbook open when responding to high-severity telemetry alerts.


1. High CPU / Load Average Spike

bash
# Top 10 CPU-consuming processes with memory and user
ps aux --sort=-%cpu | head -n 11

# Interactive thread-level CPU monitor
htop -d 10

# Check load vs CPU core count
uptime && nproc

2. Memory & OOM Killer Inspection

bash
# High memory consumers
ps aux --sort=-%mem | head -n 11

# Check for OOM killer invocations
dmesg -T | grep -i -E "oom|out of memory|killed process"

# Real-time memory split
free -h -w

3. Storage / Disk Full (100% inode or block)

bash
# Find largest directories starting from root
du -ahx / 2>/dev/null | sort -rh | head -n 20

# Check inode saturation (frequently caused by session files or mail queues)
df -i

# Delete old systemd journal logs keeping only 3 days
sudo journalctl --vacuum-time=3d

4. Network Connections & Port Listening

bash
# Active listening sockets with process PID
sudo ss -tulpn

# Active established connections by remote IP count
netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
Tags: #runbook #troubleshooting #iostat #top #perf