Member-only story
Top 10 Linux Performance Analysis Tools
Hi everyone, welcome back. Today, I will be going over Linux performance analysis tools — ten of them, from htop and vmstat through pressure stall information, iostat, pidstat, ss, perf, bpftrace, sar, and strace.
If you are newer to this, here is the vocabulary, because every tool below prints some of it. User CPU time is time running your program’s instructions; system CPU time is time inside the kernel on its behalf, usually in syscalls. Load average is the one everybody misreads: it is not a percentage. It is a decayed count of tasks runnable or blocked on disk, so 4.0 on four cores is saturated and 4.0 on sixty-four cores is idle. Resident memory is what a process physically holds in RAM; virtual memory is everything it has mapped, including files it never read. The page cache is the kernel’s copy of recent file contents in otherwise free RAM, which is why “free memory” looks alarmingly low on a healthy machine. iowait is the share of time a CPU sat idle while an I/O was outstanding — idle CPU, not a busy disk. A context switch is the kernel moving one task off a CPU for another, costing cache warmth.
Then the frame that organizes all of it, which I take from Brendan Gregg: utilization is how busy a resource is, saturation is how much work is queued behind it, and errors are failed operations. Utilization caps at 100% and stops being informative there; saturation has no ceiling. Every tool here answers “what is this machine waiting on”, and the common mistake is reaching for a profiler when the answer is a queue.
