Operating Systems Fundamentals
OS concepts are the substrate everything else runs on. When you understand how the kernel manages processes, memory, and storage, system design decisions that seemed arbitrary become obvious — why connection pools exist, why context switching is expensive, why fork() is fast even for large processes, why mmap beats read() for large files.
These topics appear in FAANG SDE interviews at the senior level: "explain what happens when you call fork()", "how does the kernel schedule threads on a multi-core machine?", "what is a page fault?", "why is reading a file from disk so much slower than reading from memory?"
But the bigger reason to learn this material is not interviews. It's that a large share of hard production incidents — the ones where the dashboards look fine and the service is still slow, or where data is lost even though every write "succeeded" — are explained one layer below the application, in the kernel. This section is written to make that layer readable.
What the OS Does
The operating system is a resource manager. It mediates access to:
- CPU time — scheduling which process runs when
- Memory — giving each process the illusion of owning all memory via virtual address spaces
- Storage — file systems that map names to data on disk
- Devices — abstracted via device drivers
It also provides two things that aren't resources but are just as fundamental:
- Isolation and protection — one process can't read another's memory or files without permission; containers are built from these same mechanisms
- Communication — pipes, sockets, signals, shared memory and the network stack, so isolated processes can still cooperate
The kernel runs in privileged mode (ring 0). User programs run in unprivileged mode (ring 3). Crossing the boundary requires a system call — a controlled gate into the kernel.
User Space: your_program → libc → system call interface
─────────────────────────────────────────────────────────
Kernel Space: syscall handler → scheduler / memory mgr / VFS / drivers
flowchart TB
subgraph US["User space (ring 3)"]
APP["Your service: Go, Java, Node, Python"]
RT["Language runtime: GC, goroutine or virtual-thread scheduler, event loop"]
LIBC["libc / syscall wrappers / vDSO"]
end
subgraph KS["Kernel space (ring 0)"]
SC["System call interface"]
PROC["Processes and scheduling"]
MEM["Virtual memory and page cache"]
VFS["VFS and file systems"]
NET["Networking stack"]
SYNC["Locks, RCU, futexes"]
CG["Namespaces and cgroups"]
DRV["Drivers, interrupts, DMA"]
end
HW["Hardware: CPUs, RAM, NVMe, NIC, timers"]
APP --> RT --> LIBC --> SC
SC --> PROC
SC --> MEM
SC --> VFS
SC --> NET
PROC --> SYNC
MEM --> DRV
VFS --> MEM
NET --> DRV
CG -.->|"limits"| PROC
CG -.->|"limits"| MEM
DRV --> HW
Every page in this section zooms into one box of that diagram.
Why This Matters for System Design
Connection pools exist because thread creation is expensive (~1ms) and context switching has overhead (~1–10μs). Understanding this makes "why use a pool?" obvious rather than cargo-culted. (A bare pthread_create is ~10–20μs; the ~1ms figure is closer to establishing a new database connection or spawning a process — either way, you don't want to pay it per request.)
fork() is fast because virtual memory uses copy-on-write — the child shares the parent's pages until it writes to them. A 1GB process forks in milliseconds rather than the seconds a full copy would take: the kernel copies only page tables (~2MB per GB of resident memory).
mmap beats read() for large files because mmap maps file pages directly into the process address space — no copy to a user-space buffer. The kernel's page cache becomes the file buffer. (It isn't a universal win: page faults, TLB pressure and unpredictable I/O stalls are why many databases still prefer explicit reads.)
epoll beats select because epoll uses kernel data structures that don't scale linearly with the number of watched file descriptors.
SSDs vs HDDs — understanding that SSDs have ~0.1ms latency (vs 5–10ms for HDD) and that both are orders of magnitude slower than RAM (~100ns) gives intuition for caching, buffering, and when disk I/O is the bottleneck.
Production incidents that are really OS problems
| What you see | What's actually happening | Where to read |
|---|---|---|
| p99 latency spikes of ~50–100ms in Kubernetes while CPU usage is "well under the limit" | CFS bandwidth throttling: the container burns its 100ms-period quota in a short multi-threaded burst, then every thread is paused until the next period | CPU Scheduling, Namespaces & cgroups |
| Pods restart with exit code 137 and no stack trace | 137 = 128 + SIGKILL: the cgroup memory limit was hit and the kernel's OOM killer terminated the process | Namespaces & cgroups, Virtual Memory |
| Redis stalls for hundreds of ms every time it snapshots | fork() of a large process copies page tables while the event loop is paused; Transparent Huge Pages turn every post-fork write into a 2MB copy | Processes & Threads |
| A database lost acknowledged writes after a disk error | fsync() error semantics: after a writeback failure, Linux may drop the dirty pages and a retried fsync can succeed — PostgreSQL's 2018 "fsyncgate" changed it to panic instead of retrying | Page Cache & fsync, File Systems |
Too many open files under load | ulimit -n defaults to 1,024 soft; every socket is a file descriptor | File Systems, Syscalls & Kernel Internals |
| Load average of 60 on a machine with idle CPUs | Tasks stuck in uninterruptible sleep (D state) on slow disk or NFS count toward load | Processes & Threads, I/O & Storage |
| Deploys always take exactly 30 seconds; shutdown is never graceful | The app is PID 1 in its container; the kernel doesn't apply default signal actions to PID 1, so SIGTERM is ignored until SIGKILL | Signals & Process Control, Processes & Threads |
Containers fail with fork: Resource temporarily unavailable | Zombie accumulation or thread explosion hitting pids.max | Processes & Threads |
| A benchmark shows 5GB/s "disk" reads | The data was in the page cache; the disk was never touched | Page Cache & fsync |
Packet drops and high ksoftirqd CPU at high request rates | Softirq processing can't keep up; RX queues, RSS/IRQ affinity and NAPI budgets matter | Linux Networking Stack, Interrupts, DMA & Drivers |
Throughput collapses when adding threads; perf shows time in spinlocks | Lock contention in the kernel or a shared futex | Kernel Synchronization |
| Container escape or privilege escalation finding in a security review | Missing seccomp profile, excess capabilities, running as root without user namespaces | Process Security |
War Story: the 2012 leap second
On 30 June 2012 a leap second was inserted at midnight UTC. A bug in the Linux kernel's high-resolution timer handling left timers firing early, so applications sleeping on futexes with timeouts — notably Java services and MySQL — woke immediately, went back to sleep, and woke again, in a loop. CPU usage jumped to 100% across large parts of the internet (Reddit, Mozilla, Gawker and others reported outages or degradation). The widely used fix was simply to reset the clock (date -s "$(date)"), which re-armed the timers. Nothing in the application code was wrong; the failure lived entirely in the timekeeping and futex layer.
Six Mental Models That Explain Most of the Kernel
Before diving into individual pages, it helps to hold a few ideas that recur everywhere. Most "weird" kernel behavior is one of these playing out.
1. Everything is a trade between latency and throughput
Batching makes systems efficient; batching also makes individual operations wait. The scheduler's time slice (longer slices = less switching overhead, worse responsiveness), the page cache's writeback delay (dirty pages sit in RAM for up to ~30 seconds so writes can be merged), NIC interrupt coalescing, TCP's Nagle algorithm and io_uring's submission batching are all the same dial. When you tune any of them, ask which side of the trade you're moving toward and whether your workload wants that.
2. Virtualize, then share lazily
The kernel gives every process the illusion of its own CPU (scheduling), its own memory (virtual address spaces), its own files (fd tables) and, in containers, its own machine (namespaces). It then backs those illusions with shared physical resources and does real work only when forced: memory is allocated on first touch, fork() copies pages only when written, file data is read only when accessed and cached afterwards. "Lazy" is why the system is fast on average — and why costs show up later, in surprising places (a page-fault storm after fork, an OOM kill long after the malloc that "succeeded").
3. The boundary crossing is the cost
Ring 3 → ring 0 transitions, user ↔ kernel copies, and CPU ↔ device round trips are where time goes. Every major performance technique in this section reduces crossings: the vDSO (no syscall for time), epoll (one syscall for many sockets), io_uring (shared rings, often zero syscalls per I/O), sendfile/splice (no user-space copy), futexes (no syscall when a lock is uncontended), and kernel-bypass networking (no kernel at all).
4. Caches everywhere, and every cache can lie
CPU caches, the TLB, the page cache, the dentry/inode caches, the drive's own write cache. They make the common case fast and make performance non-linear: the second run of a benchmark is 100× faster, a process that migrates between CPUs suddenly slows down, a write that "completed" is actually in volatile memory. Correctness questions (durability) and performance questions (warm vs cold) both start with "which cache is this in?"
5. Limits are layered, and the smallest one wins
A thread can fail to be created because of ulimit -u, pids.max, threads-max, pid_max, vm.max_map_count or memory. A container's CPU can be capped by the cgroup quota, the CPU set, the node's capacity or its weight under contention. When something fails with EAGAIN or ENOMEM, list every layer of limit and check each — the error message rarely tells you which one fired.
6. The kernel optimizes for the average; you are paged for the tail
Default settings are tuned for a broad mix of workloads. Your p99.9 lives in the cases those defaults sacrifice: a burst that trips throttling, a writeback storm, a long kernel operation on a non-preemptible kernel, an unlucky NUMA placement. That's why the observability sections across these pages focus on histograms (run-queue latency, I/O latency) rather than averages.
flowchart LR
LT["Latency vs throughput"] --> TUNE["Every tunable"]
LAZY["Virtualize, share lazily"] --> SURPRISE["Costs appear later"]
CROSS["Boundary crossings"] --> FAST["Most optimizations"]
CACHE["Caches lie"] --> DUR["Durability and benchmark bugs"]
LIMIT["Layered limits"] --> ERR["EAGAIN / ENOMEM puzzles"]
TAIL["Tail vs average"] --> HIST["Measure histograms"]
Section Map
The section has 16 pages, grouped by the part of the kernel they cover.
Execution: running code
| Page | What it covers |
|---|---|
| Processes & Threads | task_struct, fork/exec/wait/exit, zombies, clone flags, thread models (1:1, M:N, goroutines, virtual threads), thread-pool sizing |
| CPU Scheduling | Time quanta with worked numbers, FCFS/SJF/RR/MLFQ, CFS and EEVDF, real-time classes, context-switch cost, cgroup CPU throttling |
| Signals & Process Control | SIGTERM vs SIGKILL, async-signal safety, graceful shutdown |
| Syscalls & Kernel Internals | Syscall cost, vDSO, seccomp, eBPF, KPTI overhead |
| Interrupts, DMA & Drivers | Hard and soft IRQs, NAPI, DMA, how devices talk to the kernel |
| Boot Process | Firmware → bootloader → kernel → init/systemd, and how containers and microVMs shortcut it |
Memory
| Page | What it covers |
|---|---|
| Virtual Memory | Address spaces, page tables, TLB, page faults, copy-on-write, mmap |
| Memory Allocators | How malloc works, buddy/slab, fragmentation, jemalloc vs tcmalloc |
| Page Cache & fsync | Buffered vs direct I/O, dirty page writeback, fsync semantics and durability |
Storage and I/O
| Page | What it covers |
|---|---|
| File Systems | Inodes, VFS, journaling, what happens when you open() a file |
| I/O & Storage Internals | epoll vs select, io_uring, DMA, disk scheduling, zero-copy |
| Linux Networking Stack | Packet path from NIC to socket, TCP buffers, backlog queues, tuning |
Coordination and isolation
| Page | What it covers |
|---|---|
| Kernel Synchronization | Spinlocks, mutexes, RCU, futexes, memory barriers |
| Inter-Process Communication | Pipes, Unix sockets, shared memory, message queues — and their costs |
| Namespaces & cgroups | How containers work at the kernel level, OOM killer, CPU throttling |
| Process Security | Users, capabilities, seccomp, LSMs (SELinux/AppArmor), sandboxing |
flowchart LR
PT["Processes and Threads"] --> CS["CPU Scheduling"]
PT --> VM["Virtual Memory"]
PT --> SIG["Signals"]
PT --> IPC["IPC"]
CS --> KS["Kernel Synchronization"]
VM --> MA["Memory Allocators"]
VM --> PC["Page Cache and fsync"]
PC --> FS["File Systems"]
FS --> IO["I/O and Storage"]
IO --> IRQ["Interrupts, DMA, Drivers"]
IRQ --> NETS["Linux Networking Stack"]
SC["Syscalls and Kernel"] --> SEC["Process Security"]
PT --> NC["Namespaces and cgroups"]
NC --> SEC
BOOT["Boot Process"] -.-> PT
Learning Path
| Order | Page | Key Questions |
|---|---|---|
| 1 | Processes & Threads | What is a process? How does fork() work? Threads vs processes? |
| 2 | CPU Scheduling | How does the kernel decide who runs next? What is context switching? |
| 3 | Virtual Memory | What is a page fault? How does fork() not copy all memory? |
| 4 | File Systems | What is an inode? What happens when you open() a file? |
| 5 | I/O & Storage Internals | epoll vs select, io_uring, DMA, disk scheduling, zero-copy |
| 6 | Memory Allocators | How malloc works, buddy/slab, fragmentation, jemalloc vs tcmalloc |
| 7 | Signals & Process Control | SIGTERM vs SIGKILL, async-signal safety, graceful shutdown |
| 8 | Namespaces & cgroups | How containers work at the kernel level, OOM killer, CPU throttling |
| 9 | Syscalls & Kernel Internals | Syscall cost, vDSO, seccomp, eBPF, KPTI overhead |
| 10 | Kernel Synchronization | Spinlock vs mutex? What is RCU? How does a futex avoid syscalls? |
| 11 | Inter-Process Communication | Pipe vs Unix socket vs shared memory — which is fastest and why? |
| 12 | Page Cache & fsync | When is a write actually durable? Why did fsyncgate matter? |
| 13 | Interrupts, DMA & Drivers | What happens when a packet or disk completion arrives? |
| 14 | Linux Networking Stack | What does a packet go through between the NIC and recv()? |
| 15 | Process Security | Capabilities, seccomp, LSMs — how is a process confined? |
| 16 | Boot Process | What runs between power-on and PID 1? |
Paths by goal
You don't have to read in order. Pick the path that matches why you're here:
| Goal | Path |
|---|---|
| Interview prep (senior backend) | Processes & Threads → CPU Scheduling → Virtual Memory → File Systems → I/O & Storage → Syscalls |
| Running services on Kubernetes | Processes & Threads → Signals → Namespaces & cgroups → CPU Scheduling (throttling section) → Virtual Memory (OOM) → Process Security |
| Databases and storage engines | Virtual Memory → Page Cache & fsync → File Systems → I/O & Storage → Memory Allocators → Kernel Synchronization |
| High-throughput networking | Syscalls → I/O & Storage (epoll, io_uring) → Interrupts, DMA & Drivers → Linux Networking Stack → CPU Scheduling (affinity, NUMA) |
| Performance debugging / SRE | CPU Scheduling (observability) → Virtual Memory → Page Cache & fsync → Syscalls (eBPF) → Kernel Synchronization |
Numbers Every Engineer Should Know (OS Edition)
Order-of-magnitude figures for a modern x86-64 Linux server. Exact values vary 2–5× with hardware, kernel version, mitigations and virtualization — the point is which operations cost nanoseconds, microseconds or milliseconds. Measure on your own hardware before relying on any one figure.
| Operation | Typical cost | Why it matters |
|---|---|---|
| L1 cache hit | ~1ns | Baseline |
| L3 cache hit | ~10–20ns | Shared across cores on a socket |
| Main memory (local NUMA node) | ~80–100ns | A pointer chase that misses cache |
| Main memory (remote NUMA node) | ~130–200ns | Why NUMA placement matters |
clock_gettime() via vDSO | ~20ns | No kernel entry at all |
| Syscall round trip (no KPTI) | ~50–100ns | getpid-class calls |
| Syscall round trip with KPTI | ~200–500ns | Meltdown mitigation cost |
| Minor page fault (page in RAM) | ~0.5–1μs | First touch of malloced memory, CoW |
| Goroutine / green-thread switch | ~100–200ns | User-space scheduling |
| Thread context switch (same process) | ~1–2μs | Plus cache/TLB refill afterwards |
| Process context switch | ~2–5μs direct, up to ~100μs indirect | Large working sets pay the refill |
pthread_create + join | ~10–20μs | Why pools exist |
fork() small process | ~50–100μs | Page-table copy |
fork() 1GB RSS process | ~10–20ms | Page tables scale with RSS |
fork + exec + wait, small binary | ~0.5–1ms | Process-per-request is expensive |
| Pipe / Unix socket round trip | ~2–10μs | Local IPC |
| TCP loopback round trip | ~10–30μs | Same host, full TCP stack |
| Major page fault from NVMe | ~10–100μs | Page cache miss on mmap |
| NVMe 4KB random read | ~20–100μs | Device latency |
fsync on NVMe with power-loss protection | ~20–100μs | Enterprise drives can ack from protected cache |
fsync on consumer SSD | ~1–10ms | Must flush the drive cache |
| HDD seek + read | ~5–10ms | Rotational latency |
fsync on HDD | ~10–20ms | Seek + rotation |
| Scheduler tick | 1–4ms (HZ = 1000–250) | Granularity of time-slice checks |
Default fair-class slice (EEVDF base_slice) | 0.75ms base, ~3ms on 8+ CPUs | How long a CPU hog runs before others get a turn |
SCHED_RR quantum | 100ms | Real-time round robin |
| cgroup CPU quota period | 100ms | Throttling can stall you for up to this long |
| Memory copy, single core | ~10–20 GB/s (~50–100μs per MB) | Cost of extra copies (vs zero-copy) |
| Container start (runc, image cached) | ~100ms–1s | Mostly namespace + cgroup + filesystem setup |
| microVM boot (Firecracker) | ~125ms+ | Minimal kernel, minimal devices |
Using these numbers
Turn them into ratios. One NVMe read ≈ 500 DRAM accesses ≈ 1,000 syscalls ≈ 20 context switches. A design that does one extra fsync per request on a consumer SSD has capped itself at a few hundred requests per second per disk before anything else is considered. A thread pool switching 100,000 times a second at ~5μs each spends half a core just switching.
The Toolbox: Which Tool Answers Which Question
| Question | First tool | Deeper tool |
|---|---|---|
| Is the CPU saturated, and by whom? | top / htop, mpstat -P ALL 1 | perf top, perf record -g |
| Are tasks waiting for CPU? | vmstat 1 (r column), /proc/pressure/cpu | perf sched latency, runqlat (bpftrace/bcc) |
| Is the container being throttled? | cat /sys/fs/cgroup/cpu.stat | Prometheus container_cpu_cfs_throttled_periods_total |
| What is a process doing right now? | strace -p PID, cat /proc/PID/wchan | perf trace, offcputime |
| Memory pressure or leak? | free -m, /proc/meminfo, /proc/pressure/memory | smem, /proc/PID/smaps, memleak |
| Disk the bottleneck? | iostat -x 1 | biolatency, biosnoop |
| Network drops? | ss -s, nstat, ip -s link | /proc/net/softnet_stat, dropwatch, tcpretrans |
| Too many processes or threads? | ps -eo nlwp,pid,comm --sort=-nlwp | pids.current in the cgroup |
| Why can't it open more files? | cat /proc/PID/limits, ls /proc/PID/fd | wc -l | lsof -p PID |
# The 60-second triage that answers most "is it the OS?" questions
uptime # load averages: trend up or down?
dmesg -T | tail # OOM kills, hardware errors, hung tasks
vmstat 1 5 # r (run queue), b (blocked), si/so (swap), cs (switches)
mpstat -P ALL 1 3 # one hot CPU? high %iowait or %soft?
pidstat 1 3 # which processes use CPU
iostat -xz 1 3 # disk utilization and latency (r_await, w_await)
free -m # memory and page cache
sar -n DEV,TCP 1 3 # network throughput, TCP retransmits
top # sanity check
(This is the shape of Brendan Gregg's well-known "Linux performance analysis in 60 seconds" checklist.)
System Call Reference (Linux)
| Category | Key Syscalls |
|---|---|
| Process | fork(), exec(), exit(), wait(), getpid() |
| Memory | mmap(), munmap(), brk(), mprotect() |
| File I/O | open(), read(), write(), close(), lseek(), stat() |
| Sockets | socket(), bind(), listen(), accept(), connect() |
| Signals | kill(), signal(), sigaction(), pause() |
| IPC | pipe(), msgget(), shmget(), semget() |
| Threads & sync | clone(), clone3(), futex(), sched_yield(), set_tid_address() |
| Scheduling | sched_setscheduler(), sched_setattr(), sched_setaffinity(), nice() |
| Durability | fsync(), fdatasync(), sync_file_range(), msync() |
| Event I/O | epoll_create1(), epoll_ctl(), epoll_wait(), io_uring_setup(), io_uring_enter() |
| Isolation | unshare(), setns(), pivot_root(), prctl(), seccomp() |
Every time your Python code calls db.query(), your Go code calls http.Get(), or your Node.js event loop wakes up — a system call crosses the user/kernel boundary.
You can see exactly which ones, and how often:
$ strace -c -f curl -s https://example.com > /dev/null
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
31.20 0.000412 10 41 mmap
18.03 0.000238 13 18 openat
12.42 0.000164 8 20 read
9.84 0.000130 21 6 poll
...
100.00 0.001320 212 12 total
(Illustrative output — a single curl makes a couple of hundred syscalls, most of them during process startup and library loading.)
Interview Questions This Section Prepares You For
| Question | Short answer | Full treatment |
|---|---|---|
"What happens when you type ls and press Enter?" | Shell fork()s, child execve()s /bin/ls, kernel maps the ELF, ld.so loads libc, ls makes getdents64/write syscalls, exits, shell wait()s | Processes & Threads |
| "Process vs thread?" | Separate vs shared address space; in Linux both are tasks created by clone() with different sharing flags | Processes & Threads |
| "What is a time quantum and how big should it be?" | Max continuous run time before preemption; too small wastes CPU on switches, too large becomes FCFS; typically 1–10ms | CPU Scheduling |
| "How does Linux pick the next thread?" | Fair class: vruntime weighted by nice; EEVDF (6.6+) picks the earliest virtual deadline among eligible tasks | CPU Scheduling |
| "What is a page fault? Minor vs major?" | Access to an unmapped/protected page; minor = page in RAM, major = must read from disk | Virtual Memory |
| "When is a write durable?" | After fsync/fdatasync succeeds on a drive that honours flushes — not after write() returns | Page Cache & fsync |
| "Why is epoll faster than select?" | Interest list kept in the kernel; readiness delivered without scanning every fd each call | I/O & Storage |
| "Spinlock or mutex?" | Spin when the hold time is shorter than a sleep/wake cycle and you can't sleep; otherwise block | Kernel Synchronization |
| "Fastest way for two processes to share data?" | Shared memory (no copies) plus a synchronization primitive; Unix sockets when you want simplicity | IPC |
| "What is a container, really?" | A process tree with its own namespaces, constrained by cgroups, on the host kernel | Namespaces & cgroups |
"How does a packet get from the wire to recv()?" | NIC DMA → IRQ → NAPI softirq → IP/TCP → socket buffer → wakeup | Linux Networking Stack |
| "SIGTERM vs SIGKILL?" | SIGTERM can be caught for graceful shutdown; SIGKILL cannot | Signals |
Related Sections
- Concurrency — mutexes, deadlocks, lock-free structures and async patterns built on the primitives in this section
- Networking — protocols above the kernel's networking stack
- Databases — storage engines that live and die by page cache, fsync and allocator behavior
- Distributed Systems — what happens when "the OS" becomes many machines