Processes & Threads
A process is the fundamental unit of isolation in an operating system. Understanding what a process is — and how the kernel creates, manages, and switches between them — is the foundation for understanding concurrency, containers, and system design at the infrastructure level.
This page goes from first principles (why processes exist) through the exact Linux mechanics (task_struct, fork, clone, exec, wait), to the decisions you make every week in production: how many threads to run, why containers accumulate zombies, why fork() from a 50GB Redis pauses the server, and why "just add more threads" usually makes things slower.
Why Processes Exist
Early computers ran one program at a time. Two things pushed operating systems to invent the process:
- Multiplexing. A program waiting 10ms for a disk wastes ~30 million CPU cycles. If the OS can pause that program and run another, the CPU stays busy. Pausing requires saving everything needed to resume later — that bundle of state is the process.
- Protection. If two programs share a machine, a bug in one must not corrupt the other. Each process gets its own virtual address space, its own file descriptor table and its own credentials, enforced by hardware (the MMU and CPU privilege levels).
A process is therefore two things at once: a container of resources (memory, files, identity) and one or more threads of execution running inside it. Threads were split out later (1980s–90s) when people wanted multiple execution streams sharing one set of resources — concurrency without the isolation cost.
What Is a Process?
A process is a running instance of a program. It consists of:
- Address space — the virtual memory the process can access (code, heap, stack, mapped files)
- Program counter — which instruction executes next
- Registers — current CPU state
- Open file descriptors — references to open files, sockets, pipes
- Process ID (PID) — unique identifier in the kernel
- Credentials — UID, GID, permissions
- Signal handlers — registered handlers for OS signals
The kernel represents all this in a Process Control Block (PCB) — a kernel data structure that stores everything needed to pause and resume a process.
Process Address Space (virtual memory):
┌─────────────────┐ high address
│ Stack │ ← grows downward (function calls, local vars)
│ ↓ │
│ (free space) │
│ ↑ │
│ Heap │ ← grows upward (malloc/new)
├─────────────────┤
│ BSS Segment │ uninitialized global/static variables
│ Data Segment │ initialized global/static variables
│ Text Segment │ executable code (read-only)
└─────────────────┘ low address (0x0)
In reality the "free space" in the middle also holds memory-mapped regions: shared libraries (libc.so), mmaped files, thread stacks and large malloc allocations (glibc uses mmap for allocations ≥128KB by default). On x86-64 the user half of the address space is 128TB (47 bits), and the layout is randomized (ASLR) every run. You can see the real map of any process:
$ cat /proc/self/maps | head -8
5603c1a00000-5603c1a02000 r--p 00000000 08:01 1835 /usr/bin/cat # ELF header
5603c1a02000-5603c1a07000 r-xp 00002000 08:01 1835 /usr/bin/cat # text
5603c1a0b000-5603c1a0c000 rw-p 0000a000 08:01 1835 /usr/bin/cat # data
5603c2b1c000-5603c2b3d000 rw-p 00000000 00:00 0 [heap]
7f2e4c000000-7f2e4c1d5000 r-xp 00028000 08:01 2049 /usr/lib/x86_64-linux-gnu/libc.so.6
7ffd1a2f0000-7ffd1a311000 rw-p 00000000 00:00 0 [stack]
7ffd1a3c4000-7ffd1a3c8000 r--p 00000000 00:00 0 [vvar]
7ffd1a3c8000-7ffd1a3ca000 r-xp 00000000 00:00 0 [vdso]
See Virtual Memory for how these virtual ranges map to physical pages.
task_struct: Linux's PCB
In Linux the PCB is struct task_struct (in include/linux/sched.h). It's large — several kilobytes, with hundreds of fields — and most of it is pointers to shareable sub-structures. That design detail is the key to understanding Linux threads:
struct task_struct { // heavily abridged
unsigned int __state; // TASK_RUNNING, TASK_INTERRUPTIBLE, ...
void *stack; // this task's kernel stack (16KB on x86-64)
int prio, static_prio;
struct sched_entity se; // CFS/EEVDF: vruntime, weight
const struct sched_class *sched_class;
cpumask_t cpus_mask; // CPU affinity
pid_t pid; // kernel "pid" = thread ID (TID)
pid_t tgid; // thread group ID = what userspace calls the PID
struct task_struct *real_parent, *parent;
struct list_head children, sibling;
struct mm_struct *mm; // address space (shareable: CLONE_VM)
struct fs_struct *fs; // cwd, root, umask (shareable: CLONE_FS)
struct files_struct *files; // fd table (shareable: CLONE_FILES)
struct signal_struct *signal; // shared signal state (CLONE_THREAD)
struct sighand_struct *sighand; // signal handlers (CLONE_SIGHAND)
sigset_t blocked; // per-thread signal mask
const struct cred *cred; // uid, gid, capabilities
struct nsproxy *nsproxy; // namespaces (CLONE_NEW*)
struct css_set *cgroups;
int exit_state, exit_code;
// ... several hundred more fields
};
Two tasks that point at the same mm, files and sighand are threads of one process. Two tasks with their own copies are separate processes. There is no separate "thread" object in the Linux kernel — just tasks that share more or less.
Note the PID naming trap: the kernel's pid field is what userspace calls the TID, and the kernel's tgid (thread group ID) is what userspace calls the PID. getpid() returns tgid; gettid() returns pid. For a single-threaded process they're equal.
flowchart LR
subgraph P1["Process A (tgid 100)"]
T1["task 100 (main thread)"]
T2["task 101"]
T3["task 102"]
end
MM["mm_struct: page tables, VMAs"]
FD["files_struct: fd table"]
SH["sighand_struct: handlers"]
T1 --> MM
T2 --> MM
T3 --> MM
T1 --> FD
T2 --> FD
T3 --> FD
T1 --> SH
T2 --> SH
T3 --> SH
subgraph P2["Process B (tgid 200)"]
T4["task 200"]
end
MM2["own mm_struct"]
FD2["own files_struct"]
T4 --> MM2
T4 --> FD2
PIDs are a finite resource
$ cat /proc/sys/kernel/pid_max
4194304 # systemd raises it to 2^22 on 64-bit; the kernel default is 32768
$ cat /proc/sys/kernel/threads-max
251582 # derived from RAM at boot
$ ls /proc | grep -c '^[0-9]' # live processes (thread groups)
412
Every thread consumes a PID number. On older systems still at pid_max = 32768, a single Java process with 30,000 threads or a zombie leak can exhaust it, and then fork() fails system-wide with EAGAIN.
The Process Lifecycle at a Glance
sequenceDiagram
participant Sh as "Parent (shell)"
participant K as "Kernel"
participant C as "Child"
Sh->>K: "fork() / clone()"
K->>C: "new task_struct, CoW copy of address space"
K-->>Sh: "returns child PID"
K-->>C: "returns 0"
C->>K: "execve(/bin/ls)"
K->>C: "old image discarded, ELF mapped, entry point"
C->>C: "main() runs"
C->>K: "exit(0)"
K->>K: "free memory and fds, keep exit status (zombie)"
K-->>Sh: "SIGCHLD"
Sh->>K: "waitpid()"
K-->>Sh: "exit status 0, zombie reaped"
Four system calls, each with a precise job:
| Call | Job | Returns |
|---|---|---|
fork() | Duplicate the calling process | Twice: child PID in parent, 0 in child |
execve() | Replace the current program image | Never, on success |
exit() / _exit() | Terminate, leave an exit status | Never |
wait() / waitpid() | Collect a child's exit status, free its zombie | Child PID + status |
fork(): Creating a New Process
fork() creates a child process that is an exact copy of the parent. It returns twice: once in the parent (returns child PID), once in the child (returns 0).
#include <unistd.h>
#include <stdio.h>
#include <sys/wait.h>
int main() {
pid_t pid = fork();
if (pid < 0) {
// fork failed
perror("fork");
} else if (pid == 0) {
// Child process
printf("Child: PID=%d, parent PID=%d\n", getpid(), getppid());
} else {
// Parent process
printf("Parent: PID=%d, child PID=%d\n", getpid(), pid);
wait(NULL); // wait for child to finish
}
return 0;
}
What fork() copies:
- Virtual address space (pages, not physical memory — see copy-on-write)
- File descriptor table (both parent and child have open FDs pointing to the same files)
- Signal handlers
- Environment variables, working directory
Note that "pointing to the same files" is literal: parent and child share the open file description, including the file offset. If both write to an inherited log file descriptor without O_APPEND, they advance one shared offset — a classic source of interleaved or overwritten output.
What fork() does NOT copy:
- The PID (child gets a new one)
- Memory locks
- Pending signals are cleared in child
- Threads (only the calling thread is duplicated — others don't exist in child)
Also not inherited: timers (timer_create, alarm), record locks (fcntl locks), async I/O operations, and the parent's resource-usage counters (the child starts at zero).
fork() in a multithreaded program
Because only the calling thread survives, any mutex that another thread held at the moment of fork() stays locked forever in the child — with no thread left to unlock it. If the child then calls malloc() (which takes an allocator lock) or printf() (stdio lock), it can deadlock. POSIX's rule: after fork() in a multithreaded program, the child may only call async-signal-safe functions until it calls exec(). This is why Python 3.12+ warns when you os.fork() with threads running, and why multiprocessing defaults to spawn on macOS and, from Python 3.14, to forkserver (not plain fork) on Linux.
Copy-on-Write (CoW)
fork() doesn't physically copy memory. The child's page table initially points to the same physical pages as the parent — marked read-only. When either process writes to a page, the kernel:
- Catches the page fault (write to read-only page)
- Allocates a new physical page
- Copies the content
- Updates the faulting process's page table to point to the new page
- Resumes the write
This is why fork() is fast even for a 2GB process — it only copies page tables, not 2GB of data. Pages are copied lazily, only as they're written.
flowchart TD
subgraph Before["Right after fork()"]
PP["Parent page table"] -->|"read-only"| PG["Physical page X"]
CP["Child page table"] -->|"read-only"| PG
end
subgraph After["Child writes to the page"]
PP2["Parent page table"] -->|"read-write again"| PGX["Physical page X"]
CP2["Child page table"] -->|"read-write"| PGY["New page Y, copy of X"]
end
Before --> After
The numbers. "Only copies page tables" is still real work. With 4KB pages, every 1GB of resident memory needs 262,144 page-table entries × 8 bytes = 2MB of PTEs to copy, plus marking every entry read-only and a TLB flush. In practice:
| Parent RSS | Page-table copy | Typical fork() time |
|---|---|---|
| 10MB (small CLI tool) | ~20KB | ~50–100μs |
| 1GB | ~2MB | ~10–20ms |
| 25GB (big Redis) | ~50MB | ~0.3–0.5s — the whole process is paused during it |
Redis publishes this metric (INFO stats → latest_fork_usec) because the BGSAVE/AOF-rewrite fork() blocks the event loop; its latency documentation measures roughly 10–20ms per GB on modern hardware and VMs, and historically far worse on older Xen guests.
The second cost comes after fork: every page that either side writes gets copied. A Redis instance taking a snapshot while serving a write-heavy workload can see memory usage grow toward 2× during the snapshot.
War Story: Transparent Huge Pages and fork
With Transparent Huge Pages (THP) enabled, memory is mapped in 2MB pages. After fork(), a single-byte write to a 2MB huge page copies the entire 2MB instead of 4KB — a 512× amplification. Redis, MongoDB and several other databases explicitly recommend setting THP to never or madvise for this reason; Redis prints a warning at startup if THP is set to always. The symptom is latency spikes and RSS ballooning during background saves.
vfork, posix_spawn and clone3
Because most fork() calls are immediately followed by exec(), copying page tables only to throw them away is wasteful for large parents. Alternatives:
vfork()— the child borrows the parent's address space and the parent is suspended until the child callsexec()or_exit(). No page-table copy at all, but the child must not touch memory. Fast, dangerous.posix_spawn()— the portable "fork+exec in one call". glibc implements it withclone(CLONE_VM | CLONE_VFORK)and a separate small stack, so it's as fast asvforkwithout its hazards. Go'sos/exec, Rust'sstd::process::Command(in many cases), and Java's process launching on Linux all use vfork-style spawning for exactly this reason.clone3()(Linux 5.3+) — the extensible modern syscall underneath everything. glibc'sfork()andpthread_create()are thin wrappers overclone/clone3.
// Rust: spawn + wait, the idiomatic fork/exec/wait
use std::process::Command;
fn main() -> std::io::Result<()> {
let status = Command::new("ls").arg("-l").status()?; // spawn, then waitpid
println!("ls exited with {}", status); // e.g. "exit status: 0"
Ok(())
}
exec(): Replacing the Process Image
exec() replaces the current process's address space with a new program. The PID stays the same but everything else is replaced.
execve("/bin/ls", args, env);
// Code after this never runs in the current process
// (unless execve fails)
"Everything else" has precise exceptions. Across execve() the process keeps: PID and parent, open file descriptors (unless marked close-on-exec), current directory, resource limits, nice value, signal mask and ignored signals, and cgroup membership. It loses: all memory mappings, all threads except the caller, signal handlers (reset to default, since the handler code no longer exists), and mmaped regions.
Always open with O_CLOEXEC
File descriptors survive exec() by default. A server that opens a database socket and then spawns a helper process leaks that socket into the helper — which can keep connections open after the server closes them, or expose secrets to a less-trusted child. Use O_CLOEXEC / SOCK_CLOEXEC / FD_CLOEXEC (Go, Rust and Python 3.4+ do this by default).
The fork-exec pattern: Most process creation in Unix follows this pattern:
fork()— create a copy of the current process- In the child:
exec()— replace the child with the target program - In the parent:
wait()— wait for the child to finish
# When your shell runs `ls`:
# 1. shell forks a child
# 2. child exec()s /bin/ls
# 3. shell waits for the child
# 4. child exits, shell is notified
Why split creation into two calls instead of one CreateProcess() like Windows? Because the gap between fork() and exec() is where the child configures itself using ordinary system calls: redirect stdout (dup2), close fds, change directory, drop privileges, set resource limits, join namespaces. That's how shells implement cmd > out.txt 2>&1 and how container runtimes set up isolation before running your entrypoint.
// How a shell runs: ls -l > out.txt
pid_t pid = fork();
if (pid == 0) {
int fd = open("out.txt", O_WRONLY | O_CREAT | O_TRUNC, 0644);
dup2(fd, STDOUT_FILENO); // stdout now points at out.txt
close(fd);
char *argv[] = {"ls", "-l", NULL};
execvp("ls", argv);
_exit(127); // only reached if exec failed: "command not found"
}
int status;
waitpid(pid, &status, 0);
Watch it happen:
$ strace -f -e trace=clone,clone3,execve,wait4,exit_group bash -c 'ls > /dev/null'
execve("/usr/bin/bash", ["bash", "-c", "ls > /dev/null"], 0x7ffc... /* 30 vars */) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, ...) = 48211
[pid 48211] execve("/usr/bin/ls", ["ls"], 0x55d3... /* 30 vars */) = 0
[pid 48211] exit_group(0) = ?
[pid 48211] +++ exited with 0 +++
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 48211
(Bash runs the last command of -c directly via exec without forking in some cases; the redirect here forces a fork.) Note the clone(... SIGCHLD) — that's fork(): a clone with no sharing flags, and "send me SIGCHLD when this child dies".
exit() and wait(): Ending a Process
Exiting
exit(status)(libc): runsatexithandlers, flushes stdio buffers, then calls_exit._exit(status)(syscallexit_group): terminates immediately. All threads in the process die.- Returning from
mainis the same as callingexit.
Use _exit() in a forked child that fails to exec: calling exit() there would flush the parent's copied stdio buffers a second time — output duplicated.
On exit the kernel closes all file descriptors, releases memory, reparents any children to a reaper, and sends SIGCHLD to the parent. What it keeps is a tiny residue: the task_struct with the exit status and resource usage. That residue is the zombie.
Exit status conventions
| Status | Meaning |
|---|---|
| 0 | Success |
| 1–125 | Program-defined failure |
| 126 | Found but not executable (shell) |
| 127 | Command not found (shell) |
| 128 + N | Killed by signal N — e.g. 137 = 128 + 9 (SIGKILL, the OOM killer's signature in Kubernetes), 143 = 128 + 15 (SIGTERM) |
Waiting
int status;
pid_t pid = waitpid(-1, &status, WNOHANG); // any child, don't block
if (pid > 0) {
if (WIFEXITED(status)) printf("exited %d\n", WEXITSTATUS(status));
else if (WIFSIGNALED(status)) printf("killed by signal %d\n", WTERMSIG(status));
}
The standard pattern for a server that spawns children is a SIGCHLD handler that reaps in a loop — one signal can stand for many dead children, because standard signals don't queue:
static void on_sigchld(int sig) {
int saved = errno;
while (waitpid(-1, NULL, WNOHANG) > 0) { } // reap every finished child
errno = saved;
}
// install with sigaction(SIGCHLD, ...) and SA_RESTART | SA_NOCLDSTOP
Setting SIGCHLD to SIG_IGN tells the kernel to auto-reap children (no zombies, but no exit statuses either). Linux 5.3+ also offers pidfds (pidfd_open, CLONE_PIDFD): a file descriptor for a process that you can poll/epoll for exit — no signal handler, no PID-reuse race. See Signals & Process Control.
Process States
fork()
│
▼
CREATED
│ scheduler picks it up
▼
┌────────── READY ◄────────────────┐
│ │ scheduled to run │
│ ▼ │
│ RUNNING ────────────────┘ preempted (time slice expired)
│ │
│ ┌────────┴───────┐
│ │ │
│ ▼ ▼
│ WAITING exit()
│ (I/O, sleep, │
│ signal) ▼
│ │ ZOMBIE ─────────► parent wait() → DEAD
│ │ (exit code stored
│ │ until parent reads)
└────┘ I/O done / signal received
The same states as a state machine, with what Linux actually calls them:
stateDiagram-v2
[*] --> Runnable: "fork / clone"
Runnable --> Running: "picked by scheduler"
Running --> Runnable: "preempted"
Running --> Sleeping: "wait for I/O, lock, timer"
Sleeping --> Runnable: "woken up"
Running --> Stopped: "SIGSTOP / ptrace"
Stopped --> Runnable: "SIGCONT"
Running --> Zombie: "exit"
Zombie --> [*]: "parent calls wait"
The single letter you see in ps / top:
ps STAT | Kernel state | Meaning |
|---|---|---|
R | TASK_RUNNING | Running or runnable (Linux doesn't distinguish READY and RUNNING in this field) |
S | TASK_INTERRUPTIBLE | Sleeping; a signal will wake it |
D | TASK_UNINTERRUPTIBLE | Sleeping in the kernel, typically disk/NFS I/O; even SIGKILL waits |
T / t | stopped / traced | Job control stop or debugger |
Z | EXIT_ZOMBIE | Exited, waiting to be reaped |
I | idle kernel thread | Kernel thread sleeping (not counted in load) |
D matters operationally: tasks in D count toward the Linux load average even though they use no CPU. A load average of 80 on an idle-looking box is usually dozens of processes stuck in D on a hung NFS mount or a saturated disk. And kill -9 won't remove them until the I/O completes (TASK_KILLABLE sleeps are the partial exception).
Zombie process: A process that has exited but whose parent hasn't called wait() yet. The kernel keeps the PCB to store the exit code. If the parent never calls wait(), zombie entries accumulate (not a memory leak per se, but PIDs are finite).
Orphan process: A process whose parent has exited. The kernel reparents orphans to PID 1 (init/systemd), which calls wait() on them.
More precisely, an orphan is reparented to the nearest ancestor that has marked itself a child subreaper (prctl(PR_SET_CHILD_SUBREAPER, 1)) — systemd user managers, tini, and container shims use this — or to PID 1 of its PID namespace. In a container, your application is often PID 1, which is where trouble starts (see failure modes below).
You can't kill a zombie — it's already dead. You make it go away by getting its parent to call wait(), or by killing the parent so the zombie is reparented to a reaper that will.
$ ps -eo pid,ppid,stat,comm | awk '$3 ~ /Z/'
48812 48700 Z worker <defunct>
$ ps -o pid,comm -p 48700 # the parent that isn't reaping
48700 bad-supervisor
Threads vs Processes
A thread is an execution context within a process. Multiple threads share:
- Address space (heap, global variables, file descriptors)
- PID (but each thread has its own TID — Thread ID)
Each thread has its own:
- Stack
- Program counter
- Registers
- Signal mask
Each thread also has its own thread-local storage (__thread / thread_local variables, errno), its own scheduling priority and CPU affinity, and its own kernel stack.
Process:
┌──────────────────────────────────────────┐
│ Code | Data | Heap | File Descriptors │
│ │
│ Thread 1 │ Thread 2 │ Thread 3 │
│ [stack] │ [stack] │ [stack] │
│ [regs] │ [regs] │ [regs] │
│ [PC] │ [PC] │ [PC] │
└──────────────────────────────────────────┘
| Process | Thread | |
|---|---|---|
| Memory | Separate address space | Shared address space |
| Communication | IPC (pipes, sockets, shared memory) | Shared variables (needs synchronization) |
| Creation cost | Expensive (~1ms, copies page tables) | Cheap (~10μs) |
| Context switch | Expensive (TLB flush) | Cheaper (same address space) |
| Isolation | Strong (crash doesn't affect others) | Weak (crash kills all threads) |
| Use case | Isolation, security boundaries | Shared-state concurrency |
The creation-cost row is order-of-magnitude. On a modern Linux box: pthread_create + join ≈ 10–20μs; fork of a small process ≈ 50–100μs; fork + exec + wait of a small binary ≈ 0.5–1ms (most of that is exec: mapping the ELF, the dynamic linker loading libraries). Large parents make fork slower, as shown above.
Linux implementation detail: Linux implements threads as "lightweight processes" using clone(). A thread is a process that shares its address space, file descriptors, and other resources with the parent. fork() calls clone() with no sharing; pthread_create() calls clone() with full sharing.
clone() flags: the dial between process and thread
| Flag | Shares | Used by |
|---|---|---|
CLONE_VM | Address space (mm_struct) | Threads, vfork, posix_spawn |
CLONE_FS | cwd, root, umask | Threads |
CLONE_FILES | File descriptor table | Threads |
CLONE_SIGHAND | Signal handlers | Threads (requires CLONE_VM) |
CLONE_THREAD | Same thread group (same PID/tgid), shared signal delivery | Threads |
CLONE_SYSVSEM | System V semaphore undo list | Threads |
CLONE_SETTLS | Sets the new thread's TLS pointer (FS base on x86-64) | Threads |
CLONE_PARENT_SETTID / CLONE_CHILD_CLEARTID | Write TID to memory / clear it and futex-wake on exit (how pthread_join works) | Threads |
CLONE_VFORK | Suspend parent until child execs or exits | vfork, posix_spawn |
CLONE_NEWPID, CLONE_NEWNET, CLONE_NEWNS, ... | Opposite: new, unshared namespaces | Container runtimes |
CLONE_PIDFD | Return a pidfd for the child | Modern process supervisors |
strace on a pthread_create() call shows glibc asking for nearly everything:
clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|
CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
child_tid=0x7f3a1b7ff910, parent_tid=0x7f3a1b7ff910, exit_signal=0,
stack=0x7f3a1afff000, stack_size=0x7fff00, tls=0x7f3a1b7ff640} => {...}, 88) = 48233
stack_size=0x7fff00 — just under 8MB, the default thread stack. And the namespace flags at the bottom of the table are how containers are built from the same primitive: a container is a clone() with less sharing than usual.
Thread signal semantics
A process-directed signal (kill(pid, SIGTERM)) is delivered to one thread that doesn't have it blocked — any one. The robust pattern in multithreaded programs is to block signals in all threads and have one dedicated thread call sigwait() / use signalfd. A fatal signal (e.g. SIGSEGV in any thread) kills the whole process. exit() in any thread ends all threads; pthread_exit() ends only the caller.
Kernel Threads vs User Threads
Two different things share the word "thread":
- Kernel threads — tasks that run only in kernel mode, with no user address space (
mm == NULL). They do the kernel's background work. Inpsthey appear in square brackets, all descended fromkthreadd(PID 2):
$ ps -eo pid,ppid,comm | head -12
PID PPID COMMAND
1 0 systemd
2 0 kthreadd
3 2 pool_workqueue_release
15 2 ksoftirqd/0 # deferred interrupt work for CPU 0
16 2 rcu_preempt # RCU grace-period machinery
17 2 migration/0 # moves tasks between CPUs (stop class)
95 2 kswapd0 # page reclaim
311 2 kworker/3:1H-kblockd # workqueue worker
402 2 jbd2/sda1-8 # ext4 journal commit
A ksoftirqd at high CPU means the box is drowning in network or block interrupts; kswapd at high CPU means memory pressure. See Syscalls & Kernel Internals.
- User threads — threads of your program. Whether each user thread corresponds to a kernel task depends on the thread model.
Thread Models: 1:1, N:1, M:N
How many kernel-scheduled entities back your language's "threads"?
flowchart TB
subgraph OneToOne["1:1 (pthreads, Java platform threads, Rust std::thread)"]
U1["user thread"] --> K1["kernel task"]
U2["user thread"] --> K2["kernel task"]
end
subgraph MToN["M:N (Go, Erlang, Java virtual threads)"]
G1["goroutine"] --> W1["OS thread"]
G2["goroutine"] --> W1
G3["goroutine"] --> W2["OS thread"]
G4["goroutine x 100k"] --> W2
end
| Model | Examples | Pros | Cons |
|---|---|---|---|
| 1:1 — each user thread is a kernel task | Linux NPTL pthreads, Windows threads, Java platform threads, Rust std::thread, C++ std::thread | Simple; true parallelism; blocking syscalls block only one thread; kernel tools see everything | ~8MB virtual stack and a kernel task each; switches go through the kernel; tens of thousands is the practical ceiling |
| N:1 — all user threads on one kernel task | Early Java "green threads" (1.1), GNU Pth, classic coroutines | Very cheap switches | No parallelism; one blocking syscall blocks everything |
| M:N — M user threads multiplexed over N kernel threads | Go goroutines, Erlang/Elixir processes, Java virtual threads (JDK 21+), Kotlin coroutines on dispatchers | Millions of cheap tasks; user-space switches (~100–200ns); parallel across cores | Runtime must avoid blocking its OS threads; harder debugging; scheduling invisible to the kernel |
Linux itself tried M:N (NGPT, IBM, ~2002) and dropped it in favor of the simpler 1:1 NPTL, which became fast enough that the complexity wasn't worth it. M:N came back in language runtimes where the runtime knows when a task will block.
Go's G-M-P scheduler
- G — a goroutine: a stack (starts at 2KB, grows by copying, up to 1GB on 64-bit), a PC, and some state. ~2–4KB total.
- M — an OS thread (machine).
- P — a processor context with a local run queue of Gs. There are
GOMAXPROCSPs (default: number of CPUs, and since Go 1.25 the cgroup CPU limit).
An M must hold a P to run Go code. When a goroutine makes a blocking syscall, its M detaches from its P and another M takes the P, so other goroutines keep running. Network I/O doesn't block at all: it parks the goroutine and registers the fd with the runtime's epoll-based netpoller. Idle Ps steal half of another P's run queue. Since Go 1.14, goroutines are asynchronously preempted via signals after ~10ms, so a tight loop can't hog a P.
package main
import (
"fmt"
"runtime"
"sync"
)
func main() {
fmt.Println("GOMAXPROCS:", runtime.GOMAXPROCS(0)) // number of Ps
var wg sync.WaitGroup
for i := 0; i < 100_000; i++ { // 100k goroutines: ~200-400MB worst case, usually far less
wg.Add(1)
go func() { defer wg.Done() }()
}
wg.Wait()
fmt.Println("goroutines now:", runtime.NumGoroutine())
}
ps -T -p <pid> on that program shows only a handful of OS threads — the kernel never sees the 100,000 goroutines.
Java virtual threads
JDK 21 (JEP 444) added virtual threads: Thread.ofVirtual().start(...) or Executors.newVirtualThreadPerTaskExecutor(). They're mounted on a ForkJoinPool of carrier threads (default size = CPU count); a blocking JDK I/O call unmounts the virtual thread and frees the carrier. The big caveat until JDK 24: blocking inside a synchronized block pinned the carrier, so a few pinned virtual threads could stall the whole pool. JEP 491 (JDK 24) removed most pinning for synchronized. Native calls (JNI) still pin.
Stackless async (Rust, Node.js, Python asyncio)
Rust's async fn compiles each task into a state machine whose size is known at compile time (often a few hundred bytes) — no per-task stack at all. Tokio then runs these on a work-stealing pool of OS threads, much like Go's Ps. Node.js runs JavaScript on one thread with an event loop, plus a libuv threadpool (default 4 threads, UV_THREADPOOL_SIZE) for file I/O, DNS and crypto. The trade-off versus goroutines: no hidden stacks and cheaper tasks, but async "colors" every function in the call chain. See Async Patterns.
Thread Stacks and Limits
Stack sizes
| Kind of thread | Default stack | Notes |
|---|---|---|
| Linux main thread | Grows up to ulimit -s, usually 8MB | Grows on demand via page faults |
glibc pthread_create | RLIMIT_STACK if set, typically 8MB | Fixed-size mmap + guard page; set with pthread_attr_setstacksize |
| musl (Alpine) pthreads | 128KB | Deep recursion that works on glibc can segfault on Alpine |
| Java platform thread | 1MB on 64-bit Linux (-Xss) | Plus native and JVM overhead |
| Go goroutine | 2KB initial, grows by copying | Max 1GB (debug.SetMaxStack) |
Rust std::thread | 2MB (RUST_MIN_STACK, Builder::stack_size) | Main thread follows OS limit |
| Kernel stack per task | 16KB on x86-64 | Fixed; kernel stack overflow is fatal |
The 8MB is virtual, reserved not committed. Only pages actually touched consume RAM, so 1,000 threads each using 40KB of stack cost ~40MB of RSS, not 8GB. But the virtual reservation still counts against RLIMIT_AS, against overcommit accounting in strict mode (vm.overcommit_memory=2), and, on 32-bit systems, against a 3GB address space (which capped 32-bit processes at ~380 threads).
Each thread also has a guard page (an unmapped/no-access page below the stack): overflow faults with SIGSEGV instead of silently trampling the neighbouring thread's memory.
pthread_attr_t attr;
pthread_attr_init(&attr);
pthread_attr_setstacksize(&attr, 256 * 1024); // 256KB for a shallow worker
pthread_create(&tid, &attr, worker, NULL);
How many threads can a process have?
Several limits apply simultaneously; the first one you hit wins:
| Limit | Where | Typical default | Error when hit |
|---|---|---|---|
| Per-user processes+threads | ulimit -u (RLIMIT_NPROC) | Often ~63,000 or unlimited; some distros 4096 | EAGAIN from clone |
| System-wide threads | /proc/sys/kernel/threads-max | RAM-derived, ~250k on 32GB | EAGAIN |
| PID space | /proc/sys/kernel/pid_max | 32,768 or 4,194,304 | EAGAIN |
| Memory mappings per process | /proc/sys/vm/max_map_count | 65,530 (each thread stack + guard ≈ 2 maps) | ENOMEM |
| cgroup pids controller | pids.max in the cgroup; Kubernetes podPidsLimit; systemd TasksMax | Varies; systemd sets a default per unit | EAGAIN |
| Memory | RAM / container memory limit | — | ENOMEM or OOM kill |
What the errors look like in practice:
pthread_create failed: Resource temporarily unavailable # C, EAGAIN
java.lang.OutOfMemoryError: unable to create native thread # Java — usually NOT heap exhaustion
runtime/cgo: pthread_create failed: Resource temporarily unavailable # Go hitting pids.max
fork: retry: Resource temporarily unavailable # bash
# Who is using all the threads?
$ ps -eo nlwp,pid,comm --sort=-nlwp | head -5 # NLWP = number of threads
NLWP PID COMMAND
4012 21345 java
612 1803 containerd
$ cat /sys/fs/cgroup/pids.current /sys/fs/cgroup/pids.max
4031
4096
Sizing Thread Pools
Thread pools exist because creating a thread per task costs ~10–20μs plus its memory, and because an unbounded number of threads destroys performance through context switches and cache thrash (see CPU Scheduling). The question is always: how many?
CPU-bound work
threads = N_cpu (sometimes N_cpu + 1, to cover the occasional page fault or stall)
More threads than cores cannot do more CPU work — they only add switching. If the process runs in a container with a CPU limit of 4, N_cpu is 4, not the node's 64 cores.
I/O-bound work: the Goetz formula
From Java Concurrency in Practice:
threads = N_cpu × U_cpu × (1 + W / C)
N_cpu = number of cores
U_cpu = target CPU utilization (0..1)
W / C = ratio of wait time to compute time per task
Worked example: 8 cores, target 80% utilization; each request spends 45ms waiting on a database and 5ms computing (W/C = 9):
threads = 8 × 0.8 × (1 + 45/5) = 8 × 0.8 × 10 = 64 threads
Cross-check with Little's Law
Concurrency needed = throughput × latency. If that service must handle 1,200 requests/s at 50ms each:
L = λ × W = 1,200/s × 0.050s = 60 requests in flight
60 in-flight requests need ~60 threads in a thread-per-request design — consistent with the 64 above. If Little's Law says you need 600 in flight but the CPU formula says 64, the pool isn't the real limit: the downstream (the database) is, and adding threads just builds a bigger queue in front of it.
The downstream sets the ceiling
The 64 threads above all hit one database. That database's connection pool — and its own CPU — is the real bottleneck. HikariCP's well-known starting point for a database's connection pool is far smaller than people expect:
connections = (core_count_of_db × 2) + effective_spindle_count
A 16-core Postgres server with SSDs starts around 32–40 connections total across all clients — not 64 per app instance × 50 instances.
| Workload | Starting point | Watch for |
|---|---|---|
| Pure CPU (image resize, compression, crypto) | N_cpu | Run-queue latency, involuntary switches |
| Mixed request handling | Goetz formula, verify with Little's Law | p99 latency vs pool size curve |
| Blocking I/O to one downstream | Match the downstream's capacity | Downstream saturation, queueing |
| Massive concurrent I/O (10k+ connections) | Don't use a thread per connection: event loop, goroutines, virtual threads | Memory per connection |
// Bounded worker pool in Go: N workers, bounded queue = backpressure
func process(jobs []Job, workers int) {
ch := make(chan Job, workers*2) // small buffer, not unbounded
var wg sync.WaitGroup
for i := 0; i < workers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for j := range ch {
handle(j)
}
}()
}
for _, j := range jobs {
ch <- j // blocks when workers are busy: natural backpressure
}
close(ch)
wg.Wait()
}
Measure, then size
Formulas give a starting point. The real answer is a load test that plots throughput and p99 latency against pool size: throughput rises, flattens at the knee, and latency then climbs as you add threads past it. Pick the knee, and keep the queue in front of the pool bounded so overload sheds requests instead of piling them up.
Inter-Process Communication (IPC)
Processes can't directly access each other's memory (isolation). IPC mechanisms:
Pipes: Unidirectional byte stream. Parent creates before fork; child inherits. Used to chain processes (ls | grep foo).
int pipefd[2];
pipe(pipefd); // pipefd[0] = read end, pipefd[1] = write end
Unix Domain Sockets: Bidirectional, local-only. Higher throughput than TCP loopback for same-machine IPC. Used by Docker, PostgreSQL, systemd.
Shared Memory: Fastest IPC — processes map the same physical pages into their address spaces. No data copying. Requires explicit synchronization.
int shm_id = shmget(key, size, IPC_CREAT | 0666);
void* ptr = shmat(shm_id, NULL, 0);
Message Queues: Kernel-managed queues. Sender writes message, receiver reads it. POSIX (mq_open) or SysV (msgget).
Signals: Software interrupts. Asynchronous notification. Limited payload (just the signal number). Used for process control: SIGTERM (terminate gracefully), SIGKILL (terminate immediately, can't be caught), SIGUSR1/SIGUSR2 (application-defined).
For the mechanics, performance numbers and trade-offs of each mechanism, see Inter-Process Communication and Signals & Process Control.
What Happens When You Run a Program
$ python3 app.py
- Shell calls
fork()— creates child shell process - Child calls
execve("/usr/bin/python3", ["python3", "app.py"], env) - Kernel validates the executable (
ELFheader check) - Kernel maps the ELF segments into the child's address space (code, data)
- Kernel sets up the stack with
argv,envp - Kernel sets PC to the entry point
- Dynamic linker (
ld.so) runs first — loads shared libraries (libc, etc.) main()begins executing- Python interpreter starts, parses
app.py, executes bytecode - When
app.pyexits: Python callsexit()→ kernel cleans up process → sends SIGCHLD to parent shell
A few details hidden in those steps: in step 3 the kernel also handles #! scripts (it re-execs the interpreter named on the shebang line); in step 5 it also places the auxiliary vector (AT_ENTRY, AT_RANDOM, the vDSO address) on the stack; and in step 7 the entry point is actually ld.so's, which then jumps to the program's _start → __libc_start_main → main. Statically linked binaries (common in Go) skip step 7 entirely — one reason they start faster. And between steps 1 and 2 the shell sets up redirections and the process group for job control.
$ strace -e trace=execve,openat python3 -c 'pass' 2>&1 | head # exec + libraries being opened
$ strace -c python3 -c 'pass' # count them: typically 300-1,000 syscalls
$ LD_DEBUG=libs /bin/true 2>&1 | head # watch ld.so find libraries
Exploring Processes Through /proc
/proc is the kernel's live view of every task. Everything ps, top and htop show comes from here.
$ PID=$(pgrep -o nginx)
$ cat /proc/$PID/status | egrep 'Name|State|PPid|Threads|VmRSS|VmSize|voluntary'
Name: nginx
State: S (sleeping)
PPid: 1
Threads: 1
VmSize: 55120 kB # virtual size
VmRSS: 9812 kB # resident memory
voluntary_ctxt_switches: 8812
nonvoluntary_ctxt_switches: 41
$ ls -l /proc/$PID/fd | head -4 # open file descriptors
lrwx------ 1 root root 64 Nov 8 10:00 0 -> /dev/null
lrwx------ 1 root root 64 Nov 8 10:00 6 -> socket:[48211]
l-wx------ 1 root root 64 Nov 8 10:00 3 -> /var/log/nginx/access.log
$ ls /proc/$PID/task # one directory per thread (TID)
$ cat /proc/$PID/limits | egrep 'processes|open files|stack'
Max stack size 8388608 unlimited bytes
Max processes 63314 63314 processes
Max open files 1024 524288 files
$ tr '\0' ' ' < /proc/$PID/cmdline # argv
$ cat /proc/$PID/cgroup # which cgroup (container) it's in
$ readlink /proc/$PID/exe /proc/$PID/cwd # binary and working directory
$ cat /proc/$PID/wchan; echo # kernel function it's sleeping in
Useful one-liners:
$ ps -eLf | wc -l # total threads on the box
$ ps -T -p $PID # threads of one process (SPID = TID)
$ top -H -p $PID # live per-thread CPU
$ pstree -p $PID # process tree
$ ps -eo pid,stat,wchan:20,comm | awk '$2 ~ /D/' # who is stuck in uninterruptible sleep
$ cat /proc/$PID/task/*/stat | awk '{print $1, $3, $14+$15}' # TID, state, CPU ticks
Failure Modes
Fork bombs
:(){ :|:& };: # DO NOT RUN: defines ":" which calls itself twice in the background
Each invocation spawns two more; within seconds every PID or the per-user process limit is exhausted, and nothing new — including ssh, ps, or the shell you'd use to fix it — can start. Defenses, in order of strength:
ulimit -u/RLIMIT_NPROCper user (/etc/security/limits.conf).- systemd
TasksMax=per service and per user slice. - cgroup
pids.maxper container; KubernetespodPidsLimitin the kubelet config. Set this — a single misbehaving pod can otherwise exhaust the node's PIDs and break every other pod on it.
Recovery when it's happening: kill -STOP first (stop the bomb from multiplying), then kill. kill -9 -1 as the offending user kills all their processes.
Thread explosion
A thread-per-request server under a traffic spike, or a retry storm, creates threads until a limit from the table above is hit. Symptoms appear in this order: run-queue latency climbs and throughput falls (context switching), memory grows, then thread creation fails with EAGAIN / "unable to create native thread", and the process may crash. Common causes: unbounded Executors.newCachedThreadPool(), a new thread for every blocking call, a library creating a pool per client object, or GOMAXPROCS-blind thread-hungry cgo code. Fix with bounded pools and bounded queues.
War Story (illustrative)
A service created a new HTTP client object — each with its own internal thread pool — on every request instead of reusing one. Under normal load the GC cleaned them up fast enough. During a downstream slowdown, requests piled up, pools accumulated faster than they were freed, and the JVM reached the pod's pids.max of 4,096 within two minutes. Every new request failed with OutOfMemoryError: unable to create native thread while heap usage sat at 30%. ps -o nlwp on the pod showed 4,090 threads. The fix was a single shared client.
Zombie accumulation in containers (the PID 1 problem)
In a container, the entrypoint process is PID 1 of its PID namespace, which gives it two jobs most applications don't know they have:
- Reaping orphans. Any process whose parent dies gets reparented to PID 1. If your app (Node, Python, Java) is PID 1 and never calls
wait()on children it didn't create, zombies accumulate — e.g. a shell script spawningcurlhealth checks every few seconds, or a headless Chrome leaving renderer processes behind. Eventuallypids.maxis hit andfork()fails. - Signal handling. The kernel doesn't apply default signal actions to PID 1 of a namespace. A
SIGTERMto a PID 1 that hasn't installed a handler is ignored — sodocker stop/ pod termination waits the full grace period (10s Docker, 30s Kubernetes) and then sendsSIGKILL. Every deploy is slow and every shutdown is unclean.
flowchart TD
D["docker stop / pod delete"] --> T["SIGTERM to PID 1"]
T --> Q{"PID 1 installed a SIGTERM handler?"}
Q -->|"no: app is PID 1"| IG["signal ignored"]
IG --> W["wait full grace period"]
W --> K["SIGKILL: unclean shutdown"]
Q -->|"yes: tini or handler"| G["forward to app, graceful shutdown"]
G --> R["init also reaps zombies with wait"]
Fixes:
# Use a tiny init as PID 1: it reaps zombies and forwards signals
RUN apk add --no-cache tini
ENTRYPOINT ["/sbin/tini", "--"]
CMD ["node", "server.js"]
docker run --initinjectstiniautomatically.- Use the exec form
CMD ["node", "server.js"], not the shell formCMD node server.js— the shell form makes/bin/shPID 1, which may not forward signals. - In shell entrypoint scripts, end with
exec "$@"so your app replaces the shell. - In Kubernetes,
shareProcessNamespace: truemakes the pause container PID 1, and it reaps zombies.
Other common failures
| Failure | Cause | Fix |
|---|---|---|
fork() fails with ENOMEM on a box with free RAM | Strict overcommit (vm.overcommit_memory=2) must account for a full CoW copy of a large parent | Use posix_spawn/vfork for subprocesses; Redis recommends overcommit_memory=1 |
Long pause during snapshot / fork() | Copying page tables of a huge RSS | Huge pages (fewer PTEs, but beware THP CoW), smaller instances, replicas for snapshotting |
Child deadlocks right after fork() | Lock held by another thread at fork time | Exec immediately; only async-signal-safe calls before exec; pthread_atfork |
| Secrets/sockets visible in a child process | Fds inherited across exec | O_CLOEXEC everywhere |
kill -9 doesn't work | Process in D state, or it's a zombie | Fix the stuck I/O; reap via the parent |
| Load average 50 with idle CPU | Tasks in D (disk/NFS) counted in load | ps for D state, /proc/<pid>/stack (root), check storage |
| Segfault only on Alpine | musl's 128KB default thread stack | Set stack size explicitly |
Decision Guidance: Process, Thread, or Task?
flowchart TD
S["Need concurrency"] --> I{"Need isolation: crash, security, or different language or binary?"}
I -->|"yes"| P["Separate processes (prefork, workers, sidecars)"]
I -->|"no"| C{"CPU-bound?"}
C -->|"yes"| TP["Thread pool sized to cores (or processes, for Python's GIL)"]
C -->|"no: I/O-bound"| N{"How many concurrent operations?"}
N -->|"hundreds"| BTP["Bounded thread pool via Goetz formula"]
N -->|"thousands to millions"| A["Goroutines, virtual threads, or async/event loop"]
- Processes when you need fault isolation (one crash shouldn't kill the rest — Chrome's tab-per-process, PostgreSQL's backend-per-connection, nginx/gunicorn pre-fork workers), privilege separation (OpenSSH), or to sidestep a global interpreter lock (Python
multiprocessing; free-threaded CPython 3.13+ is changing this). - Threads for shared-state parallelism inside one service, sized deliberately.
- Green threads / async tasks when concurrency is huge and mostly waiting: 100k WebSocket connections, fan-out RPC.
Real systems mix all three: nginx runs one process per core (process isolation), each with one event loop (async), plus a small thread pool for blocking file I/O.
Interview Questions
"What happens when you call fork()?"
Creates a child process that is an almost-exact copy of the parent via copy-on-write. Returns 0 in child, child PID in parent. Child shares physical memory pages until either writes (CoW). The kernel does have to copy page tables (~2MB per GB of RSS), which is why forking a very large process takes milliseconds and pauses it.
"What's the difference between a process and a thread?"
A process has its own address space and resources (isolation). Threads share the address space of their parent process. Threads are cheaper to create and switch between but a crash in one thread kills them all. In Linux both are task_structs created by clone(); the flags decide what is shared.
"What is a zombie process?"
A process that has exited but whose parent hasn't called wait(). The kernel keeps the PCB to store the exit code. Zombie entries consume a PID and a small amount of kernel memory.
"How does a container differ from a process?" A container is a process (or group of processes) that uses Linux namespaces (PID, network, mount, user, UTS, IPC) to create isolation and cgroups to limit resources. The process itself runs in the host kernel — there's no separate OS.
"Why do containers sometimes ignore SIGTERM or fill up with zombies?"
The app runs as PID 1 of the PID namespace: the kernel doesn't apply default signal actions to it, and it inherits every orphan. Without a handler and a reaping loop, SIGTERM is ignored until SIGKILL and zombies accumulate. Use tini / --init, exec-form CMD, and exec in entrypoint scripts.
"How many threads should a thread pool have?"
CPU-bound: about the number of cores available to the process (respecting container limits). I/O-bound: N_cpu × U × (1 + W/C), cross-checked with Little's Law (throughput × latency), and capped by what the downstream can handle. Then load-test to find the knee.
"Why are goroutines cheaper than threads?" 2KB growable stacks instead of 8MB reserved, switching in user space (~100–200ns) without a syscall, and an M:N scheduler that parks goroutines on network I/O instead of blocking an OS thread.
"What's the difference between exit() and _exit()?"
exit() runs atexit handlers and flushes stdio buffers, then calls _exit(), which terminates immediately. A forked child whose exec fails should call _exit() so it doesn't flush the parent's duplicated buffers.