On this page

Processes & Threads

A process is the fundamental unit of isolation in an operating system. Understanding what a process is — and how the kernel creates, manages, and switches between them — is the foundation for understanding concurrency, containers, and system design at the infrastructure level.

This page goes from first principles (why processes exist) through the exact Linux mechanics (task_struct, fork, clone, exec, wait), to the decisions you make every week in production: how many threads to run, why containers accumulate zombies, why fork() from a 50GB Redis pauses the server, and why "just add more threads" usually makes things slower.


Why Processes Exist

Early computers ran one program at a time. Two things pushed operating systems to invent the process:

  1. Multiplexing. A program waiting 10ms for a disk wastes ~30 million CPU cycles. If the OS can pause that program and run another, the CPU stays busy. Pausing requires saving everything needed to resume later — that bundle of state is the process.
  2. Protection. If two programs share a machine, a bug in one must not corrupt the other. Each process gets its own virtual address space, its own file descriptor table and its own credentials, enforced by hardware (the MMU and CPU privilege levels).

A process is therefore two things at once: a container of resources (memory, files, identity) and one or more threads of execution running inside it. Threads were split out later (1980s–90s) when people wanted multiple execution streams sharing one set of resources — concurrency without the isolation cost.


What Is a Process?

A process is a running instance of a program. It consists of:

  • Address space — the virtual memory the process can access (code, heap, stack, mapped files)
  • Program counter — which instruction executes next
  • Registers — current CPU state
  • Open file descriptors — references to open files, sockets, pipes
  • Process ID (PID) — unique identifier in the kernel
  • Credentials — UID, GID, permissions
  • Signal handlers — registered handlers for OS signals

The kernel represents all this in a Process Control Block (PCB) — a kernel data structure that stores everything needed to pause and resume a process.

Process Address Space (virtual memory):
┌─────────────────┐  high address
│      Stack      │  ← grows downward (function calls, local vars)
│        ↓        │
│   (free space)  │
│        ↑        │
│      Heap       │  ← grows upward (malloc/new)
├─────────────────┤
│  BSS Segment    │  uninitialized global/static variables
│  Data Segment   │  initialized global/static variables
│  Text Segment   │  executable code (read-only)
└─────────────────┘  low address (0x0)

In reality the "free space" in the middle also holds memory-mapped regions: shared libraries (libc.so), mmaped files, thread stacks and large malloc allocations (glibc uses mmap for allocations ≥128KB by default). On x86-64 the user half of the address space is 128TB (47 bits), and the layout is randomized (ASLR) every run. You can see the real map of any process:

$ cat /proc/self/maps | head -8
5603c1a00000-5603c1a02000 r--p 00000000 08:01 1835  /usr/bin/cat        # ELF header
5603c1a02000-5603c1a07000 r-xp 00002000 08:01 1835  /usr/bin/cat        # text
5603c1a0b000-5603c1a0c000 rw-p 0000a000 08:01 1835  /usr/bin/cat        # data
5603c2b1c000-5603c2b3d000 rw-p 00000000 00:00 0     [heap]
7f2e4c000000-7f2e4c1d5000 r-xp 00028000 08:01 2049  /usr/lib/x86_64-linux-gnu/libc.so.6
7ffd1a2f0000-7ffd1a311000 rw-p 00000000 00:00 0     [stack]
7ffd1a3c4000-7ffd1a3c8000 r--p 00000000 00:00 0     [vvar]
7ffd1a3c8000-7ffd1a3ca000 r-xp 00000000 00:00 0     [vdso]

See Virtual Memory for how these virtual ranges map to physical pages.

task_struct: Linux's PCB

In Linux the PCB is struct task_struct (in include/linux/sched.h). It's large — several kilobytes, with hundreds of fields — and most of it is pointers to shareable sub-structures. That design detail is the key to understanding Linux threads:

struct task_struct {                       // heavily abridged
    unsigned int            __state;       // TASK_RUNNING, TASK_INTERRUPTIBLE, ...
    void                    *stack;        // this task's kernel stack (16KB on x86-64)
    int                     prio, static_prio;
    struct sched_entity     se;            // CFS/EEVDF: vruntime, weight
    const struct sched_class *sched_class;
    cpumask_t               cpus_mask;     // CPU affinity
    pid_t                   pid;           // kernel "pid" = thread ID (TID)
    pid_t                   tgid;          // thread group ID = what userspace calls the PID
    struct task_struct      *real_parent, *parent;
    struct list_head        children, sibling;
    struct mm_struct        *mm;           // address space        (shareable: CLONE_VM)
    struct fs_struct        *fs;           // cwd, root, umask     (shareable: CLONE_FS)
    struct files_struct     *files;        // fd table             (shareable: CLONE_FILES)
    struct signal_struct    *signal;       // shared signal state  (CLONE_THREAD)
    struct sighand_struct   *sighand;      // signal handlers      (CLONE_SIGHAND)
    sigset_t                blocked;       // per-thread signal mask
    const struct cred       *cred;         // uid, gid, capabilities
    struct nsproxy          *nsproxy;      // namespaces           (CLONE_NEW*)
    struct css_set          *cgroups;
    int                     exit_state, exit_code;
    // ... several hundred more fields
};

Two tasks that point at the same mm, files and sighand are threads of one process. Two tasks with their own copies are separate processes. There is no separate "thread" object in the Linux kernel — just tasks that share more or less.

Note the PID naming trap: the kernel's pid field is what userspace calls the TID, and the kernel's tgid (thread group ID) is what userspace calls the PID. getpid() returns tgid; gettid() returns pid. For a single-threaded process they're equal.

flowchart LR
    subgraph P1["Process A (tgid 100)"]
        T1["task 100 (main thread)"]
        T2["task 101"]
        T3["task 102"]
    end
    MM["mm_struct: page tables, VMAs"]
    FD["files_struct: fd table"]
    SH["sighand_struct: handlers"]
    T1 --> MM
    T2 --> MM
    T3 --> MM
    T1 --> FD
    T2 --> FD
    T3 --> FD
    T1 --> SH
    T2 --> SH
    T3 --> SH
    subgraph P2["Process B (tgid 200)"]
        T4["task 200"]
    end
    MM2["own mm_struct"]
    FD2["own files_struct"]
    T4 --> MM2
    T4 --> FD2

PIDs are a finite resource

$ cat /proc/sys/kernel/pid_max
4194304          # systemd raises it to 2^22 on 64-bit; the kernel default is 32768
$ cat /proc/sys/kernel/threads-max
251582           # derived from RAM at boot
$ ls /proc | grep -c '^[0-9]'      # live processes (thread groups)
412

Every thread consumes a PID number. On older systems still at pid_max = 32768, a single Java process with 30,000 threads or a zombie leak can exhaust it, and then fork() fails system-wide with EAGAIN.


The Process Lifecycle at a Glance

sequenceDiagram
    participant Sh as "Parent (shell)"
    participant K as "Kernel"
    participant C as "Child"
    Sh->>K: "fork() / clone()"
    K->>C: "new task_struct, CoW copy of address space"
    K-->>Sh: "returns child PID"
    K-->>C: "returns 0"
    C->>K: "execve(/bin/ls)"
    K->>C: "old image discarded, ELF mapped, entry point"
    C->>C: "main() runs"
    C->>K: "exit(0)"
    K->>K: "free memory and fds, keep exit status (zombie)"
    K-->>Sh: "SIGCHLD"
    Sh->>K: "waitpid()"
    K-->>Sh: "exit status 0, zombie reaped"

Four system calls, each with a precise job:

CallJobReturns
fork()Duplicate the calling processTwice: child PID in parent, 0 in child
execve()Replace the current program imageNever, on success
exit() / _exit()Terminate, leave an exit statusNever
wait() / waitpid()Collect a child's exit status, free its zombieChild PID + status

fork(): Creating a New Process

fork() creates a child process that is an exact copy of the parent. It returns twice: once in the parent (returns child PID), once in the child (returns 0).

#include <unistd.h>
#include <stdio.h>
#include <sys/wait.h>

int main() {
    pid_t pid = fork();

    if (pid < 0) {
        // fork failed
        perror("fork");
    } else if (pid == 0) {
        // Child process
        printf("Child: PID=%d, parent PID=%d\n", getpid(), getppid());
    } else {
        // Parent process
        printf("Parent: PID=%d, child PID=%d\n", getpid(), pid);
        wait(NULL);  // wait for child to finish
    }
    return 0;
}

What fork() copies:

  • Virtual address space (pages, not physical memory — see copy-on-write)
  • File descriptor table (both parent and child have open FDs pointing to the same files)
  • Signal handlers
  • Environment variables, working directory

Note that "pointing to the same files" is literal: parent and child share the open file description, including the file offset. If both write to an inherited log file descriptor without O_APPEND, they advance one shared offset — a classic source of interleaved or overwritten output.

What fork() does NOT copy:

  • The PID (child gets a new one)
  • Memory locks
  • Pending signals are cleared in child
  • Threads (only the calling thread is duplicated — others don't exist in child)

Also not inherited: timers (timer_create, alarm), record locks (fcntl locks), async I/O operations, and the parent's resource-usage counters (the child starts at zero).

fork() in a multithreaded program

Because only the calling thread survives, any mutex that another thread held at the moment of fork() stays locked forever in the child — with no thread left to unlock it. If the child then calls malloc() (which takes an allocator lock) or printf() (stdio lock), it can deadlock. POSIX's rule: after fork() in a multithreaded program, the child may only call async-signal-safe functions until it calls exec(). This is why Python 3.12+ warns when you os.fork() with threads running, and why multiprocessing defaults to spawn on macOS and, from Python 3.14, to forkserver (not plain fork) on Linux.

Copy-on-Write (CoW)

fork() doesn't physically copy memory. The child's page table initially points to the same physical pages as the parent — marked read-only. When either process writes to a page, the kernel:

  1. Catches the page fault (write to read-only page)
  2. Allocates a new physical page
  3. Copies the content
  4. Updates the faulting process's page table to point to the new page
  5. Resumes the write

This is why fork() is fast even for a 2GB process — it only copies page tables, not 2GB of data. Pages are copied lazily, only as they're written.

flowchart TD
    subgraph Before["Right after fork()"]
        PP["Parent page table"] -->|"read-only"| PG["Physical page X"]
        CP["Child page table"] -->|"read-only"| PG
    end
    subgraph After["Child writes to the page"]
        PP2["Parent page table"] -->|"read-write again"| PGX["Physical page X"]
        CP2["Child page table"] -->|"read-write"| PGY["New page Y, copy of X"]
    end
    Before --> After

The numbers. "Only copies page tables" is still real work. With 4KB pages, every 1GB of resident memory needs 262,144 page-table entries × 8 bytes = 2MB of PTEs to copy, plus marking every entry read-only and a TLB flush. In practice:

Parent RSSPage-table copyTypical fork() time
10MB (small CLI tool)~20KB~50–100μs
1GB~2MB~10–20ms
25GB (big Redis)~50MB~0.3–0.5s — the whole process is paused during it

Redis publishes this metric (INFO stats → latest_fork_usec) because the BGSAVE/AOF-rewrite fork() blocks the event loop; its latency documentation measures roughly 10–20ms per GB on modern hardware and VMs, and historically far worse on older Xen guests.

The second cost comes after fork: every page that either side writes gets copied. A Redis instance taking a snapshot while serving a write-heavy workload can see memory usage grow toward 2× during the snapshot.

War Story: Transparent Huge Pages and fork

With Transparent Huge Pages (THP) enabled, memory is mapped in 2MB pages. After fork(), a single-byte write to a 2MB huge page copies the entire 2MB instead of 4KB — a 512× amplification. Redis, MongoDB and several other databases explicitly recommend setting THP to never or madvise for this reason; Redis prints a warning at startup if THP is set to always. The symptom is latency spikes and RSS ballooning during background saves.

vfork, posix_spawn and clone3

Because most fork() calls are immediately followed by exec(), copying page tables only to throw them away is wasteful for large parents. Alternatives:

  • vfork() — the child borrows the parent's address space and the parent is suspended until the child calls exec() or _exit(). No page-table copy at all, but the child must not touch memory. Fast, dangerous.
  • posix_spawn() — the portable "fork+exec in one call". glibc implements it with clone(CLONE_VM | CLONE_VFORK) and a separate small stack, so it's as fast as vfork without its hazards. Go's os/exec, Rust's std::process::Command (in many cases), and Java's process launching on Linux all use vfork-style spawning for exactly this reason.
  • clone3() (Linux 5.3+) — the extensible modern syscall underneath everything. glibc's fork() and pthread_create() are thin wrappers over clone/clone3.
// Rust: spawn + wait, the idiomatic fork/exec/wait
use std::process::Command;

fn main() -> std::io::Result<()> {
    let status = Command::new("ls").arg("-l").status()?;   // spawn, then waitpid
    println!("ls exited with {}", status);                 // e.g. "exit status: 0"
    Ok(())
}

exec(): Replacing the Process Image

exec() replaces the current process's address space with a new program. The PID stays the same but everything else is replaced.

execve("/bin/ls", args, env);
// Code after this never runs in the current process
// (unless execve fails)

"Everything else" has precise exceptions. Across execve() the process keeps: PID and parent, open file descriptors (unless marked close-on-exec), current directory, resource limits, nice value, signal mask and ignored signals, and cgroup membership. It loses: all memory mappings, all threads except the caller, signal handlers (reset to default, since the handler code no longer exists), and mmaped regions.

Always open with O_CLOEXEC

File descriptors survive exec() by default. A server that opens a database socket and then spawns a helper process leaks that socket into the helper — which can keep connections open after the server closes them, or expose secrets to a less-trusted child. Use O_CLOEXEC / SOCK_CLOEXEC / FD_CLOEXEC (Go, Rust and Python 3.4+ do this by default).

The fork-exec pattern: Most process creation in Unix follows this pattern:

  1. fork() — create a copy of the current process
  2. In the child: exec() — replace the child with the target program
  3. In the parent: wait() — wait for the child to finish
# When your shell runs `ls`:
# 1. shell forks a child
# 2. child exec()s /bin/ls
# 3. shell waits for the child
# 4. child exits, shell is notified

Why split creation into two calls instead of one CreateProcess() like Windows? Because the gap between fork() and exec() is where the child configures itself using ordinary system calls: redirect stdout (dup2), close fds, change directory, drop privileges, set resource limits, join namespaces. That's how shells implement cmd > out.txt 2>&1 and how container runtimes set up isolation before running your entrypoint.

// How a shell runs:  ls -l > out.txt
pid_t pid = fork();
if (pid == 0) {
    int fd = open("out.txt", O_WRONLY | O_CREAT | O_TRUNC, 0644);
    dup2(fd, STDOUT_FILENO);       // stdout now points at out.txt
    close(fd);
    char *argv[] = {"ls", "-l", NULL};
    execvp("ls", argv);
    _exit(127);                    // only reached if exec failed: "command not found"
}
int status;
waitpid(pid, &status, 0);

Watch it happen:

$ strace -f -e trace=clone,clone3,execve,wait4,exit_group bash -c 'ls > /dev/null'
execve("/usr/bin/bash", ["bash", "-c", "ls > /dev/null"], 0x7ffc... /* 30 vars */) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, ...) = 48211
[pid 48211] execve("/usr/bin/ls", ["ls"], 0x55d3... /* 30 vars */) = 0
[pid 48211] exit_group(0)             = ?
[pid 48211] +++ exited with 0 +++
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 48211

(Bash runs the last command of -c directly via exec without forking in some cases; the redirect here forces a fork.) Note the clone(... SIGCHLD) — that's fork(): a clone with no sharing flags, and "send me SIGCHLD when this child dies".


exit() and wait(): Ending a Process

Exiting

  • exit(status) (libc): runs atexit handlers, flushes stdio buffers, then calls _exit.
  • _exit(status) (syscall exit_group): terminates immediately. All threads in the process die.
  • Returning from main is the same as calling exit.

Use _exit() in a forked child that fails to exec: calling exit() there would flush the parent's copied stdio buffers a second time — output duplicated.

On exit the kernel closes all file descriptors, releases memory, reparents any children to a reaper, and sends SIGCHLD to the parent. What it keeps is a tiny residue: the task_struct with the exit status and resource usage. That residue is the zombie.

Exit status conventions

StatusMeaning
0Success
1–125Program-defined failure
126Found but not executable (shell)
127Command not found (shell)
128 + NKilled by signal N — e.g. 137 = 128 + 9 (SIGKILL, the OOM killer's signature in Kubernetes), 143 = 128 + 15 (SIGTERM)

Waiting

int status;
pid_t pid = waitpid(-1, &status, WNOHANG);   // any child, don't block
if (pid > 0) {
    if (WIFEXITED(status))        printf("exited %d\n", WEXITSTATUS(status));
    else if (WIFSIGNALED(status)) printf("killed by signal %d\n", WTERMSIG(status));
}

The standard pattern for a server that spawns children is a SIGCHLD handler that reaps in a loop — one signal can stand for many dead children, because standard signals don't queue:

static void on_sigchld(int sig) {
    int saved = errno;
    while (waitpid(-1, NULL, WNOHANG) > 0) { }   // reap every finished child
    errno = saved;
}
// install with sigaction(SIGCHLD, ...) and SA_RESTART | SA_NOCLDSTOP

Setting SIGCHLD to SIG_IGN tells the kernel to auto-reap children (no zombies, but no exit statuses either). Linux 5.3+ also offers pidfds (pidfd_open, CLONE_PIDFD): a file descriptor for a process that you can poll/epoll for exit — no signal handler, no PID-reuse race. See Signals & Process Control.


Process States

            fork()
              │
              ▼
           CREATED
              │  scheduler picks it up
              ▼
┌────────── READY ◄────────────────┐
│             │  scheduled to run  │
│             ▼                    │
│          RUNNING ────────────────┘ preempted (time slice expired)
│             │
│    ┌────────┴───────┐
│    │                │
│    ▼                ▼
│  WAITING         exit()
│  (I/O, sleep,     │
│   signal)         ▼
│    │           ZOMBIE ─────────► parent wait() → DEAD
│    │           (exit code stored
│    │            until parent reads)
└────┘  I/O done / signal received

The same states as a state machine, with what Linux actually calls them:

stateDiagram-v2
    [*] --> Runnable: "fork / clone"
    Runnable --> Running: "picked by scheduler"
    Running --> Runnable: "preempted"
    Running --> Sleeping: "wait for I/O, lock, timer"
    Sleeping --> Runnable: "woken up"
    Running --> Stopped: "SIGSTOP / ptrace"
    Stopped --> Runnable: "SIGCONT"
    Running --> Zombie: "exit"
    Zombie --> [*]: "parent calls wait"

The single letter you see in ps / top:

ps STATKernel stateMeaning
RTASK_RUNNINGRunning or runnable (Linux doesn't distinguish READY and RUNNING in this field)
STASK_INTERRUPTIBLESleeping; a signal will wake it
DTASK_UNINTERRUPTIBLESleeping in the kernel, typically disk/NFS I/O; even SIGKILL waits
T / tstopped / tracedJob control stop or debugger
ZEXIT_ZOMBIEExited, waiting to be reaped
Iidle kernel threadKernel thread sleeping (not counted in load)

D matters operationally: tasks in D count toward the Linux load average even though they use no CPU. A load average of 80 on an idle-looking box is usually dozens of processes stuck in D on a hung NFS mount or a saturated disk. And kill -9 won't remove them until the I/O completes (TASK_KILLABLE sleeps are the partial exception).

Zombie process: A process that has exited but whose parent hasn't called wait() yet. The kernel keeps the PCB to store the exit code. If the parent never calls wait(), zombie entries accumulate (not a memory leak per se, but PIDs are finite).

Orphan process: A process whose parent has exited. The kernel reparents orphans to PID 1 (init/systemd), which calls wait() on them.

More precisely, an orphan is reparented to the nearest ancestor that has marked itself a child subreaper (prctl(PR_SET_CHILD_SUBREAPER, 1)) — systemd user managers, tini, and container shims use this — or to PID 1 of its PID namespace. In a container, your application is often PID 1, which is where trouble starts (see failure modes below).

You can't kill a zombie — it's already dead. You make it go away by getting its parent to call wait(), or by killing the parent so the zombie is reparented to a reaper that will.

$ ps -eo pid,ppid,stat,comm | awk '$3 ~ /Z/'
 48812  48700 Z    worker <defunct>
$ ps -o pid,comm -p 48700        # the parent that isn't reaping
 48700 bad-supervisor

Threads vs Processes

A thread is an execution context within a process. Multiple threads share:

  • Address space (heap, global variables, file descriptors)
  • PID (but each thread has its own TID — Thread ID)

Each thread has its own:

  • Stack
  • Program counter
  • Registers
  • Signal mask

Each thread also has its own thread-local storage (__thread / thread_local variables, errno), its own scheduling priority and CPU affinity, and its own kernel stack.

Process:
┌──────────────────────────────────────────┐
│ Code | Data | Heap | File Descriptors    │
│                                          │
│  Thread 1  │  Thread 2  │  Thread 3     │
│  [stack]   │  [stack]   │  [stack]      │
│  [regs]    │  [regs]    │  [regs]       │
│  [PC]      │  [PC]      │  [PC]         │
└──────────────────────────────────────────┘
ProcessThread
MemorySeparate address spaceShared address space
CommunicationIPC (pipes, sockets, shared memory)Shared variables (needs synchronization)
Creation costExpensive (~1ms, copies page tables)Cheap (~10μs)
Context switchExpensive (TLB flush)Cheaper (same address space)
IsolationStrong (crash doesn't affect others)Weak (crash kills all threads)
Use caseIsolation, security boundariesShared-state concurrency

The creation-cost row is order-of-magnitude. On a modern Linux box: pthread_create + join ≈ 10–20μs; fork of a small process ≈ 50–100μs; fork + exec + wait of a small binary ≈ 0.5–1ms (most of that is exec: mapping the ELF, the dynamic linker loading libraries). Large parents make fork slower, as shown above.

Linux implementation detail: Linux implements threads as "lightweight processes" using clone(). A thread is a process that shares its address space, file descriptors, and other resources with the parent. fork() calls clone() with no sharing; pthread_create() calls clone() with full sharing.

clone() flags: the dial between process and thread

FlagSharesUsed by
CLONE_VMAddress space (mm_struct)Threads, vfork, posix_spawn
CLONE_FScwd, root, umaskThreads
CLONE_FILESFile descriptor tableThreads
CLONE_SIGHANDSignal handlersThreads (requires CLONE_VM)
CLONE_THREADSame thread group (same PID/tgid), shared signal deliveryThreads
CLONE_SYSVSEMSystem V semaphore undo listThreads
CLONE_SETTLSSets the new thread's TLS pointer (FS base on x86-64)Threads
CLONE_PARENT_SETTID / CLONE_CHILD_CLEARTIDWrite TID to memory / clear it and futex-wake on exit (how pthread_join works)Threads
CLONE_VFORKSuspend parent until child execs or exitsvfork, posix_spawn
CLONE_NEWPID, CLONE_NEWNET, CLONE_NEWNS, ...Opposite: new, unshared namespacesContainer runtimes
CLONE_PIDFDReturn a pidfd for the childModern process supervisors

strace on a pthread_create() call shows glibc asking for nearly everything:

clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|
        CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
        child_tid=0x7f3a1b7ff910, parent_tid=0x7f3a1b7ff910, exit_signal=0,
        stack=0x7f3a1afff000, stack_size=0x7fff00, tls=0x7f3a1b7ff640} => {...}, 88) = 48233

stack_size=0x7fff00 — just under 8MB, the default thread stack. And the namespace flags at the bottom of the table are how containers are built from the same primitive: a container is a clone() with less sharing than usual.

Thread signal semantics

A process-directed signal (kill(pid, SIGTERM)) is delivered to one thread that doesn't have it blocked — any one. The robust pattern in multithreaded programs is to block signals in all threads and have one dedicated thread call sigwait() / use signalfd. A fatal signal (e.g. SIGSEGV in any thread) kills the whole process. exit() in any thread ends all threads; pthread_exit() ends only the caller.


Kernel Threads vs User Threads

Two different things share the word "thread":

  • Kernel threads — tasks that run only in kernel mode, with no user address space (mm == NULL). They do the kernel's background work. In ps they appear in square brackets, all descended from kthreadd (PID 2):
$ ps -eo pid,ppid,comm | head -12
    PID    PPID COMMAND
      1       0 systemd
      2       0 kthreadd
      3       2 pool_workqueue_release
     15       2 ksoftirqd/0          # deferred interrupt work for CPU 0
     16       2 rcu_preempt          # RCU grace-period machinery
     17       2 migration/0          # moves tasks between CPUs (stop class)
     95       2 kswapd0              # page reclaim
    311       2 kworker/3:1H-kblockd # workqueue worker
    402       2 jbd2/sda1-8          # ext4 journal commit

A ksoftirqd at high CPU means the box is drowning in network or block interrupts; kswapd at high CPU means memory pressure. See Syscalls & Kernel Internals.

  • User threads — threads of your program. Whether each user thread corresponds to a kernel task depends on the thread model.

Thread Models: 1:1, N:1, M:N

How many kernel-scheduled entities back your language's "threads"?

flowchart TB
    subgraph OneToOne["1:1 (pthreads, Java platform threads, Rust std::thread)"]
        U1["user thread"] --> K1["kernel task"]
        U2["user thread"] --> K2["kernel task"]
    end
    subgraph MToN["M:N (Go, Erlang, Java virtual threads)"]
        G1["goroutine"] --> W1["OS thread"]
        G2["goroutine"] --> W1
        G3["goroutine"] --> W2["OS thread"]
        G4["goroutine x 100k"] --> W2
    end
ModelExamplesProsCons
1:1 — each user thread is a kernel taskLinux NPTL pthreads, Windows threads, Java platform threads, Rust std::thread, C++ std::threadSimple; true parallelism; blocking syscalls block only one thread; kernel tools see everything~8MB virtual stack and a kernel task each; switches go through the kernel; tens of thousands is the practical ceiling
N:1 — all user threads on one kernel taskEarly Java "green threads" (1.1), GNU Pth, classic coroutinesVery cheap switchesNo parallelism; one blocking syscall blocks everything
M:N — M user threads multiplexed over N kernel threadsGo goroutines, Erlang/Elixir processes, Java virtual threads (JDK 21+), Kotlin coroutines on dispatchersMillions of cheap tasks; user-space switches (~100–200ns); parallel across coresRuntime must avoid blocking its OS threads; harder debugging; scheduling invisible to the kernel

Linux itself tried M:N (NGPT, IBM, ~2002) and dropped it in favor of the simpler 1:1 NPTL, which became fast enough that the complexity wasn't worth it. M:N came back in language runtimes where the runtime knows when a task will block.

Go's G-M-P scheduler

  • G — a goroutine: a stack (starts at 2KB, grows by copying, up to 1GB on 64-bit), a PC, and some state. ~2–4KB total.
  • M — an OS thread (machine).
  • P — a processor context with a local run queue of Gs. There are GOMAXPROCS Ps (default: number of CPUs, and since Go 1.25 the cgroup CPU limit).

An M must hold a P to run Go code. When a goroutine makes a blocking syscall, its M detaches from its P and another M takes the P, so other goroutines keep running. Network I/O doesn't block at all: it parks the goroutine and registers the fd with the runtime's epoll-based netpoller. Idle Ps steal half of another P's run queue. Since Go 1.14, goroutines are asynchronously preempted via signals after ~10ms, so a tight loop can't hog a P.

package main

import (
	"fmt"
	"runtime"
	"sync"
)

func main() {
	fmt.Println("GOMAXPROCS:", runtime.GOMAXPROCS(0)) // number of Ps
	var wg sync.WaitGroup
	for i := 0; i < 100_000; i++ { // 100k goroutines: ~200-400MB worst case, usually far less
		wg.Add(1)
		go func() { defer wg.Done() }()
	}
	wg.Wait()
	fmt.Println("goroutines now:", runtime.NumGoroutine())
}

ps -T -p <pid> on that program shows only a handful of OS threads — the kernel never sees the 100,000 goroutines.

Java virtual threads

JDK 21 (JEP 444) added virtual threads: Thread.ofVirtual().start(...) or Executors.newVirtualThreadPerTaskExecutor(). They're mounted on a ForkJoinPool of carrier threads (default size = CPU count); a blocking JDK I/O call unmounts the virtual thread and frees the carrier. The big caveat until JDK 24: blocking inside a synchronized block pinned the carrier, so a few pinned virtual threads could stall the whole pool. JEP 491 (JDK 24) removed most pinning for synchronized. Native calls (JNI) still pin.

Stackless async (Rust, Node.js, Python asyncio)

Rust's async fn compiles each task into a state machine whose size is known at compile time (often a few hundred bytes) — no per-task stack at all. Tokio then runs these on a work-stealing pool of OS threads, much like Go's Ps. Node.js runs JavaScript on one thread with an event loop, plus a libuv threadpool (default 4 threads, UV_THREADPOOL_SIZE) for file I/O, DNS and crypto. The trade-off versus goroutines: no hidden stacks and cheaper tasks, but async "colors" every function in the call chain. See Async Patterns.


Thread Stacks and Limits

Stack sizes

Kind of threadDefault stackNotes
Linux main threadGrows up to ulimit -s, usually 8MBGrows on demand via page faults
glibc pthread_createRLIMIT_STACK if set, typically 8MBFixed-size mmap + guard page; set with pthread_attr_setstacksize
musl (Alpine) pthreads128KBDeep recursion that works on glibc can segfault on Alpine
Java platform thread1MB on 64-bit Linux (-Xss)Plus native and JVM overhead
Go goroutine2KB initial, grows by copyingMax 1GB (debug.SetMaxStack)
Rust std::thread2MB (RUST_MIN_STACK, Builder::stack_size)Main thread follows OS limit
Kernel stack per task16KB on x86-64Fixed; kernel stack overflow is fatal

The 8MB is virtual, reserved not committed. Only pages actually touched consume RAM, so 1,000 threads each using 40KB of stack cost ~40MB of RSS, not 8GB. But the virtual reservation still counts against RLIMIT_AS, against overcommit accounting in strict mode (vm.overcommit_memory=2), and, on 32-bit systems, against a 3GB address space (which capped 32-bit processes at ~380 threads).

Each thread also has a guard page (an unmapped/no-access page below the stack): overflow faults with SIGSEGV instead of silently trampling the neighbouring thread's memory.

pthread_attr_t attr;
pthread_attr_init(&attr);
pthread_attr_setstacksize(&attr, 256 * 1024);    // 256KB for a shallow worker
pthread_create(&tid, &attr, worker, NULL);

How many threads can a process have?

Several limits apply simultaneously; the first one you hit wins:

LimitWhereTypical defaultError when hit
Per-user processes+threadsulimit -u (RLIMIT_NPROC)Often ~63,000 or unlimited; some distros 4096EAGAIN from clone
System-wide threads/proc/sys/kernel/threads-maxRAM-derived, ~250k on 32GBEAGAIN
PID space/proc/sys/kernel/pid_max32,768 or 4,194,304EAGAIN
Memory mappings per process/proc/sys/vm/max_map_count65,530 (each thread stack + guard ≈ 2 maps)ENOMEM
cgroup pids controllerpids.max in the cgroup; Kubernetes podPidsLimit; systemd TasksMaxVaries; systemd sets a default per unitEAGAIN
MemoryRAM / container memory limit—ENOMEM or OOM kill

What the errors look like in practice:

pthread_create failed: Resource temporarily unavailable           # C, EAGAIN
java.lang.OutOfMemoryError: unable to create native thread        # Java — usually NOT heap exhaustion
runtime/cgo: pthread_create failed: Resource temporarily unavailable   # Go hitting pids.max
fork: retry: Resource temporarily unavailable                     # bash
# Who is using all the threads?
$ ps -eo nlwp,pid,comm --sort=-nlwp | head -5       # NLWP = number of threads
 NLWP     PID COMMAND
 4012   21345 java
  612    1803 containerd
$ cat /sys/fs/cgroup/pids.current /sys/fs/cgroup/pids.max
4031
4096

Sizing Thread Pools

Thread pools exist because creating a thread per task costs ~10–20μs plus its memory, and because an unbounded number of threads destroys performance through context switches and cache thrash (see CPU Scheduling). The question is always: how many?

CPU-bound work

threads = N_cpu            (sometimes N_cpu + 1, to cover the occasional page fault or stall)

More threads than cores cannot do more CPU work — they only add switching. If the process runs in a container with a CPU limit of 4, N_cpu is 4, not the node's 64 cores.

I/O-bound work: the Goetz formula

From Java Concurrency in Practice:

threads = N_cpu × U_cpu × (1 + W / C)

N_cpu = number of cores
U_cpu = target CPU utilization (0..1)
W / C = ratio of wait time to compute time per task

Worked example: 8 cores, target 80% utilization; each request spends 45ms waiting on a database and 5ms computing (W/C = 9):

threads = 8 × 0.8 × (1 + 45/5) = 8 × 0.8 × 10 = 64 threads

Cross-check with Little's Law

Concurrency needed = throughput × latency. If that service must handle 1,200 requests/s at 50ms each:

L = λ × W = 1,200/s × 0.050s = 60 requests in flight

60 in-flight requests need ~60 threads in a thread-per-request design — consistent with the 64 above. If Little's Law says you need 600 in flight but the CPU formula says 64, the pool isn't the real limit: the downstream (the database) is, and adding threads just builds a bigger queue in front of it.

The downstream sets the ceiling

The 64 threads above all hit one database. That database's connection pool — and its own CPU — is the real bottleneck. HikariCP's well-known starting point for a database's connection pool is far smaller than people expect:

connections = (core_count_of_db × 2) + effective_spindle_count

A 16-core Postgres server with SSDs starts around 32–40 connections total across all clients — not 64 per app instance × 50 instances.

WorkloadStarting pointWatch for
Pure CPU (image resize, compression, crypto)N_cpuRun-queue latency, involuntary switches
Mixed request handlingGoetz formula, verify with Little's Lawp99 latency vs pool size curve
Blocking I/O to one downstreamMatch the downstream's capacityDownstream saturation, queueing
Massive concurrent I/O (10k+ connections)Don't use a thread per connection: event loop, goroutines, virtual threadsMemory per connection
// Bounded worker pool in Go: N workers, bounded queue = backpressure
func process(jobs []Job, workers int) {
	ch := make(chan Job, workers*2) // small buffer, not unbounded
	var wg sync.WaitGroup
	for i := 0; i < workers; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for j := range ch {
				handle(j)
			}
		}()
	}
	for _, j := range jobs {
		ch <- j // blocks when workers are busy: natural backpressure
	}
	close(ch)
	wg.Wait()
}

Measure, then size

Formulas give a starting point. The real answer is a load test that plots throughput and p99 latency against pool size: throughput rises, flattens at the knee, and latency then climbs as you add threads past it. Pick the knee, and keep the queue in front of the pool bounded so overload sheds requests instead of piling them up.


Inter-Process Communication (IPC)

Processes can't directly access each other's memory (isolation). IPC mechanisms:

Pipes: Unidirectional byte stream. Parent creates before fork; child inherits. Used to chain processes (ls | grep foo).

int pipefd[2];
pipe(pipefd);  // pipefd[0] = read end, pipefd[1] = write end

Unix Domain Sockets: Bidirectional, local-only. Higher throughput than TCP loopback for same-machine IPC. Used by Docker, PostgreSQL, systemd.

Shared Memory: Fastest IPC — processes map the same physical pages into their address spaces. No data copying. Requires explicit synchronization.

int shm_id = shmget(key, size, IPC_CREAT | 0666);
void* ptr = shmat(shm_id, NULL, 0);

Message Queues: Kernel-managed queues. Sender writes message, receiver reads it. POSIX (mq_open) or SysV (msgget).

Signals: Software interrupts. Asynchronous notification. Limited payload (just the signal number). Used for process control: SIGTERM (terminate gracefully), SIGKILL (terminate immediately, can't be caught), SIGUSR1/SIGUSR2 (application-defined).

For the mechanics, performance numbers and trade-offs of each mechanism, see Inter-Process Communication and Signals & Process Control.


What Happens When You Run a Program

$ python3 app.py
  1. Shell calls fork() — creates child shell process
  2. Child calls execve("/usr/bin/python3", ["python3", "app.py"], env)
  3. Kernel validates the executable (ELF header check)
  4. Kernel maps the ELF segments into the child's address space (code, data)
  5. Kernel sets up the stack with argv, envp
  6. Kernel sets PC to the entry point
  7. Dynamic linker (ld.so) runs first — loads shared libraries (libc, etc.)
  8. main() begins executing
  9. Python interpreter starts, parses app.py, executes bytecode
  10. When app.py exits: Python calls exit() → kernel cleans up process → sends SIGCHLD to parent shell

A few details hidden in those steps: in step 3 the kernel also handles #! scripts (it re-execs the interpreter named on the shebang line); in step 5 it also places the auxiliary vector (AT_ENTRY, AT_RANDOM, the vDSO address) on the stack; and in step 7 the entry point is actually ld.so's, which then jumps to the program's _start → __libc_start_main → main. Statically linked binaries (common in Go) skip step 7 entirely — one reason they start faster. And between steps 1 and 2 the shell sets up redirections and the process group for job control.

$ strace -e trace=execve,openat python3 -c 'pass' 2>&1 | head   # exec + libraries being opened
$ strace -c python3 -c 'pass'                            # count them: typically 300-1,000 syscalls
$ LD_DEBUG=libs /bin/true 2>&1 | head                    # watch ld.so find libraries

Exploring Processes Through /proc

/proc is the kernel's live view of every task. Everything ps, top and htop show comes from here.

$ PID=$(pgrep -o nginx)
$ cat /proc/$PID/status | egrep 'Name|State|PPid|Threads|VmRSS|VmSize|voluntary'
Name:   nginx
State:  S (sleeping)
PPid:   1
Threads:        1
VmSize:    55120 kB          # virtual size
VmRSS:      9812 kB          # resident memory
voluntary_ctxt_switches:        8812
nonvoluntary_ctxt_switches:     41

$ ls -l /proc/$PID/fd | head -4          # open file descriptors
lrwx------ 1 root root 64 Nov  8 10:00 0 -> /dev/null
lrwx------ 1 root root 64 Nov  8 10:00 6 -> socket:[48211]
l-wx------ 1 root root 64 Nov  8 10:00 3 -> /var/log/nginx/access.log

$ ls /proc/$PID/task                     # one directory per thread (TID)
$ cat /proc/$PID/limits | egrep 'processes|open files|stack'
Max stack size            8388608              unlimited            bytes
Max processes             63314                63314                processes
Max open files            1024                 524288               files
$ tr '\0' ' ' < /proc/$PID/cmdline       # argv
$ cat /proc/$PID/cgroup                  # which cgroup (container) it's in
$ readlink /proc/$PID/exe /proc/$PID/cwd # binary and working directory
$ cat /proc/$PID/wchan; echo             # kernel function it's sleeping in

Useful one-liners:

$ ps -eLf | wc -l                     # total threads on the box
$ ps -T -p $PID                       # threads of one process (SPID = TID)
$ top -H -p $PID                      # live per-thread CPU
$ pstree -p $PID                      # process tree
$ ps -eo pid,stat,wchan:20,comm | awk '$2 ~ /D/'   # who is stuck in uninterruptible sleep
$ cat /proc/$PID/task/*/stat | awk '{print $1, $3, $14+$15}'   # TID, state, CPU ticks

Failure Modes

Fork bombs

:(){ :|:& };:      # DO NOT RUN: defines ":" which calls itself twice in the background

Each invocation spawns two more; within seconds every PID or the per-user process limit is exhausted, and nothing new — including ssh, ps, or the shell you'd use to fix it — can start. Defenses, in order of strength:

  • ulimit -u / RLIMIT_NPROC per user (/etc/security/limits.conf).
  • systemd TasksMax= per service and per user slice.
  • cgroup pids.max per container; Kubernetes podPidsLimit in the kubelet config. Set this — a single misbehaving pod can otherwise exhaust the node's PIDs and break every other pod on it.

Recovery when it's happening: kill -STOP first (stop the bomb from multiplying), then kill. kill -9 -1 as the offending user kills all their processes.

Thread explosion

A thread-per-request server under a traffic spike, or a retry storm, creates threads until a limit from the table above is hit. Symptoms appear in this order: run-queue latency climbs and throughput falls (context switching), memory grows, then thread creation fails with EAGAIN / "unable to create native thread", and the process may crash. Common causes: unbounded Executors.newCachedThreadPool(), a new thread for every blocking call, a library creating a pool per client object, or GOMAXPROCS-blind thread-hungry cgo code. Fix with bounded pools and bounded queues.

War Story (illustrative)

A service created a new HTTP client object — each with its own internal thread pool — on every request instead of reusing one. Under normal load the GC cleaned them up fast enough. During a downstream slowdown, requests piled up, pools accumulated faster than they were freed, and the JVM reached the pod's pids.max of 4,096 within two minutes. Every new request failed with OutOfMemoryError: unable to create native thread while heap usage sat at 30%. ps -o nlwp on the pod showed 4,090 threads. The fix was a single shared client.

Zombie accumulation in containers (the PID 1 problem)

In a container, the entrypoint process is PID 1 of its PID namespace, which gives it two jobs most applications don't know they have:

  1. Reaping orphans. Any process whose parent dies gets reparented to PID 1. If your app (Node, Python, Java) is PID 1 and never calls wait() on children it didn't create, zombies accumulate — e.g. a shell script spawning curl health checks every few seconds, or a headless Chrome leaving renderer processes behind. Eventually pids.max is hit and fork() fails.
  2. Signal handling. The kernel doesn't apply default signal actions to PID 1 of a namespace. A SIGTERM to a PID 1 that hasn't installed a handler is ignored — so docker stop / pod termination waits the full grace period (10s Docker, 30s Kubernetes) and then sends SIGKILL. Every deploy is slow and every shutdown is unclean.
flowchart TD
    D["docker stop / pod delete"] --> T["SIGTERM to PID 1"]
    T --> Q{"PID 1 installed a SIGTERM handler?"}
    Q -->|"no: app is PID 1"| IG["signal ignored"]
    IG --> W["wait full grace period"]
    W --> K["SIGKILL: unclean shutdown"]
    Q -->|"yes: tini or handler"| G["forward to app, graceful shutdown"]
    G --> R["init also reaps zombies with wait"]

Fixes:

# Use a tiny init as PID 1: it reaps zombies and forwards signals
RUN apk add --no-cache tini
ENTRYPOINT ["/sbin/tini", "--"]
CMD ["node", "server.js"]
  • docker run --init injects tini automatically.
  • Use the exec form CMD ["node", "server.js"], not the shell form CMD node server.js — the shell form makes /bin/sh PID 1, which may not forward signals.
  • In shell entrypoint scripts, end with exec "$@" so your app replaces the shell.
  • In Kubernetes, shareProcessNamespace: true makes the pause container PID 1, and it reaps zombies.

Other common failures

FailureCauseFix
fork() fails with ENOMEM on a box with free RAMStrict overcommit (vm.overcommit_memory=2) must account for a full CoW copy of a large parentUse posix_spawn/vfork for subprocesses; Redis recommends overcommit_memory=1
Long pause during snapshot / fork()Copying page tables of a huge RSSHuge pages (fewer PTEs, but beware THP CoW), smaller instances, replicas for snapshotting
Child deadlocks right after fork()Lock held by another thread at fork timeExec immediately; only async-signal-safe calls before exec; pthread_atfork
Secrets/sockets visible in a child processFds inherited across execO_CLOEXEC everywhere
kill -9 doesn't workProcess in D state, or it's a zombieFix the stuck I/O; reap via the parent
Load average 50 with idle CPUTasks in D (disk/NFS) counted in loadps for D state, /proc/<pid>/stack (root), check storage
Segfault only on Alpinemusl's 128KB default thread stackSet stack size explicitly

Decision Guidance: Process, Thread, or Task?

flowchart TD
    S["Need concurrency"] --> I{"Need isolation: crash, security, or different language or binary?"}
    I -->|"yes"| P["Separate processes (prefork, workers, sidecars)"]
    I -->|"no"| C{"CPU-bound?"}
    C -->|"yes"| TP["Thread pool sized to cores (or processes, for Python's GIL)"]
    C -->|"no: I/O-bound"| N{"How many concurrent operations?"}
    N -->|"hundreds"| BTP["Bounded thread pool via Goetz formula"]
    N -->|"thousands to millions"| A["Goroutines, virtual threads, or async/event loop"]
  • Processes when you need fault isolation (one crash shouldn't kill the rest — Chrome's tab-per-process, PostgreSQL's backend-per-connection, nginx/gunicorn pre-fork workers), privilege separation (OpenSSH), or to sidestep a global interpreter lock (Python multiprocessing; free-threaded CPython 3.13+ is changing this).
  • Threads for shared-state parallelism inside one service, sized deliberately.
  • Green threads / async tasks when concurrency is huge and mostly waiting: 100k WebSocket connections, fan-out RPC.

Real systems mix all three: nginx runs one process per core (process isolation), each with one event loop (async), plus a small thread pool for blocking file I/O.


Interview Questions

"What happens when you call fork()?" Creates a child process that is an almost-exact copy of the parent via copy-on-write. Returns 0 in child, child PID in parent. Child shares physical memory pages until either writes (CoW). The kernel does have to copy page tables (~2MB per GB of RSS), which is why forking a very large process takes milliseconds and pauses it.

"What's the difference between a process and a thread?" A process has its own address space and resources (isolation). Threads share the address space of their parent process. Threads are cheaper to create and switch between but a crash in one thread kills them all. In Linux both are task_structs created by clone(); the flags decide what is shared.

"What is a zombie process?" A process that has exited but whose parent hasn't called wait(). The kernel keeps the PCB to store the exit code. Zombie entries consume a PID and a small amount of kernel memory.

"How does a container differ from a process?" A container is a process (or group of processes) that uses Linux namespaces (PID, network, mount, user, UTS, IPC) to create isolation and cgroups to limit resources. The process itself runs in the host kernel — there's no separate OS.

"Why do containers sometimes ignore SIGTERM or fill up with zombies?" The app runs as PID 1 of the PID namespace: the kernel doesn't apply default signal actions to it, and it inherits every orphan. Without a handler and a reaping loop, SIGTERM is ignored until SIGKILL and zombies accumulate. Use tini / --init, exec-form CMD, and exec in entrypoint scripts.

"How many threads should a thread pool have?" CPU-bound: about the number of cores available to the process (respecting container limits). I/O-bound: N_cpu × U × (1 + W/C), cross-checked with Little's Law (throughput × latency), and capped by what the downstream can handle. Then load-test to find the knee.

"Why are goroutines cheaper than threads?" 2KB growable stacks instead of 8MB reserved, switching in user space (~100–200ns) without a syscall, and an M:N scheduler that parks goroutines on network I/O instead of blocking an OS thread.

"What's the difference between exit() and _exit()?" exit() runs atexit handlers and flushes stdio buffers, then calls _exit(), which terminates immediately. A forked child whose exec fails should call _exit() so it doesn't flush the parent's duplicated buffers.