Virtual Memory
Virtual memory is the abstraction that lets every process believe it owns all of memory. A process running on a machine with 16GB of RAM gets a 48-bit (256TB) virtual address space. Multiple processes run concurrently, each with their own isolated address space, without knowing about or interfering with each other.
Understanding virtual memory explains why fork() is fast, why mmap is efficient, what a segfault is, and why "out of memory" can kill a process even when there's free disk space.
It also explains a long list of production mysteries that otherwise look like black magic: why a Java service with a 4GB heap shows 6GB of RSS, why a Kubernetes pod gets OOMKilled while top inside the container says there is plenty of free memory, why Redis tells you to turn off Transparent Huge Pages, why a database server with 30% free RAM is swapping, and why a 64-core box spends measurable CPU time on something called "TLB shootdowns".
This page goes from first principles (what a page is) through the hardware (page tables, the TLB), the kernel's policies (demand paging, copy-on-write, reclaim, overcommit, the OOM killer), and the operational layer (huge pages, NUMA, accounting, containers) — with commands you can run on any Linux box.
Prerequisites
You should know what a process and a thread are (Processes & Threads). The companion pages are Memory Allocators (what malloc does on top of virtual memory) and Linux Memory Management (the operator's view).
The Problem Virtual Memory Solves
Without virtual memory:
- Programs must be loaded at specific physical addresses — complicates linking
- Programs can access each other's memory — security and stability disaster
- Total memory used by all programs must fit in physical RAM
- No way to run a program larger than physical RAM
Virtual memory solves all of these by giving each process its own private address space, mapped to physical memory by the kernel.
Historically, systems tried other answers first. Base-and-bound registers relocated a program by adding a base to every address and trapping anything beyond a limit — simple, but each program needed one contiguous chunk of RAM, so memory fragmented badly. Segmentation (x86 real and protected mode) generalised this to several variable-sized segments per program — still external fragmentation, still awkward. Overlays made programmers manually swap code in and out. Paging won because it uses fixed-size units: any free frame can hold any page, so physical memory never fragments at the page level, and the mapping can be changed one page at a time.
The four things virtual memory buys you, and the mechanism behind each:
| Property | Mechanism |
|---|---|
| Isolation — process A cannot touch B's memory | Each process has its own page table; B's frames simply are not mapped in A's |
| Relocation — every program can be linked as if it starts at the same address | The same virtual address maps to different frames in different processes |
| Overcommitment — total allocated memory can exceed RAM | Pages only get frames when touched (demand paging); cold pages can be evicted to swap or dropped |
Sharing — one copy of libc for 500 processes | Many page tables point to the same physical frame (read-only or copy-on-write) |
flowchart LR
subgraph PA["Process A (virtual)"]
A1["0x400000 code"]
A2["0x7f00.. libc"]
A3["heap page"]
end
subgraph PB["Process B (virtual)"]
B1["0x400000 code"]
B2["0x7f00.. libc"]
B3["heap page"]
end
subgraph RAM["Physical RAM (frames)"]
F1["frame 0x1a2"]
F2["frame 0x3b7 (libc, shared)"]
F3["frame 0x0c9"]
F4["frame 0x5d0"]
F5["frame 0x2e4"]
end
A1 --> F1
A2 --> F2
B2 --> F2
A3 --> F3
B1 --> F4
B3 --> F5
Both processes use virtual address 0x400000 for their code, but it lands in different frames. Both map libc and land in the same frame. Neither can name the other's heap frame at all — there is no virtual address in A that resolves to 0x2e4.
Pages and Frames
Memory is divided into fixed-size chunks:
- Page: a fixed-size chunk of virtual memory (typically 4KB on x86-64)
- Frame: a fixed-size chunk of physical memory (same size as a page)
The kernel maintains a mapping from virtual pages to physical frames in a page table. When the CPU accesses a virtual address, the Memory Management Unit (MMU) translates it to a physical address using the page table.
Virtual Address 0x7fff1234:
Page number = 0x7fff1 (top bits)
Page offset = 0x234 (low 12 bits — within the 4KB page)
Page table lookup: virtual page 0x7fff1 → physical frame 0x3a2b1
Physical address: 0x3a2b1000 + 0x234 = 0x3a2b1234
The offset is never translated: a 4KB page is 2¹² bytes, so the low 12 bits pass through untouched and only the upper bits (the virtual page number) are looked up. That is why page sizes are always powers of two.
Why 4KB? It's a 1960s/70s compromise that stuck: smaller pages waste less memory at the tail of each allocation (internal fragmentation) but need more page-table entries and more TLB entries; larger pages are the reverse. Modern hardware supports several sizes at once:
| Architecture | Base page | Larger pages |
|---|---|---|
| x86-64 | 4KB | 2MB, 1GB |
| ARM64 (AArch64) | 4KB, 16KB or 64KB (chosen at kernel build) | 2MB/1GB (4KB granule), 32MB (16KB), 512MB (64KB) |
| Apple Silicon (macOS/iOS) | 16KB | — |
| POWER (ppc64le) | 64KB typical | 16MB, 16GB |
getconf PAGESIZE # 4096 on most x86-64 Linux boxes
grep Hugepagesize /proc/meminfo # Hugepagesize: 2048 kB
Don't hard-code 4096
Code that assumes 4KB pages breaks on 16KB (Apple M-series Linux VMs, some Android devices since 2024) and 64KB (many ARM servers, ppc64le) kernels. Use sysconf(_SC_PAGESIZE) in C, os.Getpagesize() in Go, page_size::get() / libc::sysconf in Rust.
The Address Space of a Linux Process
Before looking at page tables, it helps to see what a process's virtual address space actually contains. Every region is a VMA (struct vm_area_struct) — a contiguous range of virtual addresses with one set of permissions and one backing (a file, or anonymous memory). The kernel keeps them in a maple tree (since 6.1; previously a red-black tree plus linked list) per process.
$ cat /proc/self/maps
5591e0a00000-5591e0a02000 r--p 00000000 08:01 1835 /usr/bin/cat # ELF headers
5591e0a02000-5591e0a07000 r-xp 00002000 08:01 1835 /usr/bin/cat # .text (code)
5591e0a07000-5591e0a09000 r--p 00007000 08:01 1835 /usr/bin/cat # .rodata
5591e0a0a000-5591e0a0b000 rw-p 00009000 08:01 1835 /usr/bin/cat # .data/.bss
5591e1c4e000-5591e1c6f000 rw-p 00000000 00:00 0 [heap] # brk heap
7f2c8a400000-7f2c8a428000 r--p 00000000 08:01 4021 /usr/lib/x86_64-linux-gnu/libc.so.6
7f2c8a428000-7f2c8a5bd000 r-xp 00028000 08:01 4021 /usr/lib/x86_64-linux-gnu/libc.so.6
...
7ffd3b5e1000-7ffd3b602000 rw-p 00000000 00:00 0 [stack]
7ffd3b6f8000-7ffd3b6fc000 r--p 00000000 00:00 0 [vvar]
7ffd3b6fc000-7ffd3b6fe000 r-xp 00000000 00:00 0 [vdso]
The columns are: address range, permissions (r/w/x, private or shared), file offset, device, inode, path. Note that the executable's code and libc's code are separate VMAs from their data — each with exactly the permissions it needs.
0xffffffffffffffff ┌───────────────────────────┐
│ kernel space (128 TiB) │ same in every process,
0xffff800000000000 ├───────────────────────────┤ supervisor-only
│ non-canonical hole │ any access → #GP fault
0x00007fffffffffff ├───────────────────────────┤
│ stack (grows down) │
│ [vdso] [vvar] │
│ mmap region (grows down): │
│ shared libs, mmap(), │
│ large malloc, thread │
│ stacks │
│ ... │
│ heap (brk, grows up) │
│ .bss .data .rodata .text │ PIE: randomised base
0x0000000000000000 └───────────────────────────┘ (page 0 never mapped)
On x86-64 with 4-level paging, addresses are 64 bits wide but only 48 are used; bits 63–48 must be copies of bit 47 (canonical form). That splits the space into a lower half for user space and an upper half for the kernel, with an enormous unusable hole in between. ASLR randomises the base of the stack, mmap region, heap, and (for PIE binaries) the executable, so each run has different addresses — cat /proc/sys/kernel/randomize_va_space shows 2 when fully enabled.
Page Tables
A page table maps virtual page numbers to physical frame numbers plus metadata flags:
Page Table Entry (64-bit x86):
┌──────┬──────┬──────────────────────────┬───────────────────────────────────────┐
│ bit │ 63 │ 12–51 │ 11 10 9 8 7 6 5 4 3 2 1 0 │
│ │ NX │ physical frame number │ ... G PS D A ... U W P │
└──────┴──────┴──────────────────────────┴───────────────────────────────────────┘
P (present): page is in physical memory (1) or not (0) — on disk, never touched, or invalid
W (writable): page can be written (bit 1, "R/W"); if clear, the page is read-only
U (user): user mode (ring 3) may access the page (bit 2, "U/S"); kernel pages have U=0
NX (no-execute): code cannot execute from this page
A (accessed): CPU sets this when page is read
D (dirty): CPU sets this when page is written
PS (page size): in a PD/PDPT entry, "this entry maps a 2MB/1GB page directly"
G (global): don't flush this entry on CR3 reload (used for kernel mappings)
Correction
Earlier versions of this page listed an "R (read)" bit. x86 has no separate read-permission bit: a present page is always readable. Bit 1 is the R/W bit (writable when set) and bit 2 is U/S (user-accessible when set). Bits 62–59 can hold a protection key (PKU/pkey_mprotect), and bits 52–58 are ignored by hardware — Linux uses some of them for software state such as soft-dirty tracking.
When P=0, the hardware ignores the other 63 bits — so the kernel stores its own information there. For a swapped-out page, the PTE holds the swap device and offset; that's how the fault handler knows where to read it back from.
Multi-level page tables: A flat page table for a 48-bit address space would require 512GB of RAM per process. Instead, x86-64 uses a 4-level page table (PML4 → PDPT → PD → PT), where each table fits in one page (4KB). Most entries are null — the tree is sparse.
Virtual address → PML4 index → PDPT index → PD index → PT index → offset
[9 bits] [9 bits] [9 bits] [9 bits] [12 bits]
The flat-table arithmetic: 2⁴⁸ bytes / 4KB pages = 2³⁶ pages × 8-byte entries = 2³⁹ bytes = 512GB. Per process. A tree solves this because a 4KB table holds 512 (2⁹) 8-byte entries — hence 9 bits per level — and a missing upper-level entry means the entire 512GB/1GB/2MB range beneath it needs no tables at all. A small process touching a few MB needs only a handful of table pages: one PML4, a couple of PDPTs and PDs, and a PT per 2MB region in use.
The base address of the current process's PML4 table is stored in the CR3 register. Context switching includes loading CR3 with the new process's page table base — which invalidates the TLB (see below).
The levels, by name
| Level | Linux name | x86 name | Index bits (4-level) | One entry covers |
|---|---|---|---|---|
| 5 (only with LA57) | PGD | PML5 | 56–48 | 256 TiB |
| 4 | PGD (or P4D with 5-level) | PML4 | 47–39 | 512 GiB |
| 3 | PUD | PDPT | 38–30 | 1 GiB (can map a 1GB page) |
| 2 | PMD | PD | 29–21 | 2 MiB (can map a 2MB page) |
| 1 | PTE | PT | 20–12 | 4 KiB |
| — | offset | — | 11–0 | byte within page |
Linux's generic code always speaks of five levels (PGD → P4D → PUD → PMD → PTE) and "folds" the unused ones on hardware with fewer levels, so the same mm code runs on x86, ARM64, RISC-V and POWER.
Worked example: a 4-level page walk
Let's translate the user-space virtual address 0x00007F3A1B2C3D4E by hand.
Step 1 — split the address into fields. Write the low 48 bits in binary and cut them 9/9/9/9/12:
hex: 7 F 3 A 1 B 2 C 3 D 4 E
binary: 0111 1111 0011 1010 0001 1011 0010 1100 0011 1101 0100 1110
regroup: 011111110 | 011101000 | 011011001 | 011000011 | 110101001110
PML4=254 PDPT=232 PD=217 PT=195 offset=0xD4E
Bits 63–48 are all zero, matching bit 47 (also 0), so the address is canonical. Sanity check: (254<<39) + (232<<30) + (217<<21) + (195<<12) + 0xD4E = 0x7F3A1B2C3D4E.
Step 2 — walk the tree. Each entry is 8 bytes, so the entry's address is table_base + index × 8. Suppose the kernel loaded CR3 = 0x1A3B000 for this process (table addresses below are made up but internally consistent):
CR3 = 0x0001A3B000 (PML4 base)
PML4E @ 0x1A3B000 + 254*8 = 0x0001A3B7F0 → P=1, next table at 0x2C4D000
PDPTE @ 0x2C4D000 + 232*8 = 0x0002C4D740 → P=1, PS=0, next table at 0x3E5F000
PDE @ 0x3E5F000 + 217*8 = 0x0003E5F6C8 → P=1, PS=0, next table at 0x4A6B000
PTE @ 0x4A6B000 + 195*8 = 0x0004A6B618 → P=1, U=1, W=1, frame 0x9F8E7000
Physical address = 0x9F8E7000 + 0xD4E = 0x9F8E7D4E
flowchart LR
VA["VA 0x7F3A1B2C3D4E"] --> CR3["CR3 = 0x1A3B000"]
CR3 -->|"index 254"| L4["PML4 entry"]
L4 -->|"index 232"| L3["PDPT entry"]
L3 -->|"index 217"| L2["PD entry (PS=0)"]
L2 -->|"index 195"| L1["PT entry: frame 0x9F8E7000"]
L1 -->|"+ offset 0xD4E"| PA["PA 0x9F8E7D4E"]
Step 3 — check permissions at every level. The effective permission is the intersection: if any level has U=0, user mode can't access it; if any level has W=0, it's read-only. The CPU also sets the A bit in each entry it passes through (and the D bit in the final PTE on a write).
What if it were a 2MB huge page? Then the PD entry at step 2 would have PS=1 and directly hold a 2MB-aligned frame, say 0x9F800000. The walk stops one level early, and the low 21 bits (0x7F3A1B2C3D4E & 0x1FFFFF = 0xC3D4E) are the offset: physical address 0x9F8C3D4E. One fewer memory access per walk — and one TLB entry now covers 512× more memory.
5-level paging
48 bits = 256 TiB of virtual space, which some in-memory databases and HPC systems with many TB of RAM started to bump against (the physical address space was also limited). Intel's LA57 (Ice Lake server and later; AMD Zen 4 and later) adds a fifth level, giving 57-bit virtual addresses: 128 PiB total, 64 PiB for user space. Linux has supported it since 4.14 (CONFIG_X86_5LEVEL).
grep -o la57 /proc/cpuinfo | head -1 # CPU supports it
grep "address sizes" /proc/cpuinfo | head -1
# address sizes : 46 bits physical, 57 bits virtual ← 5-level active
# address sizes : 46 bits physical, 48 bits virtual ← 4-level
The cost is one more memory reference on every TLB miss. To avoid surprising programs that stuff tags into the high bits of pointers (JavaScript engines, some Lisp runtimes), Linux only hands out addresses above 47 bits if you pass an mmap hint address above that line.
What page tables cost in RAM
Page tables are real memory, accounted in /proc/meminfo as PageTables:
| Mapping | 4KB pages | 2MB pages |
|---|---|---|
| Last-level entries to map 1GB | 262,144 PTEs | 512 PDEs |
| Page-table memory for 1GB | ~2MB | ~4KB |
| Page-table memory for 64GB | ~128MB | ~256KB |
Page tables are per process, even for shared memory. A PostgreSQL server with shared_buffers = 64GB and 500 backend processes, each of which eventually touches most of the shared buffer pool, can accumulate up to ~128MB × 500 = ~64GB of page tables — as much as the buffer pool itself. This is a well-known reason PostgreSQL's docs recommend huge_pages = on for large shared_buffers. Check yours:
grep -E 'PageTables|SecPageTables' /proc/meminfo
# PageTables: 9842120 kB ← 9.4GB of page tables: time to look at huge pages
TLB: Translation Lookaside Buffer
Page table lookups require 4 memory accesses (one per level). At billions of memory accesses per second, this would be catastrophically slow. The TLB is a hardware cache in the CPU that stores recent virtual→physical translations.
CPU accesses address →
TLB hit (common case): direct physical address, 1 cycle
TLB miss: hardware page table walk (4 memory accesses) → load into TLB
In practice an L1 TLB lookup happens in parallel with the L1 data cache lookup (the cache is virtually indexed, physically tagged), so a hit adds effectively zero latency. A miss in the L1 TLB that hits the L2 TLB (STLB) costs roughly 7–10 cycles. A full miss triggers a hardware page walk; the walker has its own paging-structure caches for upper-level entries and the entries themselves are often in L1/L2 cache, so a typical walk costs ~20–40 cycles — but one that misses all the way to DRAM can cost several hundred cycles. Under virtualization it gets worse: with nested paging (Intel EPT, AMD NPT), each guest page-table access must itself be translated through the host tables, so a worst-case 2D walk takes up to 24 memory references with 4-level tables on both sides.
TLB size: Typically 64–1536 entries (L1 TLB) + larger L2 TLB. Covers 64KB–6MB of memory with 4KB pages.
Concretely, for an Intel Skylake-family server core (newer cores are somewhat larger, e.g. 2048 STLB entries on Golden Cove):
| TLB | Entries | Reach with 4KB | Reach with 2MB | Reach with 1GB |
|---|---|---|---|---|
| L1 dTLB | 64 (4KB), 32 (2MB), 4 (1GB) | 256KB | 64MB | 4GB |
| L2 STLB (shared) | 1536 (4KB/2MB), 16 (1GB) | 6MB | 3GB | 16GB |
The key concept is TLB reach = entries × page size. A Redis or JVM process with a 50GB working set and random access touches far more than 6MB of distinct pages, so with 4KB pages nearly every random access risks a TLB miss. With 2MB pages the reach jumps to ~3GB, and misses fall dramatically. This is the whole case for huge pages (below).
Measure it:
perf stat -e dTLB-loads,dTLB-load-misses,iTLB-loads,iTLB-load-misses \
-e dtlb_load_misses.walk_active,cycles -p <pid> -- sleep 10
# dTLB-load-misses as % of dTLB-loads: <1% is fine, >5% on a hot service is worth a look.
# walk_active / cycles ≈ fraction of time the page walker is busy (Intel event).
TLB shootdown: When the kernel modifies a page table entry (e.g., unmaps a page), it must invalidate the corresponding TLB entry on ALL CPU cores (INVLPG instruction + inter-processor interrupt). On NUMA machines this is expensive.
More precisely, it must invalidate it on every core that might have cached the translation — every CPU currently (or lazily) running a thread of that mm. Linux tracks this in mm_cpumask. The sequence:
sequenceDiagram
participant T0 as "CPU 0 (thread calls munmap)"
participant K as "Kernel mm code"
participant C1 as "CPU 5 (same process)"
participant C2 as "CPU 17 (same process)"
T0->>K: munmap(addr, len)
K->>K: clear PTEs, INVLPG locally
K->>C1: IPI flush TLB range
K->>C2: IPI flush TLB range
C1->>C1: interrupt current work, INVLPG
C2->>C2: interrupt current work, INVLPG
C1-->>K: ack
C2-->>K: ack
K-->>T0: munmap returns (frames can now be freed)
The initiating CPU must wait for every acknowledgement before it may reuse the freed frames — otherwise another core could still write through a stale translation into memory that now belongs to someone else. On a big box a shootdown costs from a few microseconds to tens of microseconds, and the interrupted cores pay too. Workloads that hit it hard:
- Heavily multi-threaded programs that constantly
mmap/munmap(allocators returning memory withmadvise(MADV_DONTNEED), memory-mapped file churn) mprotecttoggling (some JITs and GCs)- Page migration (automatic NUMA balancing), compaction, KSM
grep TLB /proc/interrupts # "TLB: ... TLB shootdowns" per CPU
perf stat -e tlb:tlb_flush -a -- sleep 5
Hardware is starting to help: AMD Zen 3+ has INVLPGB (broadcast invalidation without IPIs), and ARM64 has had broadcast TLBI instructions all along, which is one reason ARM servers handle shootdown-heavy workloads relatively well.
Context switch cost: When switching between processes with different address spaces, the kernel loads a new CR3 — which flushes the entire TLB. The new process starts with a cold TLB, causing many TLB misses until its working set warms up. This is why thread switches (same address space) are cheaper than process switches.
PCID and ASID: not flushing on every switch
That full flush was true on older CPUs and is still the model to reason with, but modern hardware tags TLB entries with an address-space identifier so entries from different processes can coexist:
- x86 PCID (Process-Context Identifier): 12 bits in CR3, so up to 4096 tags. Loading CR3 with bit 63 set says "switch address space but keep the TLB contents". Linux uses a small per-CPU cache of recently used address spaces (
TLB_NR_DYN_ASIDS = 6), so switching between a handful of hot processes keeps their translations warm. - ARM64 ASID: 8 or 16 bits in
TTBR0; Linux allocates them with a generation-based rollover scheme. - Global pages (
Gbit): kernel mappings are marked global so they survive CR3 reloads — except under KPTI, which un-maps most of the kernel from user page tables.
PCID matters most for KPTI (the Meltdown mitigation, see System Calls & Kernel Internals), where every syscall switches page tables twice. Without PCID, each syscall would flush the TLB; with PCID, user and kernel halves get separate tags and the cost drops to a few hundred cycles.
grep -o -w pcid /proc/cpuinfo | head -1 # pcid supported
grep -o -w invpcid /proc/cpuinfo | head -1
Huge pages: Using 2MB or 1GB pages instead of 4KB reduces TLB pressure — each TLB entry covers more memory. Important for databases, JVMs, and workloads with large working sets. Linux: madvise(MADV_HUGEPAGE). The full story, including when huge pages hurt, is in Huge Pages below.
Page Faults
A page fault occurs when the CPU tries to access a virtual address that has no valid mapping in the TLB or page table — or a mapping that violates permissions.
The CPU raises a hardware exception. The kernel's page fault handler runs:
The exception is vector 14 (#PF). The CPU pushes an error code (was it a read or write? user or kernel mode? not-present or protection violation? instruction fetch?) and puts the faulting address in the CR2 register. Linux's exc_page_fault() → handle_mm_fault() then decides which case it is:
flowchart TD
F["Page fault, address in CR2"] --> V{"Address inside a VMA?"}
V -->|"no"| SEGV["SIGSEGV (or stack growth if just below stack)"]
V -->|"yes"| P{"Access allowed by VMA permissions?"}
P -->|"no"| SEGV2["SIGSEGV"]
P -->|"yes"| PRES{"PTE present?"}
PRES -->|"no, never touched, anonymous"| ZERO["Allocate zeroed frame - minor fault"]
PRES -->|"no, file-backed"| PC{"Page in page cache?"}
PC -->|"yes"| MINOR["Map cached page - minor fault"]
PC -->|"no"| MAJOR["Read from disk - major fault"]
PRES -->|"no, swap entry"| SWAP{"In swap cache?"}
SWAP -->|"yes"| MINOR2["Map it - minor fault"]
SWAP -->|"no"| MAJOR2["Read from swap - major fault"]
PRES -->|"yes, write to read-only"| COW["Copy-on-write (or reuse if sole owner)"]
Case 1: Page not present (P=0) — valid address, page is on disk (swapped out)
- Find the page's location on the swap partition
- Allocate a free physical frame
- Read the page from disk into the frame (~5ms for HDD, ~0.1ms for SSD)
- Update the page table entry (P=1, frame number)
- Resume the faulting instruction
Case 2: Copy-on-write fault
- Process wrote to a read-only page that is shared (e.g., after fork)
- Allocate a new physical frame
- Copy the page content
- Update the page table entry to point to the new frame, set writable
- Resume the faulting instruction
Case 3: Segmentation fault (SIGSEGV)
The address is not in any valid mapping (writing to null pointer, stack overflow, reading beyond an array). The kernel sends SIGSEGV to the process, which terminates it by default.
Case 4: First touch of anonymous memory. The most common fault of all. malloc(1 << 30) returns instantly because it only creates a VMA; nothing is mapped. The first read of each page maps the shared, read-only zero page; the first write allocates a real frame, zeroes it, and maps it writable. A program that allocates 1GB and writes every byte takes 262,144 page faults to do so.
Minor vs major faults
| Minor (soft) fault | Major (hard) fault | |
|---|---|---|
| Meaning | The data is already in RAM (or needs none); only the page table must be fixed up | The data must be read from a storage device |
| Examples | First touch of anonymous memory, file page already in page cache, CoW, swap-cache hit | Swapped-out page, file page not cached (cold start, after eviction) |
| Typical cost | ~0.5–2 µs (dominated by zeroing or copying 4KB and kernel entry/exit) | ~50–200 µs on NVMe, ~5–10 ms on HDD |
| Normal? | Yes — millions per second system-wide is normal | Occasional at startup is fine; sustained major faults mean memory pressure |
# Per-process counters (cumulative)
ps -o pid,min_flt,maj_flt,rss,cmd -p <pid>
# A command's fault profile
/usr/bin/time -v ./app 2>&1 | grep -E 'page faults|Maximum resident'
# Major (requiring I/O) page faults: 12
# Minor (reclaiming a frame) page faults: 245731
# System-wide rates
perf stat -e page-faults,minor-faults,major-faults -a -- sleep 10
sar -B 1 # pgpgin/s pgpgout/s fault/s majflt/s
# Who is major-faulting right now?
perf record -e major-faults -ag -- sleep 10 && perf report
Pre-faulting for latency-sensitive code
If a trading engine or audio thread can't afford a 1 µs fault on its first touch of a buffer, fault it in up front: mmap(..., MAP_POPULATE), touch every page at startup, and mlockall(MCL_CURRENT | MCL_FUTURE) so nothing gets swapped out later. The JVM's -XX:+AlwaysPreTouch does exactly this for the heap (at the cost of slower startup and full RSS from second one).
Demand Paging
When a process starts, the kernel doesn't load all its code and data into memory immediately. Instead, it sets up page table entries marked "not present" pointing to the executable file. As the program accesses pages, page faults occur and pages are loaded on demand.
More precisely, execve() creates VMAs describing the executable's segments (backed by the ELF file), and the page tables start empty — there's nothing to "point to the file" in the PTE itself; the VMA records the file and offset, and the fault handler consults it.
Benefits:
- Faster startup (only needed pages are loaded)
- Lower memory footprint (code never executed is never loaded)
- Enables programs larger than physical RAM (swapping)
Two optimisations make demand paging less fault-heavy than it sounds:
- Fault-around: on a fault in a file-backed mapping, Linux maps not just the faulting page but up to 16 neighbouring pages that are already in the page cache (the knob is
fault_around_bytesin debugfs, default 64KB). One fault, sixteen mappings. - Readahead: when faults on a file mapping look sequential, the kernel reads ahead asynchronously, so later faults become minor faults instead of major ones.
madvise(MADV_SEQUENTIAL)/MADV_RANDOMtune this per mapping.
The flip side: a cold start after a reboot, or after the page cache got evicted, pays major faults for every code page. Big binaries (a 500MB statically linked service, a JVM loading thousands of classes) can take noticeably longer to reach steady-state latency — the "first requests are slow" effect isn't only JIT warm-up.
Copy-on-Write (CoW)
After fork(), the child's page table initially points to the same physical frames as the parent. All pages are marked read-only in both page tables.
(Strictly, only the private writable pages are marked read-only; pages that were already read-only, such as code, are simply shared, and MAP_SHARED mappings stay writable and shared.)
Before write:
Parent page table: vpage 0x1000 → frame 0xABCD (read-only)
Child page table: vpage 0x1000 → frame 0xABCD (read-only)
↑
same physical frame
Child writes to vpage 0x1000 → page fault → CoW:
1. Allocate new frame 0xEF01
2. Copy frame 0xABCD → 0xEF01
3. Child page table: vpage 0x1000 → frame 0xEF01 (writable)
4. Parent page table: vpage 0x1000 → frame 0xABCD (now writable again)
After write:
Parent: vpage 0x1000 → 0xABCD (original)
Child: vpage 0x1000 → 0xEF01 (private copy)
A detail on step 4: Linux doesn't eagerly go back and flip the parent's PTE. The parent's PTE stays read-only until the parent next writes to that page; that write faults, the handler sees the frame now has only one user, and it simply marks the PTE writable — no copy. Same end state, but done lazily.
This makes fork() O(1) in terms of memory copying — only pages that are actually written get copied.
It is not O(1) overall: fork() must still copy the page tables (or at least the upper levels and the PTEs of private mappings), which is proportional to the mapped size. Forking a process with 100GB of resident memory means copying ~200MB of page tables, which takes on the order of tens to hundreds of milliseconds — Redis reports this as latest_fork_usec in INFO stats, and it's why vfork()/posix_spawn() (which share the parent's address space until exec) exist.
Redis uses this for persistence: BGSAVE calls fork(). The child traverses the dataset and writes it to disk. Meanwhile, the parent serves requests. When the parent modifies a key, CoW copies that page. The child always sees the snapshot at the time of fork. Memory usage spikes by at most the dataset size, but in practice far less.
How much less depends on the write rate during the snapshot and on page size. With 4KB pages, updating one 100-byte key copies 4KB. With 2MB Transparent Huge Pages, the same write can copy 2MB — a 512× amplification. A write-heavy Redis with THP enabled can approach doubling its memory during BGSAVE, then get OOM-killed. Redis logs Fork CoW for RDB: current 1024 MB, peak 1024 MB so you can watch it, and the same reasoning applies to any fork-based snapshotting (Python multiprocessing with large parent heaps, PHP-FPM, Ruby preforking servers).
fork() + overcommit
Under strict overcommit (vm.overcommit_memory=2), the kernel must reserve commit charge for every private writable page the child might copy — so forking a 20GB process can fail with ENOMEM even if only a few MB will ever be copied. This is why Redis warns at startup: WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. and recommends vm.overcommit_memory = 1.
mmap: Memory-Mapped Files
mmap() maps a file (or device, or anonymous memory) directly into the process's address space. Instead of read()/write() system calls copying data through a kernel buffer, the file's pages are part of the process's address space.
// Mapping a file for reading
int fd = open("data.bin", O_RDONLY);
void* ptr = mmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0);
// Access as if it's an array — no system calls!
int value = ((int*)ptr)[1000];
How it works:
mmap()creates a virtual memory region with page table entries pointing to the file's pages- First access page-faults → kernel loads the page from the file into a physical frame
- Subsequent accesses hit the page in physical memory — no system call
- The kernel manages eviction (LRU) and write-back automatically
(As with demand paging, step 1 really creates a VMA; PTEs are filled lazily on fault.)
vs read() system call:
read(): disk →(DMA) kernel page cache →(CPU copy) user buffer (1 copy)
mmap(): disk →(DMA) kernel page cache (which IS the user mapping) (0 copies, shared)
Correction
Earlier versions showed read() as going through an extra "kernel buffer" (2 copies). On Linux, buffered read() copies directly from the page cache into your buffer — one CPU copy. The disk-to-page-cache step is DMA, not a CPU copy. mmap saves that one copy plus the syscall per access, at the price of page faults and TLB entries.
The flags that matter
| Flag / call | Meaning |
|---|---|
MAP_PRIVATE | Writes are copy-on-write and never reach the file |
MAP_SHARED | Writes go to the page cache and eventually the file; visible to other mappers |
MAP_ANONYMOUS | No file; zero-filled memory (how malloc gets large blocks) |
MAP_POPULATE | Pre-fault all pages now instead of on first touch |
MAP_FIXED_NOREPLACE | Map exactly here, fail if something is already there (safer than MAP_FIXED, which silently replaces) |
MAP_HUGETLB | Back with hugetlbfs pages |
MAP_NORESERVE | Don't reserve swap/commit charge |
madvise(MADV_SEQUENTIAL / MADV_RANDOM) | Tune readahead |
madvise(MADV_WILLNEED) | Start reading these pages in now |
madvise(MADV_DONTNEED) | Drop these pages now; next touch gets zeros (anon) or file data |
madvise(MADV_FREE) | Pages may be reclaimed lazily if memory is tight (cheaper than DONTNEED; RSS doesn't drop immediately) |
msync(MS_SYNC) | Write dirty pages of a shared mapping back and wait |
// Go: read-only mmap of a file with golang.org/x/sys/unix
f, _ := os.Open("index.bin")
st, _ := f.Stat()
data, err := unix.Mmap(int(f.Fd()), 0, int(st.Size()), unix.PROT_READ, unix.MAP_SHARED)
if err != nil { log.Fatal(err) }
defer unix.Munmap(data)
_ = unix.Madvise(data, unix.MADV_RANDOM) // B-tree style access: no readahead
fmt.Println(data[4096]) // first touch: page fault, then plain memory
mmap wins for:
- Large files accessed randomly (databases, search indexes)
- Files shared between processes (shared libraries are mmap'd)
- Files where you don't know in advance which parts you need
read()/write() wins for:
- Sequential I/O (kernel read-ahead works better)
- Small files
- When you want explicit control over buffering
Memory-mapped I/O: Databases like SQLite, RocksDB, LMDB can use mmap for their data files (LMDB always does; SQLite only with PRAGMA mmap_size; RocksDB only with allow_mmap_reads). The OS page cache becomes the database buffer pool. The downside: no control over when writes are flushed to disk (important for durability guarantees).
Where mmap bites
- I/O errors become signals. A failed read on a mapped page can't return
-EIO; the process getsSIGBUS. So does touching a mapped page beyond the end of a file that someone truncated. - No async I/O. A page fault blocks the thread. An event loop that touches a cold mmap'd page stalls all its connections for a major fault's duration.
- Eviction is the kernel's choice. The database can't pin its hot pages or prioritise them; the kernel's LRU doesn't know a B-tree root from a scan.
- TLB shootdowns on
munmap/remap in multi-threaded processes, and page-table memory proportional to the mapping. - Durability ordering. The kernel may write dirty shared pages back at any time, in any order — a nightmare for WAL protocols that require "log before data". See Page Cache & fsync.
These are the arguments of the 2022 CIDR paper "Are You Sure You Want to Use MMAP in Your Database Management System?" (Crotty, Leis, Pavlo), and history agrees: MongoDB's original MMAPv1 engine was replaced by WiredTiger (MMAPv1 removed in 4.2), and most modern databases manage their own buffer pool with pread/O_DIRECT. LMDB is the notable counterexample — read-mostly, single-writer, and designed around mmap from day one.
Overcommit and the OOM Killer
malloc(10GB) on a machine with 8GB of RAM usually succeeds. That's not a bug: it's overcommit, and it follows directly from demand paging — the kernel only promises address space, and frames are found when pages are touched. Most programs never touch everything they allocate (sparse arrays, thread stacks reserved at 8MB but using 20KB, fork() children that exec immediately), so refusing those allocations would waste RAM.
The three overcommit modes
vm.overcommit_memory | Policy | When to use |
|---|---|---|
0 (default) | Heuristic: refuse only allocations that are obviously impossible (larger than RAM + swap in one go) | General purpose |
1 | Always: never refuse | Redis, programs with large sparse mappings, scientific code; accept OOM-killer risk |
2 | Strict: total commit ≤ swap + RAM × overcommit_ratio/100 (or overcommit_kbytes) | Systems that prefer malloc returning NULL over random kills (some databases, embedded) |
sysctl vm.overcommit_memory vm.overcommit_ratio
grep -E 'CommitLimit|Committed_AS' /proc/meminfo
# CommitLimit: 20480000 kB ← only enforced in mode 2
# Committed_AS: 58123456 kB ← total promised; can exceed RAM in modes 0/1
With strict mode, applications must handle malloc failing — which most don't, gracefully. And overcommit_ratio defaults to 50, so on a swapless 64GB box mode 2 permits only ~32GB of commit: a classic way to make a healthy server start refusing fork().
When the bill comes due
Overcommit is a bet. When processes actually touch more memory than exists, the kernel goes through escalating steps:
flowchart TD
A["Page fault needs a free frame"] --> B{"Free pages above min watermark?"}
B -->|"yes"| OK["Allocate frame"]
B -->|"no"| C["Direct reclaim: drop clean page cache, write dirty pages, swap anon pages"]
C --> D{"Reclaimed enough?"}
D -->|"yes"| OK
D -->|"no, after retries"| E["Try memory compaction"]
E --> F{"Success?"}
F -->|"yes"| OK
F -->|"no"| G["Invoke OOM killer"]
G --> H["Pick victim with highest oom_score"]
H --> I["SIGKILL victim, reap its memory (oom_reaper)"]
I --> A
Before that, background reclaim by kswapd starts when free memory falls below the low watermark and runs until it's back above high; if allocation hits the min watermark, the allocating thread does direct reclaim itself — which is where latency spikes come from. cat /proc/zoneinfo | grep -E 'min|low|high' shows the watermarks, and vm.watermark_scale_factor widens the gap so kswapd starts earlier.
How the victim is chosen
Each process has an oom_score (0–1000) roughly equal to its share of memory: (RSS + swap entries + page tables) / total RAM+swap × 1000, then adjusted by oom_score_adj (−1000 to +1000). -1000 makes a process unkillable by the OOM killer; +1000 volunteers it first.
# Rank processes by OOM score
for p in /proc/[0-9]*; do
printf "%s %s %s\n" "$(cat $p/oom_score 2>/dev/null)" "${p#/proc/}" "$(cat $p/comm 2>/dev/null)"
done | sort -rn | head
# Protect sshd, sacrifice a batch job
echo -1000 > /proc/$(pidof sshd)/oom_score_adj
echo 500 > /proc/<batch_pid>/oom_score_adj
# systemd: OOMScoreAdjust=-900 in the unit file
What it looks like in dmesg:
Out of memory: Killed process 48211 (java) total-vm:18765432kB, anon-rss:15234112kB,
file-rss:1024kB, shmem-rss:0kB, UID:1001 pgtables:31240kB oom_score_adj:0
oom_reaper: reaped process 48211 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Kubernetes sets oom_score_adj by QoS class: Guaranteed pods get −997, BestEffort pods get 1000, and Burstable pods get a value between 2 and 999 based on how small their memory request is relative to node capacity. So under node-level memory pressure, BestEffort pods die first.
cgroup OOM vs global OOM
Inside a container, the limit that usually matters is the cgroup's memory.max, not the machine's RAM. When a cgroup exceeds it and reclaim within the cgroup fails, the kernel runs the OOM killer scoped to that cgroup:
Memory cgroup out of memory: Killed process 9921 (node) total-vm:2310440kB, anon-rss:510332kB, ...
The process exits with signal 9; a container runtime reports exit code 137 (128 + 9) and Kubernetes shows OOMKilled. Setting memory.oom.group = 1 kills the whole cgroup together, which is usually what you want for a multi-process service (no half-alive worker pools).
Userspace OOM handling
The kernel OOM killer acts late — often after minutes of thrashing where the box is technically alive but useless. systemd-oomd (default on Fedora and Ubuntu 22.04+) and earlyoom watch PSI memory pressure (/proc/pressure/memory) and kill earlier, per cgroup, by policy. Meta built the original oomd for exactly this reason.
Swapping
When physical memory is full, the kernel must evict some pages to make room. Evicted pages are written to the swap partition (or swap file) on disk.
Only anonymous pages (heap, stack, private CoW copies, tmpfs/shmem) need swap. Clean file-backed pages are simply dropped — they can be re-read from the file — and dirty file pages are written back to their file. So "no swap configured" doesn't mean "no paging": under pressure, a swapless system evicts code and page cache instead, which is how swapless boxes end up thrashing on their own executables.
Page replacement algorithms:
- LRU (Least Recently Used): evict the page that hasn't been accessed the longest. Optimal in theory but expensive to implement exactly.
- Clock algorithm (Linux approximation): each page has an "accessed" bit set by the CPU. The clock hand sweeps through pages; if accessed bit is set, clear it and move on; if not set, evict the page.
(Strictly, the optimal algorithm — Belady's MIN — evicts the page whose next use is furthest in the future, which requires knowing the future; LRU is the practical approximation of that.)
What Linux actually does is a refinement of the clock idea:
- Active/inactive lists (two lists each for anon and file pages). New pages enter inactive; a second access promotes them to active; reclaim scans the tail of inactive and demotes from active to keep the lists balanced. A one-off scan of a huge file therefore can't flush your hot working set.
- Workingset detection: when an evicted file page is faulted back in soon after ("refault"), the kernel notices via shadow entries and treats the workload's working set as larger than the inactive list.
- MGLRU (multi-generational LRU, merged in 6.1, default on many distros): pages are grouped into generations by age; it scans page tables for accessed bits more efficiently and makes noticeably better choices under pressure. Google reported fewer OOM kills and lower CPU spent in reclaim on Android and ChromeOS. Check
cat /sys/kernel/mm/lru_gen/enabled.
Swappiness
vm.swappiness (0–200, default 60) sets the relative cost the kernel assigns to reclaiming anonymous pages (swap) vs file pages (page cache). It is not a "% full before swapping" threshold, a common myth.
| Value | Effect |
|---|---|
| 0 | Avoid swapping anon pages until file cache is nearly exhausted (does not disable swap) |
| 1–10 | Common for databases: prefer dropping page cache, swap only in an emergency |
| 60 | Default: balanced |
| 100 | Treat anon and file pages as equally costly |
| 100–200 | Prefer swapping; sensible with fast swap such as zram |
zswap and zram
- zram: a compressed RAM block device used as swap. Pages "swapped" to it are compressed (2–3× typical with lz4/zstd) and stay in RAM. Default on Fedora, ChromeOS and Android. Effectively trades CPU for capacity.
- zswap: a compressed cache in front of a real swap device; pages are compressed in RAM first and only written to disk when the pool fills.
swapon --show
vmstat 1 # si/so columns: KB/s swapped in/out. Sustained si > 0 = trouble
cat /proc/pressure/memory
# some avg10=4.20 avg60=2.10 avg300=0.80 total=... ← % time some task stalled on memory
# full avg10=1.05 ... ← % time ALL tasks stalled
Thrashing: When a system spends more time swapping pages than executing programs. Occurs when total working set of all processes exceeds physical RAM. Symptoms: CPU utilization drops, disk I/O spikes, system becomes unresponsive.
PSI is the best thrashing detector: full avg10 above a few percent means the whole system is regularly blocked waiting for memory. Major fault rate and si (swap-in) are the other two signals.
Should production servers have swap?
| Situation | Recommendation |
|---|---|
| General-purpose servers | A modest swap (or zram) plus low swappiness: lets the kernel push truly cold anon pages out and gives you a warning window instead of instant OOM kills |
| Latency-critical databases (Redis, Postgres, Cassandra) | Swap present but swappiness=1, plus alerting on si. Some teams disable swap entirely and size RAM with headroom — then rely on the OOM killer being fast |
| Kubernetes nodes | Historically swap had to be off (kubelet refused to start); NodeSwap support has since matured (GA around 1.34) with LimitedSwap for Burstable pods. Default remains "no swap for pods" |
| Laptops/desktops | zram, swappiness 100+ |
Huge Pages: THP vs hugetlbfs
A 2MB page covers 512 normal pages with one TLB entry and one less page-table level. For large-memory workloads with poor locality — databases, JVM heaps, in-memory analytics, VMs — that can be worth 5–20% throughput. Linux offers two mechanisms with very different trade-offs.
| hugetlbfs ("HugePages") | Transparent Huge Pages (THP) | |
|---|---|---|
| Setup | Reserve a pool at boot/runtime: vm.nr_hugepages | Automatic; per-system mode in /sys/kernel/mm/transparent_hugepage/enabled |
| Application change | Must ask: mmap(MAP_HUGETLB), shmget(SHM_HUGETLB), or a file on a hugetlbfs mount | None (mode always) or madvise(MADV_HUGEPAGE) (mode madvise) |
| Sizes | 2MB and 1GB | 2MB (plus mTHP 16KB–1MB for anon memory since 6.8) |
| Swappable | No — pinned | Yes (split or swapped as a unit in recent kernels) |
| Memory | Reserved even if unused | Can inflate RSS (a 2MB page for 4KB of data) |
| Latency risk | None at runtime (pre-reserved) | Allocation may trigger compaction; khugepaged collapses pages in the background |
| Used by | Oracle, PostgreSQL huge_pages=on, DPDK, KVM guests, SAP HANA | JVM -XX:+UseTransparentHugePages, general workloads, MongoDB 8.0+ |
cat /sys/kernel/mm/transparent_hugepage/enabled # always [madvise] never
cat /sys/kernel/mm/transparent_hugepage/defrag # always defer [defer+madvise] madvise never
grep -E 'AnonHugePages|HugePages_|Hugetlb' /proc/meminfo
grep thp_ /proc/vmstat | head # thp_fault_alloc, thp_collapse_alloc, thp_split_page ...
# Reserve 1024 x 2MB = 2GB of hugetlbfs pages
sysctl -w vm.nr_hugepages=1024
# 1GB pages are best reserved at boot: hugepagesz=1G hugepages=16 on the kernel cmdline
// Ask for THP on a specific region (works when enabled=madvise or always)
size_t len = 1UL << 30;
void *p = mmap(NULL, len, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
madvise(p, len, MADV_HUGEPAGE);
// Explicit hugetlbfs pages (fails with ENOMEM if the pool is empty)
void *q = mmap(NULL, len, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_HUGETLB, -1, 0);
When THP hurts
THP's appeal is "free speed with no code change". Its failure modes are why so many database vendors have told you to turn it off:
- Allocation latency. To hand out a 2MB page, the kernel needs 512 contiguous free frames. On a long-running, fragmented machine there may be none, so with
defrag=alwaysthe faulting thread runs direct compaction — migrating pages around — and a page fault that should take 1 µs takes 10+ ms. This is the classic "random p99 spikes" symptom. - Memory bloat. A sparse access pattern that touches 4KB in each 2MB region gets a full 2MB page each time: up to 512× RSS inflation. Allocators that return memory with
MADV_DONTNEEDon 4KB granularity can't actually give back memory that sits inside a huge page until it's split. - CoW amplification after
fork()(the RedisBGSAVEproblem above): each write copies 2MB. - khugepaged CPU scanning and collapsing pages in the background, and lock contention while doing so.
Vendor guidance (as of 2026)
| Software | Guidance | Why |
|---|---|---|
| Redis / Valkey | Disable THP (never, or at least not always); Redis 6.2+ has disable-thp yes (default) which opts its own process out via prctl(PR_SET_THP_DISABLE) | Fork CoW amplification and latency; Redis prints WARNING you have Transparent Huge Pages (THP) support enabled in your kernel |
| MongoDB ≤ 7.0 | Disable THP | Sparse access + old tcmalloc bloat |
| MongoDB 8.0+ | Enable THP (always), with defrag=defer+madvise | 8.0 moved to a newer TCMalloc with per-CPU caches that is designed to use huge pages efficiently |
| PostgreSQL | THP madvise/never; use hugetlbfs via huge_pages = on sized to shared_buffers | Page-table savings across many backends without THP's runtime behaviour |
| Oracle Database | Disable THP, use HugePages | Documented Oracle requirement |
| JVM | Either -XX:+UseLargePages (hugetlbfs) or -XX:+UseTransparentHugePages with enabled=madvise | Large heaps benefit; madvise mode limits THP to the heap |
| Cassandra, Elasticsearch, Kafka | Generally madvise/never and test | Latency-sensitive, mixed results |
The safe system-wide default for a server that runs a database is enabled=madvise with defrag=defer+madvise: programs that know they benefit opt in, nobody else gets surprised, and allocation never blocks on compaction for regions that didn't ask.
# Persist it (systemd-based distros); or add transparent_hugepage=madvise to the kernel cmdline
echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
echo defer+madvise > /sys/kernel/mm/transparent_hugepage/defrag
War Story (illustrative)
Illustrative scenario — a composite of commonly reported symptoms, not a specific company.
A cache cluster ran fine for weeks after each deploy, then gradually developed p99 GET latency spikes of 20–50 ms with no change in traffic. CPU graphs showed bursts of sys time; perf top during a spike was dominated by compaction_alloc and isolate_migratepages_block. The hosts had THP set to always with defrag=always (the distro default at the time), and after weeks of churn physical memory was fragmented enough that every new 2MB fault forced synchronous compaction. Switching defrag to defer+madvise removed the spikes the same hour; switching enabled to madvise also brought RSS down by about 15% because sparse hash-table regions stopped being backed by full huge pages.
NUMA and Memory Policy
On a multi-socket server (and on single-socket chips with several memory controllers, such as AMD EPYC in NPS2/NPS4 mode), each CPU has local memory attached to its own controller and remote memory behind another socket's controller. Accessing remote memory crosses the inter-socket link (Intel UPI, AMD Infinity Fabric).
| Access | Typical latency | Typical bandwidth |
|---|---|---|
| Local DRAM | ~80–100 ns | Full per-socket bandwidth |
| Remote DRAM (1 hop) | ~130–200 ns (1.3–2× local) | Limited by the interconnect |
numactl --hardware
# available: 2 nodes (0-1)
# node 0 cpus: 0-31 64-95 node 0 size: 257790 MB node 0 free: 10234 MB
# node 1 cpus: 32-63 96-127 node 1 size: 258041 MB node 1 free: 190112 MB
# node distances:
# node 0 1
# 0: 10 21
# 1: 21 10
numastat -p <pid> # per-node memory of one process
lstopo # hwloc topology diagram
Policies
Linux's default policy is first-touch / local allocation: a page is placed on the node of the CPU that first faults it in. That's usually right — but a single-threaded loader that initialises a 200GB table puts all of it on its own node, and the 64 worker threads on the other socket then read remote memory forever.
| Policy | How to set | Use for |
|---|---|---|
| Local (default) | — | Most workloads |
| Bind | numactl --membind=0 --cpunodebind=0 ./app, mbind() | One instance per socket (run 2 Redis/Memcached/Postgres instances on a 2-socket box) |
| Preferred | numactl --preferred=1 | Soft placement, falls back when full |
| Interleave | numactl --interleave=all ./app, set_mempolicy(MPOL_INTERLEAVE) | Big shared structures accessed from all CPUs (database buffer pools, the JVM heap with -XX:+UseNUMAInterleaving) |
| Weighted interleave | MPOL_WEIGHTED_INTERLEAVE (6.9+) | Mixing DRAM with CXL-attached memory in a bandwidth ratio |
Automatic NUMA balancing (kernel.numa_balancing=1) periodically unmaps pages to sample which CPUs touch them via "NUMA hinting faults", then migrates pages and tasks to be together. Good for long-running unpinned processes; it costs faults and TLB shootdowns, so latency-sensitive pinned workloads often turn it off.
War Story (real, well documented)
In 2010 Jeremy Cole (then at Twitter) published "The MySQL 'swap insanity' problem and the effects of the NUMA architecture." A MySQL server with a buffer pool sized at most of RAM was swapping heavily while the machine showed plenty of free memory. The cause: the buffer pool was allocated by one thread, so first-touch put it almost entirely on one NUMA node; when that node ran out, the kernel's per-node reclaim (aggravated by vm.zone_reclaim_mode on some systems) swapped pages out of node 0 rather than allocating from the free memory on node 1. The fix was to start mysqld under numactl --interleave=all. MySQL later added innodb_numa_interleave, and the kernel changed zone_reclaim_mode to default to 0. The lesson outlived MySQL: "free memory" is per node, and one full node can swap while the other is empty.
cat /proc/sys/vm/zone_reclaim_mode # want 0 for most server workloads
cat /proc/sys/kernel/numa_balancing
grep -E 'numa_(hit|miss|foreign|local|other)' /proc/vmstat # numa_miss rising = allocations spilling to remote nodes
Memory Accounting: VSZ, RSS, PSS, USS
"How much memory does this process use?" has at least four correct answers:
| Metric | Definition | Overcounts? | Where |
|---|---|---|---|
| VSZ / VmSize | Total virtual address space mapped | Massively — includes reserved, never-touched, and file mappings | ps, /proc/<pid>/status |
| RSS / VmRSS | Pages currently resident in RAM and mapped by this process | Yes — shared pages (libc, shared memory) count fully in every process | ps, top (RES) |
| PSS | RSS, but each shared page divided by the number of processes sharing it | No — PSS sums correctly across processes | /proc/<pid>/smaps_rollup, smem |
| USS | Only pages private to this process | Undercounts its share of shared memory | smem |
Example: 10 PHP-FPM workers each map 40MB of shared opcache and have 20MB private. Each shows RSS = 60MB (total 600MB "used" — wrong). PSS = 20 + 40/10 = 24MB each, total 240MB (right). USS = 20MB: what you'd free by killing one worker.
$ cat /proc/<pid>/smaps_rollup
Rss: 614232 kB
Pss: 402118 kB
Pss_Anon: 381004 kB
Pss_File: 21114 kB
Shared_Clean: 212340 kB
Private_Dirty: 381004 kB
Referenced: 598120 kB
Anonymous: 381004 kB
AnonHugePages: 204800 kB
Swap: 1024 kB
Locked: 0 kB
# Per-mapping detail: which VMA is big?
awk '/^[0-9a-f]+-/ {name=$6} /^Rss:/ {rss[name]+=$2} END {for (n in rss) print rss[n], n}' \
/proc/<pid>/smaps | sort -rn | head
smem -t -k -c "pid name uss pss rss" # USS/PSS/RSS for every process
pmap -x <pid> | tail -1
/proc/<pid>/status also has RssAnon, RssFile, RssShmem and VmSwap, and VmHWM (peak RSS) — useful for "how big did it get".
Reading /proc/meminfo correctly
| Field | Meaning |
|---|---|
MemTotal | Usable RAM |
MemFree | Completely unused frames — usually small on a healthy box, and that's fine |
MemAvailable | Estimate of what can be allocated without swapping: free + reclaimable page cache + reclaimable slab. This is the number to alert on. |
Cached / Buffers | Page cache (file data) — mostly reclaimable |
AnonPages | Anonymous memory (heaps, stacks) — must be swapped to reclaim |
Shmem | tmpfs, shared memory, /dev/shm — counted in Cached but not freely reclaimable |
SReclaimable / SUnreclaim | Kernel slab caches (dentries, inodes) — see Memory Allocators |
PageTables | Memory used by page tables |
Dirty / Writeback | File data waiting to be written |
Committed_AS | Total promised by overcommit |
"Linux ate my RAM" is almost always page cache: free -h shows little "free" but large "available". That's the kernel using otherwise idle RAM as a disk cache, which it gives back on demand.
Container Memory Limits
Containers don't get a separate kernel or separate virtual memory system — they're processes in a memory cgroup whose usage the kernel tracks and limits (see Namespaces & cgroups). cgroup v2 offers four knobs:
| File | Semantics |
|---|---|
memory.max | Hard limit. Exceeding it → reclaim within the cgroup → cgroup OOM kill |
memory.high | Soft throttle. Above it, tasks are slowed and pushed into reclaim but not killed — the right place to apply back-pressure |
memory.low | Best-effort protection from reclaim by other cgroups |
memory.min | Hard protection: never reclaimed |
cd /sys/fs/cgroup/kubepods.slice/.../cri-containerd-<id>.scope
cat memory.max memory.current
cat memory.events # low 0, high 1294, max 37, oom 2, oom_kill 2
grep -E '^(anon|file|kernel|sock|shmem|file_dirty|inactive_file|active_file) ' memory.stat
What gets charged
Everything the cgroup's processes cause to be allocated: anonymous memory, page cache for files they read or write, tmpfs (emptyDir: medium: Memory in Kubernetes), kernel memory for their sockets, page tables and slab objects. This surprises people constantly:
- A container that
cats a 10GB log file fills itsmemory.currentwith page cache. That's mostly harmless — the kernel reclaims clean cache before OOM-killing — but it confuses dashboards that showmemory.current. - Kubernetes evicts and the dashboards alert on working set =
memory.current − inactive_file(cAdvisor'scontainer_memory_working_set_bytes), which still includes active page cache and so can look high for I/O-heavy pods. - Writing to tmpfs is anonymous-like memory: it can't be dropped, only swapped.
Runtimes must know about the limit
free and /proc/meminfo inside a container report the host's memory (unless LXCFS is used). Runtimes that size themselves from total RAM will overshoot the cgroup limit and get OOM-killed:
| Runtime | Container-aware sizing |
|---|---|
| JVM | Container support on by default since JDK 10 / 8u191; heap defaults to 25% of the limit — set -XX:MaxRAMPercentage=75 (and remember metaspace, thread stacks, direct buffers, and code cache live outside the heap) |
| Go | GOMEMLIMIT (Go 1.19+) gives the GC a soft memory target; set it to ~90% of the container limit. Go doesn't read cgroup memory limits by itself |
| Node.js | V8 heap limit (--max-old-space-size); the default is derived from available memory and may not match the cgroup, so set it explicitly (~75% of the limit) |
| .NET | Reads cgroup limits; GCHeapHardLimit / GCHeapHardLimitPercent |
| Python | No heap limit; size workers so N × peak RSS fits |
// Go: respect the container limit without hard-coding it
// (GOMEMLIMIT env var works too: GOMEMLIMIT=900MiB)
import (
"os"
"runtime/debug"
"strconv"
"strings"
)
func init() {
if b, err := os.ReadFile("/sys/fs/cgroup/memory.max"); err == nil {
if lim, err := strconv.ParseInt(strings.TrimSpace(string(b)), 10, 64); err == nil {
debug.SetMemoryLimit(lim * 9 / 10)
}
}
}
For Kubernetes requests/limits and QoS classes in depth, see Kubernetes Resource Management.
Segmentation Fault Deep Dive
A segfault happens when a process accesses:
- An address not in any VMA (Virtual Memory Area) — null pointer dereference, using a freed pointer
- An address in a VMA but with wrong permissions — writing to read-only text segment, executing data
Kernel's VMA list for a process:
0x400000 - 0x401000: text segment (r-x) — executable code
0x401000 - 0x402000: data segment (rw-)
0x7ffe0000 - 0x80000000: stack (rw-)
0x7f000000 - 0x7ffe0000: heap/mmap region
Access to 0x0 → not in any VMA → SIGSEGV
Write to 0x400000 → in text VMA but W=0 → SIGSEGV
(Using a freed pointer only segfaults if the allocator has actually returned that memory to the kernel; more often it silently reads or corrupts whatever now lives there — which is why use-after-free is a security bug, not just a crash.)
A few more mechanisms worth knowing:
- Page zero is never mapped (
vm.mmap_min_addr, default 65536), so null-pointer dereferences — evenptr->fieldat a small offset — reliably fault instead of reading attacker-controlled memory. - Stack growth: a fault just below the stack VMA grows the stack automatically (up to
ulimit -s, default 8MB). Beyond that, or into the guard gap (256 pages by default since the 2017 "Stack Clash" fixes), you getSIGSEGV— a stack overflow. Thread stacks created by pthreads have an explicit guard page. - The kernel logs segfaults:
app[3172]: segfault at 0 ip 000055d1c2a0113d sp 00007ffd4e8b2a30 error 6 in app[55d1c2a01000+1000]
at 0 is the faulting address (null), ip the instruction pointer, and error 6 is the page-fault error code: bit 1 (write) and bit 2 (user mode) set, bit 0 clear (page not present). Subtract the mapping base (55d1c2a01000) from ip and run addr2line -e app 0x113d to find the line.
Real Numbers
| Quantity | Value |
|---|---|
| Base page size (x86-64) | 4 KiB; huge pages 2 MiB and 1 GiB |
| Virtual address space, 4-level / 5-level | 256 TiB / 128 PiB (user half: 128 TiB / 64 PiB) |
| Entries per page-table page | 512 (9 index bits) |
| Page-table memory to map 1 GiB with 4KB pages | ~2 MiB |
| L1 dTLB / L2 STLB entries (Skylake-class) | 64 / 1536 |
| TLB reach with 4KB vs 2MB pages (STLB) | ~6 MB vs ~3 GB |
| TLB hit | ~0–1 cycles extra |
| Page walk | ~20–40 cycles typical, hundreds if it misses to DRAM |
| Minor page fault | ~0.5–2 µs |
| Major fault from NVMe / HDD | ~50–200 µs / ~5–10 ms |
| TLB shootdown on a large server | a few µs to tens of µs |
fork() of a 10 GB-RSS process (page-table copy) | ~10–100 ms |
| Local vs remote NUMA DRAM | ~80–100 ns vs ~130–200 ns |
vm.swappiness default / range | 60 / 0–200 |
oom_score_adj range | −1000 to +1000 |
| Default stack limit / guard gap | 8 MiB / 256 pages |
Decision Guide
| Symptom or goal | First thing to check | Likely fix |
|---|---|---|
Pod OOMKilled, exit 137 | memory.events (oom_kill), memory.stat anon vs file | Raise limit, set runtime limits (MaxRAMPercentage, GOMEMLIMIT), find the leak |
| Host OOM kills, wrong victim | oom_score of each process | oom_score_adj / systemd OOMScoreAdjust, cgroup limits per service |
Random multi-ms latency spikes, sys time bursts | perf top for compaction, /sys/kernel/mm/transparent_hugepage/defrag | defrag=defer+madvise, enabled=madvise |
| RSS much larger than heap | smaps_rollup (AnonHugePages), allocator stats | Disable THP for that process, tune allocator (see Memory Allocators) |
| Box slow, "plenty of free RAM" | numastat, per-node free, PSI | NUMA interleave/bind, zone_reclaim_mode=0 |
Huge PageTables in meminfo | Number of processes × shared mapping size | hugetlbfs for shared memory (huge_pages=on), fewer processes (connection pooler) |
| High dTLB miss rate on big-heap service | perf stat -e dTLB-load-misses | THP via madvise, hugetlbfs, 1GB pages for very large heaps |
fork() fails with ENOMEM | vm.overcommit_memory, CommitLimit | overcommit_memory=1 or posix_spawn |
Sustained swap-in, high PSI full | vmstat 1 si, /proc/pressure/memory | More RAM, smaller working set, systemd-oomd policy |
| Cold-start latency after deploy | Major faults (/usr/bin/time -v) | Pre-warm, MAP_POPULATE, vmtouch the binary/data files |
Advanced Topics
- KSM (Kernel Samepage Merging): a background scanner that finds identical anonymous pages (opted in via
madvise(MADV_MERGEABLE)orprctl) and merges them into one CoW page. Big wins for VM hosts running many similar guests; costs CPU and has been used as a side channel (timing a write tells you whether a page was merged). - userfaultfd: lets a user-space thread handle page faults for a region. Used for post-copy live migration in QEMU/KVM (start the VM on the destination, fetch pages on fault) and by CRIU and some GCs.
- Folios: since 5.16 the kernel has been converting from per-4KB
struct pagehandling to folios — variable-sized groups of pages — which underpins large folios in the page cache and mTHP for anonymous memory. Expect fewer, larger units of memory management going forward. - Memory tagging: ARM MTE attaches a 4-bit tag to every 16 bytes and to pointers; mismatches fault. Android uses it to catch use-after-free in production. Intel LAM and AMD UAI let user space store metadata in high pointer bits.
- Memory tiering and CXL: CXL-attached memory shows up as a CPU-less NUMA node. The kernel can demote cold pages to the slower tier and promote hot ones back (memory tiering,
numa_balancing=2). mseal()(6.10+): permanently seal a mapping's permissions so exploits can'tmprotectit writable or executable again.- Virtualization: EPT/NPT nested paging, the 24-reference worst-case walk, and why hypervisors back guest memory with huge pages almost by default.
Interview Questions
"Why is fork() fast even for a 2GB process?"
Copy-on-write. fork() only copies page table entries (a few MB of kernel data), not the actual memory. Physical pages are shared until either process writes to them, triggering per-page copying. In practice, the exec-after-fork pattern means most pages are never copied — the child immediately replaces its address space.
"What is a page fault?" A hardware exception raised when the CPU accesses a virtual address with no valid page table entry, or with permissions that don't match the access type. The kernel handles it: for a valid address, loads the page from disk or CoW-copies it; for an invalid address, sends SIGSEGV.
"What is the TLB and why does it matter for performance?" The TLB (Translation Lookaside Buffer) is a CPU cache for recent virtual→physical address translations. Without it, every memory access would require 4 additional memory accesses to walk the page table. Context switching between processes flushes the TLB, causing cache misses until the working set is re-established. (On CPUs with PCID/ASID, entries are tagged per address space and a switch need not flush everything — but the new process's translations still compete for a small number of entries.)
"When would you use mmap instead of read()?"
For random access to large files — databases, search indexes, large datasets. mmap avoids an extra copy (file content is the user mapping, not a separate user buffer), works well with the OS page cache, and enables multiple processes to share the same physical pages. Use read for sequential access where kernel read-ahead helps.
"Walk me through translating a virtual address on x86-64."
Split the 48-bit address into four 9-bit indices and a 12-bit offset. Starting from CR3, use each index to select an 8-byte entry in a 4KB table; each entry gives the physical address of the next table. The last entry gives the frame; add the offset. A PS bit at the PD or PDPT level means a 2MB or 1GB page and the walk ends early. The TLB caches the final result so this happens only on a miss.
"A container keeps getting OOMKilled but the app's heap is well under the limit. Why?"
The cgroup counts everything: heap, plus off-heap (JVM metaspace, thread stacks, direct buffers; native allocator fragmentation), tmpfs, page cache that can't be reclaimed fast enough, and kernel memory. Check memory.stat to see whether it's anon, file, shmem or kernel, and memory.events for oom_kill. Size the runtime to the limit (MaxRAMPercentage, GOMEMLIMIT) with headroom.
"Why does Redis tell you to disable Transparent Huge Pages?"
Redis snapshots with fork() + copy-on-write. With 2MB pages, each write during a snapshot copies 2MB instead of 4KB, so memory can nearly double and latency spikes; THP allocation can also trigger synchronous compaction. Redis now opts itself out with disable-thp.
"What's the difference between RSS and PSS?" RSS counts every resident page a process maps, so shared pages are counted once per process and sums across processes overcount. PSS divides each shared page by the number of sharers, so PSS values add up to real usage.
Related Pages
- Memory Allocators — malloc, arenas, fragmentation, jemalloc/tcmalloc, slab
- System Calls & Kernel Internals — KPTI,
mmap/brkas syscalls, vDSO - Page Cache & fsync — dirty pages, writeback, durability
- Namespaces & cgroups — the memory controller in context
- Linux Memory Management — operator's view
- Redis Internals — fork-based persistence in practice
- PostgreSQL Performance Tuning —
shared_buffers, huge pages