On this page

Page Cache, Write-Back & fsync

When write() returns success, your data is not on disk. It is in RAM, in the kernel's page cache, and it will reach the storage device some time in the next half-minute — unless the machine loses power first, or the device reports an error that nobody ever sees. Almost every durability bug in databases, message brokers, and "save file" code comes from a mismatch between what programmers assume write() means and what it actually means.

The File Systems page introduces the page cache and I/O & Storage covers write-back vs write-through at a high level. This page is the deep version: how the cache is organized, exactly when dirty data gets written, what each durability API does and does not guarantee, the bugs that real databases shipped for decades, and how to observe it on a live system.


Why the Page Cache Exists

Storage is slow relative to memory, and the gap is enormous:

OperationLatency
Read 4 KB from page cache (RAM)~0.5–1 µs (mostly syscall + copy)
Read 4 KB from NVMe SSD~10–100 µs
Read 4 KB from cloud block storage (EBS gp3, PD)~0.5–2 ms
Read 4 KB from 7200 RPM HDD (random)~5–10 ms
fsync on NVMe with power-loss protection~20–100 µs
fsync on consumer NVMe (volatile write cache flushed)~0.5–5 ms
fsync on HDD~5–20 ms+

Caching file data in RAM turns repeat reads into memory copies and lets writes return immediately, batching them into large sequential I/O later. Linux does this with all free memory: RAM not used by processes is used as cache, and given back when processes need it.

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            62Gi        11Gi       1.2Gi       412Mi        50Gi        50Gi
Swap:          8.0Gi          0B       8.0Gi

"Only 1.2 GB free" is not a problem. available — an estimate of memory that can be reclaimed for new allocations without swapping, mostly clean page cache — is the number that matters. The 50 GB of buff/cache is the kernel doing its job.


One Unified Cache

Before Linux 2.4 there were two caches: a buffer cache of disk blocks (for read/write) and a page cache of file pages (for mmap), with painful coherence problems between them. Since 2.4 there is one page cache, indexed by (file, offset), and everything goes through it:

graph TD
    R["read() / write()"] --> PC
    M["mmap() page faults"] --> PC
    SF["sendfile() / splice()"] --> PC
    EX["exec() — loading binaries and .so files"] --> PC
    PC["Page cache<br/>per-inode address_space<br/>(xarray of folios, keyed by file offset)"]
    PC -->|"miss: readahead / readpage"| FS["File system<br/>(ext4, XFS, btrfs)"]
    PC -->|"writeback of dirty pages"| FS
    FS --> BL["Block layer + I/O scheduler"]
    BL --> DEV["Device<br/>(and its volatile write cache)"]
    OD["O_DIRECT I/O"] -.->|"bypasses page cache"| FS

Key structures:

  • Each inode has an address_space that maps file offsets to cached pages. Since Linux 4.20 it is an xarray (previously a radix tree); since 5.16 the cached units are folios, which can be larger than 4 KB, reducing per-page overhead for big files.
  • Each cached page has flags: PG_uptodate (contents valid), PG_dirty (modified, not yet written), PG_writeback (I/O in flight), PG_locked.
  • Pages live on LRU lists (active and inactive, or the multi-generational LRU — MGLRU — available since Linux 6.1). Under memory pressure the kernel reclaims clean inactive pages first: dropping a clean page is free; a dirty one must be written first.

Because mmap and read() see the same pages, a write through one is immediately visible through the other — no coherence protocol needed. This is also why sendfile() can transmit a file to a socket without copying it into userspace: the data is already sitting in the cache.


The Read Path

sequenceDiagram
    participant App
    participant VFS as VFS / page cache
    participant FS as ext4
    participant Disk

    App->>VFS: read fd, 4 KB at offset 0
    VFS->>VFS: lookup inode address_space at index 0
    alt cache hit
        VFS-->>App: copy_to_user, about 1 µs
    else cache miss
        VFS->>FS: readahead for pages 0 to 31
        FS->>Disk: one 128 KB read via bio
        Note over App: thread sleeps in D state
        Disk-->>FS: completion interrupt
        FS-->>VFS: pages marked uptodate
        VFS-->>App: copy_to_user
    end

Readahead

On a miss, the kernel does not read just the requested page. It guesses you will want the next ones too and reads a window ahead — synchronously for the first request, then asynchronously as you consume the window (the PG_readahead marker page triggers the next batch before you need it). For sequential reads this means the application almost never waits on the disk.

  • The initial maximum window comes from the device: /sys/block/<dev>/queue/read_ahead_kb (default 128 KB), adjustable with blockdev --setra (in 512-byte sectors).
  • The window ramps up on sequential access and collapses on random access.
$ cat /sys/block/nvme0n1/queue/read_ahead_kb
128
$ sudo blockdev --setra 4096 /dev/nvme0n1     # 2 MB readahead for a streaming workload

Applications can steer it:

posix_fadvise(fd, 0, 0, POSIX_FADV_SEQUENTIAL);   // double the readahead window
posix_fadvise(fd, 0, 0, POSIX_FADV_RANDOM);       // disable readahead (index lookups)
posix_fadvise(fd, off, len, POSIX_FADV_WILLNEED); // start reading this range now
posix_fadvise(fd, off, len, POSIX_FADV_DONTNEED); // drop these clean pages from the cache
madvise(addr, len, MADV_RANDOM);                  // same idea for mmap'd regions

POSIX_FADV_DONTNEED after a one-pass job (backups, log shipping, cp of a huge file) keeps it from evicting the hot working set of every other process on the box. Note it only drops clean pages — call fdatasync first if you wrote them.


The Write Path

ssize_t n = write(fd, buf, 4096);

What happens:

  1. The VFS finds (or allocates) the page cache page for that offset. If the write covers only part of a page that is not cached, the kernel must first read the page from disk (a read-modify-write) — which is why unaligned small writes to cold files can be surprisingly slow.
  2. Data is copied from buf into the page.
  3. The page is marked dirty; the inode is put on its file system's dirty list.
  4. The file system may reserve space (ext4 and XFS use delayed allocation: physical blocks aren't chosen until writeback, which yields better, more contiguous layouts).
  5. write() returns.

That is all. No I/O has been issued. write() returning means: the kernel has accepted responsibility for your data, in volatile memory, and will try to write it later. Errors from that later attempt are not reported to this write() call — they can only surface on a later fsync() or close().

When Dirty Pages Are Written: Writeback

Per-device writeback kernel worker threads (kworker/u:…+flush-259:0 in top) flush dirty pages when any of these triggers fire:

TriggerSysctlDefaultBehaviour
Agevm.dirty_expire_centisecs3000 (30 s)Pages dirty longer than this are written on the next writeback pass
Periodic wakeupvm.dirty_writeback_centisecs500 (5 s)How often flusher threads wake to look for expired pages
Background thresholdvm.dirty_background_ratio (or _bytes)10% of available memoryFlusher starts writing in the background; writers are not slowed
Hard thresholdvm.dirty_ratio (or _bytes)20% of available memoryProcesses calling write() are throttled — they sleep in balance_dirty_pages() until writeback catches up
Memory pressure——Reclaim needs to free dirty pages, writes them first
Explicitfsync, sync, syncfs, sync_file_range—Application asks
$ sysctl vm.dirty_ratio vm.dirty_background_ratio vm.dirty_expire_centisecs vm.dirty_writeback_centisecs
vm.dirty_ratio = 20
vm.dirty_background_ratio = 10
vm.dirty_expire_centisecs = 3000
vm.dirty_writeback_centisecs = 500

So without fsync, a write can sit in RAM for up to ~35 seconds (30 s expiry + up to 5 s until the next flusher pass) before even being submitted to the device.

graph LR
    W["write()"] --> D["Dirty pages<br/>in RAM"]
    D -->|"age above 30s<br/>or above background ratio"| WB["Writeback<br/>in progress"]
    D -->|"above dirty_ratio:<br/>writer is throttled"| T["write() sleeps in<br/>balance_dirty_pages"]
    T --> WB
    WB -->|"I/O completes"| C["Clean pages<br/>(still cached)"]
    WB -->|"I/O error"| E["Error recorded on mapping,<br/>page marked clean anyway"]
    C -->|"memory pressure"| F["Evicted"]

The Big-RAM Problem

Percentages made sense when servers had 1 GB of RAM. On a 512 GB machine, 20% is ~100 GB of dirty data — which can take many minutes to drain to a single SSD and longer to network storage. Symptoms: a process copies a huge file at RAM speed, then everything writing to that device stalls for a long time when dirty_ratio is hit, or an fsync of one small file takes seconds because it queues behind gigabytes of other writeback. The kernel log may show:

INFO: task postgres:48211 blocked for more than 120 seconds.
      Not tainted 6.8.0-45-generic #45-Ubuntu
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:postgres   state:D stack:0  pid:48211 ppid:1201 flags:0x00000002
Call Trace:
 io_schedule+0x46/0x70
 folio_wait_bit_common+0x13b/0x3a0
 folio_wait_writeback+0x2b/0x80
 ...

Fix: use absolute byte limits on large-memory machines, sized to what your storage can flush in a second or two:

# /etc/sysctl.d/90-writeback.conf — e.g. for a database host with fast NVMe
vm.dirty_background_bytes = 268435456    # 256 MB: start flushing early
vm.dirty_bytes            = 1073741824   # 1 GB: throttle writers beyond this
# Setting *_bytes zeroes the corresponding *_ratio, and vice versa.

Linux also has per-device writeback throttling (/sys/block/<dev>/queue/wbt_lat_usec, default enabled on many devices) which limits background writeback when it starts hurting read latency, and cgroup v2 can attribute and limit writeback per cgroup.


The Durability APIs

fsync and fdatasync

int fsync(int fd);       // flush file data AND all metadata, then flush device cache
int fdatasync(int fd);   // flush file data and only metadata needed to read it back

fsync(fd):

  1. Writes all dirty page cache pages of the file and waits for completion.
  2. Writes the inode's metadata (size, block mapping, timestamps) — on journaling file systems, commits the journal transaction containing it.
  3. Issues a cache flush (or FUA writes) to the device so that data in the drive's volatile write cache reaches stable media.
  4. Returns 0 only if all of that succeeded.

fdatasync(fd) skips metadata that is not required to retrieve the data — typically mtime/atime. If the write extended the file or allocated blocks, the size and block map are flushed (you could not read the data back otherwise). For a database overwriting preallocated WAL segments, fdatasync avoids one journal commit per sync; that is why PostgreSQL defaults wal_sync_method to fdatasync on Linux.

# How expensive are syncs on this box? strace -T shows time per call
$ strace -f -T -e trace=fsync,fdatasync -p $(pgrep -f "postgres: walwriter")
[pid 3011] fdatasync(23)           = 0 <0.000412>
[pid 3011] fdatasync(23)           = 0 <0.000389>

# Benchmark sync latency the way databases experience it
$ fio --name=wal --rw=write --bs=8k --size=256m --fdatasync=1 --ioengine=sync --runtime=30
  fsync/fdatasync/sync_file_range:
    sync (usec): min=21, max=1880, avg=48.12, stdev=19.40     <- enterprise NVMe w/ PLP

The device write cache

fsync is only as honest as the hardware. Drives acknowledge writes into a volatile DRAM cache; the kernel's flush command forces it to stable media. Enterprise SSDs with power-loss protection (PLP) — capacitors that let them drain the cache on power failure — can safely acknowledge flushes almost immediately, which is why their sync latency is 10–50x better. Consumer drives must actually write, and some cheap or misconfigured devices (and some virtualization layers) ignore flushes entirely, silently breaking durability. Check cat /sys/block/<dev>/queue/write_cache ("write back" means a volatile cache the kernel will flush; "write through" means none or PLP-safe).

macOS: fsync is not enough

On macOS, fsync() pushes data to the drive but does not flush the drive's cache. You need fcntl(fd, F_FULLFSYNC). SQLite has PRAGMA fullfsync for this; PostgreSQL on macOS offers wal_sync_method = fsync_writethrough. Go's os.File.Sync() uses F_FULLFSYNC on Darwin; many other runtimes do not.

fsync the Directory, Too

A file's name lives in its parent directory, not in the file. Creating, renaming or unlinking a file modifies the directory. fsync(file_fd) makes the file's contents and inode durable, but not necessarily the directory entry pointing to it. After a crash you could have a perfectly intact inode with no name — garbage-collected by fsck, effectively lost.

int fd = open("/data/segment-0042.log", O_CREAT | O_WRONLY | O_EXCL, 0644);
write(fd, buf, len);
fsync(fd);                       // contents + inode durable
close(fd);

int dfd = open("/data", O_RDONLY | O_DIRECTORY);
fsync(dfd);                      // the directory entry "segment-0042.log" durable
close(dfd);

ext4 in its default data=ordered mode often makes this work by accident (the directory change tends to be committed in the same journal transaction), which is why many programs that skip it never notice — until they run on XFS, btrfs, or a different kernel version. POSIX does not promise it; write the directory fsync.

O_SYNC and O_DSYNC

Opening with O_SYNC makes every write() behave like write() + fsync(); O_DSYNC like write() + fdatasync(). Convenient, but it forces one flush per write call — terrible for many small writes, fine when each write is already a large, batched unit (a WAL group commit). PostgreSQL's wal_sync_method = open_datasync uses this.

O_DIRECT

O_DIRECT asks the kernel to bypass the page cache: DMA straight between your buffer and the device.

int fd = open("/data/ibdata1", O_RDWR | O_DIRECT);
void *buf;
posix_memalign(&buf, 4096, 1 << 20);          // buffer, offset and length must be aligned
pread(fd, buf, 1 << 20, 0);                   // to the logical block size (512 B or 4 KB)
$ strace ./direct_unaligned
pread64(3, 0x55a3c1e7a2a1, 4000, 100) = -1 EINVAL (Invalid argument)

What it gives you: no double caching, no cache pollution, predictable memory use, and the application's own buffer pool decides what to keep. What it does not give you: durability. O_DIRECT writes can still sit in the device's volatile cache, and file metadata (size, block allocation) still goes through the normal path. Databases combine O_DIRECT with fsync/fdatasync, or with O_DSYNC. You also lose readahead and write-back batching — the application must do its own I/O scheduling, usually with io_uring or Linux AIO (see I/O & Storage).

sync_file_range

sync_file_range(fd, offset, nbytes,
                SYNC_FILE_RANGE_WRITE);   // start writeback of this range, don't wait

A Linux-specific call to start (and optionally wait for) writeback of a byte range. It is not a durability primitive: it does not flush metadata and does not flush the device cache — the man page explicitly warns about this. Its real use is smoothing writeback: issue SYNC_FILE_RANGE_WRITE every few megabytes as you write so that the eventual fsync has little left to do and does not cause a latency spike. RocksDB's bytes_per_sync / wal_bytes_per_sync options do exactly this, and PostgreSQL uses it in its checkpointer's "writeback flush after" settings (checkpoint_flush_after, bgwriter_flush_after).

Summary

APIData to deviceMetadataDevice cache flushedWaitsTypical use
write()No (page cache only)NoNoNoEverything, by default
fsync(fd)YesAllYesYesMake a file durable
fdatasync(fd)YesOnly what's needed to read dataYesYesWAL / log appends into preallocated files
O_SYNC / O_DSYNCEach writeAll / neededYesEach writeLarge group-commit writes
O_DIRECTYes, bypassing cacheNoNoPer I/ODB buffer pools; pair with fsync
sync_file_rangeRange, startedNoNoOptionalSmooth writeback before fsync
syncfs(fd)Whole file systemAllYesYesCheckpoint everything on one FS
sync()All file systemsAllYesYes (on Linux)Admin / shutdown
msync(MS_SYNC)mmap'd rangeLike fsync on LinuxYesYesDurability for mmap writes

Durability Bugs in the Wild

fsyncgate: PostgreSQL, 2018

In March 2018 Craig Ringer reported on the pgsql-hackers list that PostgreSQL's handling of fsync() errors could silently lose data. The thread — which became known as "fsyncgate" — revealed a mismatch between what PostgreSQL (and most other software) assumed and what Linux did:

What applications assumed: if fsync() fails, the dirty data is still in the cache; retry fsync() later and, if it succeeds, the data is durable.

What Linux actually did: when writeback of a page fails (e.g. a transient I/O error, a thin-provisioned volume running out of space, a flaky network block device), the kernel logs the error, records it on the file's mapping, and marks the page clean anyway (otherwise memory could fill with pages that can never be written). The error is reported to the next fsync() — once. The data in that page is now neither on disk nor considered dirty. A retried fsync() returns success, because there is nothing dirty left to write.

sequenceDiagram
    participant PG as PostgreSQL checkpointer
    participant K as Kernel page cache
    participant D as Storage

    PG->>K: write dirty buffer to relation file
    K->>D: background writeback
    D-->>K: I/O error
    K->>K: log error, mark page CLEAN, remember error on mapping
    PG->>K: fsync
    K-->>PG: EIO, error now consumed
    Note over PG: log error, keep WAL, retry at next checkpoint
    PG->>K: fsync again
    K-->>PG: 0 success, nothing dirty
    Note over PG: checkpoint completes, WAL recycled, data permanently lost

Worse, before Linux 4.13 the error could be reported to whichever file descriptor happened to call fsync first — possibly in a different process — or lost entirely if the inode was evicted from memory before anyone asked. PostgreSQL's checkpointer often opens a file after backends wrote through other descriptors, so it might never see the error at all.

The fallout and fixes:

  • Linux 4.13 (2017, by Jeff Layton) introduced errseq_t per-mapping error tracking so every file descriptor open at the time of an error sees it on its next fsync; 4.16 extended reporting so a descriptor opened later still sees an error that no one has observed yet. (Errors are still lost if the inode is evicted from memory before any fsync.)
  • PostgreSQL 12, and minor releases 11.2 / 10.7 / 9.6.12 / 9.5.16 / 9.4.21 (February 2019) changed behaviour: an fsync failure on a data file now triggers a PANIC, forcing crash recovery to replay the WAL from the last good checkpoint, instead of retrying. The old behaviour is available behind data_sync_retry = on, for systems where the OS is known to keep failed pages dirty.
  • MySQL, MongoDB's WiredTiger and others audited and made similar changes. The 2020 USENIX ATC paper Can Applications Recover from fsync Failures? (Rebello et al.) tested several databases and file systems systematically and found widespread mishandling.

The lesson generalizes: after a failed fsync, you cannot know what is on disk. The only safe responses are to crash and recover from a log you know is durable, or to rewrite all the data from your own copy. Retrying the fsync is never sufficient.

The ext4 Zero-Length File Episode (2009)

When ext4 became the default in Ubuntu, users reported (Launchpad bug #317781) that after a crash, configuration files of KDE and GNOME applications were zero bytes long. The apps did the "obvious" thing — open(O_TRUNC), write(), close() — or wrote a temp file and renamed it over the original without fsync. ext3's data=ordered mode and 5-second commit interval had masked this for years. ext4's delayed allocation meant the rename (metadata) could be committed long before the new data blocks were even allocated, so a crash left the new name pointing at an empty file.

The kernel developers' position was that the applications were wrong per POSIX — but ext4 added a heuristic (auto_da_alloc, on by default) that forces block allocation when a file is renamed over an existing one or truncated to zero. Your code should not rely on that heuristic: other file systems don't have it.

The Correct Way to Replace a File

int atomic_replace(const char *dir, const char *name, const void *data, size_t len)
{
    char tmp[PATH_MAX], dst[PATH_MAX];
    snprintf(tmp, sizeof tmp, "%s/.%s.tmp.%d", dir, name, getpid());
    snprintf(dst, sizeof dst, "%s/%s", dir, name);

    int fd = open(tmp, O_WRONLY | O_CREAT | O_EXCL | O_CLOEXEC, 0644);   // 1. temp file, same dir
    if (fd < 0) return -1;
    if (write_all(fd, data, len) < 0 ||                                     // 2. write it all
        fsync(fd) < 0) {                                                    // 3. data durable BEFORE rename
        close(fd); unlink(tmp); return -1;
    }
    if (close(fd) < 0) { unlink(tmp); return -1; }                          //    close() can report errors too

    if (rename(tmp, dst) < 0) { unlink(tmp); return -1; }                   // 4. atomic name swap

    int dfd = open(dir, O_RDONLY | O_DIRECTORY | O_CLOEXEC);                // 5. make the rename durable
    if (dfd < 0) return -1;
    int r = fsync(dfd);
    close(dfd);
    return r;
}

Why each step:

  1. Same directory — rename() is only atomic within one file system; a temp file in /tmp may be on a different one (tmpfs), where rename fails with EXDEV.
  2. Write fully, handling short writes.
  3. fsync the temp file before renaming — otherwise the rename can become durable before the data, giving the empty-file outcome above.
  4. rename() atomically replaces the directory entry: any reader, and any post-crash state, sees either the complete old file or the complete new one.
  5. fsync the directory — otherwise after a crash you may have the old name still pointing at the old file (safe, but the update is lost; if you already told a client "saved," that is a durability violation).
// The same in Go — note os.WriteFile alone is NOT crash-safe
func AtomicWrite(path string, data []byte) error {
    dir := filepath.Dir(path)
    f, err := os.CreateTemp(dir, "."+filepath.Base(path)+".tmp-*")
    if err != nil { return err }
    tmp := f.Name()
    defer os.Remove(tmp) // no-op after successful rename

    if _, err := f.Write(data); err != nil { f.Close(); return err }
    if err := f.Sync(); err != nil { f.Close(); return err }
    if err := f.Close(); err != nil { return err }
    if err := os.Rename(tmp, path); err != nil { return err }

    d, err := os.Open(dir)
    if err != nil { return err }
    defer d.Close()
    return d.Sync()
}

For append-only logs (a WAL, a Kafka-style segment), the analogous rules are: preallocate or fdatasync after extending, fsync the directory when creating a new segment, and on recovery verify checksums of the tail — a torn final record after a crash is normal and must be detected and truncated, not trusted. See Write-Ahead Logging.


How Databases Use (or Avoid) the Page Cache

PostgreSQL: Buffered I/O and Double Buffering

PostgreSQL has its own buffer pool (shared_buffers) but reads and writes data files through the OS page cache with ordinary pread/pwrite. A hot page can therefore exist twice: once in shared_buffers, once in the page cache. This is double buffering.

  • Why it's done: portability, OS readahead for sequential scans, OS write-back batching, and the OS cache acting as a second-tier cache behind a modestly-sized buffer pool.
  • Consequences: the common guidance to set shared_buffers to ~25% of RAM (not 80%) and let the page cache use most of the rest; effective_cache_size tells the planner how much the OS is likely caching.
  • Durability: the WAL is fdatasynced at commit (or batched with commit_delay / synchronous_commit). Data files are written lazily and fsynced at checkpoints — exactly the place fsyncgate bit.
  • Evolution: PostgreSQL 16 added a developer-only debug_io_direct setting; PostgreSQL 18 (2025) introduced an asynchronous I/O subsystem (io_method = worker or io_uring), laying groundwork for direct I/O in the future.

MySQL InnoDB: O_DIRECT

InnoDB does the opposite. With innodb_flush_method = O_DIRECT (the standard production setting on Linux, and the default since MySQL 8.4 where supported), data files bypass the page cache entirely; the InnoDB buffer pool is the only cache, typically sized at 50–80% of RAM. InnoDB still calls fsync to flush metadata and the device cache. The redo log is written through the page cache (or with O_DSYNC, depending on configuration) and synced per innodb_flush_log_at_trx_commit.

PostgreSQLInnoDB (O_DIRECT)
Data file I/OBuffered, through page cacheDirect, bypassing page cache
Buffer pool size~25% of RAM~50–80% of RAM
Hot pages cachedTwice (shared_buffers + page cache)Once
ReadaheadOS + its own prefetch hintsInnoDB's own (linear / random read-ahead)
Write smoothingOS writeback + sync_file_range hintsInnoDB page cleaner threads
Exposure to OS writeback behaviourHighLow

See PostgreSQL Internals and Storage Engines for the engine side.

Kafka: The Page Cache Is the Cache

Kafka's design leans on the page cache deliberately. Producers' writes are appended to segment files via write() and go to the page cache; consumers that are caught up read the same pages straight back out of RAM, and Kafka uses sendfile() (zero-copy) to ship them to the socket — the bytes never enter the JVM heap. A broker with 64 GB RAM might run a 6 GB heap and leave the rest for page cache.

By default Kafka does not fsync per message (log.flush.interval.messages and log.flush.interval.ms are effectively infinite). Durability comes from replication: with acks=all and min.insync.replicas=2, a write is acknowledged once it is in the page cache of multiple brokers, betting that independent machines will not all lose power at once. That is a reasonable bet in one data center with redundant power — and a terrible one if all replicas share a rack PDU. See Kafka Internals.

SQLite

SQLite runs inside your process and is strict about syncs, which makes it a good tutor:

  • Rollback journal mode (default): before modifying the database, copy the original pages to -journal, fsync the journal, fsync the directory (on first creation), write the database, fsync it, then delete/truncate the journal. Several syncs per transaction.
  • WAL mode (PRAGMA journal_mode=WAL): append changes to -wal. With PRAGMA synchronous=FULL, fsync the WAL at every commit; with synchronous=NORMAL (the common choice in WAL mode), only at checkpoints — a power loss can roll back the last few transactions but cannot corrupt the database.
  • PRAGMA fullfsync=ON for macOS F_FULLFSYNC; PRAGMA mmap_size to read via mmap instead of read().

SQLite's documentation on atomic commit is one of the clearest explanations anywhere of how to build crash-safe storage on top of these primitives. See SQLite Internals.


mmap vs read

mmap maps page cache pages directly into your address space (see Virtual Memory). Reads become memory loads; a miss becomes a page fault that the kernel satisfies from the page cache or disk.

read() / pread()mmap()
CopyPage cache → user buffer (one copy)None — you access cache pages directly
Per-access cost on hitSyscall (~100s ns) + copyPlain memory load (once the PTE is populated)
Per-access cost on missSyscall + I/OPage fault + I/O; the thread blocks with no way to do something else meanwhile
Error handling-1 with errnoSIGBUS — e.g. file truncated by someone else, I/O error, disk full on write to a sparse file
Control over evictionYour buffer pool decidesKernel decides; madvise hints only
Write durabilityfsyncmsync(MS_SYNC), and dirty pages may be written back at any moment the kernel chooses
Large files on 64-bitFineFine; on 32-bit, address space runs out
Multi-threaded scalingGoodPage-table updates and TLB shootdowns across cores can bottleneck

LMDB and early MongoDB (MMAPv1) built their storage engines on mmap. The CIDR 2022 paper Are You Sure You Want to Use MMAP in Your Database Management System? (Crotty, Leis, Pavlo) argues that for a transactional DBMS the drawbacks — no control over write-back ordering (dirty pages can hit disk before the WAL), opaque I/O stalls inside page faults, SIGBUS error handling, and TLB-shootdown scalability limits — outweigh the convenience. MongoDB replaced MMAPv1 with WiredTiger, which uses its own cache. mmap remains excellent for read-mostly data: shared libraries, search indexes, ML model weights, and LMDB's copy-on-write B+tree design, which works with the kernel rather than against it.

// mmap read loop — no read() calls at all after setup
int fd = open("index.bin", O_RDONLY);
struct stat st; fstat(fd, &st);
const uint8_t *p = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
madvise((void *)p, st.st_size, MADV_RANDOM);   // index lookups: don't read ahead
uint64_t sum = 0;
for (off_t i = 0; i < st.st_size; i += 4096) sum += p[i];

Measuring the Page Cache

/proc/meminfo

$ grep -E '^(MemTotal|MemFree|MemAvailable|Buffers|Cached|Dirty|Writeback|Shmem|Active\(file\)|Inactive\(file\)):' /proc/meminfo
MemTotal:       65536000 kB
MemFree:         1258320 kB
MemAvailable:   52418204 kB
Buffers:          412100 kB     # block-device metadata cached via the page cache
Cached:         50113420 kB     # file page cache (includes Shmem/tmpfs!)
Dirty:            911204 kB     # modified, not yet written
Writeback:         38016 kB     # being written right now
Shmem:            421980 kB     # tmpfs, /dev/shm, shared anonymous — counted in Cached but NOT reclaimable
Active(file):   31022008 kB
Inactive(file): 19504212 kB
  • Dirty creeping toward your dirty_bytes limit means writers are about to be throttled.
  • Writeback persistently high means the device cannot keep up.
  • Cached minus Shmem is the reclaimable-ish file cache.
# Watch dirty data drain in real time
$ watch -n1 'grep -E "^(Dirty|Writeback):" /proc/meminfo'

# vmstat: bi/bo = blocks in/out per second, wa = CPU time waiting on I/O
$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  3      0 1253104 412100 50113420  0    0   120 412880 9811 15402  4  6 61 29  0

Per-File: vmtouch and fincore

# Which parts of a file are cached? (util-linux fincore)
$ fincore /var/lib/postgresql/16/main/base/16384/16397
  RES PAGES   SIZE FILE
 1.2G 314572   1.9G /var/lib/postgresql/16/main/base/16384/16397

# vmtouch: inspect, warm, or evict
$ vmtouch -v /data/index.bin
/data/index.bin
[OOOOOOOOOOooooo                                       ] 61440/262144
           Files: 1
     Directories: 0
  Resident Pages: 61440/262144  240M/1G  23.4%

$ vmtouch -t /data/index.bin     # load it all into cache (warm after restart)
$ vmtouch -e /data/index.bin     # evict it
$ vmtouch -dl /data/index.bin    # lock it in memory (mlock) as a daemon

Linux 6.5 added the cachestat() syscall, which returns cached, dirty, writeback, evicted and recently-evicted page counts for a file range — a cheap, official replacement for mincore-based tools.

Hit Ratio and Latency with BPF

# Page cache hits/misses system-wide, per second (bcc)
$ sudo cachestat-bpfcc 1
    HITS   MISSES  DIRTIES HITRATIO   BUFFERS_MB  CACHED_MB
  412004      312     8120   99.92%          402      48939
  398112    18744     7966   95.50%          402      48972   <- a cold scan started

# Block I/O latency histogram — what misses and fsyncs actually cost
$ sudo biolatency-bpfcc -D 10 1

# Which processes are doing sync calls, and how slow are they?
$ sudo bpftrace -e 'tracepoint:syscalls:sys_enter_fsync,tracepoint:syscalls:sys_enter_fdatasync { @start[tid] = nsecs; }
  tracepoint:syscalls:sys_exit_fsync,tracepoint:syscalls:sys_exit_fdatasync /@start[tid]/ {
    @us[comm] = hist((nsecs - @start[tid]) / 1000); delete(@start[tid]); }'

Dropping the Cache (for Benchmarks Only)

$ sync                                         # write dirty pages first; drop_caches skips dirty pages
$ echo 1 | sudo tee /proc/sys/vm/drop_caches   # page cache
$ echo 3 | sudo tee /proc/sys/vm/drop_caches   # page cache + dentries and inodes

Never do this on a production database host to "free memory" — the memory was already available, and now every query hits disk until the cache rewarms.


Failure Modes

FailureSymptomCauseFix
Data lost after power failureLast N seconds of writes missingRelied on write() without fsyncfsync/fdatasync at the durability point
Zero-length or garbage file after crashConfig/state file emptyRename before data fsync, or O_TRUNC + write in placeThe atomic replace pattern
New file missing after crashData was fsynced but file is goneDirectory not fsyncedfsync the parent directory
Silent corruption after I/O errorCheckpoint "succeeded", data wrong laterRetried fsync after failure (fsyncgate)Treat fsync failure as fatal; recover from WAL
Latency cliff during bulk writesAll writers stall for secondsHit dirty_ratio; huge dirty backlog on big-RAM boxdirty_background_bytes / dirty_bytes; sync_file_range smoothing
One fsync takes secondsp99 commit latency spikesShares a device/journal with a large writebackSeparate WAL device; smaller dirty limits; ext4 data=ordered journal entanglement — consider XFS
Tasks in D state, hung-task warningsblocked for more than 120 secondsWriteback stuck on slow/failing storageCheck device health, multipath, network storage
Hot data evicted by a batch jobLatency jumps when backups runOne-pass read floods LRUPOSIX_FADV_DONTNEED, O_DIRECT for the job, cgroup memory limits
Database cache "too small" despite lots of RAMHigh read IOPSDouble buffering mis-sized, or O_DIRECT with tiny buffer poolSize buffer pool for the chosen strategy
SIGBUS in an mmap readerCrash with no errnoFile truncated or I/O error under mmapUse pread for data you don't control, or handle SIGBUS
Durability liesData lost even with fsyncDevice or hypervisor ignores flushesPLP drives; verify with power-pull testing (e.g. diskchecker.pl)

War Story (illustrative)

A team ran a metrics service that wrote a 2 GB state snapshot every 10 minutes on 256 GB hosts with default writeback settings. Each snapshot went into the page cache at memory speed, then 30 seconds later the flusher began pushing it to network-attached storage at ~200 MB/s. During that ~10-second window, the service's tiny fdatasynced append log queued behind the snapshot's writeback on the same volume, and p99 write latency jumped from 2 ms to 4 seconds — every ten minutes, like clockwork. The fix was three lines: vm.dirty_background_bytes = 64MB so the snapshot started flushing immediately and in small batches, sync_file_range(SYNC_FILE_RANGE_WRITE) every 8 MB in the snapshot writer, and moving the append log to its own volume. Latency spikes disappeared without touching application logic.


Decision Guide

graph TD
    A["Writing data that must survive a crash?"] -->|no| Z0["Plain write(). Let the page cache work."]
    A -->|yes| B{"Replacing a whole file?"}
    B -->|yes| Z1["temp file, fsync, rename, fsync dir"]
    B -->|no| C{"Appending to a log?"}
    C -->|yes| D{"Preallocated / fixed-size segments?"}
    D -->|yes| Z2["fdatasync per commit or group commit"]
    D -->|no| Z3["fsync, plus fsync dir on new segments<br/>checksum records"]
    C -->|"no: random updates in place"| E{"Own buffer pool?"}
    E -->|yes| Z4["O_DIRECT + fsync,<br/>or buffered + WAL + checkpoint fsync"]
    E -->|no| Z5["Use a database. Really."]
SituationRecommendation
Application config / state fileAtomic replace pattern; never O_TRUNC in place
Append-only WAL / journalfdatasync at commit, batch commits (group commit), checksum each record
Database with its own large buffer poolO_DIRECT for data files + fsync; buffered or O_DSYNC for the log
Read-mostly large files (indexes, models)Buffered pread or mmap; fadvise/madvise hints; vmtouch -t to warm
Streaming throughput (logs, Kafka-like)Page cache + sendfile; durability via replication or periodic fsync
One-pass bulk copy / backupPOSIX_FADV_DONTNEED behind you, or O_DIRECT
Large-RAM write-heavy hostSet vm.dirty_background_bytes and vm.dirty_bytes to absolute values
You can't afford to lose an acknowledged writefsync before acknowledging, PLP storage, and test with real power loss
fsync returned an errorCrash and recover from a durable log. Do not retry and continue.

Interview Questions

"Your program called write() and got success. The machine lost power one second later. Is the data on disk?" Almost certainly not. write() copied it into the page cache and marked pages dirty. Writeback happens when pages are ~30 s old or dirty thresholds are crossed. Only a successful fsync/fdatasync (on hardware that honours flushes) guarantees durability.

"What's the difference between fsync and fdatasync?" fsync flushes data and all inode metadata; fdatasync flushes data plus only the metadata needed to read it back (e.g. file size, block allocation), skipping things like mtime. For overwrites of preallocated files it avoids a journal commit per call.

"Why fsync a directory?" A file's name is an entry in its directory. Creating or renaming a file modifies the directory; fsyncing the file does not necessarily persist that entry, so after a crash the file may not be findable. Fsync the parent directory after create/rename.

"What was fsyncgate?" PostgreSQL (and most software) assumed that after a failed fsync, retrying would eventually write the data. Linux instead marks failed pages clean and reports the error once, so a retry "succeeds" while the data is lost. PostgreSQL now PANICs on data-file fsync failure and recovers from WAL; Linux 4.13+ improved error reporting to all file descriptors.

"Why does InnoDB use O_DIRECT while PostgreSQL uses buffered I/O?" InnoDB manages its own large buffer pool and wants to avoid double caching and OS writeback interference. PostgreSQL historically chose portability and relies on the OS cache as a second-tier cache and for readahead/write-back, at the cost of double buffering and exposure to kernel writeback behaviour.

"Why does Kafka perform well without its own cache?" It appends sequentially through the page cache, consumers read recent data straight from cache, and sendfile sends pages to sockets with zero copies. The JVM heap stays small, avoiding GC pressure, and durability comes from replication rather than per-message fsync.

"When would you choose mmap over read?" Read-mostly, random access to large files where the kernel's caching is good enough — indexes, shared libraries, model weights — and you want to avoid copies and syscalls. Avoid it for transactional write paths where you need control over write ordering, explicit I/O error handling, and predictable stalls.