File Systems
A file system is the layer that translates human-readable names (/etc/passwd) into sequences of bytes on a storage device. Understanding file systems demystifies database storage engines, explains why fsync() matters for durability, and clarifies why "out of inodes" is a real error you can hit even with free disk space.
Why File Systems Exist
A disk, to the kernel, is just a giant array of fixed-size blocks: "read 4096 bytes at offset 81,920,000". That is a terrible interface for programs. Every application would have to decide where on the device its data lives, remember those locations somewhere, avoid colliding with every other application, and survive a power cut halfway through an update.
A file system solves five problems at once:
| Problem | File system answer |
|---|---|
| Naming — find data by a human name | Directories: a tree of name → object mappings |
| Allocation — which blocks are free? | Bitmaps, extent trees, B+trees of free space |
| Metadata — size, owner, permissions, times | Inodes |
| Sharing & protection — many processes, many users | Permissions, locks, a single kernel-managed namespace |
| Crash consistency — power fails mid-update | Journaling, copy-on-write, or (historically) fsck |
Every design decision in the rest of this page is a trade-off among these five, plus raw performance.
The Abstraction Stack
Application: open("/etc/passwd", O_RDONLY)
↓
System Call: sys_open()
↓
VFS Layer: Virtual File System (common interface)
↓
File System: ext4 / XFS / APFS / NTFS / tmpfs
↓
Block Layer: I/O scheduler, block device driver
↓
Storage: SSD / HDD / NVMe
The Virtual File System (VFS) is the kernel's abstraction layer that presents a uniform interface regardless of the underlying file system. It defines the objects: superblock, inode, dentry, file.
flowchart TB
APP["Application: open, read, write, stat, rename"] --> SC["Syscall layer: openat, read, write, statx, renameat2"]
SC --> VFS["VFS: path walk, permission checks, dcache, inode cache, file objects"]
VFS --> PC["Page cache (shared by all disk filesystems)"]
VFS --> EXT4["ext4"]
VFS --> XFS["XFS"]
VFS --> BTRFS["btrfs"]
VFS --> OVL["overlayfs (stacks on another fs)"]
VFS --> TMP["tmpfs (RAM only)"]
VFS --> PROC["procfs, sysfs (synthetic)"]
VFS --> NFS["NFS, FUSE (remote or userspace)"]
EXT4 --> BL["Block layer and I/O scheduler"]
XFS --> BL
BTRFS --> BL
BL --> DEV["NVMe, SATA, virtio, EBS"]
Below the filesystem, the block layer, I/O schedulers and devices are covered in I/O & Storage; the page cache, writeback and fsync() semantics in depth are in Page Cache & fsync.
The VFS Layer
The VFS is an object-oriented design written in C: each object is a struct plus a table of function pointers that a concrete filesystem fills in. Generic code (path lookup, permission checks, caching, locking) lives once in the VFS; ext4, XFS, NFS and procfs only implement the operations that differ.
| VFS object | Represents | Lifetime | Operations table |
|---|---|---|---|
super_block | One mounted filesystem instance | Mount → unmount | super_operations: alloc_inode, write_inode, sync_fs, statfs |
inode | One file object (regular file, dir, symlink, device, socket, FIFO) — the metadata | Cached while referenced or until evicted | inode_operations: lookup, create, link, unlink, mkdir, rename, setattr |
dentry | One name in a path, linking a name to an inode | In the dentry cache (dcache) | dentry_operations: d_revalidate (NFS), d_hash, d_compare (case-insensitive fs) |
file | One open of a file — current offset, flags, mode | open() → last close() | file_operations: read_iter, write_iter, mmap, fsync, llseek, ioctl |
address_space | The page-cache pages of one inode | With the inode | address_space_operations: read_folio, writepages, dirty_folio |
flowchart LR
subgraph Proc["Process A"]
FDT["fd table: fd 3"]
end
subgraph ProcB["Process B"]
FDTB["fd table: fd 7"]
end
FDT --> F1["struct file: offset 4096, O_RDONLY"]
FDTB --> F2["struct file: offset 0, O_RDWR"]
F1 --> D["dentry: passwd (parent: etc)"]
F2 --> D
D --> I["inode 2097153: size, mode, owner, extents"]
I --> AS["address_space: cached pages"]
I --> SB["super_block: ext4 on nvme0n1p2"]
Two opens of the same file produce two struct files with independent offsets, sharing one dentry and one inode (and therefore one set of cached pages). That's why two processes read()ing the same file each see their own position but share the page cache.
The objects are visible in the kernel's slab caches:
$ sudo slabtop -o -s c | head -12
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
4210816 4198123 99% 0.19K 200515 21 802060K dentry
1630272 1622001 99% 1.07K 54342 30 1738944K ext4_inode_cache
...
$ cat /proc/sys/fs/dentry-state
3891245 3712011 45 0 2104553 0
# nr_dentry nr_unused age_limit want_pages nr_negative dummy
On a busy build server or a box that stat()s millions of files, the dentry and inode caches can hold gigabytes. This is reclaimable memory (it shows up under SReclaimable in /proc/meminfo) but it surprises people who wonder why free shows little available memory.
Negative dentries
The dcache also caches failed lookups. If a program repeatedly tries to open nonexistent files (e.g. a $PATH search, or a library probing 20 locations for a config file), the kernel remembers "/usr/local/lib/libfoo.so does not exist" so the next probe costs nanoseconds. Pathological workloads that probe millions of random nonexistent names can grow negative dentries into tens of GB; nr_negative in dentry-state shows them.
Superblocks and On-Disk Layout
The superblock describes a whole filesystem: block size, total and free blocks and inodes, feature flags, UUID, mount count, state (clean/dirty). Corrupt the superblock and the filesystem is unmountable, so ext4 stores backup copies in several block groups.
ext4 divides the disk into block groups (typically 128 MiB each with 4 KiB blocks), each with its own block bitmap, inode bitmap and inode table, keeping a file's inode near its data:
ext4 on-disk layout:
| Boot | Group 0 | Group 1 ...
| Superblock | Group descriptors | Block bitmap | Inode bitmap | Inode table | Data blocks ...
$ sudo dumpe2fs -h /dev/nvme0n1p2 | grep -E 'Block size|Inode count|Free inodes|Inode size|Filesystem features|Journal size'
Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent 64bit flex_bg metadata_csum ...
Inode count: 30531584
Free inodes: 28102331
Block size: 4096
Inode size: 256
Journal size: 1024M
$ sudo dumpe2fs /dev/nvme0n1p2 | grep -i 'backup superblock' | head -3
Backup superblock at 32768, Group descriptors at 32769-32769
Backup superblock at 98304, ...
$ sudo e2fsck -b 32768 /dev/nvme0n1p2 # repair using a backup superblock
XFS instead divides the device into allocation groups (AGs) that are each managed independently with their own B+trees for free space and inodes, which is why XFS scales well with parallel writers on large arrays.
Inodes
An inode (index node) is the kernel's representation of a file. It stores all metadata about a file — everything except the filename.
What an inode contains:
Inode:
inode number: 2097153 (unique identifier within the filesystem)
file type: regular file (or directory, symlink, socket, pipe, device)
permissions: 0644 (-rw-r--r--)
owner UID/GID: 1000/1000
size: 4096 bytes
timestamps:
atime: 2026-06-12 10:30:00 (last access)
mtime: 2026-06-11 09:00:00 (last modification of content)
ctime: 2026-06-11 09:00:00 (last change of inode, incl. permissions)
link count: 1
block pointers: [block1, block2, ..., indirect block, double indirect block]
Modern filesystems store more: a birth time (btime/crtime, visible via statx() and stat on recent coreutils), nanosecond timestamps, extended attributes (xattrs — SELinux labels, ACLs, user.* metadata), a generation number (so NFS file handles don't accidentally refer to a reused inode), and a checksum (metadata_csum). ext4 inodes are 256 bytes by default; small xattrs and tiny files (inline_data) can live directly in the spare space.
What an inode does NOT contain:
- The filename — filenames live in directory entries (dentries)
- The file's content — content lives in data blocks
Checking inodes:
ls -li /etc/passwd # -i shows inode number
stat /etc/passwd # detailed inode info
df -i /dev/sda1 # inode usage on filesystem
$ stat /etc/passwd
File: /etc/passwd
Size: 2847 Blocks: 8 IO Block: 4096 regular file
Device: 259,2 Inode: 2097153 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2026-11-08 09:12:41.112233445 +0000
Modify: 2026-10-30 14:02:10.000000000 +0000
Change: 2026-10-30 14:02:10.000000000 +0000
Birth: 2026-03-01 08:00:00.000000000 +0000
Blocks: 8 is in 512-byte units — 4 KiB actually allocated for a 2,847-byte file.
Inode counts are fixed at mkfs time on ext4. The default is one inode per 16 KiB of space (-i 16384). A filesystem holding hundreds of millions of tiny files (mail spools, cache directories, node_modules farms, Maildir, image thumbnails) can run out of inodes with most of its bytes free. Plan with mkfs.ext4 -i 4096 or -N <count>, or use XFS or btrfs, which allocate inodes dynamically.
atime and Why Everyone Disables It
Updating access time on every read turns reads into writes. Linux defaults to relatime: atime is updated only if it is older than mtime/ctime or more than 24 hours old — enough for tools like mutt that check "has this been read since modified". noatime disables it entirely and is a common, safe optimization for servers. lazytime keeps all timestamps in memory and writes them lazily.
Hard links work because multiple directory entries can point to the same inode. link count tracks how many. unlink() decrements the count; when it reaches 0 (and no open file descriptors), the data blocks are freed.
Hard Links vs Symbolic Links
| Hard link | Symbolic (soft) link | |
|---|---|---|
| What it is | Another directory entry pointing at the same inode | A tiny file whose content is a path string |
ls -li | Same inode number as the target | Its own inode; type l |
| Survives target being deleted/renamed | Yes — data lives until the last link is gone | No — becomes a dangling link |
| Cross filesystems | No (EXDEV) — inode numbers are per filesystem | Yes |
| Can point to directories | No (would create cycles; only ./.. are kernel-managed) | Yes |
| Can point to nonexistent paths | No | Yes |
| Permissions | Shared with the target (it is the target) | Ignored; target's permissions apply |
| Resolution cost | None — it's just a directory entry | Extra path walk; limit of 40 nested links (ELOOP) |
$ echo hello > a.txt
$ ln a.txt hard.txt # hard link
$ ln -s a.txt soft.txt # symlink
$ ls -li
2097301 -rw-r--r-- 2 alice alice 6 Nov 8 10:00 a.txt
2097301 -rw-r--r-- 2 alice alice 6 Nov 8 10:00 hard.txt # same inode, link count 2
2097302 lrwxrwxrwx 1 alice alice 5 Nov 8 10:00 soft.txt -> a.txt
$ rm a.txt
$ cat hard.txt # hello - data still referenced
$ cat soft.txt # cat: soft.txt: No such file or directory
$ readlink -f soft.txt; namei -l /usr/bin/python3 # resolve and show each step
Where you meet them in practice:
- Atomic deploys use symlinks:
current -> releases/2026-11-08_1200; switching isln -sfn+rename(), so the swap is atomic. - Package managers and backup tools (Nix, pnpm's content-addressed store,
rsync --link-dest, Time Machine) use hard links to share identical files with zero copies. - Kubernetes ConfigMap volumes are a symlink farm:
..data -> ..2026_11_08_10_00_00.123, and each key is a symlink through..data, so an update is one atomic symlink swap. Tools that watch the file withinotifymust watch the directory or they miss the update.
Deleted but Still Open
The "link count 0 and no open file descriptors" rule produces a classic mystery: df says the disk is full, du can't find the space.
$ df -h /var
/dev/nvme1n1 100G 100G 0 100% /var
$ sudo du -sh /var
41G /var
$ sudo lsof +L1 # open files with link count < 1
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
java 4123 app 12w REG 259,3 62277025792 0 131074 /var/log/app/app.log (deleted)
Someone ran rm app.log on a 58 GB log that the JVM still had open. The directory entry is gone (so du doesn't see it), but the inode and its blocks live until the process closes the fd. Fixes: restart or signal the process to reopen logs, or truncate in place without restarting: : > /proc/4123/fd/12. Prevention: rotate logs with copytruncate or (better) a reopen signal like nginx's SIGUSR1 — see Signals.
The same trick is used deliberately: create a temp file, unlink() it immediately, keep using the fd — the space is reclaimed automatically even if the process crashes. O_TMPFILE does this atomically (and linkat() can give it a name later — a crash-safe way to publish a fully written file).
Directories
A directory is a special file whose content is a list of (filename, inode number) pairs — called directory entries (dentries).
Directory: /home/alice/
. → inode 2097152 (current directory itself)
.. → inode 1048576 (parent directory)
docs → inode 2097200
app.py → inode 2097153
Path resolution for /home/alice/app.py:
- Start at root inode (inode 2 on ext2/3/4; XFS root is usually 128)
- Look up
homein root directory → inode for/home - Look up
alicein/home→ inode for/home/alice - Look up
app.pyin/home/alice→ inode 2097153 - Read inode 2097153 to get block pointers, permissions, size
- Read data blocks to get file content
This is why deep directory trees have more path resolution overhead than shallow ones.
How Lookup Really Works: the dcache
In practice, most of those steps never touch the disk. The kernel walks the path component by component through the dentry cache, a hash table keyed by (parent dentry, name):
flowchart TD
A["openat: /home/alice/app.py"] --> B["Start at process root or cwd dentry"]
B --> C{"Next component in dcache?"}
C -->|"hit (common)"| D["Check execute permission on parent dir"]
C -->|"miss"| E["Call filesystem lookup: read directory blocks, maybe from disk"]
E --> F["Insert positive or negative dentry into dcache"]
F --> D
D --> G{"Mount point, symlink, or dot-dot?"}
G -->|"mount point"| H["Cross into the mounted filesystem root"]
G -->|"symlink"| I["Read link target, restart walk from there (max 40)"]
G -->|"no"| J{"More components?"}
H --> J
I --> J
J -->|yes| C
J -->|no| K["Final dentry and inode: permission check, allocate struct file"]
- RCU-walk: the fast path walks the dcache without taking any locks or reference counts, using RCU and sequence counters; it falls back to the slower "ref-walk" only if something changes underneath. A fully cached path lookup costs on the order of 100 ns–1 µs.
- Every directory on the path needs execute (
x, "search") permission for the caller.chmod 700 /home/alicemakes every file under it unreachable to others regardless of the files' own permissions. - Mount points are crossed transparently: if
/homeis a separate filesystem, the walk jumps from the root fs'shomedentry to the root dentry of the mounted fs.
strace shows what a program really asks for:
$ strace -e trace=openat,newfstatat,statx -f python3 -c 'import json' 2>&1 | head
openat(AT_FDCWD, "/usr/lib/python3.12/encodings/__pycache__/__init__.cpython-312.pyc", O_RDONLY|O_CLOEXEC) = 3
newfstatat(AT_FDCWD, "/usr/lib/python3.12", {st_mode=S_IFDIR|0755, ...}, 0) = 0
openat(AT_FDCWD, "/home/alice/.local/lib/python3.12/site-packages/json.py", O_RDONLY) = -1 ENOENT
...
$ strace -c -e trace=file python3 -c 'import numpy' # count file-related syscalls
Large Directories
A naive directory is a linear list: finding a name in a million-entry directory means scanning it. Modern filesystems index directories:
| Filesystem | Directory structure | Practical limit |
|---|---|---|
ext2/ext3 without dir_index | Linear list | Slow beyond ~10k entries |
| ext4 | HTree (hashed B-tree, dir_index) | ~10M entries with 2 levels; 3 levels with large_dir |
| XFS | B+tree | Effectively unlimited; handles millions well |
| btrfs | B-tree keyed by name hash | Effectively unlimited |
Even with indexing, huge directories hurt: ls sorts everything, readdir() order is hash order (so you get random disk access when you stat each entry), and backup tools crawl them. The usual fix is sharding by hash prefix: objects/ab/cdef1234... — exactly what Git, Docker's content store and most object caches do.
Block Allocation
Disk space is allocated in blocks (typically 4KB). A file's content is stored across one or more blocks. The inode contains block pointers that tell the kernel where to find the file's blocks on disk.
ext2/ext3 block pointer structure (4 KiB blocks, 4-byte block numbers → 1,024 pointers per indirect block):
Inode block pointers:
12 direct pointers → points directly to data blocks (12 × 4KB = 48KB)
1 single indirect → points to a block of pointers (4KB/4B = 1024 pointers = 4MB)
1 double indirect → pointer to block of pointers to blocks of pointers (1024 × 1024 × 4KB = 4GB)
1 triple indirect → (1024³ × 4KB = 4TB; rarely needed)
For small files (< 48KB), all blocks are directly accessible. For large files, the kernel follows indirect pointers.
ext4 uses extents instead of block pointers: An extent is a contiguous range (start_block, length). Instead of one pointer per block, one extent covers thousands of consecutive blocks. Much more efficient for large files and reduces fragmentation.
One ext4 extent covers up to 32,768 blocks (128 MiB). The inode holds up to 4 extents inline; larger or more fragmented files get an extent tree. A perfectly contiguous 1 GiB file needs just 8 extents, versus 262,144 block pointers in ext3.
$ filefrag -v bigfile.bin
ext: logical_offset: physical_offset: length: expected: flags:
0: 0.. 32767: 3440640.. 3473407: 32768:
1: 32768.. 65535: 3473408.. 3506175: 32768:
...
bigfile.bin: 8 extents found
Allocation tricks that matter for performance:
- Delayed allocation (delalloc): ext4, XFS and btrfs don't choose disk blocks at
write()time; they wait until writeback, when they know how much data there is and can allocate one big contiguous extent. Faster and less fragmented — but it changed crash behavior (see the ext4 zero-length files story). - Preallocation:
fallocate(fd, 0, 0, size)reserves blocks up front without writing them (marked "unwritten" so reads return zeros). Databases and log-structured systems (Kafka segments, PostgreSQL WAL files, SQLite) use it to guarantee contiguous space and avoidENOSPCmid-write. - Sparse files: seeking past EOF and writing leaves a hole that consumes no blocks.
ls -lshows the logical size,dushows allocated size. VM disk images and core dumps are usually sparse; copying them with a naive tool "inflates" them (usecp --sparse=always,rsync -S).
$ truncate -s 100G sparse.img; ls -lh sparse.img; du -h sparse.img
-rw-r--r-- 1 alice alice 100G Nov 8 10:00 sparse.img
0 sparse.img
$ fallocate -l 1G prealloc.bin && du -h prealloc.bin # 1.0G allocated instantly
File Descriptors
A file descriptor (fd) is an integer that identifies an open file within a process. When you call open(), the kernel:
- Looks up the path, resolves to an inode
- Creates a file table entry in the kernel (stores current position, mode, inode pointer)
- Adds an entry to the process's file descriptor table pointing to the file table entry
- Returns the index (the file descriptor integer)
Process file descriptor table:
fd 0 → stdin (file table entry → terminal inode)
fd 1 → stdout (file table entry → terminal inode)
fd 2 → stderr (file table entry → terminal inode)
fd 3 → /etc/passwd (file table entry → inode 2097153, pos=0, mode=O_RDONLY)
There are three levels, and which level a piece of state lives at explains most fd semantics:
| Level | Holds | Shared by |
|---|---|---|
| fd table (per process) | Integer → struct file pointer, plus the FD_CLOEXEC flag | Threads of the process (unless CLONE_FILES is off) |
open file description (struct file) | Offset, status flags (O_APPEND, O_NONBLOCK), access mode | fds created by dup()/dup2(), and parent + child after fork() |
| inode | Size, permissions, data, locks | Every open of the file, by anyone |
That's why O_NONBLOCK set via fcntl() on a socket inherited from a parent affects the parent too, and why dup2(fd, 1) makes stdout share an offset with fd — the basis of shell redirection.
File descriptor limits:
- Per-process: typically 1024 (soft) / 4096+ (hard). Configurable via
ulimit -n. - System-wide:
/proc/sys/fs/file-max. Running out causes "too many open files" errors.
lsof -p <pid> # list open files for a process
ls -l /proc/<pid>/fd/ # list fd numbers and what they point to
ulimit -n # current per-process fd limit
The two errors are different and point at different fixes:
| errno | Message | Meaning | Fix |
|---|---|---|---|
EMFILE | "Too many open files" | This process hit RLIMIT_NOFILE | Raise the soft limit, or find the fd leak |
ENFILE | "Too many open files in system" | The system hit fs.file-max | Raise fs.file-max (rare on modern kernels) |
Where the limits come from in practice:
$ ulimit -Sn; ulimit -Hn # soft / hard for this shell
1024
524288
$ cat /proc/$(pgrep -n nginx)/limits | grep 'open files'
Max open files 65536 65536 files
$ cat /proc/sys/fs/nr_open # ceiling for any process's hard limit
1048576
$ cat /proc/sys/fs/file-nr # allocated, unused(0), max
12480 0 9223372036854775807
- systemd services ignore
/etc/security/limits.conf(that's PAM, for logins). SetLimitNOFILE=65536in the unit. systemd's default is soft 1024 / hard 524288 — the soft limit stays at 1024 because programs usingselect()break aboveFD_SETSIZE(1024). - Docker / containerd set the container's limit from the daemon's config (
default-ulimitsor the containerd unit'sLimitNOFILE); historically it was effectively unlimited (1,048,576 or more), which made some programs that loopclose()over every possible fd take seconds to start. - Go (1.19+) automatically raises its soft limit to the hard limit at startup; Node, Python and the JVM do not by default (the JVM's
-XX:+MaxFDLimitdoes on macOS). - Every socket is an fd. A proxy handling 50,000 concurrent connections needs 100,000+ fds (client + upstream). Budget
ulimit -n≥ 2 × max connections + files + slack.
// Debugging an fd leak: count and classify what a process holds
// $ ls -l /proc/<pid>/fd | awk '{print $NF}' | sed 's/[0-9]\+/N/g' | sort | uniq -c | sort -rn
// 9214 socket:[N] <- leaked connections
// 34 /var/lib/app/data.db
// 3 pipe:[N]
After fork(): The child inherits the parent's fd table. Both parent and child share the same file table entries — including the file position pointer. If the parent reads, the position advances for both. Use O_CLOEXEC to close fds automatically after exec().
Always open with `O_CLOEXEC`
Without it, every fd your process has open leaks into every program it executes — database connections, the listening socket, secrets files. A child holding a copy of your listening socket keeps the port bound after you exit (EADDRINUSE on restart). Setting FD_CLOEXEC with fcntl() after open() races with fork() in other threads; pass O_CLOEXEC (and SOCK_CLOEXEC, pipe2(..., O_CLOEXEC), accept4(..., SOCK_CLOEXEC)) atomically. Go, Rust and Python 3.4+ do this by default; C does not.
Journaling
Without journaling, a crash during a write operation can leave the file system in an inconsistent state (partial write, updated inode pointing to stale blocks). fsck had to scan the entire disk to find and fix inconsistencies — could take hours.
The underlying problem is that one logical operation needs several independent block writes. Appending a block to a file touches: the data block, the block bitmap (mark it used), and the inode (new size, new extent). The disk can complete them in any order, and the power can fail between any two:
| Crash leaves... | Result |
|---|---|
| Bitmap updated, inode not | Leaked block (wasted space) |
| Inode updated, bitmap not | Block marked free but in use → later double allocation, data corruption |
| Inode updated, data block not | File contains garbage from whatever was there before (possibly another user's deleted data) |
Journaling maintains a write-ahead log (the journal) that records intended operations before committing them to the filesystem. On crash, the kernel replays the journal to restore consistency. Recovery takes seconds, not hours.
sequenceDiagram
participant FS as "ext4 (JBD2)"
participant J as "Journal area"
participant M as "Metadata home locations"
participant D as "Data blocks"
Note over FS: "data=ordered: write data first"
FS->>D: "1. write data blocks to final location"
FS->>J: "2. write descriptor plus modified metadata blocks"
FS->>J: "3. write commit block (after a cache flush or FUA)"
Note over J: "transaction is now durable, replay will redo it"
FS->>M: "4. checkpoint: write metadata to home locations, later"
FS->>J: "5. free journal space"
On mount after a crash, ext4 scans the journal: transactions with a valid commit block are replayed (copied to their home locations); incomplete ones are discarded. The filesystem is consistent again in seconds, regardless of disk size.
ext4 journaling modes:
- journal (safest): data AND metadata written to journal first, then to disk. Survives crashes but highest I/O overhead.
- ordered (default): metadata journaled, data written to disk first before metadata is committed. Data written before crash may be lost, but filesystem is consistent.
- writeback (fastest): only metadata journaled. Data may appear after metadata update, so a crash can expose old data. Not safe for databases.
| Mode | Journaled | Write cost | After a crash you may see | Typical use |
|---|---|---|---|---|
data=journal | Data + metadata | ~2× for data (written twice) | At most the last few seconds missing; never garbage | Small, sync-heavy workloads; can actually help small fsyncs (sequential journal writes) |
data=ordered (default) | Metadata only; data flushed before its metadata commits | Low | Recent writes missing; never stale/garbage blocks | Almost everything |
data=writeback | Metadata only, no ordering | Lowest | Files containing stale blocks from deleted files | Scratch space; databases that fsync everything themselves |
$ grep ' / ' /proc/mounts
/dev/nvme0n1p2 / ext4 rw,relatime,errors=remount-ro 0 0 # data=ordered is implied
$ sudo tune2fs -l /dev/nvme0n1p2 | grep -i 'journal'
Journal features: journal_incompat_revoke journal_64bit journal_checksum_v3
$ mount -o remount,commit=5 / # JBD2 commits every 5 s by default
The commit interval (default 5 s) is why, without fsync(), up to ~5 s of metadata changes (and up to ~30 s of data, via writeback timers) can vanish in a crash even though the filesystem itself is perfectly consistent. Consistent is not the same as durable.
War Story: the ext4 zero-length file problem (2009)
When ext4 introduced delayed allocation, users of early Ubuntu releases reported that after a crash, desktop configuration files were zero bytes long. Applications updated config files with the pattern open("file.new") → write() → close() → rename("file.new", "file") — without fsync(). With delayed allocation, the rename (metadata) was committed to the journal within 5 seconds, but the data blocks weren't allocated or written for up to 60 seconds. A crash in that window left a renamed, empty file — the old contents gone. POSIX allowed it; ext3's ordered mode had simply hidden the bug. The fix was twofold: ext4 added the auto_da_alloc heuristic (force data out before a rename-over or truncate commits), and application developers were told to use the correct pattern: write temp → fsync(temp) → rename() → fsync(dir). The full durability protocol is in Page Cache & fsync.
Why this matters for databases: Databases implement their own WAL (Write-Ahead Log) precisely because they can't rely on file system ordering guarantees. PostgreSQL, MySQL InnoDB, SQLite all use WAL to ensure crash recovery without depending on the file system journal.
A filesystem journal protects the filesystem's structures; it says nothing about whether your multi-page update was atomic. A database page write of 8 KiB can be torn (half old, half new) by a crash on 4 KiB-sector hardware — which is why PostgreSQL writes full page images to WAL after each checkpoint (full_page_writes) and InnoDB has its doublewrite buffer. See Write-Ahead Logging and Storage Engines.
Copy-on-Write File Systems
Journaling writes changes twice (journal, then home location) and updates blocks in place. Copy-on-write (CoW) filesystems — btrfs, ZFS, APFS, bcachefs — never overwrite live data. A modified block is written to a new location, then the pointer to it is updated, then the pointer to that, all the way up to the root. The root pointer update (the "superblock" or "uberblock") is a single atomic write.
flowchart TB
subgraph Before["Before: tree rooted at R1"]
R1["root R1"] --> A1["node A"]
R1 --> B1["node B"]
A1 --> D1["data 1"]
A1 --> D2["data 2"]
B1 --> D3["data 3"]
end
subgraph After["After modifying data 2: new root R2"]
R2["root R2"] --> A2["node A copy"]
R2 --> B1b["node B (shared, unchanged)"]
A2 --> D1b["data 1 (shared)"]
A2 --> D2n["data 2 NEW"]
B1b --> D3b["data 3 (shared)"]
end
Consequences:
| Property | Why CoW gives it |
|---|---|
| Always consistent on disk | The old tree is untouched until the new root is committed; a crash just leaves the old root. No fsck, no journal replay. |
| Cheap snapshots | A snapshot is "keep the old root alive". O(1) to create; space is only consumed as blocks diverge. |
| Clones and reflinks | cp --reflink=always big.img copy.img shares all blocks instantly (also supported by XFS). |
| End-to-end checksums | Each pointer stores the checksum of the block it points to, so silent corruption (bit rot, misdirected writes) is detected — and repaired from a mirror/parity copy. |
| Send/receive replication | Diff two snapshots and stream only changed blocks (zfs send -i, btrfs send -p). |
| Transparent compression | Since blocks are rewritten anyway, compress them (zstd, lz4). |
And the costs:
- Fragmentation of randomly-overwritten files. A database file or VM image that is updated in place becomes scattered across the disk, because every overwrite goes elsewhere. Sequential scans of it turn into random I/O. Mitigations: btrfs
chattr +C(nodatacow) on database directories, ZFSrecordsize=8K/16Kmatched to the database page size, and SSDs. - Write amplification — a 4 KiB change rewrites the data block plus every tree node up to the root (batched per transaction group, but still more than in-place).
- Space accounting is hard. "How much will deleting this free?" depends on snapshots. CoW filesystems can hit
ENOSPCwith free space reported, because even a delete needs to write new metadata. Keep 10–20 % free. - ZFS ARC is a separate cache from the Linux page cache, sized by
zfs_arc_max; on Linux it competes awkwardly with applications for memory.
File System Comparison
| ext4 | XFS | btrfs | ZFS (OpenZFS) | |
|---|---|---|---|---|
| Design | Journaling, in-place | Journaling (metadata), in-place, allocation groups | Copy-on-write B-trees | Copy-on-write, pooled storage + volume manager |
| Max file size | 16 TiB (4 KiB blocks) | 8 EiB | 16 EiB | 16 EiB |
| Max volume | 1 EiB (vendors support ~50–100 TiB) | 8 EiB | 16 EiB | 256 ZiB-scale |
| Inodes | Fixed at mkfs | Dynamic | Dynamic | Dynamic |
| Data checksums | No (metadata only) | No (metadata only, v5) | Yes | Yes |
| Snapshots | No (use LVM) | No (use LVM) | Yes | Yes |
| Compression | No | No | zstd, lzo, zlib | lz4, zstd, gzip |
| Built-in RAID | No (mdadm) | No (mdadm) | RAID 0/1/10 solid; 5/6 not recommended | Mirrors, RAID-Z1/2/3 — mature |
| Shrink online | Offline shrink only | Cannot shrink | Yes | Limited (device removal) |
| Parallel write scaling | Good | Excellent | Good | Good |
| fsync-heavy performance | Very good | Very good | Weaker (CoW tree updates) | Good with a SLOG device |
| Default on | Debian, Ubuntu | RHEL, Amazon Linux 2023, Rocky | Fedora Workstation, SUSE | TrueNAS, many storage appliances (out-of-tree on Linux) |
Quick picks:
- General servers, VMs, cloud volumes: ext4 or XFS. Boring, fast, well understood; leave integrity, snapshots and replication to the layer above (EBS snapshots, database replication).
- Large files, many parallel writers, big arrays: XFS.
- Databases: ext4 or XFS with
noatime. Avoid CoW for the data directory, or tune it (nodatacow / recordsize). - Workstations, snapshots before upgrades, homelab: btrfs.
- Storage servers where data integrity is the product: ZFS.
- Flash-only embedded/phones: F2FS (log-structured, flash-friendly).
- Scratch and secrets: tmpfs (RAM-backed, gone on reboot; counts against memory cgroup limits).
OverlayFS and Container Image Layers
Container images are stacks of read-only layers; a running container needs a writable view of them. OverlayFS (in-kernel since 3.18) merges directories:
mkdir -p lower1 lower2 upper work merged
echo "from base" > lower1/a.txt
echo "from app" > lower2/b.txt
sudo mount -t overlay overlay \
-o lowerdir=lower2:lower1,upperdir=upper,workdir=work merged
ls merged # a.txt b.txt
echo change >> merged/a.txt
ls upper # a.txt <- copied up on first write
rm merged/b.txt
ls -l upper # c--------- b.txt 0,0 <- whiteout (char device 0/0)
flowchart TB
M["merged (what the container sees as /)"]
U["upperdir: container writable layer (copy-ups, new files, whiteouts)"]
L1["lowerdir 3: app layer (read-only)"]
L2["lowerdir 2: dependencies layer (read-only)"]
L3["lowerdir 1: base image, e.g. debian (read-only)"]
M --> U
U --> L1
L1 --> L2
L2 --> L3
Rules of the merge:
- Lookup goes top-down: upper first, then each lower; the first match wins.
- Writing a lower file triggers copy-up: the entire file is copied to upper before the first byte is modified. Appending one line to a 2 GB file in an image copies 2 GB.
- Deleting a lower file creates a whiteout (a 0/0 character device) in upper that hides it. The lower data still exists — deleting a file in a later Dockerfile layer doesn't shrink the image.
- Replacing a directory creates an opaque directory (
trusted.overlay.opaque=yxattr) that hides everything below it. - Renaming a lower directory historically returned
EXDEV; tools fall back to copy + delete (enableredirect_dirto avoid it).
Docker's overlay2 driver keeps layers under /var/lib/docker/overlay2/<id>/diff and uses up to 128 lower layers per mount (shortened via symlinks in l/). containerd keeps them in /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/.
$ docker inspect web --format '{{json .GraphDriver.Data}}' | jq
{
"LowerDir": "/var/lib/docker/overlay2/7c1.../diff:/var/lib/docker/overlay2/a9f.../diff",
"MergedDir": "/var/lib/docker/overlay2/3e2.../merged",
"UpperDir": "/var/lib/docker/overlay2/3e2.../diff",
"WorkDir": "/var/lib/docker/overlay2/3e2.../work"
}
$ docker diff web # what the container changed in its upper layer
C /etc
A /etc/app.conf
D /tmp/install.log
Don't run databases on the container's writable layer
Copy-up latency, no O_DIRECT guarantees on some kernels, inflated layer sizes, and data that disappears with the container. Mount a volume (bind mount or a Kubernetes PersistentVolume) for anything that writes heavily or must persist. The writable layer is for small, disposable changes.
War Story (illustrative)
A CI job ran chmod -R on a 3 GB dependency directory baked into the image. chmod changes only metadata — but on overlayfs, changing metadata of a lower file triggers a full copy-up of that file (later kernels added metacopy=on to copy only metadata, but it was off). The step took 4 minutes and the container's writable layer grew by 3 GB, filling the node's disk and triggering kubelet DiskPressure evictions. The fix was setting permissions in the Dockerfile at COPY --chmod=... time, inside the layer itself.
Page Cache
The kernel doesn't read directly from disk on every read() call. It maintains a page cache — recently read file pages cached in RAM. read() checks the cache first; only on a miss does it read from disk.
read(fd, buf, size):
1. Check page cache for requested pages
Cache hit: copy from cache to buf → return (microseconds)
Cache miss: read from disk → load into cache → copy to buf (milliseconds)
write(fd, buf, size):
1. Copy buf to page cache (write is immediate from app's view)
2. Mark page dirty
3. Return (async — data not on disk yet)
4. Kernel flusher threads write back dirty pages once they are older than
vm.dirty_expire_centisecs (30s by default, checked every 5s)
or when dirty ratio threshold is exceeded
fsync(fd): Forces all dirty pages for this fd to be flushed to disk immediately. Returns only after the storage device confirms the write. Essential for database durability — without fsync, a crash after write() but before the kernel flush loses data.
O_DIRECT: Bypass the page cache entirely. Data goes directly between user buffer and disk. Used by databases that want to manage their own caching (more efficient than double-caching: db buffer pool + page cache).
The details — fsync vs fdatasync, why you must also fsync the directory after a create or rename, what happens when fsync reports an error (PostgreSQL's 2018 "fsyncgate", where retrying a failed fsync on Linux silently reported success after the dirty pages had been dropped), writeback tuning, and O_DIRECT alignment rules — are covered in Page Cache & fsync.
What Happens When You open() a File
int fd = open("/etc/passwd", O_RDONLY);
open()→sys_open()syscall (glibc actually issuesopenat(AT_FDCWD, ...))- Kernel resolves path: root inode →
/etcdirectory →passwddentry → inode 2097153 - Permission check: does the calling process's UID/GID have read permission?
- Allocate a
struct file(file table entry): stores inode pointer, current position=0, mode=O_RDONLY - Add to the process's fd table; find the lowest available index (e.g., 3)
- Return 3
ssize_t n = read(fd, buf, 4096);
read()→sys_read()syscall- Look up fd 3 → file table entry → current position, inode
- Check page cache for this inode's pages at the current position
- Cache miss: read 4KB block from disk, load into page cache
- Copy 4096 bytes from page cache to
bufin user space - Advance file position by 4096
- Return number of bytes read
On a cache miss the kernel also starts readahead: it detects sequential access and reads ahead asynchronously (128 KiB by default, growing with the pattern — tunable per device in /sys/block/<dev>/queue/read_ahead_kb or per file with posix_fadvise(POSIX_FADV_SEQUENTIAL)). That's why sequential reads of a cold file run at device bandwidth rather than one round trip per 4 KiB.
ssize_t w = write(fd2, buf, 4096); // fd2 opened O_WRONLY on an existing file
write()→sys_write()→ext4_file_write_iter()- Take the inode lock (writes to one file are serialized at this level)
- For each affected page: find or allocate it in the page cache; if the write is partial and the page isn't cached, read it from disk first (read-modify-write)
- Copy user data into the page, mark it dirty; update
i_sizeandmtimein the in-memory inode - With delayed allocation, no disk blocks are chosen yet
- Return 4096 — nothing is on disk yet; flusher threads write it back later
$ strace -T -e trace=openat,read,write,fsync,close ./app
openat(AT_FDCWD, "data.bin", O_WRONLY|O_CREAT|O_CLOEXEC, 0644) = 3 <0.000021>
write(3, "..."..., 4096) = 4096 <0.000009> # 9 µs - just a memcpy
fsync(3) = 0 <0.002140> # 2.1 ms - the real cost
close(3) = 0 <0.000004>
Common File System Operations
# Inode usage
df -i / # inode usage for root filesystem
find / -xdev -type f | wc -l # count files (uses inodes)
# File descriptor debugging
lsof | wc -l # total open file descriptors system-wide (overcounts: threads, mmaps)
lsof -u username # per user
cat /proc/sys/fs/file-nr # [used, 0, max] system-wide fd count
# Block allocation
filefrag -v file.txt # show extent map (fragmentation)
tune2fs -l /dev/sda1 # ext4 filesystem info
# Journaling and sync
sync # flush all dirty pages
sync -f /data # syncfs(): flush one filesystem
# fsync(fd) is a syscall, not a shell command: flush one file's dirty pages
echo 3 > /proc/sys/vm/drop_caches # drop page cache (testing only)
More you'll reach for:
findmnt # mount tree with fs types and options
findmnt -T /var/lib/postgresql # which filesystem holds this path?
du -xh --max-depth=1 / | sort -h # where's the space (one filesystem only)
ncdu -x / # interactive version
lsof +L1 # deleted-but-open files holding space
find /var -xdev -type d -size +1M # directories with huge entry lists
xfs_info /data; xfs_db -r -c frag /dev/nvme1n1 # XFS geometry, fragmentation
btrfs filesystem usage /; btrfs scrub start / # btrfs space, verify checksums
zpool status -v; zfs list -o name,used,refer,compressratio
inotifywait -m -r /etc/app # watch for changes (inotify-tools)
getfattr -d -m - file; setfacl -m u:bob:r file # xattrs, ACLs
Failure Modes
| Symptom | Cause | Diagnose | Fix |
|---|---|---|---|
No space left on device with free space in df -h | Inodes exhausted | df -i at 100 % | Delete small files; reformat with more inodes or use XFS |
df full, du much smaller | Deleted file still open | lsof +L1 | Restart or signal the holder; truncate via /proc/<pid>/fd/N |
df full, du much smaller (other cause) | Files hidden under a mount point (written before the mount) | Bind-mount / elsewhere and du it | Unmount, clean, remount |
Too many open files (EMFILE) | fd leak or low soft limit | ls /proc/<pid>/fd | wc -l vs /proc/<pid>/limits | Fix leak; LimitNOFILE= |
| Files zero-length or old after crash | No fsync before rename | Code review | Temp + fsync + rename + fsync(dir) |
| Filesystem remounted read-only | ext4 hit an error (errors=remount-ro), often a failing disk | dmesg | grep -i ext4 | Check disk (smartctl, nvme smart-log), e2fsck offline |
EXDEV on rename() | Source and target on different filesystems (incl. overlay lower dirs) | findmnt -T both paths | Copy + fsync + unlink; keep temp files on the same fs as the target |
ENOSPC on btrfs/ZFS with space shown | CoW metadata needs free chunks; snapshots pin space | btrfs fi usage, zfs list -t snapshot | Delete snapshots, btrfs balance, keep 15 %+ free |
Directory operations slow, ls hangs | Millions of entries in one directory | ls -f | wc -l | Shard by hash prefix |
Huge SReclaimable, apps short of RAM | Dentry/inode cache bloat (incl. negative dentries) | slabtop, dentry-state | Usually self-resolves under pressure; fix the probing workload |
| Container image layers grow unexpectedly | overlay copy-up on metadata changes | docker diff, layer sizes | Set permissions/ownership in the layer that creates files |
Real Numbers
| Measurement | Typical value |
|---|---|
| Block size (ext4, XFS default) | 4 KiB |
| ext4 inode size / default inode ratio | 256 B / one per 16 KiB |
| Max extent length (ext4) | 32,768 blocks = 128 MiB |
| Max symlink nesting | 40 (ELOOP) |
| Max filename / path length | 255 bytes / 4,096 bytes (PATH_MAX) |
| Max hard links per inode (ext4) | 65,000 |
| JBD2 commit interval | 5 s |
| Dirty page expiry / flusher wakeup | 30 s / 5 s |
Cached stat() / open() syscall | ~0.3–1 µs / ~1–2 µs |
| Uncached metadata lookup on NVMe | ~50–100 µs; on HDD 5–10 ms |
fsync of a small file on NVMe (with FUA/flush) | ~0.1–2 ms; consumer SSD without PLP can be several ms |
fsync on HDD | ~5–20 ms (≈ 50–200 fsyncs/s per spindle) |
fsck of a multi-TB ext3 without journal (historical) | Hours |
| Journal replay after crash | Seconds |
| Default per-process fd soft / hard limit (systemd) | 1,024 / 524,288 |
FD_SETSIZE for select() | 1,024 |
Decision Guide
- Choosing a filesystem for a server: ext4 or XFS unless you specifically need snapshots, checksums or compression. XFS for very large volumes and many parallel writers; ext4 when you may need to shrink.
- Many small files? Use XFS/btrfs (dynamic inodes) or tune ext4's inode ratio — and shard directories.
- Need durability for a file you write? Never trust
write()+close(). Use the temp +fsync+rename+fsync(dir)pattern, or let a database do it. - Need to replace a file atomically?
rename()within one filesystem is atomic; across filesystems it isn't possible. - Running in containers? Persistent or write-heavy data goes on volumes, never the overlay writable layer.
- Serving many connections? Raise
LimitNOFILEdeliberately and alert on fd count near the limit. - Need snapshots or data integrity? btrfs or ZFS — or use ext4/XFS and get snapshots from LVM, cloud volume snapshots, or the database.
Advanced Topics
- FUSE (Filesystem in Userspace): the kernel forwards VFS operations to a user process — how
sshfs,s3fs,rclone mount, gcsfuse and many container lazy-loading snapshotters work. Flexible but each operation costs user/kernel round trips. - Network filesystems (NFS, SMB, CephFS, EFS) relax POSIX: close-to-open consistency instead of instant visibility, attribute caching, and locks that depend on a server. See Distributed File Systems.
- inotify / fanotify — change notification. inotify watches are per-directory, not recursive, and limited by
fs.inotify.max_user_watches; fanotify can watch whole filesystems (used by antivirus and audit tools). - DAX (direct access) maps persistent memory straight into user space, bypassing the page cache.
- fscrypt — per-directory encryption in ext4/F2FS (how Android does file-based encryption).
- Case-insensitive directories (ext4
casefold) — for Windows/Steam compatibility; a reasond_hash/d_compareexist in the VFS. io_uringfor file I/O andcopy_file_range()— server-side copies and reflinks without moving data through user space. See I/O & Storage.
Interview Questions
"What is an inode?" A kernel data structure that stores all metadata about a file (permissions, timestamps, size, block pointers) except the filename. The filename lives in a directory entry that points to the inode. Hard links work by having multiple directory entries point to the same inode.
"What happens when you run out of inodes?" The file system can't create new files even if disk space is free. Each file (and directory) consumes one inode. Filesystems like ext4 allocate a fixed number of inodes at creation time. Common on systems with millions of small files (mail servers, temp file directories). Fix: delete files or reformat with more inodes (or use a filesystem with dynamic inodes like XFS).
"Why does fsync() exist? Why not rely on write()?"
write() only writes to the kernel's page cache. The kernel flushes dirty pages to disk asynchronously (up to 30 seconds later). A crash between write() and the flush loses data. fsync() blocks until the storage device confirms the write is on persistent storage. Databases call fsync() after writing WAL entries to guarantee durability.
"What is the page cache?"
The kernel's in-RAM cache of recently read/written file pages. read() checks the cache first; write() goes to the cache (returns immediately), with pages flushed to disk asynchronously. This is why Linux memory usage appears high — the kernel aggressively caches file data, releasing it only when applications need RAM.
"What's the difference between a hard link and a symlink?" A hard link is another name for the same inode — same data, same permissions, survives deletion of the "original", can't cross filesystems or point to directories. A symlink is a separate file containing a path; it can cross filesystems and point anywhere, but dangles if the target moves.
"df says the disk is full but du disagrees. Why?"
Most often a deleted file that a process still has open (lsof +L1), or files written into a directory before another filesystem was mounted on top of it. Less often, filesystem reserved blocks (ext4 reserves 5 % for root by default) or CoW snapshots holding space.
"What does a journal protect, and what doesn't it?"
It protects filesystem metadata consistency across crashes — no corrupt bitmaps or dangling inodes, no long fsck. It does not make your recent writes durable (that needs fsync) and doesn't make multi-block application updates atomic (that needs a database WAL or the rename pattern).
"How do container image layers work?" Each layer is a directory of files; overlayfs stacks them read-only with a writable upper layer per container. Reads go top-down; the first write to a lower file copies it up entirely; deletions are whiteouts, so deleting in a later layer doesn't shrink the image.
"Copy-on-write vs journaling?" Journaling writes changes to a log first, then in place. CoW never overwrites: it writes new blocks and atomically swaps the root pointer, which gives free snapshots and checksums but fragments randomly-overwritten files like databases.
Related
- Virtual Memory — mmap, page cache and memory pressure
- Page Cache & fsync — durability semantics in depth
- I/O & Storage — block layer, schedulers, SSDs, io_uring
- Namespaces & cgroups — mount namespaces and containers
- Write-Ahead Logging and Storage Engines
- Docker Internals — overlay2 storage driver