I/O & Storage Internals
I/O is almost always the bottleneck. CPU runs at GHz. RAM at ~100ns. NVMe SSD at ~0.1ms. SATA SSD at ~0.5ms. HDD at ~5–10ms. Network within a datacenter at ~0.5ms. Understanding how the OS mediates I/O — buffering, scheduling, notification — is the foundation for understanding why databases, web servers, and file systems are designed the way they are.
Why I/O Is Hard
Two facts drive everything on this page:
- The latency gap is enormous. A 3 GHz core executes roughly 300,000 instructions in the 100 µs an NVMe read takes, and ~30 million during a 10 ms HDD seek. Any design that makes a CPU wait for storage wastes that capacity — so the OS must let the CPU do other work while I/O is in flight.
- Devices are inherently asynchronous and parallel. A disk is not a function you call; it is a separate computer you send commands to. It accepts many commands at once (an NVMe SSD wants dozens to hundreds in flight to reach its rated IOPS), completes them out of order, and tells you via an interrupt.
The kernel's I/O stack exists to bridge a programmer's mental model ("read() returns my bytes") to that reality: buffer and cache so most I/O never reaches the device, batch and reorder what does, keep the device's queues full, and hand completions back efficiently.
The I/O Path, End to End
flowchart TB
APP["Application: read, write, io_uring_enter"] --> VFS["VFS and filesystem (ext4, XFS)"]
VFS --> PC{"Page cache hit?"}
PC -->|"hit"| RET["Copy to user buffer, return in microseconds"]
PC -->|"miss, writeback, or O_DIRECT"| BIO["Build bio: device, sector, pages"]
BIO --> BL["Block layer (blk-mq): plug, merge, cgroup throttling"]
BL --> SCHED["I/O scheduler: none, mq-deadline, bfq, kyber"]
SCHED --> HWQ["Hardware dispatch queue, tag allocated"]
HWQ --> DRV["Driver: nvme, ahci, virtio-blk, ena-backed EBS"]
DRV -->|"doorbell write, DMA from RAM"| DEV["Device executes command"]
DEV -->|"DMA data into RAM, then interrupt or polled completion"| IRQ["Completion handler"]
IRQ --> WAKE["Wake waiting task or post io_uring CQE"]
Each layer adds a little latency and a lot of leverage. On a fast NVMe device the software stack (syscall, filesystem, block layer, interrupt, wakeup) can be 5–15 µs of a ~80 µs read — significant enough that io_uring polling and SPDK exist to shave it. On an HDD the stack is noise next to a 5 ms seek.
The Block Layer
Filesystems describe I/O as a bio: "this device, starting at this sector, these memory pages, read or write". The block layer turns bios into requests sent to the driver.
blk-mq: Multi-Queue Block Layer
The original block layer had one request queue per device protected by one lock — fine for a 200 IOPS disk, a disaster for a 1,000,000 IOPS NVMe drive with 64 CPUs contending on that lock. Since Linux 3.13 (and exclusively since 5.0) the block layer is blk-mq:
flowchart TB
subgraph CPUs["Per-CPU software staging queues"]
C0["CPU 0 ctx"]
C1["CPU 1 ctx"]
C2["CPU 2 ctx"]
C3["CPU 3 ctx"]
end
C0 --> H0["Hardware queue 0"]
C1 --> H0
C2 --> H1["Hardware queue 1"]
C3 --> H1
H0 --> D["Device: NVMe submission and completion queue pairs"]
H1 --> D
- Each CPU submits into its own software context, so no global lock.
- Software contexts map onto hardware dispatch queues — one per device queue (NVMe: commonly one per CPU; SATA: one).
- Each in-flight request holds a tag; the number of tags is the effective queue depth (
/sys/block/<dev>/queue/nr_requests).
Merging and Plugging
Small adjacent requests are expensive individually. The block layer:
- Plugs: a task submitting several bios holds them briefly in a per-task plug list (
blk_start_plug), then releases them together, so they can be sorted and merged. - Merges: a 4 KiB write at sector 1000 and another at sector 1008 become a single 8 KiB request ("back merge"). Visible as
rrqm/s/wrqm/siniostat. - Splits: requests larger than
max_sectors_kbare split to what the device supports.
Useful knobs:
$ grep -H . /sys/block/nvme0n1/queue/{scheduler,nr_requests,rotational,max_sectors_kb,read_ahead_kb,nomerges,io_poll,wbt_lat_usec}
/sys/block/nvme0n1/queue/scheduler:[none] mq-deadline kyber bfq
/sys/block/nvme0n1/queue/nr_requests:1023
/sys/block/nvme0n1/queue/rotational:0
/sys/block/nvme0n1/queue/max_sectors_kb:1280
/sys/block/nvme0n1/queue/read_ahead_kb:128
/sys/block/nvme0n1/queue/nomerges:0
/sys/block/nvme0n1/queue/io_poll:0
/sys/block/nvme0n1/queue/wbt_lat_usec:2000
rotational is used by filesystems and schedulers to decide whether seeks matter — virtual disks sometimes report it wrongly (1 for an SSD-backed cloud volume), which can enable HDD-oriented heuristics. wbt_lat_usec is writeback throttling: the block layer limits background writeback depth when read latency rises, so a big cp doesn't make the machine unresponsive.
NVMe vs SATA
| SATA (AHCI) | NVMe | |
|---|---|---|
| Designed for | Spinning disks (2004) | Flash (2011) |
| Bus | SATA 6 Gb/s (~550 MB/s usable) | PCIe: Gen3 x4 ~3.5 GB/s, Gen4 x4 ~7 GB/s, Gen5 x4 ~14 GB/s |
| Queues | 1 queue, 32 commands deep (NCQ) | Up to 65,535 queues × 65,536 commands each |
| Per-command overhead | Several register reads/writes (uncached MMIO) | One doorbell write to submit, one to acknowledge completion |
| Interrupts | One, shared | MSI-X — one per queue, steered to the submitting CPU |
| Software stack latency | ~6 µs+ | ~2–3 µs |
| Typical 4 KiB random read IOPS | ~90–100k | ~500k–1.5M+ (high-end Gen4/Gen5) |
flowchart LR
subgraph Host["Host memory"]
SQ0["SQ 0 (CPU 0)"]
CQ0["CQ 0 (CPU 0)"]
SQ1["SQ 1 (CPU 1)"]
CQ1["CQ 1 (CPU 1)"]
end
subgraph Ctrl["NVMe controller"]
DB["Doorbell registers"]
FTL["Firmware and FTL"]
end
SQ0 -->|"CPU 0 writes command, rings SQ0 tail doorbell"| DB
DB --> FTL
FTL -->|"DMA completion entry"| CQ0
CQ0 -->|"MSI-X interrupt to CPU 0"| Host
SQ1 --> DB
FTL --> CQ1
The NVMe design removes the bottleneck the OS used to work around: each CPU has its own submission/completion queue pair in host memory, submits without locks, and gets its completion interrupt on the same CPU, keeping caches warm. This is why "scheduling" for NVMe is mostly about not getting in the way.
$ sudo nvme list
Node Model Namespace Usage Format FW Rev
/dev/nvme0n1 Samsung SSD 990 PRO 2TB 1 1.20 TB / 2.00 TB 512 B + 0 B 4B2QJXD7
$ sudo nvme smart-log /dev/nvme0n1 | grep -E 'percentage_used|data_units_written|media_errors|critical_warning'
critical_warning : 0
percentage_used : 3%
data_units_written : 98,765,432 # x 512,000 bytes = ~50 TB written
media_errors : 0
$ ls /sys/block/nvme0n1/mq/ | wc -l # hardware queues
16
Device Performance Numbers
| Device | 4 KiB random read latency | 4 KiB random read IOPS | Sequential throughput | Notes |
|---|---|---|---|---|
| 7,200 RPM HDD | 5–10 ms (seek + half rotation ≈ 4.2 ms) | ~80–200 | ~150–250 MB/s | IOPS limited by mechanics; fine for sequential |
| 15k RPM SAS HDD (legacy) | 3–5 ms | ~200–300 | ~200–300 MB/s | Largely replaced by SSD |
| SATA SSD | ~80–150 µs | ~90–100k | ~550 MB/s | Interface-bound |
| NVMe Gen3 SSD | ~80–100 µs | ~400–700k | ~3.5 GB/s | |
| NVMe Gen4/Gen5 SSD | ~50–90 µs | ~1–2M+ (at high queue depth) | ~7–14 GB/s | |
| Low-latency SCM (e.g. Optane, discontinued) | ~10 µs | ~500k+ at low QD | ~2.5–7 GB/s | Shows what software overhead matters |
| Local instance NVMe (AWS i4i, etc.) | ~100 µs | 100k–1M+ per device | ~1–4 GB/s | Ephemeral — lost on stop |
| Network block (EBS gp3) | ~0.5–2 ms | 3,000 baseline (provisionable) | 125 MB/s baseline | Network hop; limits per volume and per instance |
| Network block (EBS io2 Block Express) | Sub-ms, ~0.2–0.5 ms | Up to 256,000 | Up to 4,000 MB/s | Expensive |
Two relationships to keep in your head:
- Little's Law:
IOPS = queue depth / latency. A device with 100 µs latency at queue depth 1 can do at most 10,000 IOPS from a single synchronous thread. To get 500,000 IOPS you need ~50 requests in flight. This is why single-threaded blocking I/O never reaches a fast SSD's rated numbers, and why async I/O matters. Throughput = IOPS × I/O size. 3,000 IOPS × 16 KiB = 48 MB/s; to use 125 MB/s at 3,000 IOPS you need ≥ 42 KiB I/Os.
SSD Internals
An SSD looks like a disk (read/write a 512 B or 4 KiB sector anywhere) but NAND flash physically works nothing like that:
| NAND rule | Consequence |
|---|---|
| Read and program in pages (4–16 KiB) | Small writes still program a whole page |
| A page can only be programmed when erased | No in-place overwrite |
| Erase works on blocks of hundreds of pages (several MiB) | Can't erase one page |
| Each block survives a limited number of program/erase cycles (TLC ~1,000–3,000, QLC ~100–1,000, SLC ~50,000–100,000) | Writes wear the device out |
The Flash Translation Layer (FTL)
The SSD's firmware runs an FTL that maps logical block addresses (LBAs) to physical pages, exactly like a log-structured filesystem:
flowchart LR
W["Host overwrites LBA 100"] --> FTL["FTL mapping table: LBA to physical page"]
FTL --> NEW["Write new data to a fresh, already-erased page"]
FTL --> OLD["Mark old page for LBA 100 as invalid"]
OLD --> GC["Garbage collection: pick a block with many invalid pages"]
GC --> COPY["Copy its still-valid pages elsewhere"]
COPY --> ERASE["Erase the whole block, return to free pool"]
- Garbage collection (GC) runs in the background, but when the drive runs out of pre-erased blocks it must GC inline with your writes. That's the write cliff: a new SSD benchmarks at 500k write IOPS; after hours of sustained random writes it settles to a "steady state" that can be 3–10× lower, with ugly latency spikes. Always benchmark enterprise workloads at steady state.
- Wear leveling spreads writes across all blocks, including moving cold data so its blocks get used.
- The mapping table lives in DRAM on the SSD (~1 GB of DRAM per 1 TB of flash is the classic ratio). DRAM-less consumer drives cache part of it in host memory (HMB) and degrade with large random working sets.
- SLC caching in consumer TLC/QLC drives writes bursts in fast single-bit mode; once the cache fills, sustained write speed can drop from 5 GB/s to a few hundred MB/s.
- Power-loss protection (PLP) — capacitors that let the drive flush its DRAM on power loss — is what lets enterprise drives acknowledge a flush quickly. Consumer drives without PLP must actually program NAND on every
fsync, sofsync-heavy databases can be 10× slower on them despite similar spec-sheet numbers.
Write Amplification
Write amplification factor (WAF) = bytes written to NAND ÷ bytes written by the host. GC copying valid pages is the main source; a WAF of 1 is ideal, 2–5 is common for random writes on a full drive.
The stack multiplies amplifications: an application writing a 100-byte row → database page (8–16 KiB) + WAL record + full-page image → filesystem journal → SSD GC. An end-to-end amplification of 10–50× is not unusual for small random updates, which is one reason LSM-tree storage engines (RocksDB, Cassandra) — which turn random writes into large sequential ones — are popular on flash. See Storage Engines.
Reduce WAF with:
- Over-provisioning — space the FTL can use but you can't. Enterprise drives reserve 7–28 %; you can add more by leaving part of a drive unpartitioned (after a full TRIM).
- TRIM / discard — telling the drive which LBAs no longer hold data, so GC doesn't copy dead pages.
- Large, sequential, aligned writes — whole-block writes leave nothing valid to copy.
TRIM
When a filesystem deletes a file, it only updates its own metadata; the SSD still thinks those LBAs hold live data and will preserve them through GC forever. TRIM (ATA) / Deallocate (NVMe) / UNMAP (SCSI) tells it otherwise.
# Periodic TRIM (recommended): systemd timer runs fstrim weekly
systemctl status fstrim.timer
sudo fstrim -av # trim all mounted filesystems that support it
/: 112.4 GiB (120691986432 bytes) trimmed on /dev/nvme0n1p2
# Continuous TRIM: discard mount option (ext4, XFS) - issues discards on every delete
# btrfs uses discard=async by default on SSDs since 6.2, which batches them
lsblk --discard # DISC-GRAN / DISC-MAX non-zero = device supports it
Periodic fstrim is usually preferred over synchronous discard with ext4/XFS: some drives handle discards slowly, and issuing them inline with deletes can stall the delete path. On cloud volumes, TRIM also matters for thin provisioning and for snapshot sizes.
Endurance
Endurance is quoted as TBW (terabytes written) or DWPD (drive writes per day over the warranty period):
DWPD = TBW / (capacity_TB × warranty_days)
Example: 3.84 TB drive, 7,000 TBW, 5-year warranty
7000 / (3.84 × 1825) ≈ 1.0 DWPD
| Class | Typical DWPD | Use |
|---|---|---|
| Consumer QLC | ~0.1–0.3 | Read-heavy, media |
| Consumer TLC | ~0.3–0.6 | Desktops, light servers |
| Enterprise read-intensive | ~1 | Most databases and services |
| Enterprise mixed-use | ~3 | Write-heavy OLTP, logging |
| Enterprise write-intensive | 10+ | Caches, journals, Kafka-heavy |
Monitor percentage_used in nvme smart-log (or Wear_Leveling_Count/Media_Wearout_Indicator via smartctl -a) and alert well before 100 %.
I/O Modes
Blocking I/O
The calling thread blocks until the I/O operation completes. The OS puts the thread in the WAITING state and runs other threads. Simple to program but one blocked thread per connection.
# Thread blocks here until all bytes arrive or connection closes
data = socket.recv(4096)
# Resumes here only after data is available
process(data)
When to use: Simple scripts, low-concurrency services, database drivers where each connection has its own thread.
(Note recv(4096) returns as soon as some data is available — up to 4,096 bytes — not necessarily all of it. Blocking calls on sockets return partial results; loop until you have what you need.)
A thread blocked on a disk read shows state D (uninterruptible) in ps; one blocked on a socket shows S (interruptible). Lots of D-state threads mean storage is the bottleneck — and they count toward the Linux load average, which is why a box with idle CPUs can show load 40 during an I/O storm.
Non-Blocking I/O
The syscall returns immediately with EAGAIN if no data is available. The application must poll (keep calling recv until it returns data). Avoids blocking but burns CPU on polling.
int flags = fcntl(fd, F_GETFL);
fcntl(fd, F_SETFL, flags | O_NONBLOCK);
ssize_t n = read(fd, buf, sizeof(buf));
if (n < 0 && errno == EAGAIN) {
// No data ready yet, try later
}
`O_NONBLOCK` does nothing for regular files
For sockets, pipes and terminals, "ready" is well defined: is there data in the buffer? For regular files, the kernel always reports them as readable and writable, and a read() that misses the page cache simply blocks in the filesystem — O_NONBLOCK is ignored. That's why event loops like Node's libuv run file operations on a thread pool, and why true async disk I/O needed AIO and then io_uring.
Multiplexed I/O (select/poll)
Monitor many file descriptors simultaneously; wake up when any of them is ready. Avoids the one-thread-per-connection limitation.
fd_set readfds;
FD_ZERO(&readfds);
FD_SET(sock1, &readfds);
FD_SET(sock2, &readfds);
int ready = select(max_fd + 1, &readfds, NULL, NULL, &timeout);
if (FD_ISSET(sock1, &readfds)) {
read(sock1, buf, sizeof(buf));
}
select and poll problems: O(N) — they scan all watched file descriptors on every call. At 10,000+ connections, this becomes the bottleneck. select additionally can't handle fd numbers ≥ FD_SETSIZE (1024) — writing past it corrupts the stack — and the sets must be rebuilt before every call because the kernel overwrites them.
epoll (Linux)
Event-based. Register interest once; kernel maintains the watched set. Returns only the FDs that are ready — O(ready events), not O(total FDs).
int epfd = epoll_create1(0);
struct epoll_event ev = {.events = EPOLLIN, .data.fd = sock};
epoll_ctl(epfd, EPOLL_CTL_ADD, sock, &ev);
// Blocks until events arrive; returns only ready fds
int n = epoll_wait(epfd, events, MAX_EVENTS, timeout_ms);
for (int i = 0; i < n; i++) {
handle(events[i].data.fd);
}
Edge-triggered vs level-triggered:
- Level-triggered (default):
epoll_waitkeeps returning the fd while data is available. Easier, and forgiving if you don't drain the buffer in one go — you'll simply be told again. - Edge-triggered (
EPOLLET): returns only when new data arrives (state transition). More efficient — kernel wakes you up once — but you must drain the entire buffer (read untilEAGAIN) or you miss data: no new edge will come for bytes already buffered.
A minimal edge-triggered echo loop shows the full pattern:
#include <sys/epoll.h>
#include <sys/socket.h>
#include <fcntl.h>
#include <errno.h>
#include <unistd.h>
static void set_nonblock(int fd) { fcntl(fd, F_SETFL, fcntl(fd, F_GETFL) | O_NONBLOCK); }
void serve(int listen_fd) {
int ep = epoll_create1(EPOLL_CLOEXEC);
set_nonblock(listen_fd);
struct epoll_event ev = { .events = EPOLLIN, .data.fd = listen_fd };
epoll_ctl(ep, EPOLL_CTL_ADD, listen_fd, &ev);
struct epoll_event events[256];
char buf[16384];
for (;;) {
int n = epoll_wait(ep, events, 256, -1);
if (n < 0 && errno == EINTR) continue; // signals interrupt epoll_wait
for (int i = 0; i < n; i++) {
int fd = events[i].data.fd;
if (fd == listen_fd) { // accept everything pending
int c;
while ((c = accept4(listen_fd, NULL, NULL, SOCK_NONBLOCK | SOCK_CLOEXEC)) >= 0) {
struct epoll_event cev = { .events = EPOLLIN | EPOLLET | EPOLLRDHUP, .data.fd = c };
epoll_ctl(ep, EPOLL_CTL_ADD, c, &cev);
}
continue;
}
for (;;) { // edge-triggered: drain until EAGAIN
ssize_t r = read(fd, buf, sizeof buf);
if (r > 0) { write(fd, buf, r); continue; } // real code: buffer partial writes
if (r == 0 || (errno != EAGAIN && errno != EINTR)) close(fd); // EOF or error
break;
}
}
}
}
Other epoll details worth knowing: EPOLLONESHOT disarms the fd after one event (useful when handing fds to worker threads), EPOLLEXCLUSIVE avoids the thundering herd when many processes wait on one listening socket, and SO_REUSEPORT gives each worker its own accept queue instead.
Node.js, Nginx, Redis all use epoll under the hood. So does the Go runtime: goroutines look like they do blocking I/O, but the netpoller parks the goroutine and registers the socket with epoll; the OS thread moves on to other goroutines.
// Looks blocking, is epoll underneath: 10,000 goroutines, a handful of OS threads.
ln, _ := net.Listen("tcp", ":8080")
for {
conn, _ := ln.Accept()
go func(c net.Conn) {
defer c.Close()
io.Copy(c, c) // Read parks the goroutine on EAGAIN, epoll wakes it
}(conn)
}
// File reads are different: os.File.Read on a regular file really blocks an OS thread,
// so the runtime hands the P to another thread while it waits.
Linux AIO (libaio)
Before io_uring, Linux had native AIO (io_setup, io_submit, io_getevents). It is genuinely asynchronous only for O_DIRECT file I/O; for buffered I/O it silently becomes synchronous inside io_submit. It can also block in submission (e.g. when the filesystem needs to allocate blocks or read metadata). Databases that manage their own cache with O_DIRECT — MySQL InnoDB, ScyllaDB/Seastar, older Ceph — used it successfully; almost nobody else could. (POSIX AIO in glibc, aio_read(), is a user-space thread pool — not kernel AIO at all.)
io_uring (Linux 5.1+)
The modern Linux async I/O interface. Two ring buffers shared between kernel and user space:
- Submission Queue (SQ): user writes I/O requests here
- Completion Queue (CQ): kernel writes results here
No syscall is required per I/O operation — user space writes many requests into the SQ and makes one io_uring_enter() call to submit them all (or none at all with SQPOLL, where a kernel thread polls the SQ), and reads completions from the CQ directly from shared memory. The kernel processes submissions in batches. Supports true async for disk I/O (buffered and direct), sockets, pipes, and more — over 50 operation types including accept, connect, send/recv, openat, statx, fsync, splice and timeouts.
struct io_uring ring;
io_uring_queue_init(32, &ring, 0);
// Submit a read without a syscall (after setup)
struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
io_uring_prep_read(sqe, fd, buf, sizeof(buf), 0);
io_uring_submit(&ring);
// Poll for completion
struct io_uring_cqe *cqe;
io_uring_wait_cqe(&ring, &cqe);
// cqe->res contains bytes read or error code
sequenceDiagram
participant U as "User space"
participant SQ as "SQ ring (shared memory)"
participant K as Kernel
participant CQ as "CQ ring (shared memory)"
U->>SQ: "write 64 SQEs (reads at different offsets)"
U->>K: "one io_uring_enter syscall (skipped with SQPOLL)"
K->>K: "issue to block layer or network stack, async workers if needed"
K-->>CQ: "post CQEs as each I/O completes (any order)"
U->>CQ: "reap CQEs from shared memory, no syscall"
A batched reader with liburing — 64 reads in flight from one thread:
#include <liburing.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#define QD 64
#define BS 4096
int main(int argc, char **argv) {
int fd = open(argv[1], O_RDONLY | O_DIRECT); // O_DIRECT: needs aligned buffers
struct io_uring ring;
io_uring_queue_init(QD, &ring, 0);
void *bufs[QD];
for (int i = 0; i < QD; i++) posix_memalign(&bufs[i], 4096, BS);
for (int i = 0; i < QD; i++) { // queue 64 reads...
struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
io_uring_prep_read(sqe, fd, bufs[i], BS, (off_t)i * BS * 1000);
io_uring_sqe_set_data64(sqe, i); // tag to match completions
}
io_uring_submit(&ring); // ...one syscall
for (int done = 0; done < QD; done++) {
struct io_uring_cqe *cqe;
io_uring_wait_cqe(&ring, &cqe);
if (cqe->res < 0) fprintf(stderr, "read %llu failed: %d\n",
(unsigned long long)io_uring_cqe_get_data64(cqe), cqe->res);
io_uring_cqe_seen(&ring, cqe); // advance CQ head
}
io_uring_queue_exit(&ring);
}
io_uring outperforms epoll at high RPS because it eliminates syscall overhead per operation and supports disk I/O that epoll cannot. Further speedups come from registered files and buffers (skip per-I/O fd lookup and page pinning), multishot accept/recv (one SQE, many completions), provided buffer rings (kernel picks a buffer only when data arrives), and IORING_SETUP_IOPOLL (busy-poll NVMe completions instead of taking interrupts).
io_uring and security
io_uring exposes a large, fast-moving kernel attack surface. Google reported in 2023 that 60 % of kernel exploits submitted to its bug bounty targeted io_uring, and disabled it on ChromeOS, for Android apps, and on its production servers. Many container platforms block the io_uring_* syscalls in their default seccomp profiles, and Linux 6.6+ has kernel.io_uring_disabled to turn it off. Check whether it's available in your environment before designing around it. See Process Security.
Choosing an I/O Model
| Model | Disk files | Sockets | Syscalls per op | Concurrency model | Used by |
|---|---|---|---|---|---|
| Blocking | Yes (blocks) | Yes (blocks) | 1 | Thread per operation | Simple services, JDBC, classic Apache |
| Non-blocking + busy poll | No effect | Yes | Many | Spin | Rare (DPDK-style) |
select / poll | Always "ready" | Yes | 1 + O(N) scan | Event loop | Legacy, portable code |
epoll / kqueue | Always "ready" | Yes | ~1–2 | Event loop + thread pool for files | Nginx, Redis, Node, Go, Netty |
| Linux AIO | O_DIRECT only | No | 1 per batch | Completion-based | InnoDB, Seastar (historical) |
| io_uring | Yes (buffered + direct) | Yes | ~0–1 per batch | Completion-based | RocksDB MultiGet, ScyllaDB, TigerBeetle, QEMU, newer runtimes (tokio-uring, Netty incubator, liburing) |
Readiness-based models (epoll) say "you can now read without blocking"; completion-based models (AIO, io_uring, Windows IOCP) say "your read has finished — here are the bytes". Completion is the only model that works for regular files.
Direct Memory Access (DMA)
Without DMA, the CPU must copy every byte between a device and memory — burning CPU cycles on data movement instead of computation.
DMA offloads this to dedicated hardware:
Without DMA:
Device → CPU registers → Memory (CPU fully involved, ~100 cycles per word)
With DMA:
CPU tells DMA controller: "copy N bytes from disk buffer to memory address X"
DMA copies data directly, CPU does other work
DMA raises interrupt when done
CPU resumes handling the transfer
Why this matters: NVMe SSDs can sustain 7GB/s. Without DMA, the CPU would burn most of a core (or several) just moving data. With DMA, CPU utilization from I/O is near zero.
Modern devices are bus masters: the NVMe controller itself reads commands and data from host RAM and writes results back — there is no separate "DMA controller" chip. The IOMMU restricts which memory each device may touch. Interrupt handling, MSI-X and driver structure are covered in Interrupts, DMA & Drivers.
Zero-copy transfers: sendfile() syscall sends data from a file descriptor directly to a socket without copying to user space. Data path: disk → kernel page cache → NIC buffer, all via DMA. Used by Nginx for static file serving, Kafka for log segment transfers.
// Send file without copying to user space
ssize_t sent = sendfile(out_fd, in_fd, &offset, count);
Related zero-copy tools: splice()/tee() (move pages between pipes and fds), copy_file_range() (file-to-file, can become a reflink or server-side copy on NFS), MSG_ZEROCOPY (send from user pages without copying), and kTLS (lets sendfile work with TLS by encrypting in the kernel or NIC). See Linux Networking Stack.
Direct I/O
O_DIRECT asks the kernel to DMA straight between your buffer and the device, bypassing the page cache.
#define _GNU_SOURCE
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
int fd = open("/data/table.ibd", O_RDWR | O_DIRECT | O_CLOEXEC);
void *buf;
posix_memalign(&buf, 4096, 16384); // buffer address aligned to logical block size
ssize_t n = pread(fd, buf, 16384, 65536); // length and offset also multiples of block size
// Misaligned buffer, length or offset → EINVAL
| Why databases use it | The catch |
|---|---|
| No double caching (buffer pool + page cache holding the same pages) | You must implement caching, readahead and write batching yourself |
| Predictable memory: the kernel won't evict your cache under pressure | Alignment rules (statx(STATX_DIOALIGN) reports them on 6.1+) |
| Avoids writeback storms and dirty-page throttling stalls | Every read is a device read — terrible for apps that relied on the page cache |
| Works with AIO/io_uring for deep queues | Not a durability guarantee: the device may still hold the data in its volatile cache. You still need fsync/fdatasync, or O_DSYNC, or RWF_DSYNC per write (which can map to an NVMe FUA write) |
PostgreSQL historically relied on buffered I/O plus the page cache (direct I/O support is a work-in-progress debug_io_direct option); MySQL InnoDB defaults to innodb_flush_method = O_DIRECT on Linux; ScyllaDB and most specialised engines use direct I/O exclusively. Durability semantics are covered in Page Cache & fsync.
Disk Scheduling Algorithms
For HDDs, disk head movement is the bottleneck (~5ms per seek). The I/O scheduler determines the order in which pending requests are served to minimize seeks.
FCFS (First-Come, First-Served)
Serve requests in arrival order. Simple but worst-case head movement.
Requests (cylinder): 98, 183, 37, 122, 14, 124, 65, 67
Head at: 53
FCFS order: 53→98→183→37→122→14→124→65→67
Total movement: 640 cylinders
SSTF (Shortest Seek Time First)
Serve the pending request closest to current head position. Minimizes head movement but can starve requests at extremes.
Head at 53, requests: 98, 183, 37, 122, 14, 124, 65, 67
SSTF: 53→65→67→37→14→98→122→124→183
Total movement: 236 cylinders
SCAN (Elevator Algorithm)
Head moves in one direction servicing all requests, then reverses. Like an elevator.
Head at 53, moving toward low cylinders (LOOK variant - reverses at the last request):
53→37→14→(reverse)→65→67→98→122→124→183
Total movement: 39 + 169 = 208 cylinders
Same requests, moving toward high cylinders first:
53→65→67→98→122→124→183→(reverse)→37→14
Total movement: 130 + 169 = 299 cylinders
Pure SCAN travels all the way to the edge of the disk (cylinder 0 or the last cylinder) before reversing; LOOK reverses at the last pending request, which is what real implementations do. The direction you happen to be moving matters a lot for any given batch, which is why schedulers care about fairness over time rather than optimal movement for one batch.
C-SCAN (Circular SCAN): Moves in one direction only; after reaching the end, jumps back to the start without servicing. More uniform wait times.
| Algorithm | Head movement | Starvation risk | Wait-time variance |
|---|---|---|---|
| FCFS | Worst | None | Low but uniformly slow |
| SSTF | Good | High for edge requests | High |
| SCAN / LOOK | Good | None | Middle-cylinder requests favored |
| C-SCAN / C-LOOK | Slightly more | None | Most uniform |
Modern HDDs do their own reordering internally (NCQ, 32 commands), using knowledge the OS lacks — rotational position. The OS scheduler's job shifted from "minimize seek distance" to "keep latency bounded and share bandwidth fairly".
Linux I/O Schedulers
For NVMe SSDs, disk scheduling is largely irrelevant — random access latency is ~0.1ms regardless of "seek order." Linux defaults to the none scheduler for NVMe.
For HDDs, Linux offers:
- mq-deadline: simple, prevents starvation with request deadlines
- BFQ (Budget Fair Queueing): complex, provides fair I/O bandwidth across cgroups — used for desktop responsiveness
- Kyber: latency-optimized for fast SSDs
| Scheduler | How it works | Overhead | Pick it for |
|---|---|---|---|
none | FIFO pass-through; blk-mq still merges | Lowest | NVMe, fast SSDs, virtual disks where the hypervisor/backend schedules (EBS, virtio on fast storage) |
mq-deadline | Sorted by sector, with read (500 ms) and write (5 s) deadlines and read preference; batches dispatch | Low | HDDs, SATA SSDs, RAID arrays; the usual default for SATA/SCSI |
bfq | Per-process/cgroup queues with sector budgets, idling to preserve sequential streams; honours io.weight | Highest (CPU per I/O) — caps out around ~0.5–1M IOPS on fast CPUs | Desktops, slow devices, and when you need proportional I/O sharing between cgroups |
kyber | Token-based: throttles queue depth per class (reads, sync writes, other) to hit target latencies (default 2 ms read, 10 ms sync write) | Low | Fast multi-queue devices where reads must stay low-latency under mixed load |
cat /sys/block/sda/queue/scheduler # none [mq-deadline] kyber bfq
echo bfq | sudo tee /sys/block/sda/queue/scheduler # change at runtime
# Make it persistent with a udev rule
cat <<'EOF' | sudo tee /etc/udev/rules.d/60-ioschedulers.rules
ACTION=="add|change", KERNEL=="nvme[0-9]*n[0-9]*", ATTR{queue/scheduler}="none"
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="mq-deadline"
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="mq-deadline"
EOF
The schedulers of the old single-queue era — noop, deadline, cfq — were removed in Linux 5.0. Advice that mentions cfq or elevator= boot parameters is out of date.
Buffering and Buffered I/O
Kernel Buffering
The kernel's page cache acts as a write buffer. write() returns immediately after copying to page cache — the kernel flushes to disk asynchronously (dirty pages older than 30s, or when dirty pages exceed a threshold: vm.dirty_background_ratio starts background writeback, vm.dirty_ratio makes writers block and do writeback themselves).
Application write():
app buffer → page cache → (async) → disk
↑ fast (microseconds) ↑ slow (milliseconds)
That dirty_ratio cliff is a classic latency bug: on a 256 GB machine, the default dirty_ratio=20 allows ~50 GB of dirty data before writers are throttled — then a single write() can stall for seconds while the kernel flushes. Large-memory database and logging hosts often set absolute limits (vm.dirty_background_bytes, vm.dirty_bytes) instead. Details in Page Cache & fsync.
Write-Back vs Write-Through
- Write-back (default): write to cache, flush to disk later. Fast writes, risk of data loss on crash.
- Write-through: write to cache AND disk synchronously. Slow but safe.
Databases use write-back (page cache or their own buffer pool) combined with a WAL to guarantee durability without synchronous disk writes on every operation.
The same distinction exists inside the device: SSDs and RAID controllers have volatile write caches. The kernel sends a cache flush (or uses FUA — Force Unit Access — writes) on fsync so the data reaches non-volatile media. Disabling those flushes (nobarrier, a RAID controller in write-back mode without a battery) makes benchmarks look great and loses data on power failure.
Buffered vs Unbuffered (stdio vs syscalls)
fwrite() (libc) maintains a user-space buffer. Accumulates small writes and flushes as larger chunks. write() (syscall) goes directly to the kernel. For many small writes, fwrite() dramatically outperforms write() by amortizing syscall overhead.
// Slow: 1000 syscalls
for (int i = 0; i < 1000; i++) write(fd, &byte, 1);
// Fast: 1 syscall (libc buffers 1000 bytes, then flushes)
for (int i = 0; i < 1000; i++) fwrite(&byte, 1, 1, fp);
fflush(fp); // Ensure flush
The same idea everywhere: bufio.Writer in Go, BufferedOutputStream in Java, Python's default buffered file objects. The trap is the flip side — data in a user-space buffer is lost if the process is killed (SIGKILL, OOM) before it's flushed, even though the kernel would have kept anything already passed to write(). See Signals.
I/O Latency Numbers
| Operation | Latency |
|---|---|
| L1 cache | 1 ns |
| L2 cache | 4 ns |
| RAM access | 100 ns |
Syscall round trip (getpid-class) | ~50–100 ns (more with mitigations) |
Page cache hit read() of 4 KiB | ~1 µs |
| NVMe SSD random read | 0.1 ms |
| SATA SSD random read | 0.5 ms (device ~0.1 ms + stack and interface; varies widely) |
NVMe flush / fsync (drive with PLP) | ~20–100 µs |
NVMe flush / fsync (consumer, no PLP) | ~1–10 ms |
| EBS gp3 4 KiB read | ~0.5–1 ms |
| HDD random read | 5–10 ms |
| Network round trip (same DC) | 0.5–1 ms |
| Network round trip (cross-region) | 50–150 ms |
| Read 1MB from NVMe | 0.2–0.5 ms |
| Read 1MB from HDD | 10–30 ms (seek + transfer) |
These numbers are why: caching exists, connection pooling exists, batching exists, and async I/O matters at scale.
I/O Observability
iostat -x
$ iostat -xz 1
Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s r_await w_await aqu-sz rareq-sz wareq-sz %util
nvme0n1 8421.0 2210.0 33684.0 70720.0 0.0 312.0 0.11 0.45 1.92 4.0 32.0 71.2
sda 42.0 180.0 168.0 14400.0 0.0 20.0 18.40 92.10 17.35 4.0 80.0 99.8
| Column | Meaning | Read it as |
|---|---|---|
r/s, w/s | Completed requests per second (after merging) | IOPS |
rkB/s, wkB/s | Throughput | Compare with device/volume limits |
rrqm/s, wrqm/s | Requests merged per second | High = sequential-ish workload |
r_await, w_await | Average ms from submission to completion including time queued in the OS | The user-visible latency |
aqu-sz | Average number of requests in flight (queue depth) | Little's Law: aqu-sz ≈ IOPS × await |
rareq-sz, wareq-sz | Average request size (kB) | 4 = random small I/O; 128+ = streaming |
%util | Percentage of time the device had at least one request in flight | Saturation for HDDs; misleading for SSDs |
`%util` lies on parallel devices
%util was meaningful when a disk could do one thing at a time: 100 % meant saturated. An NVMe SSD processes dozens of requests concurrently, so it can show 100 % util at 10 % of its capacity — a single thread doing back-to-back reads keeps "at least one request in flight" all the time. For SSDs and cloud volumes, judge saturation by await rising, aqu-sz growing, and IOPS/throughput approaching the device's or volume's documented limits. (svctm was removed from sysstat for the same reason — it was never measurable from the host.)
In the example above, sda is a saturated HDD: 222 IOPS, 92 ms write latency, 17 requests queued. nvme0n1 at "71 % util" is nowhere near its limit.
Who Is Doing the I/O?
sudo iotop -oPa # processes doing I/O, accumulated
pidstat -d 1 # per-process kB_rd/s, kB_wr/s, iodelay
cat /proc/<pid>/io # read_bytes / write_bytes actually hitting storage
cat /proc/pressure/io # PSI: % time tasks stalled on I/O
cat /sys/fs/cgroup/<cg>/io.stat # per-cgroup bytes and IOs per device
vmstat 1 # 'b' = tasks blocked (D state), 'wa' = iowait %
iowait (wa in top/vmstat) is the percentage of time a CPU was idle while some task had I/O outstanding. It's a hint, not a measurement of disk load: a busy CPU hides iowait, and a single slow I/O on an idle machine can show high iowait. PSI (/proc/pressure/io) is the better signal — see Namespaces & cgroups.
Latency Distributions with eBPF
Averages hide the tail. The BCC/bpftrace tools trace every block I/O:
$ sudo biolatency -D 10 1 # histogram per disk, 10 s
disk = nvme0n1
usecs : count distribution
16 -> 31 : 1204 |** |
32 -> 63 : 18832 |****************************************|
64 -> 127 : 9120 |******************* |
128 -> 255 : 822 |* |
2048 -> 4095 : 41 | | <- tail: GC or flush stalls
16384 -> 32767 : 7 | |
$ sudo biosnoop | head # every I/O: process, disk, sector, bytes, latency
TIME(s) COMM PID DISK T SECTOR BYTES LAT(ms)
0.000 postgres 2210 nvme0n1 W 81203200 8192 0.41
0.001 kworker/u8 311 nvme0n1 W 99102208 524288 2.95
# bpftrace one-liner: I/O size histogram by process
sudo bpftrace -e 'tracepoint:block:block_rq_issue { @[comm] = hist(args->bytes); }'
blktrace/blkparse give per-request timelines through every block-layer stage (queued → merged → dispatched → completed) when you need to know where in the stack latency is added. See eBPF for the tooling.
Benchmarking with fio
# 4 KiB random reads, 32 deep, 4 jobs, direct I/O - the "IOPS" number
fio --name=randread --filename=/dev/nvme1n1 --rw=randread --bs=4k --iodepth=32 \
--numjobs=4 --direct=1 --ioengine=io_uring --runtime=60 --time_based --group_reporting
# fsync latency - what a database commit feels like
fio --name=fsync --filename=/data/fio.test --size=256M --rw=write --bs=4k \
--fdatasync=1 --ioengine=sync --runtime=30 --time_based
# Sequential throughput
fio --name=seq --filename=/data/fio.test --size=10G --rw=read --bs=1M --iodepth=8 --direct=1 --ioengine=libaio
Report percentiles (clat percentiles p99, p99.9), not just averages; pre-condition SSDs (write the whole device twice) before steady-state write tests; never run write tests against a raw device holding data.
Cloud Block Storage
A cloud volume (AWS EBS, GCP Persistent Disk/Hyperdisk, Azure Managed Disks) is not a disk — it's a distributed, replicated storage service reached over the network, presented to the VM as an NVMe or virtio device. That changes the performance model:
| Physical NVMe | Cloud volume |
|---|---|
| Latency ~0.1 ms | Latency ~0.5–2 ms (network + replication) |
| IOPS limited by hardware | IOPS and throughput are provisioned limits enforced by the provider |
| Limits per device | Limits per volume and per instance — the lower one wins |
| Survives nothing if the host dies | Replicated within an AZ; snapshots to object storage |
AWS EBS Volume Types
| Type | Media | Baseline | Max per volume | Pricing model | Use |
|---|---|---|---|---|---|
| gp3 | SSD | 3,000 IOPS + 125 MiB/s regardless of size | 16,000 IOPS / 1,000 MiB/s historically; raised in 2025 for large volumes (up to 80,000 IOPS / 2,000 MiB/s) — check current docs | Pay for GB + extra IOPS + extra throughput separately | Default for almost everything |
| gp2 (legacy) | SSD | 3 IOPS per GiB (min 100) | 16,000 IOPS / 250 MiB/s | Pay per GB; burst credits below 1 TiB | Migrate to gp3 |
| io2 Block Express | SSD | Provisioned, up to 1,000 IOPS per GiB | 256,000 IOPS / 4,000 MiB/s, sub-ms latency, 99.999 % durability | Pay per provisioned IOPS | Large OLTP databases |
| st1 | HDD | 40 MiB/s per TiB baseline, burst to 250 MiB/s per TiB | 500 MiB/s | Throughput credits | Big sequential: logs, Kafka, ETL |
| sc1 | HDD | 12 MiB/s per TiB | 250 MiB/s | Throughput credits | Cold data |
EBS counts SSD I/Os of up to 256 KiB as one operation (merging smaller sequential ones), HDD I/Os up to 1 MiB.
Burst Credits
gp2 (and st1/sc1, and smaller EC2 instances' EBS bandwidth) use a token bucket:
gp2 volume of 200 GiB:
baseline = 3 IOPS/GiB × 200 = 600 IOPS
burst = 3,000 IOPS
bucket = 5.4 million I/O credits (full at creation)
drain rate = 3,000 − 600 = 2,400 credits/s while bursting
burst time = 5,400,000 / 2,400 ≈ 37.5 minutes, then you drop to 600 IOPS
flowchart LR
A["Volume busy above baseline"] --> B["Burst credits drain"]
B --> C{"Credits left?"}
C -->|"yes"| D["Up to 3,000 IOPS"]
C -->|"no"| E["Throttled to baseline: latency and queue depth explode"]
F["Volume below baseline"] --> G["Credits refill at baseline rate"]
Monitor BurstBalance in CloudWatch — when it hits zero, so does your performance. gp3 has no burst model (baseline is always available), which is the main reason to migrate.
Instance-Level Limits
Every EC2 instance type also has an EBS bandwidth and IOPS ceiling across all attached volumes. Many smaller instances (e.g. most .large and below in current families) have a baseline EBS bandwidth and can burst to their maximum for only ~30 minutes per 24 hours. Attaching a 16,000-IOPS gp3 volume to a small instance doesn't give you 16,000 IOPS. Check EBSIOBalance% and EBSByteBalance% CloudWatch metrics, and the instance's documented "EBS-optimized" baseline vs maximum.
Things That Surprise People
- First-read penalty from snapshots. A volume restored from a snapshot is lazily loaded from S3: the first read of each block can take tens of milliseconds. A freshly restored database replica can be 10× slower for hours. Fix: Fast Snapshot Restore (paid), or pre-warm with
fio --rw=read --bs=1Mover the whole device. - Queue depth is required to reach provisioned IOPS. At ~0.5 ms latency, one synchronous thread gets ~2,000 IOPS at best (Little's Law). To use 16,000 IOPS you need ~8+ requests in flight.
- Local instance store NVMe is far faster (100k+ IOPS, ~100 µs) but ephemeral — data is lost on stop/terminate or host failure. Fine for caches, scratch, and replicated systems (Cassandra, Kafka with replication, Elasticsearch replicas).
- NVMe timeouts. EBS volumes appear as NVMe devices; the Linux default
nvme_core.io_timeoutshould be set high (AWS recommends the maximum) so a transient EBS hiccup doesn't make the kernel declare the device dead and remount filesystems read-only.
# Is this an EBS volume and what's its ID?
sudo nvme id-ctrl -v /dev/nvme1n1 | grep -E '^mn|^sn'
mn : Amazon Elastic Block Store
sn : vol0a1b2c3d4e5f67890
cat /sys/module/nvme_core/parameters/io_timeout
War Story (illustrative)
A team's PostgreSQL primary on a 500 GiB gp2 volume ran fine for months, then started timing out every night at 02:00. The nightly batch job plus autovacuum pushed I/O above the 1,500 IOPS baseline; burst credits lasted about an hour, then latency jumped from 1 ms to 40 ms and connections piled up. BurstBalance in CloudWatch showed a sawtooth hitting 0 % at 02:50 each night. Migrating the volume to gp3 (online, via ModifyVolume) with 6,000 provisioned IOPS cost less per month than the gp2 volume and removed the cliff.
Failure Modes
| Symptom | Likely cause | Confirm | Fix |
|---|---|---|---|
| Load average high, CPUs idle | Tasks in D state waiting on storage | vmstat b column; ps -eo stat,wchan:30,cmd | grep ^D | Find the slow device; reduce I/O; faster storage |
| Periodic multi-second write stalls | Dirty page throttling (dirty_ratio) or SSD GC | grep -E 'Dirty|Writeback' /proc/meminfo; biolatency tail | Lower dirty_bytes/dirty_background_bytes; over-provision SSD |
| Cloud DB fast, then suddenly slow for hours | Burst credits exhausted (gp2, st1, small instance EBS bandwidth) | BurstBalance, EBSIOBalance% | gp3 / provisioned IOPS / bigger instance |
%util 100 % but low IOPS on NVMe | Misreading %util | Compare IOPS with device spec; check await | Ignore %util for SSDs |
| Database commit latency 5–10 ms on "fast" SSD | Consumer SSD without PLP; each fsync programs NAND | fio --fdatasync=1 | Enterprise SSD with PLP; group commit |
| SSD write throughput collapses after minutes of writes | SLC cache exhausted / steady-state GC | Long fio run | Size for steady state; more over-provisioning; TRIM |
| Filesystem goes read-only on a VM | Device I/O timeout or media error | dmesg | grep -iE 'nvme|I/O error|remount' | Raise nvme_core.io_timeout; check device health |
| Event loop stalls despite epoll | Blocking file I/O or fsync on the loop thread | strace -T -p, profiler shows read/fsync on loop | Move to thread pool or io_uring |
io_uring unavailable / EPERM in containers | Disabled by seccomp or io_uring_disabled | sysctl kernel.io_uring_disabled | Fall back to epoll + thread pool |
| Restored replica slow for hours | Lazy snapshot loading | Read latency high only on first touch | Fast Snapshot Restore or pre-warm |
Decision Guide
- Choosing storage: NVMe local for latency and IOPS (if you can tolerate ephemerality or replicate above it); cloud SSD volumes (gp3) for general durable storage; provisioned-IOPS volumes only when measured need justifies the price; HDD or object storage for cold, sequential data.
- Choosing a scheduler:
nonefor NVMe and most cloud volumes;mq-deadlinefor SATA SSD/HDD;bfqwhen you need per-cgroup fairness on slower devices;kyberif mixed workloads hurt read latency on fast devices. - Choosing an I/O model: blocking with threads for modest concurrency (simplest, often fast enough); epoll (or your runtime's netpoller) for many sockets; io_uring for very high IOPS file workloads or syscall-bound servers if your platform allows it; a thread pool for file I/O inside an event loop otherwise.
- Buffered vs direct: buffered unless you are building a database-like system with its own cache.
- Capacity planning: compute required IOPS and throughput, then required queue depth (
IOPS × latency), then check both volume and instance limits. - Monitoring: alert on
awaitand PSI I/O, queue depth, burst balances, SSD wear (percentage_used) and device errors — not on%util.
Advanced Topics
- Polled I/O (
io_poll,IORING_SETUP_IOPOLL,nvme poll_queues=) trades a busy CPU for ~2–5 µs lower latency per I/O by spinning on the completion queue instead of taking an interrupt. - SPDK moves the NVMe driver entirely into user space (polling, no syscalls, no interrupts) for multi-million IOPS per core — at the cost of dedicating cores and devices.
- Zoned storage (ZNS NVMe, SMR HDDs) exposes erase blocks/zones directly to the host, which must write sequentially per zone; this eliminates device GC and over-provisioning when the filesystem (f2fs, btrfs zoned mode) or database cooperates.
- NVMe over Fabrics (NVMe/TCP, NVMe/RDMA) — the NVMe queue model across a network; the basis of many disaggregated storage systems.
- Multipath I/O (
dm-multipath, native NVMe multipath) for redundant paths to SAN storage. - Writeback throttling and
io.latency/io.costcgroup controllers for protecting latency-sensitive workloads — see Namespaces & cgroups. - Device mapper stacks (LVM, dm-crypt, dm-raid, dm-cache/bcache) — each adds a layer and sometimes changes flush/discard behavior; verify
fsyncand TRIM pass through.
Interview Questions
"What is epoll and why is it better than select?"
epoll maintains a kernel-side set of watched file descriptors. epoll_wait() returns only the descriptors that are ready — O(ready events). select scans all watched FDs on every call — O(total FDs). At 10K connections, select scans 10K FDs per wakeup; epoll returns only the 1–5 that have data.
"What is a zero-copy transfer?"
Using sendfile() or DMA to move data from a file to a network socket without copying it to user space. Data path: disk → page cache → NIC via DMA. The CPU sets up the transfer and returns; DMA hardware does the actual data movement.
"Why is HDD disk scheduling important but SSD disk scheduling isn't?" HDDs have a physical read head that must physically move to the cylinder — seek time is 5–10ms and depends on distance. SSDs have no moving parts; all blocks are equally fast to access (~0.1ms). Reordering requests saves 0ms on SSD; on HDD it can reduce total seek distance by 5×.
"What is io_uring and when would you use it?" io_uring uses two shared ring buffers between user and kernel space to submit and complete I/O operations without a syscall per operation. Ideal for high-RPS servers that are syscall-bound, or for disk I/O that epoll can't handle. Used by modern database engines (RocksDB, Scylla) and network runtimes (tokio-uring, liburing-based servers).
"Why can't epoll make disk reads asynchronous?"
Regular files are always reported ready; a read that misses the page cache blocks inside the filesystem regardless of O_NONBLOCK. Readiness doesn't apply to files — you need a completion-based interface (io_uring, AIO with O_DIRECT) or a thread pool.
"What is write amplification on an SSD?" NAND can't be overwritten in place and erases in large blocks, so the FTL writes new data elsewhere and garbage-collects old blocks by copying their still-valid pages. Bytes written to flash ÷ bytes written by the host is the WAF. TRIM, over-provisioning and large sequential writes reduce it; high WAF reduces both performance and lifetime.
"Your NVMe disk shows 100% util in iostat. Is it saturated?"
Not necessarily. %util is the fraction of time at least one request was outstanding; NVMe handles many in parallel. Check await, aqu-sz, and IOPS/throughput against the device's limits.
"Why does your database get slow every night on its cloud volume?"
Likely burst credits (gp2/st1 or instance EBS burst) running out under batch load, dropping to baseline IOPS. Check BurstBalance/EBSIOBalance%; move to gp3 or provisioned IOPS, or a larger instance.
"How many IOPS can one thread doing synchronous 4 KiB reads get from an EBS volume?" Little's Law: queue depth 1 ÷ ~0.5–1 ms latency ≈ 1,000–2,000 IOPS, no matter what is provisioned. Use concurrency or async I/O to reach the limit.
Related
- File Systems — VFS, journaling, overlayfs
- Page Cache & fsync — writeback, durability,
O_DIRECTsemantics - Interrupts, DMA & Drivers — how devices signal completion
- Linux Networking Stack — sockets, zero-copy, NIC queues
- System Calls & the Kernel — syscall cost and the user/kernel boundary
- Storage Engines and Write-Ahead Logging
- Kafka Internals — sequential I/O and
sendfilein practice