On this page

I/O & Storage Internals

I/O is almost always the bottleneck. CPU runs at GHz. RAM at ~100ns. NVMe SSD at ~0.1ms. SATA SSD at ~0.5ms. HDD at ~5–10ms. Network within a datacenter at ~0.5ms. Understanding how the OS mediates I/O — buffering, scheduling, notification — is the foundation for understanding why databases, web servers, and file systems are designed the way they are.


Why I/O Is Hard

Two facts drive everything on this page:

  1. The latency gap is enormous. A 3 GHz core executes roughly 300,000 instructions in the 100 µs an NVMe read takes, and ~30 million during a 10 ms HDD seek. Any design that makes a CPU wait for storage wastes that capacity — so the OS must let the CPU do other work while I/O is in flight.
  2. Devices are inherently asynchronous and parallel. A disk is not a function you call; it is a separate computer you send commands to. It accepts many commands at once (an NVMe SSD wants dozens to hundreds in flight to reach its rated IOPS), completes them out of order, and tells you via an interrupt.

The kernel's I/O stack exists to bridge a programmer's mental model ("read() returns my bytes") to that reality: buffer and cache so most I/O never reaches the device, batch and reorder what does, keep the device's queues full, and hand completions back efficiently.


The I/O Path, End to End

flowchart TB
    APP["Application: read, write, io_uring_enter"] --> VFS["VFS and filesystem (ext4, XFS)"]
    VFS --> PC{"Page cache hit?"}
    PC -->|"hit"| RET["Copy to user buffer, return in microseconds"]
    PC -->|"miss, writeback, or O_DIRECT"| BIO["Build bio: device, sector, pages"]
    BIO --> BL["Block layer (blk-mq): plug, merge, cgroup throttling"]
    BL --> SCHED["I/O scheduler: none, mq-deadline, bfq, kyber"]
    SCHED --> HWQ["Hardware dispatch queue, tag allocated"]
    HWQ --> DRV["Driver: nvme, ahci, virtio-blk, ena-backed EBS"]
    DRV -->|"doorbell write, DMA from RAM"| DEV["Device executes command"]
    DEV -->|"DMA data into RAM, then interrupt or polled completion"| IRQ["Completion handler"]
    IRQ --> WAKE["Wake waiting task or post io_uring CQE"]

Each layer adds a little latency and a lot of leverage. On a fast NVMe device the software stack (syscall, filesystem, block layer, interrupt, wakeup) can be 5–15 µs of a ~80 µs read — significant enough that io_uring polling and SPDK exist to shave it. On an HDD the stack is noise next to a 5 ms seek.


The Block Layer

Filesystems describe I/O as a bio: "this device, starting at this sector, these memory pages, read or write". The block layer turns bios into requests sent to the driver.

blk-mq: Multi-Queue Block Layer

The original block layer had one request queue per device protected by one lock — fine for a 200 IOPS disk, a disaster for a 1,000,000 IOPS NVMe drive with 64 CPUs contending on that lock. Since Linux 3.13 (and exclusively since 5.0) the block layer is blk-mq:

flowchart TB
    subgraph CPUs["Per-CPU software staging queues"]
        C0["CPU 0 ctx"]
        C1["CPU 1 ctx"]
        C2["CPU 2 ctx"]
        C3["CPU 3 ctx"]
    end
    C0 --> H0["Hardware queue 0"]
    C1 --> H0
    C2 --> H1["Hardware queue 1"]
    C3 --> H1
    H0 --> D["Device: NVMe submission and completion queue pairs"]
    H1 --> D
  • Each CPU submits into its own software context, so no global lock.
  • Software contexts map onto hardware dispatch queues — one per device queue (NVMe: commonly one per CPU; SATA: one).
  • Each in-flight request holds a tag; the number of tags is the effective queue depth (/sys/block/<dev>/queue/nr_requests).

Merging and Plugging

Small adjacent requests are expensive individually. The block layer:

  • Plugs: a task submitting several bios holds them briefly in a per-task plug list (blk_start_plug), then releases them together, so they can be sorted and merged.
  • Merges: a 4 KiB write at sector 1000 and another at sector 1008 become a single 8 KiB request ("back merge"). Visible as rrqm/s / wrqm/s in iostat.
  • Splits: requests larger than max_sectors_kb are split to what the device supports.

Useful knobs:

$ grep -H . /sys/block/nvme0n1/queue/{scheduler,nr_requests,rotational,max_sectors_kb,read_ahead_kb,nomerges,io_poll,wbt_lat_usec}
/sys/block/nvme0n1/queue/scheduler:[none] mq-deadline kyber bfq
/sys/block/nvme0n1/queue/nr_requests:1023
/sys/block/nvme0n1/queue/rotational:0
/sys/block/nvme0n1/queue/max_sectors_kb:1280
/sys/block/nvme0n1/queue/read_ahead_kb:128
/sys/block/nvme0n1/queue/nomerges:0
/sys/block/nvme0n1/queue/io_poll:0
/sys/block/nvme0n1/queue/wbt_lat_usec:2000

rotational is used by filesystems and schedulers to decide whether seeks matter — virtual disks sometimes report it wrongly (1 for an SSD-backed cloud volume), which can enable HDD-oriented heuristics. wbt_lat_usec is writeback throttling: the block layer limits background writeback depth when read latency rises, so a big cp doesn't make the machine unresponsive.


NVMe vs SATA

SATA (AHCI)NVMe
Designed forSpinning disks (2004)Flash (2011)
BusSATA 6 Gb/s (~550 MB/s usable)PCIe: Gen3 x4 ~3.5 GB/s, Gen4 x4 ~7 GB/s, Gen5 x4 ~14 GB/s
Queues1 queue, 32 commands deep (NCQ)Up to 65,535 queues × 65,536 commands each
Per-command overheadSeveral register reads/writes (uncached MMIO)One doorbell write to submit, one to acknowledge completion
InterruptsOne, sharedMSI-X — one per queue, steered to the submitting CPU
Software stack latency~6 µs+~2–3 µs
Typical 4 KiB random read IOPS~90–100k~500k–1.5M+ (high-end Gen4/Gen5)
flowchart LR
    subgraph Host["Host memory"]
        SQ0["SQ 0 (CPU 0)"]
        CQ0["CQ 0 (CPU 0)"]
        SQ1["SQ 1 (CPU 1)"]
        CQ1["CQ 1 (CPU 1)"]
    end
    subgraph Ctrl["NVMe controller"]
        DB["Doorbell registers"]
        FTL["Firmware and FTL"]
    end
    SQ0 -->|"CPU 0 writes command, rings SQ0 tail doorbell"| DB
    DB --> FTL
    FTL -->|"DMA completion entry"| CQ0
    CQ0 -->|"MSI-X interrupt to CPU 0"| Host
    SQ1 --> DB
    FTL --> CQ1

The NVMe design removes the bottleneck the OS used to work around: each CPU has its own submission/completion queue pair in host memory, submits without locks, and gets its completion interrupt on the same CPU, keeping caches warm. This is why "scheduling" for NVMe is mostly about not getting in the way.

$ sudo nvme list
Node          Model                    Namespace Usage                 Format       FW Rev
/dev/nvme0n1  Samsung SSD 990 PRO 2TB  1         1.20 TB / 2.00 TB     512 B + 0 B  4B2QJXD7
$ sudo nvme smart-log /dev/nvme0n1 | grep -E 'percentage_used|data_units_written|media_errors|critical_warning'
critical_warning   : 0
percentage_used    : 3%
data_units_written : 98,765,432   # x 512,000 bytes = ~50 TB written
media_errors       : 0
$ ls /sys/block/nvme0n1/mq/ | wc -l   # hardware queues
16

Device Performance Numbers

Device4 KiB random read latency4 KiB random read IOPSSequential throughputNotes
7,200 RPM HDD5–10 ms (seek + half rotation ≈ 4.2 ms)~80–200~150–250 MB/sIOPS limited by mechanics; fine for sequential
15k RPM SAS HDD (legacy)3–5 ms~200–300~200–300 MB/sLargely replaced by SSD
SATA SSD~80–150 µs~90–100k~550 MB/sInterface-bound
NVMe Gen3 SSD~80–100 µs~400–700k~3.5 GB/s
NVMe Gen4/Gen5 SSD~50–90 µs~1–2M+ (at high queue depth)~7–14 GB/s
Low-latency SCM (e.g. Optane, discontinued)~10 µs~500k+ at low QD~2.5–7 GB/sShows what software overhead matters
Local instance NVMe (AWS i4i, etc.)~100 µs100k–1M+ per device~1–4 GB/sEphemeral — lost on stop
Network block (EBS gp3)~0.5–2 ms3,000 baseline (provisionable)125 MB/s baselineNetwork hop; limits per volume and per instance
Network block (EBS io2 Block Express)Sub-ms, ~0.2–0.5 msUp to 256,000Up to 4,000 MB/sExpensive

Two relationships to keep in your head:

  • Little's Law: IOPS = queue depth / latency. A device with 100 µs latency at queue depth 1 can do at most 10,000 IOPS from a single synchronous thread. To get 500,000 IOPS you need ~50 requests in flight. This is why single-threaded blocking I/O never reaches a fast SSD's rated numbers, and why async I/O matters.
  • Throughput = IOPS × I/O size. 3,000 IOPS × 16 KiB = 48 MB/s; to use 125 MB/s at 3,000 IOPS you need ≥ 42 KiB I/Os.

SSD Internals

An SSD looks like a disk (read/write a 512 B or 4 KiB sector anywhere) but NAND flash physically works nothing like that:

NAND ruleConsequence
Read and program in pages (4–16 KiB)Small writes still program a whole page
A page can only be programmed when erasedNo in-place overwrite
Erase works on blocks of hundreds of pages (several MiB)Can't erase one page
Each block survives a limited number of program/erase cycles (TLC ~1,000–3,000, QLC ~100–1,000, SLC ~50,000–100,000)Writes wear the device out

The Flash Translation Layer (FTL)

The SSD's firmware runs an FTL that maps logical block addresses (LBAs) to physical pages, exactly like a log-structured filesystem:

flowchart LR
    W["Host overwrites LBA 100"] --> FTL["FTL mapping table: LBA to physical page"]
    FTL --> NEW["Write new data to a fresh, already-erased page"]
    FTL --> OLD["Mark old page for LBA 100 as invalid"]
    OLD --> GC["Garbage collection: pick a block with many invalid pages"]
    GC --> COPY["Copy its still-valid pages elsewhere"]
    COPY --> ERASE["Erase the whole block, return to free pool"]
  • Garbage collection (GC) runs in the background, but when the drive runs out of pre-erased blocks it must GC inline with your writes. That's the write cliff: a new SSD benchmarks at 500k write IOPS; after hours of sustained random writes it settles to a "steady state" that can be 3–10× lower, with ugly latency spikes. Always benchmark enterprise workloads at steady state.
  • Wear leveling spreads writes across all blocks, including moving cold data so its blocks get used.
  • The mapping table lives in DRAM on the SSD (~1 GB of DRAM per 1 TB of flash is the classic ratio). DRAM-less consumer drives cache part of it in host memory (HMB) and degrade with large random working sets.
  • SLC caching in consumer TLC/QLC drives writes bursts in fast single-bit mode; once the cache fills, sustained write speed can drop from 5 GB/s to a few hundred MB/s.
  • Power-loss protection (PLP) — capacitors that let the drive flush its DRAM on power loss — is what lets enterprise drives acknowledge a flush quickly. Consumer drives without PLP must actually program NAND on every fsync, so fsync-heavy databases can be 10× slower on them despite similar spec-sheet numbers.

Write Amplification

Write amplification factor (WAF) = bytes written to NAND ÷ bytes written by the host. GC copying valid pages is the main source; a WAF of 1 is ideal, 2–5 is common for random writes on a full drive.

The stack multiplies amplifications: an application writing a 100-byte row → database page (8–16 KiB) + WAL record + full-page image → filesystem journal → SSD GC. An end-to-end amplification of 10–50× is not unusual for small random updates, which is one reason LSM-tree storage engines (RocksDB, Cassandra) — which turn random writes into large sequential ones — are popular on flash. See Storage Engines.

Reduce WAF with:

  • Over-provisioning — space the FTL can use but you can't. Enterprise drives reserve 7–28 %; you can add more by leaving part of a drive unpartitioned (after a full TRIM).
  • TRIM / discard — telling the drive which LBAs no longer hold data, so GC doesn't copy dead pages.
  • Large, sequential, aligned writes — whole-block writes leave nothing valid to copy.

TRIM

When a filesystem deletes a file, it only updates its own metadata; the SSD still thinks those LBAs hold live data and will preserve them through GC forever. TRIM (ATA) / Deallocate (NVMe) / UNMAP (SCSI) tells it otherwise.

# Periodic TRIM (recommended): systemd timer runs fstrim weekly
systemctl status fstrim.timer
sudo fstrim -av                # trim all mounted filesystems that support it
/: 112.4 GiB (120691986432 bytes) trimmed on /dev/nvme0n1p2

# Continuous TRIM: discard mount option (ext4, XFS) - issues discards on every delete
# btrfs uses discard=async by default on SSDs since 6.2, which batches them
lsblk --discard                # DISC-GRAN / DISC-MAX non-zero = device supports it

Periodic fstrim is usually preferred over synchronous discard with ext4/XFS: some drives handle discards slowly, and issuing them inline with deletes can stall the delete path. On cloud volumes, TRIM also matters for thin provisioning and for snapshot sizes.

Endurance

Endurance is quoted as TBW (terabytes written) or DWPD (drive writes per day over the warranty period):

DWPD = TBW / (capacity_TB × warranty_days)
Example: 3.84 TB drive, 7,000 TBW, 5-year warranty
         7000 / (3.84 × 1825) ≈ 1.0 DWPD
ClassTypical DWPDUse
Consumer QLC~0.1–0.3Read-heavy, media
Consumer TLC~0.3–0.6Desktops, light servers
Enterprise read-intensive~1Most databases and services
Enterprise mixed-use~3Write-heavy OLTP, logging
Enterprise write-intensive10+Caches, journals, Kafka-heavy

Monitor percentage_used in nvme smart-log (or Wear_Leveling_Count/Media_Wearout_Indicator via smartctl -a) and alert well before 100 %.


I/O Modes

Blocking I/O

The calling thread blocks until the I/O operation completes. The OS puts the thread in the WAITING state and runs other threads. Simple to program but one blocked thread per connection.

# Thread blocks here until all bytes arrive or connection closes
data = socket.recv(4096)
# Resumes here only after data is available
process(data)

When to use: Simple scripts, low-concurrency services, database drivers where each connection has its own thread.

(Note recv(4096) returns as soon as some data is available — up to 4,096 bytes — not necessarily all of it. Blocking calls on sockets return partial results; loop until you have what you need.)

A thread blocked on a disk read shows state D (uninterruptible) in ps; one blocked on a socket shows S (interruptible). Lots of D-state threads mean storage is the bottleneck — and they count toward the Linux load average, which is why a box with idle CPUs can show load 40 during an I/O storm.

Non-Blocking I/O

The syscall returns immediately with EAGAIN if no data is available. The application must poll (keep calling recv until it returns data). Avoids blocking but burns CPU on polling.

int flags = fcntl(fd, F_GETFL);
fcntl(fd, F_SETFL, flags | O_NONBLOCK);

ssize_t n = read(fd, buf, sizeof(buf));
if (n < 0 && errno == EAGAIN) {
    // No data ready yet, try later
}

`O_NONBLOCK` does nothing for regular files

For sockets, pipes and terminals, "ready" is well defined: is there data in the buffer? For regular files, the kernel always reports them as readable and writable, and a read() that misses the page cache simply blocks in the filesystem — O_NONBLOCK is ignored. That's why event loops like Node's libuv run file operations on a thread pool, and why true async disk I/O needed AIO and then io_uring.

Multiplexed I/O (select/poll)

Monitor many file descriptors simultaneously; wake up when any of them is ready. Avoids the one-thread-per-connection limitation.

fd_set readfds;
FD_ZERO(&readfds);
FD_SET(sock1, &readfds);
FD_SET(sock2, &readfds);

int ready = select(max_fd + 1, &readfds, NULL, NULL, &timeout);
if (FD_ISSET(sock1, &readfds)) {
    read(sock1, buf, sizeof(buf));
}

select and poll problems: O(N) — they scan all watched file descriptors on every call. At 10,000+ connections, this becomes the bottleneck. select additionally can't handle fd numbers ≥ FD_SETSIZE (1024) — writing past it corrupts the stack — and the sets must be rebuilt before every call because the kernel overwrites them.

epoll (Linux)

Event-based. Register interest once; kernel maintains the watched set. Returns only the FDs that are ready — O(ready events), not O(total FDs).

int epfd = epoll_create1(0);

struct epoll_event ev = {.events = EPOLLIN, .data.fd = sock};
epoll_ctl(epfd, EPOLL_CTL_ADD, sock, &ev);

// Blocks until events arrive; returns only ready fds
int n = epoll_wait(epfd, events, MAX_EVENTS, timeout_ms);
for (int i = 0; i < n; i++) {
    handle(events[i].data.fd);
}

Edge-triggered vs level-triggered:

  • Level-triggered (default): epoll_wait keeps returning the fd while data is available. Easier, and forgiving if you don't drain the buffer in one go — you'll simply be told again.
  • Edge-triggered (EPOLLET): returns only when new data arrives (state transition). More efficient — kernel wakes you up once — but you must drain the entire buffer (read until EAGAIN) or you miss data: no new edge will come for bytes already buffered.

A minimal edge-triggered echo loop shows the full pattern:

#include <sys/epoll.h>
#include <sys/socket.h>
#include <fcntl.h>
#include <errno.h>
#include <unistd.h>

static void set_nonblock(int fd) { fcntl(fd, F_SETFL, fcntl(fd, F_GETFL) | O_NONBLOCK); }

void serve(int listen_fd) {
    int ep = epoll_create1(EPOLL_CLOEXEC);
    set_nonblock(listen_fd);
    struct epoll_event ev = { .events = EPOLLIN, .data.fd = listen_fd };
    epoll_ctl(ep, EPOLL_CTL_ADD, listen_fd, &ev);

    struct epoll_event events[256];
    char buf[16384];
    for (;;) {
        int n = epoll_wait(ep, events, 256, -1);
        if (n < 0 && errno == EINTR) continue;           // signals interrupt epoll_wait
        for (int i = 0; i < n; i++) {
            int fd = events[i].data.fd;
            if (fd == listen_fd) {                        // accept everything pending
                int c;
                while ((c = accept4(listen_fd, NULL, NULL, SOCK_NONBLOCK | SOCK_CLOEXEC)) >= 0) {
                    struct epoll_event cev = { .events = EPOLLIN | EPOLLET | EPOLLRDHUP, .data.fd = c };
                    epoll_ctl(ep, EPOLL_CTL_ADD, c, &cev);
                }
                continue;
            }
            for (;;) {                                    // edge-triggered: drain until EAGAIN
                ssize_t r = read(fd, buf, sizeof buf);
                if (r > 0) { write(fd, buf, r); continue; }  // real code: buffer partial writes
                if (r == 0 || (errno != EAGAIN && errno != EINTR)) close(fd);  // EOF or error
                break;
            }
        }
    }
}

Other epoll details worth knowing: EPOLLONESHOT disarms the fd after one event (useful when handing fds to worker threads), EPOLLEXCLUSIVE avoids the thundering herd when many processes wait on one listening socket, and SO_REUSEPORT gives each worker its own accept queue instead.

Node.js, Nginx, Redis all use epoll under the hood. So does the Go runtime: goroutines look like they do blocking I/O, but the netpoller parks the goroutine and registers the socket with epoll; the OS thread moves on to other goroutines.

// Looks blocking, is epoll underneath: 10,000 goroutines, a handful of OS threads.
ln, _ := net.Listen("tcp", ":8080")
for {
    conn, _ := ln.Accept()
    go func(c net.Conn) {
        defer c.Close()
        io.Copy(c, c)       // Read parks the goroutine on EAGAIN, epoll wakes it
    }(conn)
}
// File reads are different: os.File.Read on a regular file really blocks an OS thread,
// so the runtime hands the P to another thread while it waits.

Linux AIO (libaio)

Before io_uring, Linux had native AIO (io_setup, io_submit, io_getevents). It is genuinely asynchronous only for O_DIRECT file I/O; for buffered I/O it silently becomes synchronous inside io_submit. It can also block in submission (e.g. when the filesystem needs to allocate blocks or read metadata). Databases that manage their own cache with O_DIRECT — MySQL InnoDB, ScyllaDB/Seastar, older Ceph — used it successfully; almost nobody else could. (POSIX AIO in glibc, aio_read(), is a user-space thread pool — not kernel AIO at all.)

io_uring (Linux 5.1+)

The modern Linux async I/O interface. Two ring buffers shared between kernel and user space:

  • Submission Queue (SQ): user writes I/O requests here
  • Completion Queue (CQ): kernel writes results here

No syscall is required per I/O operation — user space writes many requests into the SQ and makes one io_uring_enter() call to submit them all (or none at all with SQPOLL, where a kernel thread polls the SQ), and reads completions from the CQ directly from shared memory. The kernel processes submissions in batches. Supports true async for disk I/O (buffered and direct), sockets, pipes, and more — over 50 operation types including accept, connect, send/recv, openat, statx, fsync, splice and timeouts.

struct io_uring ring;
io_uring_queue_init(32, &ring, 0);

// Submit a read without a syscall (after setup)
struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
io_uring_prep_read(sqe, fd, buf, sizeof(buf), 0);
io_uring_submit(&ring);

// Poll for completion
struct io_uring_cqe *cqe;
io_uring_wait_cqe(&ring, &cqe);
// cqe->res contains bytes read or error code
sequenceDiagram
    participant U as "User space"
    participant SQ as "SQ ring (shared memory)"
    participant K as Kernel
    participant CQ as "CQ ring (shared memory)"
    U->>SQ: "write 64 SQEs (reads at different offsets)"
    U->>K: "one io_uring_enter syscall (skipped with SQPOLL)"
    K->>K: "issue to block layer or network stack, async workers if needed"
    K-->>CQ: "post CQEs as each I/O completes (any order)"
    U->>CQ: "reap CQEs from shared memory, no syscall"

A batched reader with liburing — 64 reads in flight from one thread:

#include <liburing.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>

#define QD 64
#define BS 4096

int main(int argc, char **argv) {
    int fd = open(argv[1], O_RDONLY | O_DIRECT);          // O_DIRECT: needs aligned buffers
    struct io_uring ring;
    io_uring_queue_init(QD, &ring, 0);

    void *bufs[QD];
    for (int i = 0; i < QD; i++) posix_memalign(&bufs[i], 4096, BS);

    for (int i = 0; i < QD; i++) {                        // queue 64 reads...
        struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
        io_uring_prep_read(sqe, fd, bufs[i], BS, (off_t)i * BS * 1000);
        io_uring_sqe_set_data64(sqe, i);                  // tag to match completions
    }
    io_uring_submit(&ring);                               // ...one syscall

    for (int done = 0; done < QD; done++) {
        struct io_uring_cqe *cqe;
        io_uring_wait_cqe(&ring, &cqe);
        if (cqe->res < 0) fprintf(stderr, "read %llu failed: %d\n",
                                  (unsigned long long)io_uring_cqe_get_data64(cqe), cqe->res);
        io_uring_cqe_seen(&ring, cqe);                    // advance CQ head
    }
    io_uring_queue_exit(&ring);
}

io_uring outperforms epoll at high RPS because it eliminates syscall overhead per operation and supports disk I/O that epoll cannot. Further speedups come from registered files and buffers (skip per-I/O fd lookup and page pinning), multishot accept/recv (one SQE, many completions), provided buffer rings (kernel picks a buffer only when data arrives), and IORING_SETUP_IOPOLL (busy-poll NVMe completions instead of taking interrupts).

io_uring and security

io_uring exposes a large, fast-moving kernel attack surface. Google reported in 2023 that 60 % of kernel exploits submitted to its bug bounty targeted io_uring, and disabled it on ChromeOS, for Android apps, and on its production servers. Many container platforms block the io_uring_* syscalls in their default seccomp profiles, and Linux 6.6+ has kernel.io_uring_disabled to turn it off. Check whether it's available in your environment before designing around it. See Process Security.

Choosing an I/O Model

ModelDisk filesSocketsSyscalls per opConcurrency modelUsed by
BlockingYes (blocks)Yes (blocks)1Thread per operationSimple services, JDBC, classic Apache
Non-blocking + busy pollNo effectYesManySpinRare (DPDK-style)
select / pollAlways "ready"Yes1 + O(N) scanEvent loopLegacy, portable code
epoll / kqueueAlways "ready"Yes~1–2Event loop + thread pool for filesNginx, Redis, Node, Go, Netty
Linux AIOO_DIRECT onlyNo1 per batchCompletion-basedInnoDB, Seastar (historical)
io_uringYes (buffered + direct)Yes~0–1 per batchCompletion-basedRocksDB MultiGet, ScyllaDB, TigerBeetle, QEMU, newer runtimes (tokio-uring, Netty incubator, liburing)

Readiness-based models (epoll) say "you can now read without blocking"; completion-based models (AIO, io_uring, Windows IOCP) say "your read has finished — here are the bytes". Completion is the only model that works for regular files.


Direct Memory Access (DMA)

Without DMA, the CPU must copy every byte between a device and memory — burning CPU cycles on data movement instead of computation.

DMA offloads this to dedicated hardware:

Without DMA:
  Device → CPU registers → Memory   (CPU fully involved, ~100 cycles per word)

With DMA:
  CPU tells DMA controller: "copy N bytes from disk buffer to memory address X"
  DMA copies data directly, CPU does other work
  DMA raises interrupt when done
  CPU resumes handling the transfer

Why this matters: NVMe SSDs can sustain 7GB/s. Without DMA, the CPU would burn most of a core (or several) just moving data. With DMA, CPU utilization from I/O is near zero.

Modern devices are bus masters: the NVMe controller itself reads commands and data from host RAM and writes results back — there is no separate "DMA controller" chip. The IOMMU restricts which memory each device may touch. Interrupt handling, MSI-X and driver structure are covered in Interrupts, DMA & Drivers.

Zero-copy transfers: sendfile() syscall sends data from a file descriptor directly to a socket without copying to user space. Data path: disk → kernel page cache → NIC buffer, all via DMA. Used by Nginx for static file serving, Kafka for log segment transfers.

// Send file without copying to user space
ssize_t sent = sendfile(out_fd, in_fd, &offset, count);

Related zero-copy tools: splice()/tee() (move pages between pipes and fds), copy_file_range() (file-to-file, can become a reflink or server-side copy on NFS), MSG_ZEROCOPY (send from user pages without copying), and kTLS (lets sendfile work with TLS by encrypting in the kernel or NIC). See Linux Networking Stack.


Direct I/O

O_DIRECT asks the kernel to DMA straight between your buffer and the device, bypassing the page cache.

#define _GNU_SOURCE
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>

int fd = open("/data/table.ibd", O_RDWR | O_DIRECT | O_CLOEXEC);
void *buf;
posix_memalign(&buf, 4096, 16384);       // buffer address aligned to logical block size
ssize_t n = pread(fd, buf, 16384, 65536); // length and offset also multiples of block size
// Misaligned buffer, length or offset → EINVAL
Why databases use itThe catch
No double caching (buffer pool + page cache holding the same pages)You must implement caching, readahead and write batching yourself
Predictable memory: the kernel won't evict your cache under pressureAlignment rules (statx(STATX_DIOALIGN) reports them on 6.1+)
Avoids writeback storms and dirty-page throttling stallsEvery read is a device read — terrible for apps that relied on the page cache
Works with AIO/io_uring for deep queuesNot a durability guarantee: the device may still hold the data in its volatile cache. You still need fsync/fdatasync, or O_DSYNC, or RWF_DSYNC per write (which can map to an NVMe FUA write)

PostgreSQL historically relied on buffered I/O plus the page cache (direct I/O support is a work-in-progress debug_io_direct option); MySQL InnoDB defaults to innodb_flush_method = O_DIRECT on Linux; ScyllaDB and most specialised engines use direct I/O exclusively. Durability semantics are covered in Page Cache & fsync.


Disk Scheduling Algorithms

For HDDs, disk head movement is the bottleneck (~5ms per seek). The I/O scheduler determines the order in which pending requests are served to minimize seeks.

FCFS (First-Come, First-Served)

Serve requests in arrival order. Simple but worst-case head movement.

Requests (cylinder): 98, 183, 37, 122, 14, 124, 65, 67
Head at: 53
FCFS order: 53→98→183→37→122→14→124→65→67
Total movement: 640 cylinders

SSTF (Shortest Seek Time First)

Serve the pending request closest to current head position. Minimizes head movement but can starve requests at extremes.

Head at 53, requests: 98, 183, 37, 122, 14, 124, 65, 67
SSTF: 53→65→67→37→14→98→122→124→183
Total movement: 236 cylinders

SCAN (Elevator Algorithm)

Head moves in one direction servicing all requests, then reverses. Like an elevator.

Head at 53, moving toward low cylinders (LOOK variant - reverses at the last request):
53→37→14→(reverse)→65→67→98→122→124→183
Total movement: 39 + 169 = 208 cylinders

Same requests, moving toward high cylinders first:
53→65→67→98→122→124→183→(reverse)→37→14
Total movement: 130 + 169 = 299 cylinders

Pure SCAN travels all the way to the edge of the disk (cylinder 0 or the last cylinder) before reversing; LOOK reverses at the last pending request, which is what real implementations do. The direction you happen to be moving matters a lot for any given batch, which is why schedulers care about fairness over time rather than optimal movement for one batch.

C-SCAN (Circular SCAN): Moves in one direction only; after reaching the end, jumps back to the start without servicing. More uniform wait times.

AlgorithmHead movementStarvation riskWait-time variance
FCFSWorstNoneLow but uniformly slow
SSTFGoodHigh for edge requestsHigh
SCAN / LOOKGoodNoneMiddle-cylinder requests favored
C-SCAN / C-LOOKSlightly moreNoneMost uniform

Modern HDDs do their own reordering internally (NCQ, 32 commands), using knowledge the OS lacks — rotational position. The OS scheduler's job shifted from "minimize seek distance" to "keep latency bounded and share bandwidth fairly".

Linux I/O Schedulers

For NVMe SSDs, disk scheduling is largely irrelevant — random access latency is ~0.1ms regardless of "seek order." Linux defaults to the none scheduler for NVMe.

For HDDs, Linux offers:

  • mq-deadline: simple, prevents starvation with request deadlines
  • BFQ (Budget Fair Queueing): complex, provides fair I/O bandwidth across cgroups — used for desktop responsiveness
  • Kyber: latency-optimized for fast SSDs
SchedulerHow it worksOverheadPick it for
noneFIFO pass-through; blk-mq still mergesLowestNVMe, fast SSDs, virtual disks where the hypervisor/backend schedules (EBS, virtio on fast storage)
mq-deadlineSorted by sector, with read (500 ms) and write (5 s) deadlines and read preference; batches dispatchLowHDDs, SATA SSDs, RAID arrays; the usual default for SATA/SCSI
bfqPer-process/cgroup queues with sector budgets, idling to preserve sequential streams; honours io.weightHighest (CPU per I/O) — caps out around ~0.5–1M IOPS on fast CPUsDesktops, slow devices, and when you need proportional I/O sharing between cgroups
kyberToken-based: throttles queue depth per class (reads, sync writes, other) to hit target latencies (default 2 ms read, 10 ms sync write)LowFast multi-queue devices where reads must stay low-latency under mixed load
cat /sys/block/sda/queue/scheduler            # none [mq-deadline] kyber bfq
echo bfq | sudo tee /sys/block/sda/queue/scheduler   # change at runtime

# Make it persistent with a udev rule
cat <<'EOF' | sudo tee /etc/udev/rules.d/60-ioschedulers.rules
ACTION=="add|change", KERNEL=="nvme[0-9]*n[0-9]*", ATTR{queue/scheduler}="none"
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="mq-deadline"
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="mq-deadline"
EOF

The schedulers of the old single-queue era — noop, deadline, cfq — were removed in Linux 5.0. Advice that mentions cfq or elevator= boot parameters is out of date.


Buffering and Buffered I/O

Kernel Buffering

The kernel's page cache acts as a write buffer. write() returns immediately after copying to page cache — the kernel flushes to disk asynchronously (dirty pages older than 30s, or when dirty pages exceed a threshold: vm.dirty_background_ratio starts background writeback, vm.dirty_ratio makes writers block and do writeback themselves).

Application write():
  app buffer → page cache → (async) → disk
              ↑ fast (microseconds)    ↑ slow (milliseconds)

That dirty_ratio cliff is a classic latency bug: on a 256 GB machine, the default dirty_ratio=20 allows ~50 GB of dirty data before writers are throttled — then a single write() can stall for seconds while the kernel flushes. Large-memory database and logging hosts often set absolute limits (vm.dirty_background_bytes, vm.dirty_bytes) instead. Details in Page Cache & fsync.

Write-Back vs Write-Through

  • Write-back (default): write to cache, flush to disk later. Fast writes, risk of data loss on crash.
  • Write-through: write to cache AND disk synchronously. Slow but safe.

Databases use write-back (page cache or their own buffer pool) combined with a WAL to guarantee durability without synchronous disk writes on every operation.

The same distinction exists inside the device: SSDs and RAID controllers have volatile write caches. The kernel sends a cache flush (or uses FUA — Force Unit Access — writes) on fsync so the data reaches non-volatile media. Disabling those flushes (nobarrier, a RAID controller in write-back mode without a battery) makes benchmarks look great and loses data on power failure.

Buffered vs Unbuffered (stdio vs syscalls)

fwrite() (libc) maintains a user-space buffer. Accumulates small writes and flushes as larger chunks. write() (syscall) goes directly to the kernel. For many small writes, fwrite() dramatically outperforms write() by amortizing syscall overhead.

// Slow: 1000 syscalls
for (int i = 0; i < 1000; i++) write(fd, &byte, 1);

// Fast: 1 syscall (libc buffers 1000 bytes, then flushes)
for (int i = 0; i < 1000; i++) fwrite(&byte, 1, 1, fp);
fflush(fp);  // Ensure flush

The same idea everywhere: bufio.Writer in Go, BufferedOutputStream in Java, Python's default buffered file objects. The trap is the flip side — data in a user-space buffer is lost if the process is killed (SIGKILL, OOM) before it's flushed, even though the kernel would have kept anything already passed to write(). See Signals.


I/O Latency Numbers

OperationLatency
L1 cache1 ns
L2 cache4 ns
RAM access100 ns
Syscall round trip (getpid-class)~50–100 ns (more with mitigations)
Page cache hit read() of 4 KiB~1 µs
NVMe SSD random read0.1 ms
SATA SSD random read0.5 ms (device ~0.1 ms + stack and interface; varies widely)
NVMe flush / fsync (drive with PLP)~20–100 µs
NVMe flush / fsync (consumer, no PLP)~1–10 ms
EBS gp3 4 KiB read~0.5–1 ms
HDD random read5–10 ms
Network round trip (same DC)0.5–1 ms
Network round trip (cross-region)50–150 ms
Read 1MB from NVMe0.2–0.5 ms
Read 1MB from HDD10–30 ms (seek + transfer)

These numbers are why: caching exists, connection pooling exists, batching exists, and async I/O matters at scale.


I/O Observability

iostat -x

$ iostat -xz 1
Device  r/s     w/s     rkB/s    wkB/s   rrqm/s wrqm/s r_await w_await aqu-sz rareq-sz wareq-sz %util
nvme0n1 8421.0  2210.0  33684.0  70720.0   0.0   312.0    0.11    0.45   1.92     4.0     32.0  71.2
sda       42.0   180.0    168.0  14400.0   0.0    20.0   18.40   92.10  17.35     4.0     80.0  99.8
ColumnMeaningRead it as
r/s, w/sCompleted requests per second (after merging)IOPS
rkB/s, wkB/sThroughputCompare with device/volume limits
rrqm/s, wrqm/sRequests merged per secondHigh = sequential-ish workload
r_await, w_awaitAverage ms from submission to completion including time queued in the OSThe user-visible latency
aqu-szAverage number of requests in flight (queue depth)Little's Law: aqu-sz ≈ IOPS × await
rareq-sz, wareq-szAverage request size (kB)4 = random small I/O; 128+ = streaming
%utilPercentage of time the device had at least one request in flightSaturation for HDDs; misleading for SSDs

`%util` lies on parallel devices

%util was meaningful when a disk could do one thing at a time: 100 % meant saturated. An NVMe SSD processes dozens of requests concurrently, so it can show 100 % util at 10 % of its capacity — a single thread doing back-to-back reads keeps "at least one request in flight" all the time. For SSDs and cloud volumes, judge saturation by await rising, aqu-sz growing, and IOPS/throughput approaching the device's or volume's documented limits. (svctm was removed from sysstat for the same reason — it was never measurable from the host.)

In the example above, sda is a saturated HDD: 222 IOPS, 92 ms write latency, 17 requests queued. nvme0n1 at "71 % util" is nowhere near its limit.

Who Is Doing the I/O?

sudo iotop -oPa                    # processes doing I/O, accumulated
pidstat -d 1                       # per-process kB_rd/s, kB_wr/s, iodelay
cat /proc/<pid>/io                 # read_bytes / write_bytes actually hitting storage
cat /proc/pressure/io              # PSI: % time tasks stalled on I/O
cat /sys/fs/cgroup/<cg>/io.stat    # per-cgroup bytes and IOs per device
vmstat 1                           # 'b' = tasks blocked (D state), 'wa' = iowait %

iowait (wa in top/vmstat) is the percentage of time a CPU was idle while some task had I/O outstanding. It's a hint, not a measurement of disk load: a busy CPU hides iowait, and a single slow I/O on an idle machine can show high iowait. PSI (/proc/pressure/io) is the better signal — see Namespaces & cgroups.

Latency Distributions with eBPF

Averages hide the tail. The BCC/bpftrace tools trace every block I/O:

$ sudo biolatency -D 10 1          # histogram per disk, 10 s
disk = nvme0n1
     usecs               : count     distribution
        16 -> 31         : 1204     |**                                      |
        32 -> 63         : 18832    |****************************************|
        64 -> 127        : 9120     |*******************                     |
       128 -> 255        : 822      |*                                       |
      2048 -> 4095       : 41       |                                        |   <- tail: GC or flush stalls
     16384 -> 32767      : 7        |                                        |

$ sudo biosnoop | head              # every I/O: process, disk, sector, bytes, latency
TIME(s)  COMM        PID   DISK     T SECTOR     BYTES  LAT(ms)
0.000    postgres    2210  nvme0n1  W 81203200   8192     0.41
0.001    kworker/u8  311   nvme0n1  W 99102208   524288   2.95

# bpftrace one-liner: I/O size histogram by process
sudo bpftrace -e 'tracepoint:block:block_rq_issue { @[comm] = hist(args->bytes); }'

blktrace/blkparse give per-request timelines through every block-layer stage (queued → merged → dispatched → completed) when you need to know where in the stack latency is added. See eBPF for the tooling.

Benchmarking with fio

# 4 KiB random reads, 32 deep, 4 jobs, direct I/O - the "IOPS" number
fio --name=randread --filename=/dev/nvme1n1 --rw=randread --bs=4k --iodepth=32 \
    --numjobs=4 --direct=1 --ioengine=io_uring --runtime=60 --time_based --group_reporting

# fsync latency - what a database commit feels like
fio --name=fsync --filename=/data/fio.test --size=256M --rw=write --bs=4k \
    --fdatasync=1 --ioengine=sync --runtime=30 --time_based

# Sequential throughput
fio --name=seq --filename=/data/fio.test --size=10G --rw=read --bs=1M --iodepth=8 --direct=1 --ioengine=libaio

Report percentiles (clat percentiles p99, p99.9), not just averages; pre-condition SSDs (write the whole device twice) before steady-state write tests; never run write tests against a raw device holding data.


Cloud Block Storage

A cloud volume (AWS EBS, GCP Persistent Disk/Hyperdisk, Azure Managed Disks) is not a disk — it's a distributed, replicated storage service reached over the network, presented to the VM as an NVMe or virtio device. That changes the performance model:

Physical NVMeCloud volume
Latency ~0.1 msLatency ~0.5–2 ms (network + replication)
IOPS limited by hardwareIOPS and throughput are provisioned limits enforced by the provider
Limits per deviceLimits per volume and per instance — the lower one wins
Survives nothing if the host diesReplicated within an AZ; snapshots to object storage

AWS EBS Volume Types

TypeMediaBaselineMax per volumePricing modelUse
gp3SSD3,000 IOPS + 125 MiB/s regardless of size16,000 IOPS / 1,000 MiB/s historically; raised in 2025 for large volumes (up to 80,000 IOPS / 2,000 MiB/s) — check current docsPay for GB + extra IOPS + extra throughput separatelyDefault for almost everything
gp2 (legacy)SSD3 IOPS per GiB (min 100)16,000 IOPS / 250 MiB/sPay per GB; burst credits below 1 TiBMigrate to gp3
io2 Block ExpressSSDProvisioned, up to 1,000 IOPS per GiB256,000 IOPS / 4,000 MiB/s, sub-ms latency, 99.999 % durabilityPay per provisioned IOPSLarge OLTP databases
st1HDD40 MiB/s per TiB baseline, burst to 250 MiB/s per TiB500 MiB/sThroughput creditsBig sequential: logs, Kafka, ETL
sc1HDD12 MiB/s per TiB250 MiB/sThroughput creditsCold data

EBS counts SSD I/Os of up to 256 KiB as one operation (merging smaller sequential ones), HDD I/Os up to 1 MiB.

Burst Credits

gp2 (and st1/sc1, and smaller EC2 instances' EBS bandwidth) use a token bucket:

gp2 volume of 200 GiB:
  baseline   = 3 IOPS/GiB × 200 = 600 IOPS
  burst      = 3,000 IOPS
  bucket     = 5.4 million I/O credits (full at creation)
  drain rate = 3,000 − 600 = 2,400 credits/s while bursting
  burst time = 5,400,000 / 2,400 ≈ 37.5 minutes, then you drop to 600 IOPS
flowchart LR
    A["Volume busy above baseline"] --> B["Burst credits drain"]
    B --> C{"Credits left?"}
    C -->|"yes"| D["Up to 3,000 IOPS"]
    C -->|"no"| E["Throttled to baseline: latency and queue depth explode"]
    F["Volume below baseline"] --> G["Credits refill at baseline rate"]

Monitor BurstBalance in CloudWatch — when it hits zero, so does your performance. gp3 has no burst model (baseline is always available), which is the main reason to migrate.

Instance-Level Limits

Every EC2 instance type also has an EBS bandwidth and IOPS ceiling across all attached volumes. Many smaller instances (e.g. most .large and below in current families) have a baseline EBS bandwidth and can burst to their maximum for only ~30 minutes per 24 hours. Attaching a 16,000-IOPS gp3 volume to a small instance doesn't give you 16,000 IOPS. Check EBSIOBalance% and EBSByteBalance% CloudWatch metrics, and the instance's documented "EBS-optimized" baseline vs maximum.

Things That Surprise People

  • First-read penalty from snapshots. A volume restored from a snapshot is lazily loaded from S3: the first read of each block can take tens of milliseconds. A freshly restored database replica can be 10× slower for hours. Fix: Fast Snapshot Restore (paid), or pre-warm with fio --rw=read --bs=1M over the whole device.
  • Queue depth is required to reach provisioned IOPS. At ~0.5 ms latency, one synchronous thread gets ~2,000 IOPS at best (Little's Law). To use 16,000 IOPS you need ~8+ requests in flight.
  • Local instance store NVMe is far faster (100k+ IOPS, ~100 µs) but ephemeral — data is lost on stop/terminate or host failure. Fine for caches, scratch, and replicated systems (Cassandra, Kafka with replication, Elasticsearch replicas).
  • NVMe timeouts. EBS volumes appear as NVMe devices; the Linux default nvme_core.io_timeout should be set high (AWS recommends the maximum) so a transient EBS hiccup doesn't make the kernel declare the device dead and remount filesystems read-only.
# Is this an EBS volume and what's its ID?
sudo nvme id-ctrl -v /dev/nvme1n1 | grep -E '^mn|^sn'
mn        : Amazon Elastic Block Store
sn        : vol0a1b2c3d4e5f67890
cat /sys/module/nvme_core/parameters/io_timeout

War Story (illustrative)

A team's PostgreSQL primary on a 500 GiB gp2 volume ran fine for months, then started timing out every night at 02:00. The nightly batch job plus autovacuum pushed I/O above the 1,500 IOPS baseline; burst credits lasted about an hour, then latency jumped from 1 ms to 40 ms and connections piled up. BurstBalance in CloudWatch showed a sawtooth hitting 0 % at 02:50 each night. Migrating the volume to gp3 (online, via ModifyVolume) with 6,000 provisioned IOPS cost less per month than the gp2 volume and removed the cliff.


Failure Modes

SymptomLikely causeConfirmFix
Load average high, CPUs idleTasks in D state waiting on storagevmstat b column; ps -eo stat,wchan:30,cmd | grep ^DFind the slow device; reduce I/O; faster storage
Periodic multi-second write stallsDirty page throttling (dirty_ratio) or SSD GCgrep -E 'Dirty|Writeback' /proc/meminfo; biolatency tailLower dirty_bytes/dirty_background_bytes; over-provision SSD
Cloud DB fast, then suddenly slow for hoursBurst credits exhausted (gp2, st1, small instance EBS bandwidth)BurstBalance, EBSIOBalance%gp3 / provisioned IOPS / bigger instance
%util 100 % but low IOPS on NVMeMisreading %utilCompare IOPS with device spec; check awaitIgnore %util for SSDs
Database commit latency 5–10 ms on "fast" SSDConsumer SSD without PLP; each fsync programs NANDfio --fdatasync=1Enterprise SSD with PLP; group commit
SSD write throughput collapses after minutes of writesSLC cache exhausted / steady-state GCLong fio runSize for steady state; more over-provisioning; TRIM
Filesystem goes read-only on a VMDevice I/O timeout or media errordmesg | grep -iE 'nvme|I/O error|remount'Raise nvme_core.io_timeout; check device health
Event loop stalls despite epollBlocking file I/O or fsync on the loop threadstrace -T -p, profiler shows read/fsync on loopMove to thread pool or io_uring
io_uring unavailable / EPERM in containersDisabled by seccomp or io_uring_disabledsysctl kernel.io_uring_disabledFall back to epoll + thread pool
Restored replica slow for hoursLazy snapshot loadingRead latency high only on first touchFast Snapshot Restore or pre-warm

Decision Guide

  • Choosing storage: NVMe local for latency and IOPS (if you can tolerate ephemerality or replicate above it); cloud SSD volumes (gp3) for general durable storage; provisioned-IOPS volumes only when measured need justifies the price; HDD or object storage for cold, sequential data.
  • Choosing a scheduler: none for NVMe and most cloud volumes; mq-deadline for SATA SSD/HDD; bfq when you need per-cgroup fairness on slower devices; kyber if mixed workloads hurt read latency on fast devices.
  • Choosing an I/O model: blocking with threads for modest concurrency (simplest, often fast enough); epoll (or your runtime's netpoller) for many sockets; io_uring for very high IOPS file workloads or syscall-bound servers if your platform allows it; a thread pool for file I/O inside an event loop otherwise.
  • Buffered vs direct: buffered unless you are building a database-like system with its own cache.
  • Capacity planning: compute required IOPS and throughput, then required queue depth (IOPS × latency), then check both volume and instance limits.
  • Monitoring: alert on await and PSI I/O, queue depth, burst balances, SSD wear (percentage_used) and device errors — not on %util.

Advanced Topics

  • Polled I/O (io_poll, IORING_SETUP_IOPOLL, nvme poll_queues=) trades a busy CPU for ~2–5 µs lower latency per I/O by spinning on the completion queue instead of taking an interrupt.
  • SPDK moves the NVMe driver entirely into user space (polling, no syscalls, no interrupts) for multi-million IOPS per core — at the cost of dedicating cores and devices.
  • Zoned storage (ZNS NVMe, SMR HDDs) exposes erase blocks/zones directly to the host, which must write sequentially per zone; this eliminates device GC and over-provisioning when the filesystem (f2fs, btrfs zoned mode) or database cooperates.
  • NVMe over Fabrics (NVMe/TCP, NVMe/RDMA) — the NVMe queue model across a network; the basis of many disaggregated storage systems.
  • Multipath I/O (dm-multipath, native NVMe multipath) for redundant paths to SAN storage.
  • Writeback throttling and io.latency/io.cost cgroup controllers for protecting latency-sensitive workloads — see Namespaces & cgroups.
  • Device mapper stacks (LVM, dm-crypt, dm-raid, dm-cache/bcache) — each adds a layer and sometimes changes flush/discard behavior; verify fsync and TRIM pass through.

Interview Questions

"What is epoll and why is it better than select?" epoll maintains a kernel-side set of watched file descriptors. epoll_wait() returns only the descriptors that are ready — O(ready events). select scans all watched FDs on every call — O(total FDs). At 10K connections, select scans 10K FDs per wakeup; epoll returns only the 1–5 that have data.

"What is a zero-copy transfer?" Using sendfile() or DMA to move data from a file to a network socket without copying it to user space. Data path: disk → page cache → NIC via DMA. The CPU sets up the transfer and returns; DMA hardware does the actual data movement.

"Why is HDD disk scheduling important but SSD disk scheduling isn't?" HDDs have a physical read head that must physically move to the cylinder — seek time is 5–10ms and depends on distance. SSDs have no moving parts; all blocks are equally fast to access (~0.1ms). Reordering requests saves 0ms on SSD; on HDD it can reduce total seek distance by 5×.

"What is io_uring and when would you use it?" io_uring uses two shared ring buffers between user and kernel space to submit and complete I/O operations without a syscall per operation. Ideal for high-RPS servers that are syscall-bound, or for disk I/O that epoll can't handle. Used by modern database engines (RocksDB, Scylla) and network runtimes (tokio-uring, liburing-based servers).

"Why can't epoll make disk reads asynchronous?" Regular files are always reported ready; a read that misses the page cache blocks inside the filesystem regardless of O_NONBLOCK. Readiness doesn't apply to files — you need a completion-based interface (io_uring, AIO with O_DIRECT) or a thread pool.

"What is write amplification on an SSD?" NAND can't be overwritten in place and erases in large blocks, so the FTL writes new data elsewhere and garbage-collects old blocks by copying their still-valid pages. Bytes written to flash ÷ bytes written by the host is the WAF. TRIM, over-provisioning and large sequential writes reduce it; high WAF reduces both performance and lifetime.

"Your NVMe disk shows 100% util in iostat. Is it saturated?" Not necessarily. %util is the fraction of time at least one request was outstanding; NVMe handles many in parallel. Check await, aqu-sz, and IOPS/throughput against the device's limits.

"Why does your database get slow every night on its cloud volume?" Likely burst credits (gp2/st1 or instance EBS burst) running out under batch load, dropping to baseline IOPS. Check BurstBalance/EBSIOBalance%; move to gp3 or provisioned IOPS, or a larger instance.

"How many IOPS can one thread doing synchronous 4 KiB reads get from an EBS volume?" Little's Law: queue depth 1 ÷ ~0.5–1 ms latency ≈ 1,000–2,000 IOPS, no matter what is provisioned. Use concurrency or async I/O to reach the limit.