Interrupts, DMA & Device Drivers
The CPU is fast and alone. Everything else — disks, NICs, GPUs, keyboards, timers — is a separate piece of hardware running at its own pace. The kernel needs two things from that hardware: a way for the device to say "something happened" without the CPU constantly asking, and a way to move data between device and memory without the CPU copying every byte. Those two mechanisms are interrupts and DMA. The code that knows how to speak to a particular piece of hardware through them is a device driver.
These sound like kernel-developer concerns, but they leak into every production system. A database whose p99 jumps when one core hits 100% %soft. A VM that loses 30% of its network throughput after a live migration to a host without SR-IOV. An NVMe-backed service that gets 2x more IOPS when you switch to polling. A "noisy" core that turns out to be handling every interrupt for every queue of a 100 GbE NIC. Knowing how interrupts and DMA work tells you where to look.
Where this fits
This page is the hardware layer under The Linux Networking Stack and I/O & Storage Internals. The CPU-time side — how interrupt work competes with your threads — connects to CPU Scheduling. The privilege boundary these mechanisms rely on is described in Syscalls & Kernel Internals.
Why Interrupts Exist: The Polling Problem
Suppose a NIC receives a packet. How does the CPU find out?
Option 1: polling. The CPU reads a device status register in a loop. Simple, and for a device that is always busy it is actually optimal — no wasted work. But a device that is idle 99.9% of the time turns a CPU core into a space heater, and if you poll every millisecond instead of continuously, you add up to a millisecond of latency to every event.
Option 2: interrupts. The device sends a signal; the CPU stops what it's doing, runs a handler, and resumes. Zero cost while idle, immediate reaction when something happens. But each interrupt has a fixed cost — saving state, jumping into the kernel, polluting caches and branch predictors, returning — and at very high event rates that fixed cost dominates.
Every design on this page is a point on that trade-off curve:
| Event rate | Best mechanism | Why |
|---|---|---|
| Rare (keyboard, timer, idle NIC) | Pure interrupts | No CPU spent waiting |
| Moderate (busy disk, loaded NIC) | Interrupts + coalescing | Batch several events per interrupt |
| Very high (10–100 GbE, NVMe at 1M IOPS) | Interrupt to start, poll while busy (NAPI, blk-mq polling) | Amortise interrupt cost over hundreds of events |
| Extreme, latency-critical | Pure polling on dedicated cores (DPDK, SPDK, io_uring SQPOLL) | Zero interrupt latency, one core burned permanently |
Three Kinds of "Interrupt"
The word covers three different things that share the same CPU machinery.
| Kind | Source | Synchronous? | Examples |
|---|---|---|---|
| Hardware interrupt (IRQ) | External device via the interrupt controller | No — arrives at any point | NIC packet received, disk I/O complete, timer tick |
| Exception | The CPU itself, caused by the instruction it is executing | Yes — tied to a specific instruction | Page fault (#PF), divide by zero (#DE), general protection fault (#GP), invalid opcode (#UD), breakpoint (#BP) |
| Software interrupt (trap) | An explicit instruction | Yes | int 0x80 (legacy 32-bit syscalls), int3 (debugger breakpoint) |
Then there are two special cases:
- NMI (Non-Maskable Interrupt) cannot be disabled. Linux uses it for the hard-lockup watchdog and for
perfsampling, which is howperfcan profile code that runs with interrupts disabled. - IPI (Inter-Processor Interrupt) is one CPU interrupting another: "reschedule", "flush your TLB" (after
munmapormprotect), "run this function". TLB shootdown IPIs are a real cost on many-core machines with heavy memory mapping churn.
And separately, softirqs — despite the name — are not hardware interrupts at all. They are a kernel mechanism for deferred work, covered below.
Syscalls on x86-64 don't use interrupts
Modern 64-bit Linux enters the kernel with the syscall instruction, which jumps to an address stored in a model-specific register (MSR_LSTAR) without consulting the interrupt table. It is significantly faster than int 0x80. Exceptions and hardware interrupts still go through the IDT.
The Interrupt Descriptor Table
On x86, the CPU finds the handler for any interrupt or exception through the Interrupt Descriptor Table (IDT): an array of 256 entries, each pointing to a handler and specifying the privilege level and stack to use. The IDTR register holds its address.
| Vector range | Use |
|---|---|
| 0–31 | Reserved for CPU exceptions: 0 divide error, 3 breakpoint, 6 invalid opcode, 8 double fault, 13 general protection, 14 page fault, 18 machine check |
| 32–255 | Available for external interrupts, IPIs, and the local APIC timer — the kernel allocates these to devices |
flowchart LR
DEV["Device - NIC, NVMe, timer"] -- "MSI write or IRQ line" --> APIC["Local APIC of target CPU"]
APIC -- "vector number 32-255" --> CPU["CPU core"]
EXC["Faulting instruction"] -- "vector 0-31" --> CPU
CPU --> IDT["IDT lookup via IDTR"]
IDT --> ENTRY["Kernel entry stub: save registers, switch stack"]
ENTRY --> H["Generic IRQ code then driver handler"]
H --> RET["Raise softirqs if needed, IRET back to interrupted code"]
When an interrupt arrives, the CPU (in hardware): finishes the current instruction, saves the instruction pointer, flags, and stack pointer onto the kernel stack (switching from the user stack if the CPU was in ring 3), clears the interrupt-enable flag if the gate type requires it, and jumps to the handler. The kernel's entry stub saves the remaining registers, figures out which device the vector belongs to, and calls the driver's registered handler.
On ARM the equivalent is the exception vector table plus the GIC (Generic Interrupt Controller); the concepts — vectors, priority, per-CPU routing — map across directly.
The interrupt controller
Old PCs had a single 8259 PIC with 15 lines. Modern x86 machines have a Local APIC in every core (handling timers, IPIs, and delivery to that core) plus one or more I/O APICs that route legacy wired interrupt lines to cores. But modern PCIe devices don't use wires at all.
MSI and MSI-X: Interrupts as Memory Writes
Message Signaled Interrupts replace physical interrupt lines with a memory write. The device writes a specific value to a specific address (in the 0xFEExxxxx range on x86); the chipset recognises that address and turns it into an interrupt delivered to a specific core with a specific vector.
| Legacy INTx | MSI | MSI-X | |
|---|---|---|---|
| Mechanism | Shared physical wire | Memory write | Memory write |
| Vectors per device | 1 (shared with other devices) | Up to 32, contiguous | Up to 2048, independent |
| Per-vector CPU targeting | No | Limited | Yes — each vector has its own address/data |
| Sharing with other devices | Common — every handler must check "was it me?" | No | No |
MSI-X is what makes modern high-performance I/O possible. A 32-queue NIC gets 32+ vectors; an NVMe drive gets one per submission/completion queue pair, typically one per CPU. Each queue's interrupt can be pinned to a different core, so completions are handled on the core that issued the I/O — good for cache locality and no cross-core locking.
# Which interrupt mode a device uses
sudo lspci -vv -s 0000:3b:00.0 | grep -E 'MSI|MSI-X'
# Capabilities: [a0] MSI-X: Enable+ Count=64 Masked-
# All interrupt vectors, per-CPU counts
cat /proc/interrupts | head -5
# CPU0 CPU1 CPU2 CPU3
# 0: 35 0 0 0 IR-IO-APIC 2-edge timer
# 124: 88214321 0 0 0 IR-PCI-MSIX-0000:3b:00.0 1-edge eth0-TxRx-0
# 125: 0 90123554 0 0 IR-PCI-MSIX-0000:3b:00.0 2-edge eth0-TxRx-1
# 140: 412880 409117 411203 410998 IR-PCI-MSIX-0000:5e:00.0 0-edge nvme0q0
Note the bottom of /proc/interrupts too: lines like LOC (local timer), RES (rescheduling IPIs), CAL (function-call IPIs), TLB (TLB shootdowns), and NMI. A high TLB rate points at memory-mapping churn; a high RES rate points at threads constantly waking each other across cores.
Top Half and Bottom Half
An interrupt handler runs with that interrupt line (and often all interrupts on that CPU) disabled, on whatever stack and context it interrupted, and cannot sleep — there is no process to put to sleep, so taking a mutex, allocating memory with GFP_KERNEL, or waiting for I/O is forbidden. While it runs, other interrupts on that core wait. Long handlers mean latency for everyone.
So Linux splits interrupt handling in two:
- Top half (the hard IRQ handler): acknowledge the device, grab the minimum state, schedule the rest. Microseconds or less.
- Bottom half: everything else — protocol processing, waking processes, completing requests — run later with interrupts enabled.
flowchart TD
IRQ["Hardware IRQ arrives"] --> TOP["Top half: ack device, read status, mask queue"]
TOP --> CHOICE{"How is deferred work scheduled?"}
CHOICE -- "high rate, per-CPU, cannot sleep" --> SIRQ["Softirq - NET_RX, BLOCK, TIMER, RCU"]
CHOICE -- "driver-specific, cannot sleep" --> TL["Tasklet - built on softirq, being phased out"]
CHOICE -- "needs to sleep, process context" --> WQ["Workqueue - kworker threads"]
CHOICE -- "whole handler in a thread" --> TH["Threaded IRQ - irq/N-name kernel thread"]
SIRQ --> KSD["If overloaded: ksoftirqd/N thread"]
Softirqs
A fixed set of ten statically defined, per-CPU deferred handlers, run on the way out of a hard IRQ, when bottom halves are re-enabled, or in the ksoftirqd/N thread:
cat /proc/softirqs
# CPU0 CPU1 CPU2 CPU3
# HI: 1 0 0 2
# TIMER: 12011837 10988771 11203347 11455201
# NET_TX: 41823 38211 40017 39866
# NET_RX: 981234511 12003321 8812211 9002118 <- CPU0 is doing all the packet work
# BLOCK: 2203411 2190033 2210988 2187655
# IRQ_POLL: 0 0 0 0
# TASKLET: 12201 8834 9102 9311
# SCHED: 8812331 8711234 8698772 8733901
# HRTIMER: 1123 998 1045 1011
# RCU: 9921387 9877312 9902211 9888714
The same softirq can run concurrently on several CPUs, so handlers must use per-CPU data or locking. Softirqs cannot sleep. If they keep re-raising themselves (a packet flood), the kernel stops running them inline after a bounded time and wakes ksoftirqd/N so that the scheduler can balance packet processing against user threads. Networking's NET_RX softirq — and its NAPI polling loop — is walked through in detail in The Linux Networking Stack.
Tasklets
Tasklets were a dynamically allocatable layer on top of softirqs, with one guarantee: a given tasklet never runs on two CPUs at once, which made them easier to write. They have been deprecated in spirit for years, and recent kernels (6.9+) added BH workqueues — workqueues that execute in softirq context — as the intended replacement. New drivers should not use tasklets.
Workqueues
When deferred work needs to sleep — take a mutex, allocate memory that may block, talk to a slow bus like I2C — it goes into a workqueue, executed by kworker/N:M kernel threads in normal process context. Workqueues are scheduled like any other thread, so their latency depends on the scheduler.
Threaded IRQs
request_threaded_irq() registers two functions: a tiny hard-IRQ handler that only checks "is this mine?" and masks the device, and a thread function that runs the rest in a dedicated irq/N-name kernel thread. That thread has a priority (SCHED_FIFO 50 by default), can be pinned and reprioritised with chrt and taskset, and can sleep.
The PREEMPT_RT real-time patch set — merged into mainline in 6.12 — forces nearly all interrupt handlers into threads. That is how it achieves bounded latency: a high-priority real-time task can preempt interrupt processing that it doesn't care about. You can get the same forced-threading behaviour on a normal kernel with the threadirqs boot parameter.
| Mechanism | Context | Can sleep | Concurrency | Typical use |
|---|---|---|---|---|
| Hard IRQ (top half) | Interrupt | No | One per IRQ line at a time | Ack device, schedule bottom half |
| Softirq | Softirq | No | Same softirq on many CPUs at once | Networking, block completion, timers, RCU |
| Tasklet | Softirq | No | Serialised per tasklet | Legacy drivers |
| BH workqueue | Softirq | No | Workqueue semantics | Tasklet replacement (6.9+) |
| Workqueue | Process (kworker) | Yes | Configurable | Driver housekeeping, slow buses, firmware loading |
| Threaded IRQ | Process (irq/N thread) | Yes | One thread per IRQ | RT systems, I2C/SPI devices, slow handlers |
Interrupt Cost and Latency
What does one interrupt actually cost? The direct cost — vector delivery, register save, handler dispatch, IRET — is on the order of a few hundred nanoseconds to a microsecond on modern x86. The indirect cost is usually bigger: the interrupted code loses L1/L2 cache contents and branch-predictor state to the handler, and pays for that in cache misses for several microseconds afterward.
| Event | Rough cost on modern x86 |
|---|---|
| Hardware interrupt, direct entry/exit | ~0.3–1 µs |
| Interrupt including cache and pipeline disruption | ~1–5 µs effective |
| IPI from one core to another | ~1–2 µs to deliver and handle |
| Syscall entry/exit (with KPTI mitigations) | ~0.1–0.3 µs |
| Thread context switch | ~2–5 µs including cache effects |
| Wakeup latency: interrupt → blocked thread running | ~5–50 µs depending on load and C-states |
| Exit from deep CPU C-state (C6) | ~50–100+ µs |
| Busy-poll check of a device ring in L1/L2 | ~10–100 ns |
That last two rows explain a lot of real latency. An idle CPU that has dropped into a deep C-state needs tens of microseconds just to wake up before it can run the interrupt handler. Latency-sensitive systems often cap C-states (intel_idle.max_cstate=1, processor.max_cstate=1, or the /dev/cpu_dma_latency interface) at the price of power.
Interrupt latency vs interrupt overhead
- Latency — time from the device raising the interrupt to useful work happening. Hurt by: interrupts disabled for long stretches, deep C-states, the handler's CPU being busy with another softirq, the woken thread waiting in a run queue.
- Overhead — total CPU time spent servicing interrupts. Hurt by: high event rates without coalescing or NAPI.
You usually tune one at the expense of the other. Coalescing reduces overhead and adds latency. Polling removes latency and costs a core.
Interrupt storms
An interrupt storm is a device firing interrupts faster than the CPU can handle them — from a hardware fault, a misconfigured level-triggered line that is never deasserted, or a firmware bug. The CPU spends 100% of its time in the handler and nothing else runs on that core. Linux has a safety valve: if a shared IRQ line keeps firing and no handler claims it, the kernel eventually logs
irq 16: nobody cared (try booting with the "irqpoll" option)
Disabling IRQ #16
and turns the line off. That stops the storm and also breaks whatever device was on it. Seeing this message means "investigate the hardware or the driver", not "add irqpoll".
A softer, more common version is an IRQ hotspot: every queue's interrupt lands on CPU 0 because affinity was never set, and CPU 0 shows 100% %irq+%soft while the other cores idle.
War Story (illustrative)
A team migrated a Kafka broker fleet to new instances with twice the cores and saw worse tail latency. mpstat -P ALL 1 showed CPU 0 at 95% %soft, every other core under 20%. The image had irqbalance disabled and the new NIC driver created 8 queues, but all 8 MSI-X vectors still had the default affinity mask, and on that platform the effective target was CPU 0. Spreading the vectors across the cores of the NIC's NUMA node — one echo per IRQ into smp_affinity_list — dropped p99 produce latency by more than half. Nothing in the application had changed.
Interrupt Coalescing
Coalescing (also called moderation) tells the device: don't interrupt for every event; wait until N events have accumulated or T microseconds have passed, whichever comes first.
# Current settings
ethtool -c eth0
# Adaptive RX: on TX: on
# rx-usecs: 50
# rx-frames: 0
# tx-usecs: 50
# Fixed coalescing: interrupt after 64 frames or 100 µs
sudo ethtool -C eth0 adaptive-rx off rx-usecs 100 rx-frames 64
# Latency-first: interrupt immediately
sudo ethtool -C eth0 adaptive-rx off rx-usecs 0 rx-frames 1
# Let the driver adapt to traffic (Linux DIM on many drivers)
sudo ethtool -C eth0 adaptive-rx on adaptive-tx on
| Setting | Interrupt rate | Added latency | CPU overhead | Good for |
|---|---|---|---|---|
rx-usecs 0 | Up to one per packet | ~0 | High | Low-rate latency-critical traffic |
rx-usecs 50–100 | Bounded ~10–20k/s per queue | Up to 50–100 µs | Low | Bulk throughput |
| Adaptive (DIM) | Varies with load | Low at low load, higher under load | Balanced | Most servers |
Adaptive coalescing — the kernel's Dynamic Interrupt Moderation (DIM) library, used by mlx5, ENA, and many others — watches packet rate and bytes per interrupt and moves the settings automatically. It is the right default unless you have measured a reason to override it.
NVMe has the same idea in its spec (aggregation threshold and time), but Linux typically leaves NVMe coalescing off and relies on per-CPU queues plus the block layer's batching.
Polling vs Interrupts in Practice
Linux has converged on hybrids that interrupt when idle and poll when busy.
NAPI (networking)
One interrupt arrives; the driver masks the queue's interrupt and schedules a softirq poll; the poll processes up to a budget of packets per run and keeps the interrupt masked as long as there is work. When the ring is drained, it re-enables the interrupt. Under load, interrupt rate falls to near zero while throughput rises. Details are in The Linux Networking Stack.
Block layer polling (NVMe)
A fast NVMe SSD completes a 4 KB read in ~10–80 µs; low-latency devices (Optane-class, some SLC drives) in under 10 µs. At that scale, interrupt delivery + wakeup + context switch (5–20 µs in total) is a significant fraction of the I/O latency. Linux supports polled I/O on dedicated poll queues:
# Allocate poll queues in the NVMe driver (module parameter, set at load/boot)
cat /sys/module/nvme/parameters/poll_queues
# Kernel cmdline: nvme.poll_queues=4
# fio comparison: interrupt-driven vs polled completions
fio --name=irq --filename=/dev/nvme0n1 --direct=1 --rw=randread --bs=4k \
--ioengine=io_uring --iodepth=1 --runtime=30 --time_based
fio --name=poll --filename=/dev/nvme0n1 --direct=1 --rw=randread --bs=4k \
--ioengine=io_uring --hipri=1 --iodepth=1 --runtime=30 --time_based
--hipri maps to io_uring's IORING_SETUP_IOPOLL: instead of sleeping until a completion interrupt, the submitting thread spins on the completion queue. Typical results on a fast drive: a few microseconds shaved off per I/O at queue depth 1, at the cost of 100% CPU on the polling thread. At high queue depths the gain shrinks because interrupts are already amortised.
io_uring SQPOLL
IORING_SETUP_SQPOLL creates a kernel thread that polls the submission queue, so the application can submit I/O by writing to shared memory with zero syscalls. Combined with IOPOLL on the completion side you get a fully polled path: no syscalls, no interrupts. SQPOLL threads go to sleep after an idle timeout (sq_thread_idle), so the cost is bounded when traffic stops. The io_uring APIs themselves are described in I/O & Storage Internals.
Full user-space polling
DPDK (network) and SPDK (storage) take the device away from the kernel, map it into a process, and poll from dedicated cores that never sleep. Lowest latency and highest throughput available — at the cost of a permanently busy core per poller and losing kernel tooling.
flowchart LR
A["Pure interrupts"] --> B["Interrupt coalescing"]
B --> C["NAPI or hybrid polling"]
C --> D["io_uring IOPOLL and SQPOLL"]
D --> E["DPDK or SPDK user-space polling"]
A -. "lowest CPU when idle" .-> A
E -. "lowest latency, burns cores" .-> E
DMA: Letting Devices Write Memory Directly
Copying a 64 KB network buffer one word at a time through the CPU would cost tens of microseconds and evict useful data from cache. Direct Memory Access lets the device read and write host RAM itself.
Bus mastering
On PCIe, any device can become a bus master and issue memory read/write transactions to host addresses. The driver's job is to tell the device which addresses to use. The CPU sets up the transfer, the device moves the data, and an interrupt signals completion.
sequenceDiagram
participant Drv as Driver on CPU
participant Mem as Host RAM
participant Dev as PCIe device
Drv->>Mem: allocate buffer, map it for DMA
Drv->>Mem: write descriptor with buffer bus address
Drv->>Dev: MMIO doorbell write - new work queued
Dev->>Mem: DMA read descriptor
Dev->>Mem: DMA write data into buffer
Dev->>Mem: DMA write completion status
Dev->>Drv: MSI-X interrupt
Drv->>Mem: unmap buffer, hand data up the stack
Descriptor rings and scatter-gather
Devices and drivers communicate through descriptor rings — circular arrays in host memory, one entry per buffer, each holding an address, a length, and status flags. NVMe has submission and completion queues; NICs have RX and TX rings; virtio has virtqueues. Same idea everywhere.
Real data is rarely physically contiguous: a 64 KB skb might be a header in one page plus payload fragments in four others; a user-space buffer spans many non-adjacent physical pages. Scatter-gather DMA lets one operation describe a list of (address, length) segments, and the device walks the list. Without it, the kernel would have to copy everything into a contiguous bounce buffer first.
The kernel DMA API
Drivers never hand raw physical addresses to devices. They use the DMA API, which handles IOMMU translation, cache coherence on non-coherent architectures, and bounce buffering for devices that can only address low memory:
#include <linux/dma-mapping.h>
/* Long-lived, coherent memory shared with the device: descriptor rings */
dma_addr_t ring_dma;
void *ring = dma_alloc_coherent(dev, RING_BYTES, &ring_dma, GFP_KERNEL);
/* write ring_dma into the device's ring-base register */
/* Streaming mapping for one transfer: a packet buffer */
dma_addr_t buf_dma = dma_map_single(dev, buf, len, DMA_FROM_DEVICE);
if (dma_mapping_error(dev, buf_dma))
return -ENOMEM;
/* put buf_dma in a descriptor, ring the doorbell, wait for completion */
dma_unmap_single(dev, buf_dma, len, DMA_FROM_DEVICE);
/* only now is it safe for the CPU to read buf */
/* Scatter-gather: map a list of pages in one go */
int nents = dma_map_sg(dev, sgl, nr_pages, DMA_TO_DEVICE);
The direction flag matters: on architectures without hardware cache coherence (many ARM SoCs), DMA_FROM_DEVICE mapping invalidates CPU cache lines so that the CPU doesn't read stale cached data after the device writes RAM, and DMA_TO_DEVICE flushes dirty lines so the device sees what the CPU wrote. x86 is cache-coherent for DMA, which is why x86-only driver bugs in this area tend to show up later on ARM.
The IOMMU
A bus-mastering device can, in principle, write anywhere in physical memory. A buggy driver, a malicious Thunderbolt peripheral, or a compromised device firmware could overwrite the kernel. The IOMMU (Intel VT-d, AMD-Vi, ARM SMMU) is an MMU for devices: each device gets its own page tables translating I/O virtual addresses (IOVAs) to physical addresses, and anything not mapped faults.
# Is the IOMMU on?
dmesg | grep -iE 'DMAR|IOMMU|AMD-Vi' | head
ls /sys/kernel/iommu_groups/ | wc -l
# Kernel cmdline options
# intel_iommu=on enable translation for all devices
# iommu=pt passthrough for host-owned devices, translation for VFIO-assigned ones
# iommu.strict=0 lazy IOTLB invalidation, faster, small safety window
What the IOMMU gives you:
- Isolation — a device can only DMA to buffers the driver mapped for it. Required for safely passing a device into a VM or a user-space process.
- Contiguous illusion — scattered physical pages can be mapped to a contiguous IOVA range.
- Addressing — a 32-bit-only device can reach memory above 4 GB through remapping.
- Interrupt remapping — MSI writes go through the IOMMU too, so a device assigned to a VM can't forge interrupts to arbitrary vectors.
What it costs: every map/unmap updates IOMMU page tables and requires IOTLB invalidation. On high packet-rate workloads with strict mode, IOMMU overhead used to be measured in tens of percent; modern kernels batch invalidations, and iommu=pt (identity mapping for host drivers) removes the cost for devices the host trusts. That is why many bare-metal servers run iommu=pt — it keeps the IOMMU available for VFIO while not translating ordinary host DMA.
Zero-copy
DMA removes the device ↔ kernel copy. The remaining copy is usually kernel ↔ user. Zero-copy techniques remove that too by making the device DMA directly into (or out of) pages the application can see:
| Technique | Direction | How |
|---|---|---|
sendfile(), splice() | File → socket | Page-cache pages handed to the NIC by reference |
MSG_ZEROCOPY, IORING_OP_SEND_ZC | User buffer → NIC | Pin user pages, DMA from them, notify when done |
O_DIRECT | Disk ↔ user buffer | DMA straight to/from the aligned user buffer, bypass page cache |
| io_uring registered buffers | Disk/NIC ↔ user | Pre-pinned, pre-mapped buffers, no per-I/O pin cost |
| AF_XDP, io_uring zcrx, DPDK | NIC → user | Device DMAs into user-registered memory |
| RDMA | Remote memory ↔ local memory | NIC DMAs directly between machines' user buffers |
The catch is always the same: the buffer must stay pinned and unmodified until the device is done, which means asynchronous completion notifications and more complex buffer lifecycle management. For small transfers (under ~10 KB), page pinning and notification cost more than just copying.
The Linux Device Driver Model
Three classic driver types
| Type | Interface to user space | Unit | Examples | Lives in |
|---|---|---|---|---|
| Character | /dev node, read/write/ioctl/mmap | Byte stream, device-defined | Terminals, /dev/null, /dev/random, GPUs (/dev/dri/*), /dev/kvm, /dev/vfio/* | drivers/char, many subsystems |
| Block | /dev node, used via file systems or raw I/O | Fixed-size sectors through the block layer (blk-mq) | NVMe (/dev/nvme0n1), SCSI/SATA (/dev/sda), virtio-blk (/dev/vda), loop | drivers/block, drivers/nvme |
| Network | No /dev node — interfaces (eth0) via sockets and netlink | Packets (sk_buff) | ixgbe, mlx5_core, ena, virtio_net | drivers/net |
Network drivers are the odd one out: you never open() a NIC. User space reaches them through the socket API and configures them with netlink (ip) and ethtool ioctls.
Each /dev node is identified by a major number (which driver) and a minor number (which instance):
ls -l /dev/nvme0n1 /dev/null /dev/sda
# brw-rw---- 1 root disk 259, 0 Nov 8 09:12 /dev/nvme0n1 <- b = block, major 259
# crw-rw-rw- 1 root root 1, 3 Nov 8 09:12 /dev/null <- c = char, major 1 minor 3
cat /proc/devices | head
Buses, devices, drivers, and sysfs
The kernel models hardware as buses (PCI, USB, platform, virtio), devices found on those buses, and drivers that claim devices by matching IDs. When a PCI device with vendor 8086 and device 1572 is discovered, the PCI core asks each registered PCI driver "do you handle this ID?"; the first match gets its probe() callback called, which maps BARs, sets up DMA, requests IRQs, and registers with the relevant subsystem (netdev, block, etc.).
All of this is visible in sysfs:
# Which driver owns a PCI device
lspci -k -s 3b:00.0
# 3b:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710
# Kernel driver in use: i40e
# Kernel modules: i40e
readlink /sys/class/net/eth0/device/driver
cat /sys/class/net/eth0/device/numa_node # which socket the NIC hangs off
cat /sys/block/nvme0n1/queue/scheduler # block layer scheduler for this device
# Manually unbind and bind (what VFIO setup does)
echo 0000:3b:00.0 | sudo tee /sys/bus/pci/drivers/i40e/unbind
udev and the /dev tree
/dev is a devtmpfs populated by the kernel itself, and udev (part of systemd) reacts to kernel uevents to set permissions, create stable symlinks, rename interfaces, and load firmware or modules. Persistent names like /dev/disk/by-uuid/... and predictable interface names like enp59s0f0 come from udev rules.
# Watch events as you plug in a USB device or hot-add a disk
udevadm monitor --kernel --udev
# All the properties udev knows for a device
udevadm info /dev/nvme0n1
# Example rule: stable name and permissions for a serial adapter
# /etc/udev/rules.d/99-gps.rules
SUBSYSTEM=="tty", ATTRS{idVendor}=="1546", ATTRS{idProduct}=="01a8", SYMLINK+="gps0", MODE="0660", GROUP="dialout"
Kernel modules
Most drivers are loadable modules (.ko files) loaded on demand when udev sees a matching device alias.
lsmod | head
modinfo ena # version, parameters, aliases, firmware
sudo modprobe nvme poll_queues=4 # load with a parameter
cat /sys/module/nvme/parameters/poll_queues
echo "options nvme poll_queues=4" | sudo tee /etc/modprobe.d/nvme.conf
A minimal character driver shows the shape — register a device, implement file_operations:
// hello_dev.c — /dev/hello returns a greeting on read
#include <linux/module.h>
#include <linux/miscdevice.h>
#include <linux/fs.h>
static const char msg[] = "hello from kernel space\n";
static ssize_t hello_read(struct file *f, char __user *buf, size_t len, loff_t *off)
{
return simple_read_from_buffer(buf, len, off, msg, sizeof(msg) - 1);
}
static const struct file_operations hello_fops = {
.owner = THIS_MODULE,
.read = hello_read,
};
static struct miscdevice hello_dev = {
.minor = MISC_DYNAMIC_MINOR,
.name = "hello", /* udev creates /dev/hello */
.fops = &hello_fops,
.mode = 0444,
};
module_misc_device(hello_dev);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Minimal misc character device");
# Makefile: obj-m += hello_dev.o
make -C /lib/modules/$(uname -r)/build M=$PWD modules
sudo insmod hello_dev.ko
cat /dev/hello
sudo rmmod hello_dev
A real hardware driver adds a bus driver (pci_driver with an ID table and probe/remove), BAR mapping (pci_iomap), DMA setup, and request_irq/pci_alloc_irq_vectors — but the structure is the same: register callbacks with a subsystem and let the kernel call you.
Drivers run with full kernel privilege
A driver bug is a kernel bug: a null pointer dereference is an oops or a panic, a buffer overflow is a privilege escalation, a DMA to the wrong address corrupts arbitrary memory (unless the IOMMU catches it). Drivers make up the majority of the kernel's source and historically a large share of its bugs. That is a big part of the motivation for Rust in the kernel, for user-space drivers, and for keeping the IOMMU on.
User-Space Drivers: UIO and VFIO
Sometimes you want a process, not the kernel, to drive a device: DPDK and SPDK for performance, QEMU for passing a physical device to a VM, or a vendor that ships a user-space stack.
| UIO | VFIO | |
|---|---|---|
| What it exposes | Device memory regions + an interrupt fd | Device regions, DMA mapping, interrupts, config space, reset |
| DMA safety | None — user space hands raw physical addresses to the device | IOMMU-enforced — process can only DMA to memory it mapped through VFIO |
| Interrupts | Legacy/MSI via read() on the fd | MSI/MSI-X via eventfds |
| Privilege needed | Effectively root, can corrupt any memory | Can be granted to an unprivileged user per IOMMU group |
| Use today | Legacy, embedded, cases without an IOMMU | DPDK, SPDK, QEMU/KVM device passthrough, cloud hypervisors |
# Hand a NIC to VFIO for DPDK or a VM
sudo modprobe vfio-pci
sudo dpdk-devbind.py --bind=vfio-pci 0000:3b:00.1
# or by hand:
echo 0000:3b:00.1 | sudo tee /sys/bus/pci/devices/0000:3b:00.1/driver/unbind
echo vfio-pci | sudo tee /sys/bus/pci/devices/0000:3b:00.1/driver_override
echo 0000:3b:00.1 | sudo tee /sys/bus/pci/drivers_probe
ls /dev/vfio/ # one node per IOMMU group, plus /dev/vfio/vfio
The unit of isolation is the IOMMU group — the set of devices that the hardware cannot isolate from each other (e.g. functions behind the same PCIe switch without ACS). You must assign a whole group, which is why some consumer motherboards make GPU passthrough painful.
How This Looks in Cloud VMs
Inside a virtual machine, "the NIC" and "the disk" are usually not real hardware. How they are virtualised determines most of the I/O performance gap between VMs and bare metal.
flowchart TD
subgraph EMU["Full emulation"]
G1["Guest e1000 driver"] -- "every register access traps" --> Q1["QEMU emulates device"]
end
subgraph PV["Paravirtual virtio"]
G2["Guest virtio-net driver"] -- "shared virtqueues, one notify per batch" --> V2["vhost-net in host kernel"]
end
subgraph SRIOV["SR-IOV or Nitro"]
G3["Guest ENA or mlx5 VF driver"] -- "direct DMA via IOMMU, posted interrupts" --> HW3["Physical NIC virtual function"]
end
Full emulation
The hypervisor emulates a real device (an Intel e1000 NIC, an IDE disk). Every register access from the guest driver is a VM exit — the CPU leaves the guest, the hypervisor emulates the access, and re-enters. VM exits cost roughly a microsecond or more each, and a single packet can need several. Compatible with any guest OS, painfully slow. Used for boot and legacy guests.
virtio (paravirtualisation)
The guest knows it is virtualised and uses a device designed for it. virtio devices (virtio-net, virtio-blk, virtio-scsi) communicate through virtqueues — descriptor rings in guest memory shared with the host, structurally the same as a real NIC's rings. The guest batches many buffers and sends one notification ("kick"); the host backend (QEMU, or vhost-net / vhost-user for speed) processes them and injects one interrupt back. Far fewer exits than emulation, but every packet still passes through the host's software path.
SR-IOV
Single Root I/O Virtualization lets one physical PCIe device present itself as a Physical Function (PF, for the host) plus many Virtual Functions (VFs) — lightweight PCIe functions, each with its own queues, MAC, and MSI-X vectors. A VF is passed into a guest with VFIO; the guest driver DMAs directly to and from guest memory through the IOMMU, with no hypervisor on the data path. Combined with posted interrupts (Intel APICv / AMD AVIC), which deliver device interrupts into a running guest without a VM exit, I/O performance approaches bare metal.
The trade-off: the VM is tied to specific hardware, which complicates live migration — part of why clouds that use SR-IOV-style designs build their own migration-friendly devices.
AWS Nitro and ENA (and peers)
AWS's Nitro system moves networking, EBS storage, and management off the host CPU onto dedicated Nitro cards. Guests see an ENA (Elastic Network Adapter) device for networking and NVMe devices for EBS volumes and instance storage — both are PCIe devices presented directly to the guest, SR-IOV style, with the virtualisation logic implemented in the Nitro card's hardware and firmware rather than in host software. That is why EBS volumes show up as /dev/nvme1n1 on Nitro instances, and why network performance scales up to 100–200 Gbit/s on larger instance types. Google Cloud (gVNIC), Azure (Accelerated Networking with Mellanox/MANA VFs), and others have analogous designs.
What this means operationally inside a cloud VM:
- All the earlier tuning still applies. ENA has multiple queues with MSI-X vectors, RSS, NAPI, adaptive coalescing.
ethtool -S eth0exposes ENA-specific counters, including allowance-exceeded counters (bw_in_allowance_exceeded,pps_allowance_exceeded,conntrack_allowance_exceeded) that tell you the cloud shaped your traffic, not the kernel. - Queue count is set by the instance type. Small instances get few queues, which caps how far RSS can spread packet work.
- "Steal" time (
%stealinmpstat,top) shows the hypervisor running something else while your vCPU wanted to run — which delays interrupt handling too. - Interrupts can still be expensive on older or emulated paths. If
lspcishows an emulated device or the driver ise1000, you are leaving a lot of performance on the table.
# What kind of devices does this VM have?
lspci | grep -iE 'ethernet|nvme|virtio'
ethtool -i eth0 # driver: ena / virtio_net / mlx5_core / gve
ethtool -S eth0 | grep allowance # ENA: cloud-side shaping
mpstat 1 | awk 'NR<4 || /all/' # watch %steal alongside %irq and %soft
Observability: Watching Interrupts
Which device, which CPU, how often
# Per-IRQ, per-CPU counts; diff twice to get rates
watch -d -n1 'grep -E "eth0|nvme|CPU" /proc/interrupts'
# Interrupt rate per second for one IRQ
irq=124; a=$(awk -v i="$irq:" '$1==i {s=0; for(n=2;n<=NF;n++) if($n~/^[0-9]+$/) s+=$n; print s}' /proc/interrupts)
sleep 1
b=$(awk -v i="$irq:" '$1==i {s=0; for(n=2;n<=NF;n++) if($n~/^[0-9]+$/) s+=$n; print s}' /proc/interrupts)
echo "IRQ $irq: $((b-a))/s"
# Softirq counts per CPU
watch -d -n1 cat /proc/softirqs
# CPU time split: %irq = hard interrupts, %soft = softirqs
mpstat -P ALL 1
# CPU %usr %nice %sys %iowait %irq %soft %steal %idle
# 0 4.1 0.0 3.2 0.0 2.1 61.8 0.0 28.8 <- hot core
# 1 22.0 0.0 6.1 0.0 0.0 1.2 0.0 70.7
vmstat 1 shows total interrupts per second in the in column and context switches in cs — a quick sanity check for interrupt storms.
Where the time goes
# Count hard IRQs by handler name over 10 s
sudo perf stat -e 'irq:irq_handler_entry' -a -- sleep 10
sudo perf record -e 'irq:irq_handler_entry' -a -- sleep 10 && sudo perf report --sort comm,sym
# Latency distribution of hard IRQ handlers, by IRQ name (bcc-tools)
sudo /usr/share/bcc/tools/hardirqs -d 10 1
# Time spent in each softirq type
sudo /usr/share/bcc/tools/softirqs -d 10 1
# bpftrace: histogram of time spent in each softirq vector, in µs
sudo bpftrace -e '
tracepoint:irq:softirq_entry { @start[cpu] = nsecs; }
tracepoint:irq:softirq_exit /@start[cpu]/ {
@us[args->vec] = hist((nsecs - @start[cpu]) / 1000);
delete(@start[cpu]);
}'
# Which functions run in interrupt context the most
sudo perf top -C 0 # profile the hot core
The softirq vector numbers in that bpftrace output follow the /proc/softirqs order: 0 HI, 1 TIMER, 2 NET_TX, 3 NET_RX, 4 BLOCK, 5 IRQ_POLL, 6 TASKLET, 7 SCHED, 8 HRTIMER, 9 RCU.
Interrupt affinity
# Where IRQ 124 is allowed to go, and where it actually went
cat /proc/irq/124/smp_affinity_list
cat /proc/irq/124/effective_affinity_list
# Pin it
echo 4 | sudo tee /proc/irq/124/smp_affinity_list
# Keep general IRQs off latency-critical cores (irqbalance honours this)
# /etc/sysconfig/irqbalance or /etc/default/irqbalance:
# IRQBALANCE_BANNED_CPULIST=8-15
# Kernel cmdline alternatives: irqaffinity=0-7, isolcpus=, nohz_full=
Many modern drivers (NVMe, mlx5) use managed interrupts: the kernel spreads vectors across CPUs automatically at probe time and won't let you move them. That's by design — the queue-to-CPU mapping is part of how the driver avoids locking — and those IRQs show "managed" behaviour by rejecting writes to smp_affinity.
Failure Modes
| Symptom | Likely cause | Check | Fix |
|---|---|---|---|
One core at ~100% %soft or %irq, others idle | All device vectors on one CPU, or a single RX queue | /proc/interrupts, mpstat -P ALL | Spread IRQ affinity, enable irqbalance, more queues, RPS |
ksoftirqd/N high in top, app latency spikes on that core | Softirq overload deferring to thread | /proc/softirqs, softnet_stat | Spread load, reduce per-packet work, move app threads off that core |
irq N: nobody cared ... Disabling IRQ in dmesg | Interrupt storm, unclaimed shared line | dmesg, /proc/interrupts rate | Fix driver/firmware/hardware; check for legacy INTx sharing |
| High tail latency on an idle-ish server | Deep C-state exit latency, coalescing delay | turbostat, cpupower idle-info, ethtool -c | Cap C-states, reduce coalescing, busy polling |
DMA errors, DMAR: [DMA Read] Request device ... fault addr | Device DMA to an unmapped IOVA — driver bug or firmware bug | dmesg | Update driver/firmware; never "fix" by disabling the IOMMU in production without understanding why |
| Throughput drop after enabling IOMMU | Map/unmap and IOTLB invalidation overhead | perf top showing iommu functions | iommu=pt for host devices, newer kernel, larger batched mappings |
Many TLB shootdown IPIs | Heavy munmap/mprotect churn across many threads | /proc/interrupts TLB line | Fewer threads touching the mapping, avoid churny allocators, hugepages |
| VM network throughput much lower than expected | Emulated or plain virtio device, cloud allowance limits | ethtool -i, ethtool -S allowance counters, %steal | Use the enhanced/SR-IOV NIC type, larger instance, spread flows |
| NVMe IOPS lower than spec at QD1 | Interrupt + wakeup latency dominates | fio with and without --hipri | Poll queues + IOPOLL for the hot path, if a core can be spared |
| Kernel oops in a driver after an update | Driver bug | dmesg, journalctl -k | Roll back module, report upstream, pin kernel version in the fleet |
Decision Guide
flowchart TD
Q["How should this device notify the CPU?"] --> A{"Event rate?"}
A -- "Low, under ~10k per second" --> I["Interrupts, coalescing off or adaptive"]
A -- "High" --> B{"Can you dedicate a core?"}
B -- "No" --> N["Hybrid: NAPI or adaptive coalescing, spread vectors with MSI-X affinity"]
B -- "Yes" --> C{"Need kernel features - TCP stack, file systems, tooling?"}
C -- "Yes" --> P["Kernel polling: busy_poll, NVMe poll queues, io_uring IOPOLL and SQPOLL"]
C -- "No" --> U["User-space polling: DPDK or SPDK via VFIO"]
Practical defaults:
- Use MSI-X with one vector per queue per CPU. It is the default for every serious modern device; if
lspci -vvshows INTx, something is wrong. - Keep adaptive coalescing on unless you have measured that fixed values help your workload.
- Spread interrupt vectors across the cores of the device's NUMA node. Either let
irqbalancedo it or pin deliberately — not both. - Isolate latency-critical application cores from interrupt work (
irqaffinity=, irqbalance banned CPUs) when p99 matters more than total throughput. - Keep the IOMMU on, use
iommu=ptif host-side translation overhead is measurable. - Prefer VFIO over UIO for user-space drivers; UIO has no DMA protection.
- In the cloud, pick instance types with the enhanced NIC and NVMe storage, and watch the provider's allowance counters before blaming the kernel.
- Poll only where the latency win is measurable and the burned core is affordable.
Interview Questions
What's the difference between an interrupt, an exception, and a trap?
A hardware interrupt is asynchronous, from a device, through the interrupt controller. An exception is synchronous, raised by the CPU because of the current instruction (page fault, divide by zero). A trap is an exception triggered deliberately by an instruction (int3, legacy int 0x80). All three are dispatched through the IDT on x86.
Why split interrupt handling into a top half and bottom half? The top half runs with interrupts masked, in no process context, and cannot sleep — every microsecond it runs delays other interrupts on that core. So it does the minimum (acknowledge, capture state) and defers the rest to softirqs, workqueues, or a threaded IRQ, which run with interrupts enabled and, in the workqueue/thread case, can sleep.
What is MSI-X and why does it matter for performance? Interrupts delivered as memory writes rather than physical lines, with up to 2048 independent vectors per device, each targetable to a specific CPU. It lets a multi-queue NIC or NVMe drive have one interrupt per queue per core, so completions are handled on the core that owns the queue — no shared lines, no cross-core locking.
When is polling better than interrupts? When events are frequent enough that interrupt overhead dominates, or latency requirements are tighter than interrupt-plus-wakeup latency. NAPI and NVMe polling are hybrids; DPDK/SPDK poll permanently on dedicated cores. At low event rates polling wastes a core.
What does the IOMMU protect against, and what does it cost?
It restricts each device to DMA only into memory explicitly mapped for it, preventing buggy or malicious devices from writing arbitrary memory, and it makes safe device passthrough to VMs and user-space drivers possible. The cost is map/unmap and IOTLB invalidation overhead, which matters at very high I/O rates; iommu=pt and batching reduce it.
Why are cloud VMs with SR-IOV or Nitro NICs so much faster than emulated NICs? Emulated devices cause a VM exit for every register access; virtio batches but still goes through host software. SR-IOV and Nitro give the guest a real PCIe function that DMAs directly into guest memory through the IOMMU, with posted interrupts avoiding exits — the hypervisor is off the data path.
Related
- The Linux Networking Stack — NAPI, softirqs, and RSS applied to packets end to end
- I/O & Storage Internals — epoll, io_uring, and the block layer above these devices
- CPU Scheduling — how ksoftirqd, kworkers, and IRQ threads compete with your code
- Syscalls & Kernel Internals — the other way into the kernel
- Virtual Memory — page tables, which the IOMMU mirrors for devices
- eBPF — tracing interrupt and driver paths without recompiling the kernel