On this page

Interrupts, DMA & Device Drivers

The CPU is fast and alone. Everything else — disks, NICs, GPUs, keyboards, timers — is a separate piece of hardware running at its own pace. The kernel needs two things from that hardware: a way for the device to say "something happened" without the CPU constantly asking, and a way to move data between device and memory without the CPU copying every byte. Those two mechanisms are interrupts and DMA. The code that knows how to speak to a particular piece of hardware through them is a device driver.

These sound like kernel-developer concerns, but they leak into every production system. A database whose p99 jumps when one core hits 100% %soft. A VM that loses 30% of its network throughput after a live migration to a host without SR-IOV. An NVMe-backed service that gets 2x more IOPS when you switch to polling. A "noisy" core that turns out to be handling every interrupt for every queue of a 100 GbE NIC. Knowing how interrupts and DMA work tells you where to look.

Where this fits

This page is the hardware layer under The Linux Networking Stack and I/O & Storage Internals. The CPU-time side — how interrupt work competes with your threads — connects to CPU Scheduling. The privilege boundary these mechanisms rely on is described in Syscalls & Kernel Internals.


Why Interrupts Exist: The Polling Problem

Suppose a NIC receives a packet. How does the CPU find out?

Option 1: polling. The CPU reads a device status register in a loop. Simple, and for a device that is always busy it is actually optimal — no wasted work. But a device that is idle 99.9% of the time turns a CPU core into a space heater, and if you poll every millisecond instead of continuously, you add up to a millisecond of latency to every event.

Option 2: interrupts. The device sends a signal; the CPU stops what it's doing, runs a handler, and resumes. Zero cost while idle, immediate reaction when something happens. But each interrupt has a fixed cost — saving state, jumping into the kernel, polluting caches and branch predictors, returning — and at very high event rates that fixed cost dominates.

Every design on this page is a point on that trade-off curve:

Event rateBest mechanismWhy
Rare (keyboard, timer, idle NIC)Pure interruptsNo CPU spent waiting
Moderate (busy disk, loaded NIC)Interrupts + coalescingBatch several events per interrupt
Very high (10–100 GbE, NVMe at 1M IOPS)Interrupt to start, poll while busy (NAPI, blk-mq polling)Amortise interrupt cost over hundreds of events
Extreme, latency-criticalPure polling on dedicated cores (DPDK, SPDK, io_uring SQPOLL)Zero interrupt latency, one core burned permanently

Three Kinds of "Interrupt"

The word covers three different things that share the same CPU machinery.

KindSourceSynchronous?Examples
Hardware interrupt (IRQ)External device via the interrupt controllerNo — arrives at any pointNIC packet received, disk I/O complete, timer tick
ExceptionThe CPU itself, caused by the instruction it is executingYes — tied to a specific instructionPage fault (#PF), divide by zero (#DE), general protection fault (#GP), invalid opcode (#UD), breakpoint (#BP)
Software interrupt (trap)An explicit instructionYesint 0x80 (legacy 32-bit syscalls), int3 (debugger breakpoint)

Then there are two special cases:

  • NMI (Non-Maskable Interrupt) cannot be disabled. Linux uses it for the hard-lockup watchdog and for perf sampling, which is how perf can profile code that runs with interrupts disabled.
  • IPI (Inter-Processor Interrupt) is one CPU interrupting another: "reschedule", "flush your TLB" (after munmap or mprotect), "run this function". TLB shootdown IPIs are a real cost on many-core machines with heavy memory mapping churn.

And separately, softirqs — despite the name — are not hardware interrupts at all. They are a kernel mechanism for deferred work, covered below.

Syscalls on x86-64 don't use interrupts

Modern 64-bit Linux enters the kernel with the syscall instruction, which jumps to an address stored in a model-specific register (MSR_LSTAR) without consulting the interrupt table. It is significantly faster than int 0x80. Exceptions and hardware interrupts still go through the IDT.


The Interrupt Descriptor Table

On x86, the CPU finds the handler for any interrupt or exception through the Interrupt Descriptor Table (IDT): an array of 256 entries, each pointing to a handler and specifying the privilege level and stack to use. The IDTR register holds its address.

Vector rangeUse
0–31Reserved for CPU exceptions: 0 divide error, 3 breakpoint, 6 invalid opcode, 8 double fault, 13 general protection, 14 page fault, 18 machine check
32–255Available for external interrupts, IPIs, and the local APIC timer — the kernel allocates these to devices
flowchart LR
    DEV["Device - NIC, NVMe, timer"] -- "MSI write or IRQ line" --> APIC["Local APIC of target CPU"]
    APIC -- "vector number 32-255" --> CPU["CPU core"]
    EXC["Faulting instruction"] -- "vector 0-31" --> CPU
    CPU --> IDT["IDT lookup via IDTR"]
    IDT --> ENTRY["Kernel entry stub: save registers, switch stack"]
    ENTRY --> H["Generic IRQ code then driver handler"]
    H --> RET["Raise softirqs if needed, IRET back to interrupted code"]

When an interrupt arrives, the CPU (in hardware): finishes the current instruction, saves the instruction pointer, flags, and stack pointer onto the kernel stack (switching from the user stack if the CPU was in ring 3), clears the interrupt-enable flag if the gate type requires it, and jumps to the handler. The kernel's entry stub saves the remaining registers, figures out which device the vector belongs to, and calls the driver's registered handler.

On ARM the equivalent is the exception vector table plus the GIC (Generic Interrupt Controller); the concepts — vectors, priority, per-CPU routing — map across directly.

The interrupt controller

Old PCs had a single 8259 PIC with 15 lines. Modern x86 machines have a Local APIC in every core (handling timers, IPIs, and delivery to that core) plus one or more I/O APICs that route legacy wired interrupt lines to cores. But modern PCIe devices don't use wires at all.


MSI and MSI-X: Interrupts as Memory Writes

Message Signaled Interrupts replace physical interrupt lines with a memory write. The device writes a specific value to a specific address (in the 0xFEExxxxx range on x86); the chipset recognises that address and turns it into an interrupt delivered to a specific core with a specific vector.

Legacy INTxMSIMSI-X
MechanismShared physical wireMemory writeMemory write
Vectors per device1 (shared with other devices)Up to 32, contiguousUp to 2048, independent
Per-vector CPU targetingNoLimitedYes — each vector has its own address/data
Sharing with other devicesCommon — every handler must check "was it me?"NoNo

MSI-X is what makes modern high-performance I/O possible. A 32-queue NIC gets 32+ vectors; an NVMe drive gets one per submission/completion queue pair, typically one per CPU. Each queue's interrupt can be pinned to a different core, so completions are handled on the core that issued the I/O — good for cache locality and no cross-core locking.

# Which interrupt mode a device uses
sudo lspci -vv -s 0000:3b:00.0 | grep -E 'MSI|MSI-X'
#   Capabilities: [a0] MSI-X: Enable+ Count=64 Masked-

# All interrupt vectors, per-CPU counts
cat /proc/interrupts | head -5
#            CPU0       CPU1       CPU2       CPU3
#   0:         35          0          0          0  IR-IO-APIC    2-edge      timer
# 124:   88214321          0          0          0  IR-PCI-MSIX-0000:3b:00.0  1-edge  eth0-TxRx-0
# 125:          0   90123554          0          0  IR-PCI-MSIX-0000:3b:00.0  2-edge  eth0-TxRx-1
# 140:     412880     409117     411203     410998  IR-PCI-MSIX-0000:5e:00.0  0-edge  nvme0q0

Note the bottom of /proc/interrupts too: lines like LOC (local timer), RES (rescheduling IPIs), CAL (function-call IPIs), TLB (TLB shootdowns), and NMI. A high TLB rate points at memory-mapping churn; a high RES rate points at threads constantly waking each other across cores.


Top Half and Bottom Half

An interrupt handler runs with that interrupt line (and often all interrupts on that CPU) disabled, on whatever stack and context it interrupted, and cannot sleep — there is no process to put to sleep, so taking a mutex, allocating memory with GFP_KERNEL, or waiting for I/O is forbidden. While it runs, other interrupts on that core wait. Long handlers mean latency for everyone.

So Linux splits interrupt handling in two:

  • Top half (the hard IRQ handler): acknowledge the device, grab the minimum state, schedule the rest. Microseconds or less.
  • Bottom half: everything else — protocol processing, waking processes, completing requests — run later with interrupts enabled.
flowchart TD
    IRQ["Hardware IRQ arrives"] --> TOP["Top half: ack device, read status, mask queue"]
    TOP --> CHOICE{"How is deferred work scheduled?"}
    CHOICE -- "high rate, per-CPU, cannot sleep" --> SIRQ["Softirq - NET_RX, BLOCK, TIMER, RCU"]
    CHOICE -- "driver-specific, cannot sleep" --> TL["Tasklet - built on softirq, being phased out"]
    CHOICE -- "needs to sleep, process context" --> WQ["Workqueue - kworker threads"]
    CHOICE -- "whole handler in a thread" --> TH["Threaded IRQ - irq/N-name kernel thread"]
    SIRQ --> KSD["If overloaded: ksoftirqd/N thread"]

Softirqs

A fixed set of ten statically defined, per-CPU deferred handlers, run on the way out of a hard IRQ, when bottom halves are re-enabled, or in the ksoftirqd/N thread:

cat /proc/softirqs
#                     CPU0       CPU1       CPU2       CPU3
#           HI:          1          0          0          2
#        TIMER:   12011837   10988771   11203347   11455201
#       NET_TX:      41823      38211      40017      39866
#       NET_RX:  981234511   12003321    8812211    9002118   <- CPU0 is doing all the packet work
#        BLOCK:    2203411    2190033    2210988    2187655
#     IRQ_POLL:          0          0          0          0
#      TASKLET:      12201       8834       9102       9311
#        SCHED:    8812331    8711234    8698772    8733901
#      HRTIMER:       1123        998       1045       1011
#          RCU:    9921387    9877312    9902211    9888714

The same softirq can run concurrently on several CPUs, so handlers must use per-CPU data or locking. Softirqs cannot sleep. If they keep re-raising themselves (a packet flood), the kernel stops running them inline after a bounded time and wakes ksoftirqd/N so that the scheduler can balance packet processing against user threads. Networking's NET_RX softirq — and its NAPI polling loop — is walked through in detail in The Linux Networking Stack.

Tasklets

Tasklets were a dynamically allocatable layer on top of softirqs, with one guarantee: a given tasklet never runs on two CPUs at once, which made them easier to write. They have been deprecated in spirit for years, and recent kernels (6.9+) added BH workqueues — workqueues that execute in softirq context — as the intended replacement. New drivers should not use tasklets.

Workqueues

When deferred work needs to sleep — take a mutex, allocate memory that may block, talk to a slow bus like I2C — it goes into a workqueue, executed by kworker/N:M kernel threads in normal process context. Workqueues are scheduled like any other thread, so their latency depends on the scheduler.

Threaded IRQs

request_threaded_irq() registers two functions: a tiny hard-IRQ handler that only checks "is this mine?" and masks the device, and a thread function that runs the rest in a dedicated irq/N-name kernel thread. That thread has a priority (SCHED_FIFO 50 by default), can be pinned and reprioritised with chrt and taskset, and can sleep.

The PREEMPT_RT real-time patch set — merged into mainline in 6.12 — forces nearly all interrupt handlers into threads. That is how it achieves bounded latency: a high-priority real-time task can preempt interrupt processing that it doesn't care about. You can get the same forced-threading behaviour on a normal kernel with the threadirqs boot parameter.

MechanismContextCan sleepConcurrencyTypical use
Hard IRQ (top half)InterruptNoOne per IRQ line at a timeAck device, schedule bottom half
SoftirqSoftirqNoSame softirq on many CPUs at onceNetworking, block completion, timers, RCU
TaskletSoftirqNoSerialised per taskletLegacy drivers
BH workqueueSoftirqNoWorkqueue semanticsTasklet replacement (6.9+)
WorkqueueProcess (kworker)YesConfigurableDriver housekeeping, slow buses, firmware loading
Threaded IRQProcess (irq/N thread)YesOne thread per IRQRT systems, I2C/SPI devices, slow handlers

Interrupt Cost and Latency

What does one interrupt actually cost? The direct cost — vector delivery, register save, handler dispatch, IRET — is on the order of a few hundred nanoseconds to a microsecond on modern x86. The indirect cost is usually bigger: the interrupted code loses L1/L2 cache contents and branch-predictor state to the handler, and pays for that in cache misses for several microseconds afterward.

EventRough cost on modern x86
Hardware interrupt, direct entry/exit~0.3–1 µs
Interrupt including cache and pipeline disruption~1–5 µs effective
IPI from one core to another~1–2 µs to deliver and handle
Syscall entry/exit (with KPTI mitigations)~0.1–0.3 µs
Thread context switch~2–5 µs including cache effects
Wakeup latency: interrupt → blocked thread running~5–50 µs depending on load and C-states
Exit from deep CPU C-state (C6)~50–100+ µs
Busy-poll check of a device ring in L1/L2~10–100 ns

That last two rows explain a lot of real latency. An idle CPU that has dropped into a deep C-state needs tens of microseconds just to wake up before it can run the interrupt handler. Latency-sensitive systems often cap C-states (intel_idle.max_cstate=1, processor.max_cstate=1, or the /dev/cpu_dma_latency interface) at the price of power.

Interrupt latency vs interrupt overhead

  • Latency — time from the device raising the interrupt to useful work happening. Hurt by: interrupts disabled for long stretches, deep C-states, the handler's CPU being busy with another softirq, the woken thread waiting in a run queue.
  • Overhead — total CPU time spent servicing interrupts. Hurt by: high event rates without coalescing or NAPI.

You usually tune one at the expense of the other. Coalescing reduces overhead and adds latency. Polling removes latency and costs a core.

Interrupt storms

An interrupt storm is a device firing interrupts faster than the CPU can handle them — from a hardware fault, a misconfigured level-triggered line that is never deasserted, or a firmware bug. The CPU spends 100% of its time in the handler and nothing else runs on that core. Linux has a safety valve: if a shared IRQ line keeps firing and no handler claims it, the kernel eventually logs

irq 16: nobody cared (try booting with the "irqpoll" option)
Disabling IRQ #16

and turns the line off. That stops the storm and also breaks whatever device was on it. Seeing this message means "investigate the hardware or the driver", not "add irqpoll".

A softer, more common version is an IRQ hotspot: every queue's interrupt lands on CPU 0 because affinity was never set, and CPU 0 shows 100% %irq+%soft while the other cores idle.

War Story (illustrative)

A team migrated a Kafka broker fleet to new instances with twice the cores and saw worse tail latency. mpstat -P ALL 1 showed CPU 0 at 95% %soft, every other core under 20%. The image had irqbalance disabled and the new NIC driver created 8 queues, but all 8 MSI-X vectors still had the default affinity mask, and on that platform the effective target was CPU 0. Spreading the vectors across the cores of the NIC's NUMA node — one echo per IRQ into smp_affinity_list — dropped p99 produce latency by more than half. Nothing in the application had changed.


Interrupt Coalescing

Coalescing (also called moderation) tells the device: don't interrupt for every event; wait until N events have accumulated or T microseconds have passed, whichever comes first.

# Current settings
ethtool -c eth0
# Adaptive RX: on  TX: on
# rx-usecs: 50
# rx-frames: 0
# tx-usecs: 50

# Fixed coalescing: interrupt after 64 frames or 100 µs
sudo ethtool -C eth0 adaptive-rx off rx-usecs 100 rx-frames 64

# Latency-first: interrupt immediately
sudo ethtool -C eth0 adaptive-rx off rx-usecs 0 rx-frames 1

# Let the driver adapt to traffic (Linux DIM on many drivers)
sudo ethtool -C eth0 adaptive-rx on adaptive-tx on
SettingInterrupt rateAdded latencyCPU overheadGood for
rx-usecs 0Up to one per packet~0HighLow-rate latency-critical traffic
rx-usecs 50–100Bounded ~10–20k/s per queueUp to 50–100 µsLowBulk throughput
Adaptive (DIM)Varies with loadLow at low load, higher under loadBalancedMost servers

Adaptive coalescing — the kernel's Dynamic Interrupt Moderation (DIM) library, used by mlx5, ENA, and many others — watches packet rate and bytes per interrupt and moves the settings automatically. It is the right default unless you have measured a reason to override it.

NVMe has the same idea in its spec (aggregation threshold and time), but Linux typically leaves NVMe coalescing off and relies on per-CPU queues plus the block layer's batching.


Polling vs Interrupts in Practice

Linux has converged on hybrids that interrupt when idle and poll when busy.

NAPI (networking)

One interrupt arrives; the driver masks the queue's interrupt and schedules a softirq poll; the poll processes up to a budget of packets per run and keeps the interrupt masked as long as there is work. When the ring is drained, it re-enables the interrupt. Under load, interrupt rate falls to near zero while throughput rises. Details are in The Linux Networking Stack.

Block layer polling (NVMe)

A fast NVMe SSD completes a 4 KB read in ~10–80 µs; low-latency devices (Optane-class, some SLC drives) in under 10 µs. At that scale, interrupt delivery + wakeup + context switch (5–20 µs in total) is a significant fraction of the I/O latency. Linux supports polled I/O on dedicated poll queues:

# Allocate poll queues in the NVMe driver (module parameter, set at load/boot)
cat /sys/module/nvme/parameters/poll_queues
# Kernel cmdline: nvme.poll_queues=4

# fio comparison: interrupt-driven vs polled completions
fio --name=irq  --filename=/dev/nvme0n1 --direct=1 --rw=randread --bs=4k \
    --ioengine=io_uring --iodepth=1 --runtime=30 --time_based
fio --name=poll --filename=/dev/nvme0n1 --direct=1 --rw=randread --bs=4k \
    --ioengine=io_uring --hipri=1 --iodepth=1 --runtime=30 --time_based

--hipri maps to io_uring's IORING_SETUP_IOPOLL: instead of sleeping until a completion interrupt, the submitting thread spins on the completion queue. Typical results on a fast drive: a few microseconds shaved off per I/O at queue depth 1, at the cost of 100% CPU on the polling thread. At high queue depths the gain shrinks because interrupts are already amortised.

io_uring SQPOLL

IORING_SETUP_SQPOLL creates a kernel thread that polls the submission queue, so the application can submit I/O by writing to shared memory with zero syscalls. Combined with IOPOLL on the completion side you get a fully polled path: no syscalls, no interrupts. SQPOLL threads go to sleep after an idle timeout (sq_thread_idle), so the cost is bounded when traffic stops. The io_uring APIs themselves are described in I/O & Storage Internals.

Full user-space polling

DPDK (network) and SPDK (storage) take the device away from the kernel, map it into a process, and poll from dedicated cores that never sleep. Lowest latency and highest throughput available — at the cost of a permanently busy core per poller and losing kernel tooling.

flowchart LR
    A["Pure interrupts"] --> B["Interrupt coalescing"]
    B --> C["NAPI or hybrid polling"]
    C --> D["io_uring IOPOLL and SQPOLL"]
    D --> E["DPDK or SPDK user-space polling"]
    A -. "lowest CPU when idle" .-> A
    E -. "lowest latency, burns cores" .-> E

DMA: Letting Devices Write Memory Directly

Copying a 64 KB network buffer one word at a time through the CPU would cost tens of microseconds and evict useful data from cache. Direct Memory Access lets the device read and write host RAM itself.

Bus mastering

On PCIe, any device can become a bus master and issue memory read/write transactions to host addresses. The driver's job is to tell the device which addresses to use. The CPU sets up the transfer, the device moves the data, and an interrupt signals completion.

sequenceDiagram
    participant Drv as Driver on CPU
    participant Mem as Host RAM
    participant Dev as PCIe device
    Drv->>Mem: allocate buffer, map it for DMA
    Drv->>Mem: write descriptor with buffer bus address
    Drv->>Dev: MMIO doorbell write - new work queued
    Dev->>Mem: DMA read descriptor
    Dev->>Mem: DMA write data into buffer
    Dev->>Mem: DMA write completion status
    Dev->>Drv: MSI-X interrupt
    Drv->>Mem: unmap buffer, hand data up the stack

Descriptor rings and scatter-gather

Devices and drivers communicate through descriptor rings — circular arrays in host memory, one entry per buffer, each holding an address, a length, and status flags. NVMe has submission and completion queues; NICs have RX and TX rings; virtio has virtqueues. Same idea everywhere.

Real data is rarely physically contiguous: a 64 KB skb might be a header in one page plus payload fragments in four others; a user-space buffer spans many non-adjacent physical pages. Scatter-gather DMA lets one operation describe a list of (address, length) segments, and the device walks the list. Without it, the kernel would have to copy everything into a contiguous bounce buffer first.

The kernel DMA API

Drivers never hand raw physical addresses to devices. They use the DMA API, which handles IOMMU translation, cache coherence on non-coherent architectures, and bounce buffering for devices that can only address low memory:

#include <linux/dma-mapping.h>

/* Long-lived, coherent memory shared with the device: descriptor rings */
dma_addr_t ring_dma;
void *ring = dma_alloc_coherent(dev, RING_BYTES, &ring_dma, GFP_KERNEL);
/* write ring_dma into the device's ring-base register */

/* Streaming mapping for one transfer: a packet buffer */
dma_addr_t buf_dma = dma_map_single(dev, buf, len, DMA_FROM_DEVICE);
if (dma_mapping_error(dev, buf_dma))
    return -ENOMEM;
/* put buf_dma in a descriptor, ring the doorbell, wait for completion */
dma_unmap_single(dev, buf_dma, len, DMA_FROM_DEVICE);
/* only now is it safe for the CPU to read buf */

/* Scatter-gather: map a list of pages in one go */
int nents = dma_map_sg(dev, sgl, nr_pages, DMA_TO_DEVICE);

The direction flag matters: on architectures without hardware cache coherence (many ARM SoCs), DMA_FROM_DEVICE mapping invalidates CPU cache lines so that the CPU doesn't read stale cached data after the device writes RAM, and DMA_TO_DEVICE flushes dirty lines so the device sees what the CPU wrote. x86 is cache-coherent for DMA, which is why x86-only driver bugs in this area tend to show up later on ARM.

The IOMMU

A bus-mastering device can, in principle, write anywhere in physical memory. A buggy driver, a malicious Thunderbolt peripheral, or a compromised device firmware could overwrite the kernel. The IOMMU (Intel VT-d, AMD-Vi, ARM SMMU) is an MMU for devices: each device gets its own page tables translating I/O virtual addresses (IOVAs) to physical addresses, and anything not mapped faults.

# Is the IOMMU on?
dmesg | grep -iE 'DMAR|IOMMU|AMD-Vi' | head
ls /sys/kernel/iommu_groups/ | wc -l

# Kernel cmdline options
#   intel_iommu=on          enable translation for all devices
#   iommu=pt                passthrough for host-owned devices, translation for VFIO-assigned ones
#   iommu.strict=0          lazy IOTLB invalidation, faster, small safety window

What the IOMMU gives you:

  • Isolation — a device can only DMA to buffers the driver mapped for it. Required for safely passing a device into a VM or a user-space process.
  • Contiguous illusion — scattered physical pages can be mapped to a contiguous IOVA range.
  • Addressing — a 32-bit-only device can reach memory above 4 GB through remapping.
  • Interrupt remapping — MSI writes go through the IOMMU too, so a device assigned to a VM can't forge interrupts to arbitrary vectors.

What it costs: every map/unmap updates IOMMU page tables and requires IOTLB invalidation. On high packet-rate workloads with strict mode, IOMMU overhead used to be measured in tens of percent; modern kernels batch invalidations, and iommu=pt (identity mapping for host drivers) removes the cost for devices the host trusts. That is why many bare-metal servers run iommu=pt — it keeps the IOMMU available for VFIO while not translating ordinary host DMA.

Zero-copy

DMA removes the device ↔ kernel copy. The remaining copy is usually kernel ↔ user. Zero-copy techniques remove that too by making the device DMA directly into (or out of) pages the application can see:

TechniqueDirectionHow
sendfile(), splice()File → socketPage-cache pages handed to the NIC by reference
MSG_ZEROCOPY, IORING_OP_SEND_ZCUser buffer → NICPin user pages, DMA from them, notify when done
O_DIRECTDisk ↔ user bufferDMA straight to/from the aligned user buffer, bypass page cache
io_uring registered buffersDisk/NIC ↔ userPre-pinned, pre-mapped buffers, no per-I/O pin cost
AF_XDP, io_uring zcrx, DPDKNIC → userDevice DMAs into user-registered memory
RDMARemote memory ↔ local memoryNIC DMAs directly between machines' user buffers

The catch is always the same: the buffer must stay pinned and unmodified until the device is done, which means asynchronous completion notifications and more complex buffer lifecycle management. For small transfers (under ~10 KB), page pinning and notification cost more than just copying.


The Linux Device Driver Model

Three classic driver types

TypeInterface to user spaceUnitExamplesLives in
Character/dev node, read/write/ioctl/mmapByte stream, device-definedTerminals, /dev/null, /dev/random, GPUs (/dev/dri/*), /dev/kvm, /dev/vfio/*drivers/char, many subsystems
Block/dev node, used via file systems or raw I/OFixed-size sectors through the block layer (blk-mq)NVMe (/dev/nvme0n1), SCSI/SATA (/dev/sda), virtio-blk (/dev/vda), loopdrivers/block, drivers/nvme
NetworkNo /dev node — interfaces (eth0) via sockets and netlinkPackets (sk_buff)ixgbe, mlx5_core, ena, virtio_netdrivers/net

Network drivers are the odd one out: you never open() a NIC. User space reaches them through the socket API and configures them with netlink (ip) and ethtool ioctls.

Each /dev node is identified by a major number (which driver) and a minor number (which instance):

ls -l /dev/nvme0n1 /dev/null /dev/sda
# brw-rw---- 1 root disk 259, 0 Nov  8 09:12 /dev/nvme0n1   <- b = block, major 259
# crw-rw-rw- 1 root root   1, 3 Nov  8 09:12 /dev/null      <- c = char, major 1 minor 3
cat /proc/devices | head

Buses, devices, drivers, and sysfs

The kernel models hardware as buses (PCI, USB, platform, virtio), devices found on those buses, and drivers that claim devices by matching IDs. When a PCI device with vendor 8086 and device 1572 is discovered, the PCI core asks each registered PCI driver "do you handle this ID?"; the first match gets its probe() callback called, which maps BARs, sets up DMA, requests IRQs, and registers with the relevant subsystem (netdev, block, etc.).

All of this is visible in sysfs:

# Which driver owns a PCI device
lspci -k -s 3b:00.0
# 3b:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710
#   Kernel driver in use: i40e
#   Kernel modules: i40e

readlink /sys/class/net/eth0/device/driver
cat /sys/class/net/eth0/device/numa_node     # which socket the NIC hangs off
cat /sys/block/nvme0n1/queue/scheduler       # block layer scheduler for this device

# Manually unbind and bind (what VFIO setup does)
echo 0000:3b:00.0 | sudo tee /sys/bus/pci/drivers/i40e/unbind

udev and the /dev tree

/dev is a devtmpfs populated by the kernel itself, and udev (part of systemd) reacts to kernel uevents to set permissions, create stable symlinks, rename interfaces, and load firmware or modules. Persistent names like /dev/disk/by-uuid/... and predictable interface names like enp59s0f0 come from udev rules.

# Watch events as you plug in a USB device or hot-add a disk
udevadm monitor --kernel --udev

# All the properties udev knows for a device
udevadm info /dev/nvme0n1

# Example rule: stable name and permissions for a serial adapter
# /etc/udev/rules.d/99-gps.rules
SUBSYSTEM=="tty", ATTRS{idVendor}=="1546", ATTRS{idProduct}=="01a8", SYMLINK+="gps0", MODE="0660", GROUP="dialout"

Kernel modules

Most drivers are loadable modules (.ko files) loaded on demand when udev sees a matching device alias.

lsmod | head
modinfo ena                        # version, parameters, aliases, firmware
sudo modprobe nvme poll_queues=4   # load with a parameter
cat /sys/module/nvme/parameters/poll_queues
echo "options nvme poll_queues=4" | sudo tee /etc/modprobe.d/nvme.conf

A minimal character driver shows the shape — register a device, implement file_operations:

// hello_dev.c — /dev/hello returns a greeting on read
#include <linux/module.h>
#include <linux/miscdevice.h>
#include <linux/fs.h>

static const char msg[] = "hello from kernel space\n";

static ssize_t hello_read(struct file *f, char __user *buf, size_t len, loff_t *off)
{
    return simple_read_from_buffer(buf, len, off, msg, sizeof(msg) - 1);
}

static const struct file_operations hello_fops = {
    .owner = THIS_MODULE,
    .read  = hello_read,
};

static struct miscdevice hello_dev = {
    .minor = MISC_DYNAMIC_MINOR,
    .name  = "hello",          /* udev creates /dev/hello */
    .fops  = &hello_fops,
    .mode  = 0444,
};

module_misc_device(hello_dev);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Minimal misc character device");
# Makefile:  obj-m += hello_dev.o
make -C /lib/modules/$(uname -r)/build M=$PWD modules
sudo insmod hello_dev.ko
cat /dev/hello
sudo rmmod hello_dev

A real hardware driver adds a bus driver (pci_driver with an ID table and probe/remove), BAR mapping (pci_iomap), DMA setup, and request_irq/pci_alloc_irq_vectors — but the structure is the same: register callbacks with a subsystem and let the kernel call you.

Drivers run with full kernel privilege

A driver bug is a kernel bug: a null pointer dereference is an oops or a panic, a buffer overflow is a privilege escalation, a DMA to the wrong address corrupts arbitrary memory (unless the IOMMU catches it). Drivers make up the majority of the kernel's source and historically a large share of its bugs. That is a big part of the motivation for Rust in the kernel, for user-space drivers, and for keeping the IOMMU on.


User-Space Drivers: UIO and VFIO

Sometimes you want a process, not the kernel, to drive a device: DPDK and SPDK for performance, QEMU for passing a physical device to a VM, or a vendor that ships a user-space stack.

UIOVFIO
What it exposesDevice memory regions + an interrupt fdDevice regions, DMA mapping, interrupts, config space, reset
DMA safetyNone — user space hands raw physical addresses to the deviceIOMMU-enforced — process can only DMA to memory it mapped through VFIO
InterruptsLegacy/MSI via read() on the fdMSI/MSI-X via eventfds
Privilege neededEffectively root, can corrupt any memoryCan be granted to an unprivileged user per IOMMU group
Use todayLegacy, embedded, cases without an IOMMUDPDK, SPDK, QEMU/KVM device passthrough, cloud hypervisors
# Hand a NIC to VFIO for DPDK or a VM
sudo modprobe vfio-pci
sudo dpdk-devbind.py --bind=vfio-pci 0000:3b:00.1
# or by hand:
echo 0000:3b:00.1 | sudo tee /sys/bus/pci/devices/0000:3b:00.1/driver/unbind
echo vfio-pci     | sudo tee /sys/bus/pci/devices/0000:3b:00.1/driver_override
echo 0000:3b:00.1 | sudo tee /sys/bus/pci/drivers_probe

ls /dev/vfio/        # one node per IOMMU group, plus /dev/vfio/vfio

The unit of isolation is the IOMMU group — the set of devices that the hardware cannot isolate from each other (e.g. functions behind the same PCIe switch without ACS). You must assign a whole group, which is why some consumer motherboards make GPU passthrough painful.


How This Looks in Cloud VMs

Inside a virtual machine, "the NIC" and "the disk" are usually not real hardware. How they are virtualised determines most of the I/O performance gap between VMs and bare metal.

flowchart TD
    subgraph EMU["Full emulation"]
        G1["Guest e1000 driver"] -- "every register access traps" --> Q1["QEMU emulates device"]
    end
    subgraph PV["Paravirtual virtio"]
        G2["Guest virtio-net driver"] -- "shared virtqueues, one notify per batch" --> V2["vhost-net in host kernel"]
    end
    subgraph SRIOV["SR-IOV or Nitro"]
        G3["Guest ENA or mlx5 VF driver"] -- "direct DMA via IOMMU, posted interrupts" --> HW3["Physical NIC virtual function"]
    end

Full emulation

The hypervisor emulates a real device (an Intel e1000 NIC, an IDE disk). Every register access from the guest driver is a VM exit — the CPU leaves the guest, the hypervisor emulates the access, and re-enters. VM exits cost roughly a microsecond or more each, and a single packet can need several. Compatible with any guest OS, painfully slow. Used for boot and legacy guests.

virtio (paravirtualisation)

The guest knows it is virtualised and uses a device designed for it. virtio devices (virtio-net, virtio-blk, virtio-scsi) communicate through virtqueues — descriptor rings in guest memory shared with the host, structurally the same as a real NIC's rings. The guest batches many buffers and sends one notification ("kick"); the host backend (QEMU, or vhost-net / vhost-user for speed) processes them and injects one interrupt back. Far fewer exits than emulation, but every packet still passes through the host's software path.

SR-IOV

Single Root I/O Virtualization lets one physical PCIe device present itself as a Physical Function (PF, for the host) plus many Virtual Functions (VFs) — lightweight PCIe functions, each with its own queues, MAC, and MSI-X vectors. A VF is passed into a guest with VFIO; the guest driver DMAs directly to and from guest memory through the IOMMU, with no hypervisor on the data path. Combined with posted interrupts (Intel APICv / AMD AVIC), which deliver device interrupts into a running guest without a VM exit, I/O performance approaches bare metal.

The trade-off: the VM is tied to specific hardware, which complicates live migration — part of why clouds that use SR-IOV-style designs build their own migration-friendly devices.

AWS Nitro and ENA (and peers)

AWS's Nitro system moves networking, EBS storage, and management off the host CPU onto dedicated Nitro cards. Guests see an ENA (Elastic Network Adapter) device for networking and NVMe devices for EBS volumes and instance storage — both are PCIe devices presented directly to the guest, SR-IOV style, with the virtualisation logic implemented in the Nitro card's hardware and firmware rather than in host software. That is why EBS volumes show up as /dev/nvme1n1 on Nitro instances, and why network performance scales up to 100–200 Gbit/s on larger instance types. Google Cloud (gVNIC), Azure (Accelerated Networking with Mellanox/MANA VFs), and others have analogous designs.

What this means operationally inside a cloud VM:

  • All the earlier tuning still applies. ENA has multiple queues with MSI-X vectors, RSS, NAPI, adaptive coalescing. ethtool -S eth0 exposes ENA-specific counters, including allowance-exceeded counters (bw_in_allowance_exceeded, pps_allowance_exceeded, conntrack_allowance_exceeded) that tell you the cloud shaped your traffic, not the kernel.
  • Queue count is set by the instance type. Small instances get few queues, which caps how far RSS can spread packet work.
  • "Steal" time (%steal in mpstat, top) shows the hypervisor running something else while your vCPU wanted to run — which delays interrupt handling too.
  • Interrupts can still be expensive on older or emulated paths. If lspci shows an emulated device or the driver is e1000, you are leaving a lot of performance on the table.
# What kind of devices does this VM have?
lspci | grep -iE 'ethernet|nvme|virtio'
ethtool -i eth0                       # driver: ena / virtio_net / mlx5_core / gve
ethtool -S eth0 | grep allowance      # ENA: cloud-side shaping
mpstat 1 | awk 'NR<4 || /all/'        # watch %steal alongside %irq and %soft

Observability: Watching Interrupts

Which device, which CPU, how often

# Per-IRQ, per-CPU counts; diff twice to get rates
watch -d -n1 'grep -E "eth0|nvme|CPU" /proc/interrupts'

# Interrupt rate per second for one IRQ
irq=124; a=$(awk -v i="$irq:" '$1==i {s=0; for(n=2;n<=NF;n++) if($n~/^[0-9]+$/) s+=$n; print s}' /proc/interrupts)
sleep 1
b=$(awk -v i="$irq:" '$1==i {s=0; for(n=2;n<=NF;n++) if($n~/^[0-9]+$/) s+=$n; print s}' /proc/interrupts)
echo "IRQ $irq: $((b-a))/s"

# Softirq counts per CPU
watch -d -n1 cat /proc/softirqs

# CPU time split: %irq = hard interrupts, %soft = softirqs
mpstat -P ALL 1
# CPU  %usr %nice %sys %iowait %irq %soft %steal %idle
# 0    4.1  0.0   3.2  0.0     2.1  61.8  0.0    28.8   <- hot core
# 1    22.0 0.0   6.1  0.0     0.0  1.2   0.0    70.7

vmstat 1 shows total interrupts per second in the in column and context switches in cs — a quick sanity check for interrupt storms.

Where the time goes

# Count hard IRQs by handler name over 10 s
sudo perf stat -e 'irq:irq_handler_entry' -a -- sleep 10
sudo perf record -e 'irq:irq_handler_entry' -a -- sleep 10 && sudo perf report --sort comm,sym

# Latency distribution of hard IRQ handlers, by IRQ name (bcc-tools)
sudo /usr/share/bcc/tools/hardirqs -d 10 1
# Time spent in each softirq type
sudo /usr/share/bcc/tools/softirqs -d 10 1

# bpftrace: histogram of time spent in each softirq vector, in µs
sudo bpftrace -e '
tracepoint:irq:softirq_entry { @start[cpu] = nsecs; }
tracepoint:irq:softirq_exit /@start[cpu]/ {
  @us[args->vec] = hist((nsecs - @start[cpu]) / 1000);
  delete(@start[cpu]);
}'

# Which functions run in interrupt context the most
sudo perf top -C 0       # profile the hot core

The softirq vector numbers in that bpftrace output follow the /proc/softirqs order: 0 HI, 1 TIMER, 2 NET_TX, 3 NET_RX, 4 BLOCK, 5 IRQ_POLL, 6 TASKLET, 7 SCHED, 8 HRTIMER, 9 RCU.

Interrupt affinity

# Where IRQ 124 is allowed to go, and where it actually went
cat /proc/irq/124/smp_affinity_list
cat /proc/irq/124/effective_affinity_list

# Pin it
echo 4 | sudo tee /proc/irq/124/smp_affinity_list

# Keep general IRQs off latency-critical cores (irqbalance honours this)
# /etc/sysconfig/irqbalance or /etc/default/irqbalance:
#   IRQBALANCE_BANNED_CPULIST=8-15
# Kernel cmdline alternatives: irqaffinity=0-7, isolcpus=, nohz_full=

Many modern drivers (NVMe, mlx5) use managed interrupts: the kernel spreads vectors across CPUs automatically at probe time and won't let you move them. That's by design — the queue-to-CPU mapping is part of how the driver avoids locking — and those IRQs show "managed" behaviour by rejecting writes to smp_affinity.


Failure Modes

SymptomLikely causeCheckFix
One core at ~100% %soft or %irq, others idleAll device vectors on one CPU, or a single RX queue/proc/interrupts, mpstat -P ALLSpread IRQ affinity, enable irqbalance, more queues, RPS
ksoftirqd/N high in top, app latency spikes on that coreSoftirq overload deferring to thread/proc/softirqs, softnet_statSpread load, reduce per-packet work, move app threads off that core
irq N: nobody cared ... Disabling IRQ in dmesgInterrupt storm, unclaimed shared linedmesg, /proc/interrupts rateFix driver/firmware/hardware; check for legacy INTx sharing
High tail latency on an idle-ish serverDeep C-state exit latency, coalescing delayturbostat, cpupower idle-info, ethtool -cCap C-states, reduce coalescing, busy polling
DMA errors, DMAR: [DMA Read] Request device ... fault addrDevice DMA to an unmapped IOVA — driver bug or firmware bugdmesgUpdate driver/firmware; never "fix" by disabling the IOMMU in production without understanding why
Throughput drop after enabling IOMMUMap/unmap and IOTLB invalidation overheadperf top showing iommu functionsiommu=pt for host devices, newer kernel, larger batched mappings
Many TLB shootdown IPIsHeavy munmap/mprotect churn across many threads/proc/interrupts TLB lineFewer threads touching the mapping, avoid churny allocators, hugepages
VM network throughput much lower than expectedEmulated or plain virtio device, cloud allowance limitsethtool -i, ethtool -S allowance counters, %stealUse the enhanced/SR-IOV NIC type, larger instance, spread flows
NVMe IOPS lower than spec at QD1Interrupt + wakeup latency dominatesfio with and without --hipriPoll queues + IOPOLL for the hot path, if a core can be spared
Kernel oops in a driver after an updateDriver bugdmesg, journalctl -kRoll back module, report upstream, pin kernel version in the fleet

Decision Guide

flowchart TD
    Q["How should this device notify the CPU?"] --> A{"Event rate?"}
    A -- "Low, under ~10k per second" --> I["Interrupts, coalescing off or adaptive"]
    A -- "High" --> B{"Can you dedicate a core?"}
    B -- "No" --> N["Hybrid: NAPI or adaptive coalescing, spread vectors with MSI-X affinity"]
    B -- "Yes" --> C{"Need kernel features - TCP stack, file systems, tooling?"}
    C -- "Yes" --> P["Kernel polling: busy_poll, NVMe poll queues, io_uring IOPOLL and SQPOLL"]
    C -- "No" --> U["User-space polling: DPDK or SPDK via VFIO"]

Practical defaults:

  • Use MSI-X with one vector per queue per CPU. It is the default for every serious modern device; if lspci -vv shows INTx, something is wrong.
  • Keep adaptive coalescing on unless you have measured that fixed values help your workload.
  • Spread interrupt vectors across the cores of the device's NUMA node. Either let irqbalance do it or pin deliberately — not both.
  • Isolate latency-critical application cores from interrupt work (irqaffinity=, irqbalance banned CPUs) when p99 matters more than total throughput.
  • Keep the IOMMU on, use iommu=pt if host-side translation overhead is measurable.
  • Prefer VFIO over UIO for user-space drivers; UIO has no DMA protection.
  • In the cloud, pick instance types with the enhanced NIC and NVMe storage, and watch the provider's allowance counters before blaming the kernel.
  • Poll only where the latency win is measurable and the burned core is affordable.

Interview Questions

What's the difference between an interrupt, an exception, and a trap? A hardware interrupt is asynchronous, from a device, through the interrupt controller. An exception is synchronous, raised by the CPU because of the current instruction (page fault, divide by zero). A trap is an exception triggered deliberately by an instruction (int3, legacy int 0x80). All three are dispatched through the IDT on x86.

Why split interrupt handling into a top half and bottom half? The top half runs with interrupts masked, in no process context, and cannot sleep — every microsecond it runs delays other interrupts on that core. So it does the minimum (acknowledge, capture state) and defers the rest to softirqs, workqueues, or a threaded IRQ, which run with interrupts enabled and, in the workqueue/thread case, can sleep.

What is MSI-X and why does it matter for performance? Interrupts delivered as memory writes rather than physical lines, with up to 2048 independent vectors per device, each targetable to a specific CPU. It lets a multi-queue NIC or NVMe drive have one interrupt per queue per core, so completions are handled on the core that owns the queue — no shared lines, no cross-core locking.

When is polling better than interrupts? When events are frequent enough that interrupt overhead dominates, or latency requirements are tighter than interrupt-plus-wakeup latency. NAPI and NVMe polling are hybrids; DPDK/SPDK poll permanently on dedicated cores. At low event rates polling wastes a core.

What does the IOMMU protect against, and what does it cost? It restricts each device to DMA only into memory explicitly mapped for it, preventing buggy or malicious devices from writing arbitrary memory, and it makes safe device passthrough to VMs and user-space drivers possible. The cost is map/unmap and IOTLB invalidation overhead, which matters at very high I/O rates; iommu=pt and batching reduce it.

Why are cloud VMs with SR-IOV or Nitro NICs so much faster than emulated NICs? Emulated devices cause a VM exit for every register access; virtio batches but still goes through host software. SR-IOV and Nitro give the guest a real PCIe function that DMAs directly into guest memory through the IOMMU, with posted interrupts avoiding exits — the hypervisor is off the data path.