The Linux Networking Stack: From NIC to Socket
Your service calls read() on a socket and gets bytes. Between the electrical signal arriving at the network card and those bytes landing in your buffer, the packet crosses a DMA engine, a ring buffer, a hardware interrupt, a softirq, a polling loop, a packet-merging layer, a firewall, a connection tracker, the IP layer, the TCP state machine, a socket queue, and an event-notification system. Every one of those stages has a queue, and every queue can overflow.
That is why "the network is slow" is almost never a useful diagnosis. When a p99 latency spike or a mysterious connection reset shows up, the real question is which queue dropped or delayed the packet. Linux exposes a counter for nearly every one of them — but only if you know the path well enough to know which counter to read.
This page walks the receive path end to end, then the send path, then the knobs (multi-queue, steering, backlogs, buffer sizes), then the escape hatches (busy polling, io_uring, XDP, AF_XDP, DPDK), and finishes with a troubleshooting table that maps symptoms to the exact counters that explain them.
What you should already know
How a syscall crosses into the kernel (Syscalls & Kernel Internals), and the TCP basics — handshake, windows, retransmission (TCP/IP Deep Dive). The hardware side of interrupts and DMA is covered in depth in Interrupts, DMA & Device Drivers.
Why the Stack Looks the Way It Does
Start from the hardware budget. A 10 GbE link carrying minimum-size frames (64 bytes plus 20 bytes of preamble and inter-frame gap) delivers 14.88 million packets per second. That is one packet every 67 nanoseconds. A 100 GbE link carrying the same traffic delivers 148.8 Mpps — one packet every 6.7 ns. For comparison:
| Operation | Approximate cost |
|---|---|
| L1 cache hit | ~1 ns |
| L3 cache hit | ~10–15 ns |
| DRAM access (cache miss) | ~80–100 ns |
| Hardware interrupt entry + exit | ~1–2 µs including cache disruption |
| Syscall round trip (post-KPTI) | ~100–300 ns |
| Context switch | ~2–5 µs |
A single cache miss costs more than the entire per-packet budget at 10 GbE line rate with small packets. One interrupt per packet would cost 15–30x the budget. So the design of the Linux receive path follows directly from arithmetic:
- Never let the CPU touch packet bytes it doesn't have to. The NIC writes packets directly into memory with DMA.
- Never take one interrupt per packet. Take one interrupt, disable further interrupts, and poll until the queue is drained (NAPI).
- Do as little as possible in interrupt context. Defer protocol processing to a softirq.
- Amortise per-packet cost over many packets. Merge consecutive TCP segments into one large buffer (GRO) on receive; hand one large buffer to the NIC and let it split (TSO) on send.
- Spread the work across cores. Multiple hardware queues, each with its own interrupt and its own CPU (RSS).
Every tuning knob on this page is one of those five ideas made adjustable.
The Receive Path at a Glance
flowchart TD
A["Frame arrives on the wire"] --> B["NIC: checksum, RSS hash, picks RX queue"]
B --> C["DMA write into RX ring buffer slot"]
C --> D["Hardware interrupt - MSI-X vector for that queue"]
D --> E["IRQ handler: disable queue IRQ, schedule NAPI"]
E --> F["NET_RX softirq runs net_rx_action"]
F --> G["Driver poll: build sk_buff per descriptor, refill ring"]
G --> H["GRO: merge same-flow TCP segments"]
H --> I["Optional RPS: hand to another CPU backlog"]
I --> J["netif_receive_skb: taps, tc ingress"]
J --> K["Netfilter PREROUTING: conntrack, iptables or nftables"]
K --> L["IP: routing decision, local delivery"]
L --> M["Netfilter INPUT"]
M --> N["TCP: socket lookup, state machine, ACK, receive queue"]
N --> O["sk_data_ready wakes epoll waiters"]
O --> P["Application read copies bytes to user space"]
Nine distinct places can drop that packet: the NIC FIFO, the RX ring, the per-CPU backlog, conntrack, netfilter rules, the IP layer, the TCP SYN queue, the accept queue, and the socket receive buffer. We'll walk through each.
Stage 1: NIC, Ring Buffers, and DMA
A modern NIC is a small computer. When a frame arrives it validates the Ethernet FCS, optionally verifies IP/TCP/UDP checksums (RX checksum offload), computes a hash over the flow tuple (RSS hash), and uses that hash to pick a receive queue.
Each RX queue is backed by a ring buffer of descriptors in host RAM. At driver load time, the driver allocates a page or buffer for each descriptor and writes its physical (or IOMMU-mapped) address into the descriptor. The NIC owns the descriptors at the "head"; the driver owns the ones at the "tail".
flowchart LR
subgraph RING["RX descriptor ring - host RAM"]
D0["desc 0: filled"] --> D1["desc 1: filled"] --> D2["desc 2: empty, owned by NIC"] --> D3["desc 3: empty, owned by NIC"] --> D0
end
NIC["NIC DMA engine"] -- "writes packet bytes" --> D2
DRV["Driver NAPI poll"] -- "reads filled, refills buffers" --> D0
The NIC writes the packet into the next free buffer via DMA — the CPU is not involved — and marks the descriptor as done. If the driver doesn't refill buffers fast enough and the ring fills up, the NIC has nowhere to put the next frame and drops it in hardware. That drop is invisible to every kernel counter except the driver's own statistics:
# Ring size: current vs hardware max
ethtool -g eth0
# Ring parameters for eth0:
# Pre-set maximums:
# RX: 4096
# TX: 4096
# Current hardware settings:
# RX: 512
# TX: 512
# Grow the RX ring to absorb bursts
sudo ethtool -G eth0 rx 4096
# Driver-level drop counters (names vary by driver)
ethtool -S eth0 | grep -Ei 'drop|miss|fifo|no_buf|discard'
# rx_missed_errors: 18324 <- ring was full, NIC dropped
# rx_no_dma_resources: 0
# rx_queue_3_drops: 18324
Bigger rings are not free
A 4096-entry ring with 2 KB buffers per queue across 32 queues pins ~256 MB of DMA memory, and a deeper ring means a packet can sit longer before it is processed — added latency under load. Grow rings to absorb bursts, not to hide a CPU that is permanently too slow to keep up. If the ring is full at steady state, you need more queues or more CPUs, not a bigger ring.
The interaction between DMA, the IOMMU, and device memory ownership is covered in Interrupts, DMA & Device Drivers.
Stage 2: Interrupts and NAPI
Once the packet is in RAM, the NIC raises an interrupt — specifically an MSI-X message tied to that RX queue. The CPU that the vector is routed to jumps into the driver's interrupt handler.
The handler does almost nothing. It acknowledges the interrupt, masks further interrupts for that queue, and calls napi_schedule(), which puts the queue's NAPI structure on the current CPU's poll list and raises the NET_RX_SOFTIRQ. Then it returns. The whole thing takes well under a microsecond of handler time.
This is NAPI (the "New API", from 2001): interrupt to start, poll to continue. Under light load you get one interrupt per packet or small batch and low latency. Under heavy load, interrupts stay masked and the kernel just keeps polling the ring — interrupt cost goes to near zero per packet.
stateDiagram-v2
[*] --> IRQ_ENABLED
IRQ_ENABLED --> POLLING: packet arrives, IRQ fires, napi_schedule
POLLING --> POLLING: budget used up, ring still has packets
POLLING --> IRQ_ENABLED: ring drained, napi_complete_done re-arms IRQ
The softirq and its budget
The softirq handler net_rx_action() walks the CPU's poll list and calls each driver's poll() function with a weight (usually 64 packets). It keeps going until either the list is empty or it exhausts a global budget:
sysctl net.core.netdev_budget # default 300 packets per softirq run
sysctl net.core.netdev_budget_usecs # default 2000 µs per softirq run
If the budget runs out while there is still work, net_rx_action stops, re-raises the softirq, and increments the time_squeeze counter. This is a fairness mechanism: without it, a flood of packets would starve every user process on that CPU.
Softirqs normally run on the way out of the hardware interrupt. If they keep re-raising themselves, the kernel hands the work to a per-CPU kernel thread called ksoftirqd/N, which is scheduled like any other thread. When you see ksoftirqd/3 near the top of top, CPU 3 is spending most of its time processing packets and the scheduler is now arbitrating between that and your application. See CPU Scheduling for how CFS/EEVDF weighs those threads.
Reading /proc/net/softnet_stat
One line per CPU, hex values. The first three columns are the ones that matter:
cat /proc/net/softnet_stat
# 0a3f8c21 00000000 0000012c 00000000 ...
# 0b11d004 000004d2 00002f1a 00000000 ...
# ^ ^ ^
# processed dropped time_squeeze
- Column 1 — processed: packets handled by this CPU.
- Column 2 — dropped: packets dropped because the per-CPU input backlog (
net.core.netdev_max_backlog, default 1000) was full. This only happens on the RPS/backlog path or with non-NAPI drivers. - Column 3 — time_squeeze: the softirq ran out of budget or time with work remaining.
A readable version:
awk '{ printf "cpu%-3d processed=%d dropped=%d squeezed=%d\n", NR-1,
strtonum("0x"$1), strtonum("0x"$2), strtonum("0x"$3) }' /proc/net/softnet_stat
Rising time_squeeze means the CPU is falling behind. Fixes, in order of preference: spread load over more queues/CPUs (RSS, RPS), make each packet cheaper (fewer iptables rules, GRO enabled), then raise netdev_budget / netdev_budget_usecs — the last one trades application CPU time for packet processing time.
Stage 3: sk_buff and GRO
sk_buff — the packet's passport
In the driver's poll function, each completed descriptor becomes a struct sk_buff ("skb"). The skb is metadata — around 230 bytes of pointers, lengths, flags, timestamps, the receiving device, the protocol, and the socket it will eventually belong to — that points at the packet data rather than containing it. Each layer of the stack moves the data pointer rather than copying bytes: Ethernet strips 14 bytes by advancing skb->data, IP advances past its header, and so on.
head data tail end
| headroom | Eth | IP | TCP | payload | tailroom |
+-------------+-----+----+-----+---------+-------------+
^ each layer "pulls" its header by moving data forward
Modern drivers build the skb around a page that already holds the DMA'd packet (build_skb / napi_build_skb), so no copy happens here either. The only copy on the whole receive path, in the default configuration, is the final one into your user-space buffer.
GRO — Generic Receive Offload
Processing 45 separate 1448-byte TCP segments through netfilter, IP, and TCP costs 45x the per-packet overhead. GRO merges consecutive segments of the same flow, arriving in the same NAPI poll, into one large skb (up to 64 KB, more with BIG TCP on recent kernels) before they go up the stack. The TCP layer then sees one "super-segment" and does its per-packet work once.
ethtool -k eth0 | grep -E 'generic-receive-offload|large-receive-offload|rx-gro-hw'
# generic-receive-offload: on
# large-receive-offload: off [fixed]
# rx-gro-hw: off
GRO is software and is safe — it preserves enough information to re-segment exactly (important when forwarding). LRO (Large Receive Offload) is the older hardware variant that merges lossy-ly and breaks routing and bridging; it is usually off. Hardware GRO (rx-gro-hw) on newer NICs does GRO-correct merging in silicon.
Why GRO matters for throughput
On a typical bulk TCP transfer, GRO cuts the number of skbs reaching the TCP layer by 10–40x. Turning GRO off on a busy 25 GbE host can easily double softirq CPU. If someone "disabled offloads to debug" and forgot to turn them back on, this is the first thing to check.
Stage 4: Spreading Load — RSS, RPS, RFS, aRFS
A single CPU core can push roughly 1–3 million packets per second through the full kernel TCP/IP stack, depending on packet size, netfilter rules, and CPU generation. A 100 GbE NIC can deliver 50x that. So receive processing has to be parallel, but packets of one TCP flow must stay ordered and ideally stay on one core (for cache locality and to avoid lock contention on the socket).
| Mechanism | Where | What it does |
|---|---|---|
| RSS — Receive Side Scaling | NIC hardware | Hash flow tuple → indirection table → one of N RX queues, each with its own MSI-X IRQ |
| RPS — Receive Packet Steering | Kernel, software | After the driver poll, hash the skb and enqueue to another CPU's backlog via IPI. For NICs with too few queues |
| RFS — Receive Flow Steering | Kernel, software | Like RPS, but steer to the CPU where the consuming application thread last ran |
| aRFS — accelerated RFS | NIC + kernel | Program the NIC's flow table so hardware delivers the flow to the queue whose IRQ is on the app's CPU |
| XPS — Transmit Packet Steering | Kernel | Map CPUs to TX queues so a CPU sends on "its" queue and TX completions land locally |
flowchart LR
W["Packets from wire"] --> H["NIC RSS hash on 5-tuple"]
H --> T["Indirection table - 128 or more entries"]
T --> Q0["RX queue 0 - IRQ on CPU 0"]
T --> Q1["RX queue 1 - IRQ on CPU 1"]
T --> Q2["RX queue 2 - IRQ on CPU 2"]
T --> Q3["RX queue 3 - IRQ on CPU 3"]
Q1 -.-> R["RFS: forward to CPU 6 where the app thread runs"]
Commands
# How many queues, and how many are in use
ethtool -l eth0
sudo ethtool -L eth0 combined 16
# RSS indirection table and hash key
ethtool -x eth0
# Weight queues 0-7 only, keep 8-15 for something else
sudo ethtool -X eth0 equal 8
# Which tuple fields feed the hash for UDP over IPv4 (s=src ip, d=dst ip, f=src port, n=dst port)
sudo ethtool -N eth0 rx-flow-hash udp4 sdfn
# IRQ per queue and which CPU it lands on
grep eth0 /proc/interrupts
cat /proc/irq/123/smp_affinity_list
# Pin queue IRQ 123 to CPU 4 (stop irqbalance first or it will move it back)
sudo systemctl stop irqbalance
echo 4 | sudo tee /proc/irq/123/smp_affinity_list
# RPS: let CPUs 0-7 (mask ff) process packets from rx-0
echo ff | sudo tee /sys/class/net/eth0/queues/rx-0/rps_cpus
# RFS: global flow table + per-queue share
sudo sysctl -w net.core.rps_sock_flow_entries=32768
echo 2048 | sudo tee /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
# XPS: CPU 0 transmits on tx-0
echo 1 | sudo tee /sys/class/net/eth0/queues/tx-0/xps_cpus
Picking a layout
- NIC has at least as many queues as cores you want doing network work → RSS only, one queue per core, pin IRQs to cores on the NIC's NUMA node (
cat /sys/class/net/eth0/device/numa_node). Leave RPS off; it only adds IPIs. - NIC has few queues (cheap NICs, some virtual NICs) → RSS across what you have plus RPS to fan out.
- Latency-sensitive app with pinned threads → RFS or aRFS so the packet is processed on the same core (same L1/L2) as the thread that reads it.
- Mixed workload where app threads and IRQs compete → separate them: IRQs on cores 0–7, application pinned to 8–31. Measure; sometimes co-location wins because of cache locality.
The single-flow ceiling
RSS hashes flows. One elephant TCP connection — a replication stream, a backup, a big gRPC channel — always lands on one queue and one CPU, no matter how many queues you have. If one core is at 100% %soft while the rest are idle, you have a single-flow bottleneck, and the fix is at the application level (multiple connections, or the protocol's own striping), not in RSS settings.
Stage 5: Netfilter, iptables, and conntrack
After GRO and steering, __netif_receive_skb delivers the packet to packet taps (this is where tcpdump sees it — after the NIC and ring, so tcpdump cannot see ring drops), to tc ingress filters, and then to the protocol handler, which for IPv4 is ip_rcv. Before routing, the packet hits the netfilter hooks:
flowchart LR
IN["ip_rcv"] --> PRE["PREROUTING - raw, conntrack, mangle, DNAT"]
PRE --> RT{"Routing decision"}
RT -- "local address" --> INP["INPUT - filter"]
INP --> TCP["tcp_v4_rcv"]
RT -- "forward" --> FWD["FORWARD"]
FWD --> POST["POSTROUTING - SNAT, masquerade"]
POST --> OUT["dev_queue_xmit"]
Two costs hide here.
Rule evaluation. Classic iptables evaluates rules linearly per chain. A Kubernetes node in iptables-mode kube-proxy with 5,000 Services can have tens of thousands of rules; every new connection walks a long chain. nftables with sets/maps, IPVS, or eBPF-based dataplanes (Cilium) replace that walk with hash lookups. See eBPF for how eBPF dataplanes avoid the chain walk entirely.
Connection tracking. If nf_conntrack is loaded (it is on any host doing NAT, including every Docker or Kubernetes node), every packet is looked up in a hash table of flows, and every new flow inserts an entry. The table has a hard size:
sysctl net.netfilter.nf_conntrack_count # current entries
sysctl net.netfilter.nf_conntrack_max # limit
dmesg | grep conntrack
# nf_conntrack: nf_conntrack: table full, dropping packet
# Per-CPU stats: look at insert_failed, drop, early_drop
conntrack -S
When the table is full, new connections are silently dropped — clients see SYN timeouts, not resets. Raise nf_conntrack_max (each entry is roughly 300 bytes), shorten nf_conntrack_tcp_timeout_time_wait and _established if you have many short-lived flows, or exempt high-volume traffic that doesn't need NAT with NOTRACK rules in the raw table. NAT itself is covered in NAT.
War Story (illustrative)
A DNS caching tier on Kubernetes started returning intermittent 5-second lookup delays every evening. Application logs showed nothing; the pods were at 20% CPU. The cause was conntrack: each UDP DNS query created a tracked flow with a 30–120 second timeout, and at peak the node's table hit nf_conntrack_max. New flows were dropped, the resolver's 5-second retry kicked in, and the second attempt usually succeeded. conntrack -S showed drop climbing exactly at peak. Raising the limit stopped the bleeding; adding NOTRACK rules for port 53 fixed it properly.
Stage 6: TCP — Lookup, State, and the Receive Queue
tcp_v4_rcv looks up the socket in the established hash table by the 4-tuple, falling back to the listening table. Then it runs the state machine: validate sequence numbers, process ACKs (which frees data in the send queue), update the congestion window, and queue payload.
The receive side has three queues per socket:
| Queue | Holds | When used |
|---|---|---|
Receive queue (sk_receive_queue) | In-order data waiting for the app | Normal path |
| Out-of-order queue (rbtree) | Segments after a gap | Until the gap is filled by retransmit |
| Backlog queue | Packets that arrived while a user thread held the socket lock | Processed when the app releases the lock |
TCP decides when to ACK (delayed ACKs, quick-ACK mode), and advertises a receive window based on free space in the socket buffer. If the app stops reading, the buffer fills, the advertised window shrinks to zero, and the sender stops — this is flow control, distinct from congestion control. You can see it as Recv-Q growing on an established socket in ss.
When memory runs out despite that — sender ignoring the window, or many sockets pressuring global TCP memory — the kernel first tries to collapse the queue (re-pack skbs to reduce overhead), then prunes the out-of-order queue, and finally drops. All of those show up in nstat:
nstat -az | grep -E 'TcpExtTCPRcvCollapsed|TcpExtPruneCalled|TcpExtRcvPruned|TcpExtTCPOFOQueue|TcpExtTCPRcvQDrop'
Stage 7: Wakeup — sk_data_ready, epoll, and read()
When data is queued, TCP calls the socket's sk_data_ready callback. For a blocking reader that wakes the thread sleeping in read(). For an epoll-based server, the socket's wait queue has an entry installed by epoll_ctl(EPOLL_CTL_ADD) whose callback, ep_poll_callback, adds the file to the epoll instance's ready list and wakes whichever thread is blocked in epoll_wait().
sequenceDiagram
participant NIC
participant SoftIRQ as NET_RX softirq
participant TCP as TCP layer
participant EP as epoll ready list
participant App as App thread
App->>EP: epoll_wait blocks
NIC->>SoftIRQ: IRQ then NAPI poll
SoftIRQ->>TCP: tcp_v4_rcv with skb
TCP->>TCP: queue data, send ACK
TCP->>EP: sk_data_ready calls ep_poll_callback
EP->>App: wake up, fd is readable
App->>TCP: read copies bytes to user buffer
TCP->>NIC: window update may trigger ACK
The read() (or recv()) is a syscall that runs tcp_recvmsg: lock the socket, copy bytes from skbs to your buffer (skb_copy_datagram_iter), free consumed skbs, maybe send a window update, unlock. That copy is the main per-byte cost of receive.
A minimal epoll server in C, showing the shape that every high-performance event loop (nginx, Redis, libuv, Go's netpoller) is built on:
#include <sys/epoll.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#include <stdio.h>
#define MAX_EVENTS 256
static void set_nonblock(int fd) {
fcntl(fd, F_SETFL, fcntl(fd, F_GETFL) | O_NONBLOCK);
}
int main(void) {
int lfd = socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0);
int one = 1;
setsockopt(lfd, SOL_SOCKET, SO_REUSEPORT, &one, sizeof one);
struct sockaddr_in addr = { .sin_family = AF_INET,
.sin_port = htons(8080),
.sin_addr.s_addr = htonl(INADDR_ANY) };
bind(lfd, (struct sockaddr *)&addr, sizeof addr);
listen(lfd, 4096); /* capped by net.core.somaxconn */
int ep = epoll_create1(0);
struct epoll_event ev = { .events = EPOLLIN, .data.fd = lfd };
epoll_ctl(ep, EPOLL_CTL_ADD, lfd, &ev);
struct epoll_event events[MAX_EVENTS];
char buf[64 * 1024];
for (;;) {
int n = epoll_wait(ep, events, MAX_EVENTS, -1);
for (int i = 0; i < n; i++) {
int fd = events[i].data.fd;
if (fd == lfd) {
/* Drain the accept queue completely */
for (;;) {
int c = accept4(lfd, NULL, NULL, SOCK_NONBLOCK);
if (c < 0) break; /* EAGAIN: queue empty */
struct epoll_event cev = { .events = EPOLLIN | EPOLLRDHUP,
.data.fd = c };
epoll_ctl(ep, EPOLL_CTL_ADD, c, &cev);
}
} else {
ssize_t r;
while ((r = read(fd, buf, sizeof buf)) > 0) {
write(fd, buf, r); /* echo; real code handles short writes */
}
if (r == 0 || (r < 0 && errno != EAGAIN)) {
close(fd); /* close also removes it from epoll */
}
}
}
}
}
Two details matter for the kernel side. First, the accept loop drains the queue until EAGAIN — accepting one connection per wakeup lets the accept queue grow under load. Second, SO_REUSEPORT lets you run one listener per worker thread or process; the kernel hashes incoming connections across them, which avoids the thundering-herd and lock contention of many threads sharing one listening socket. The epoll API itself is covered in I/O & Storage Internals.
The Accept Queue and SYN Backlog
Connection setup has its own two-queue pipeline, and its failure modes cause some of the most confusing production problems: connection timeouts with a healthy-looking server.
flowchart LR
C["Client SYN"] --> S1["SYN queue - half-open, SYN_RECV state"]
S1 -- "SYN-ACK sent, client ACK received" --> S2["Accept queue - fully established"]
S2 -- "app calls accept" --> APP["Application owns the socket"]
S1 -. "full: drop SYN or send SYN cookie" .-> X1["TcpExtListenDrops"]
S2 -. "full: drop final ACK" .-> X2["TcpExtListenOverflows"]
SYN queue (half-open connections). Sized by net.ipv4.tcp_max_syn_backlog, bounded by the listen backlog. When it fills — usually a SYN flood — the kernel falls back to SYN cookies (net.ipv4.tcp_syncookies=1, the default), which encode the connection state in the SYN-ACK's sequence number so no memory is held. Cookies lose some TCP options on older kernels, so frequent TcpExtSyncookiesSent on a non-attacked server means the queue is too small.
Accept queue (established, not yet accept()ed). Its length is min(backlog argument to listen(), net.core.somaxconn). somaxconn defaulted to 128 for decades and was raised to 4096 in kernel 5.4. Many runtimes and frameworks still pass a small backlog — check what yours does.
When the accept queue is full and the client's final ACK arrives, Linux by default silently drops that ACK (tcp_abort_on_overflow=0). The client thinks the connection is established and may send a request; the server retransmits its SYN-ACK; eventually either the app catches up and accepts, or the client times out. The client sees latency spikes in exact multiples of the SYN-ACK retransmit timeout (1 s, then 3 s, then 7 s). If you set tcp_abort_on_overflow=1, the client gets an immediate RST instead — faster failure, but harder to tell apart from a dead server.
Seeing it
# For LISTEN sockets: Recv-Q = current accept queue length, Send-Q = max backlog
ss -ltn
# State Recv-Q Send-Q Local Address:Port
# LISTEN 0 4096 0.0.0.0:8080
# LISTEN 129 128 0.0.0.0:9000 <- full: app is not accepting fast enough
# Cumulative counters
nstat -az TcpExtListenOverflows TcpExtListenDrops TcpExtSyncookiesSent TcpExtTCPReqQFullDrop
netstat -s | grep -iE 'listen|SYNs to LISTEN'
# 4211 times the listen queue of a socket overflowed
# 4211 SYNs to LISTEN sockets dropped
Fixes, in order: make the app accept faster (dedicated accept thread, drain-to-EAGAIN loop, SO_REUSEPORT sharding); raise the backlog passed to listen() and somaxconn; only then consider the SYN queue. A Go service controls the backlog through somaxconn — the runtime reads it at startup and passes it to listen():
package main
import (
"context"
"net"
"net/http"
"syscall"
"golang.org/x/sys/unix"
)
func main() {
lc := net.ListenConfig{
Control: func(network, address string, c syscall.RawConn) error {
var serr error
err := c.Control(func(fd uintptr) {
// One listener per process, kernel load-balances new connections
serr = unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_REUSEPORT, 1)
if serr == nil {
// Larger receive buffer inherited by accepted sockets.
// Note: setting SO_RCVBUF disables receive autotuning for those sockets.
serr = unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_RCVBUF, 4<<20)
}
})
if err != nil {
return err
}
return serr
},
}
ln, err := lc.Listen(context.Background(), "tcp", ":8080")
if err != nil {
panic(err)
}
http.Serve(ln, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("ok\n"))
}))
}
SO_RCVBUF turns off autotuning
Setting SO_RCVBUF or SO_SNDBUF explicitly pins that socket's buffer and disables the kernel's dynamic autotuning for it. For most services, leave them unset and tune the tcp_rmem/tcp_wmem ceilings instead. The snippet above shows the mechanism, not a recommendation.
Socket Buffer Sizing and the Bandwidth-Delay Product
TCP can only have as many unacknowledged bytes in flight as the receiver's window allows, and the window is bounded by the receive buffer. To keep a pipe full you need:
buffer >= bandwidth x round-trip time (the bandwidth-delay product, BDP)
| Link | RTT | BDP |
|---|---|---|
| 10 Gbit/s, same rack | 0.1 ms | 125 KB |
| 10 Gbit/s, cross-AZ | 1 ms | 1.25 MB |
| 1 Gbit/s, cross-continent | 80 ms | 10 MB |
| 10 Gbit/s, cross-continent | 80 ms | 100 MB |
Linux autotunes each socket's buffers between a min and max, growing the receive buffer as it observes the sender's rate:
sysctl net.ipv4.tcp_rmem # 4096 131072 6291456 -> min, default, max (bytes)
sysctl net.ipv4.tcp_wmem # 4096 16384 4194304
sysctl net.core.rmem_max # cap for explicit SO_RCVBUF
sysctl net.core.wmem_max
sysctl net.ipv4.tcp_mem # global TCP memory in PAGES: low, pressure, high
With a 6 MB receive max, a single connection over an 80 ms path is capped at roughly 6 MB / 0.08 s = 75 MB/s ≈ 600 Mbit/s, no matter how fast the link is. For bulk transfer across regions:
sudo sysctl -w net.ipv4.tcp_rmem="4096 131072 67108864"
sudo sysctl -w net.ipv4.tcp_wmem="4096 16384 67108864"
sudo sysctl -w net.core.rmem_max=67108864
sudo sysctl -w net.core.wmem_max=67108864
Note that part of the buffer is reserved for skb overhead (tcp_adv_win_scale), so the advertised window is less than the buffer size. And tcp_mem is a global ceiling: 100,000 connections each autotuned to a few MB will hit memory pressure, at which point the kernel shrinks buffers for everyone. For a service with many mostly-idle connections (chat, push notification gateways), smaller defaults are right. For a few fat cross-region pipes, larger maxima are right. There is no single correct value.
On the send side, TCP_NOTSENT_LOWAT (or net.ipv4.tcp_notsent_lowat) limits how much unsent data sits in the socket. With large send buffers, an app can queue megabytes that then cannot be reprioritised — the HTTP/2 head-of-line problem. Setting it to 16–128 KB keeps the queue short and makes EPOLLOUT fire only when the data actually needs replenishing.
The Send Path
Transmit is the receive path mirrored, with its own queues:
flowchart TD
A["App write or sendmsg"] --> B["tcp_sendmsg: copy into skbs in socket send queue"]
B --> C["TCP: cwnd, rwnd, pacing, TSQ decide what may leave"]
C --> D["IP: route lookup, netfilter OUTPUT and POSTROUTING"]
D --> E["Qdisc on chosen TX queue - fq_codel, fq, pfifo"]
E --> F["GSO: segment in software if NIC cannot"]
F --> G["Driver: map skb for DMA, write TX descriptors, ring doorbell"]
G --> H["NIC: TSO splits into MTU frames, checksums, sends"]
H --> I["TX completion IRQ, NAPI frees skbs"]
I --> J["Socket send space freed, EPOLLOUT may fire"]
Key stages
Socket send queue. write() copies your bytes into skbs on the socket's write queue and returns as soon as they are copied — not when they are sent, and certainly not when they are acknowledged. A successful write() means "the kernel has it", nothing more. That's why a crashed peer is discovered on the next write or by a timeout.
TCP output. The congestion window, the receiver's window, pacing (with BBR or fq), and TCP Small Queues (net.ipv4.tcp_limit_output_bytes) decide how many bytes go down to the device layer. TSQ caps how much one socket may have sitting in the qdisc and device queues, which keeps latency for other flows low.
Qdisc. Each TX queue has a queueing discipline. Modern distros default to fq_codel (fair queueing with active queue management); hosts running BBR typically use fq, which also implements pacing. Drops here show up in tc -s qdisc.
tc -s qdisc show dev eth0
# qdisc mq 0: root
# qdisc fq_codel 0: parent :1 limit 10240p flows 1024 ...
# Sent 98234811 bytes 81230 pkt (dropped 0, overlimits 0 requeues 12)
Byte Queue Limits (BQL) sit between qdisc and driver and limit how many bytes are queued in the NIC's TX ring, so that the bulk of queueing happens in the smart qdisc rather than in a dumb hardware FIFO. That is why giant TX rings don't add latency on BQL-capable drivers.
TSO and GSO. TCP builds skbs up to 64 KB (larger with BIG TCP). With TSO (TCP Segmentation Offload) the NIC splits that into MTU-sized frames and computes checksums in hardware — one trip down the stack for ~45 packets. If the NIC cannot (or a tunnel encapsulation gets in the way), GSO performs the split in software as late as possible, still saving most of the per-packet stack cost.
ethtool -k eth0 | grep -E 'tcp-segmentation-offload|generic-segmentation-offload|tx-checksum'
Doorbell and completion. The driver writes TX descriptors, then writes to a NIC register (the "doorbell", an MMIO write costing hundreds of nanoseconds) to say "new work". After transmission the NIC raises a TX completion interrupt, and NAPI frees the skbs. Only then does the socket's send buffer space become available again — which is what eventually triggers EPOLLOUT for a writer blocked on a full buffer.
Zero-copy send
For large payloads, sendfile() (file → socket), splice(), and MSG_ZEROCOPY (user buffer → NIC via page pinning, completion reported on the socket error queue) skip the copy into kernel skbs. MSG_ZEROCOPY only pays off for writes of roughly 10 KB or more; for smaller writes the page-pinning and completion notification cost more than the copy. io_uring's IORING_OP_SEND_ZC is the same idea with lower overhead.
epoll vs io_uring for Networking
Both are ways for the kernel to tell you sockets are ready (or done). They differ in how many syscalls you pay per operation.
epoll is a readiness API. epoll_wait tells you "fd 17 is readable"; you then call read() yourself. Per request, a typical server pays epoll_wait (amortised over many fds) + read + write — two or three syscalls per request, each ~100–300 ns of entry/exit overhead plus the actual work.
io_uring is a completion API. You place submission entries ("recv into this buffer", "send this buffer", "accept on this socket") in a shared-memory ring, submit with one io_uring_enter (or none, with SQPOLL), and later reap completion entries from another ring. For networking, the features that matter are:
| Feature | Kernel | What it buys |
|---|---|---|
| Multishot accept | 5.19 | One SQE produces a completion per accepted connection |
| Multishot recv + provided buffer rings | 6.0 | One SQE keeps receiving; kernel picks buffers from a pool, so idle connections don't pin buffers |
IORING_OP_SEND_ZC | 6.0 | Zero-copy send without MSG_ZEROCOPY's error-queue dance |
| Zero-copy receive (zcrx) | 6.15+ | NIC writes payload straight into user-registered memory, needs header/data split support in the NIC |
| SQPOLL | 5.1+ | A kernel thread polls the SQ, so submission needs zero syscalls — at the cost of a spinning core |
/* liburing sketch: multishot recv with a provided buffer ring */
struct io_uring ring;
io_uring_queue_init(4096, &ring, 0);
struct io_uring_buf_ring *br;
int ret;
br = io_uring_setup_buf_ring(&ring, NBUFS, BGID, 0, &ret);
for (int i = 0; i < NBUFS; i++)
io_uring_buf_ring_add(br, bufs[i], BUF_SZ, i, io_uring_buf_ring_mask(NBUFS), i);
io_uring_buf_ring_advance(br, NBUFS);
struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
io_uring_prep_recv_multishot(sqe, conn_fd, NULL, 0, 0);
sqe->flags |= IOSQE_BUFFER_SELECT;
sqe->buf_group = BGID;
io_uring_submit(&ring);
/* Each CQE: res = bytes received, flags >> IORING_CQE_BUFFER_SHIFT = buffer id.
IORING_CQE_F_MORE set means the multishot request is still armed. */
When io_uring wins: many small requests per connection, where syscall overhead dominates (proxies, caches, RPC servers), and designs that want a thread-per-core model without readiness callbacks. Gains in published benchmarks are commonly in the 10–30% throughput range for small-message workloads, larger with zero-copy on big payloads.
When it doesn't: when your time is spent in application logic, TLS, or serialization, saving a few hundred nanoseconds of syscall overhead per request disappears into noise. Go's runtime, for instance, uses epoll internally, and most Go services would see little benefit from a hypothetical io_uring netpoller. io_uring has also been the source of many kernel security bugs; several large platforms and container runtimes disable it via seccomp or the kernel.io_uring_disabled sysctl — check whether your environment allows it at all before designing around it.
Busy Polling
Between "interrupt + softirq" and "full kernel bypass" sits busy polling: the application thread itself spins on the NIC queue for a short time instead of sleeping and waiting for an interrupt.
sudo sysctl -w net.core.busy_read=50 # µs to busy-poll in blocking read/recv
sudo sysctl -w net.core.busy_poll=50 # µs to busy-poll in poll/select/epoll
Per socket: SO_BUSY_POLL. With SO_PREFER_BUSY_POLL plus napi_defer_hard_irqs and gro_flush_timeout on the device, the kernel can keep interrupts off entirely while the app is busy-polling, and fall back to interrupts when it goes idle. This removes interrupt and wakeup latency — typically shaving 5–20 µs off a request round trip — at the cost of burning CPU while spinning. It is a latency tool for trading, market data, and tight-SLO key-value stores, not a throughput tool.
Kernel Bypass: XDP, AF_XDP, and DPDK
The full stack costs on the order of hundreds of nanoseconds to a few microseconds per packet. When your workload is "classify and drop, rewrite, or forward millions of packets per second", most of that stack is work you don't need. There are three escape hatches, in increasing order of how much they bypass.
flowchart TD
NIC["NIC RX ring"] --> XDP{"XDP program in driver - before sk_buff"}
XDP -- "XDP_DROP" --> DROP["Dropped - tens of Mpps per core"]
XDP -- "XDP_TX or XDP_REDIRECT" --> OUT["Out another queue or NIC"]
XDP -- "XDP_REDIRECT to XSKMAP" --> AFX["AF_XDP socket - user space, zero copy UMEM"]
XDP -- "XDP_PASS" --> STACK["Normal stack - sk_buff, GRO, netfilter, TCP"]
NIC2["NIC bound to vfio-pci"] --> DPDK["DPDK poll-mode driver in user space - kernel never sees packets"]
XDP — eXpress Data Path
An eBPF program attached to the driver's receive path runs on each packet before an skb is allocated, with direct access to the DMA buffer. It returns a verdict: XDP_DROP, XDP_PASS (continue to the normal stack), XDP_TX (bounce out the same NIC), XDP_REDIRECT (to another NIC, CPU, or AF_XDP socket), or XDP_ABORTED.
// xdp_drop_udp_port.c — drop UDP to port 9999, pass everything else
#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/ip.h>
#include <linux/udp.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_endian.h>
SEC("xdp")
int drop_9999(struct xdp_md *ctx) {
void *data = (void *)(long)ctx->data;
void *end = (void *)(long)ctx->data_end;
struct ethhdr *eth = data;
if ((void *)(eth + 1) > end) return XDP_PASS;
if (eth->h_proto != bpf_htons(ETH_P_IP)) return XDP_PASS;
struct iphdr *ip = (void *)(eth + 1);
if ((void *)(ip + 1) > end) return XDP_PASS;
if (ip->protocol != IPPROTO_UDP) return XDP_PASS;
struct udphdr *udp = (void *)ip + ip->ihl * 4;
if ((void *)(udp + 1) > end) return XDP_PASS;
return udp->dest == bpf_htons(9999) ? XDP_DROP : XDP_PASS;
}
char LICENSE[] SEC("license") = "GPL";
clang -O2 -g -target bpf -c xdp_drop_udp_port.c -o xdp_drop.o
sudo ip link set dev eth0 xdpdrv obj xdp_drop.o sec xdp # native mode
ip link show eth0 | grep xdp
sudo ip link set dev eth0 xdp off
Native (driver) mode is the fast path; xdpgeneric mode works on any NIC but runs after skb allocation and loses most of the benefit. Published measurements put XDP drop at ~20–25 Mpps per core on modern hardware, versus roughly 2–5 Mpps for the cheapest iptables raw-table drop. This is why DDoS scrubbing, L4 load balancers (Meta's Katran, Cloudflare's edge), and Cilium's dataplane use XDP. eBPF itself is covered in eBPF and eBPF in Production.
AF_XDP
An XDP program can redirect frames into an AF_XDP socket, delivering raw frames into a user-space memory region (UMEM) shared with the kernel via fill/completion/RX/TX rings. In zero-copy mode the NIC DMAs directly into UMEM. You get near-DPDK packet rates while the NIC stays under kernel control: other traffic still flows through the normal stack, ip, ethtool, and monitoring keep working.
DPDK
DPDK unbinds the NIC from the kernel driver entirely (binding it to vfio-pci), maps its registers and rings into a user-space process, and runs poll-mode drivers on dedicated cores that spin forever. No interrupts, no syscalls, no skbs, hugepage-backed buffers, batch processing of 32+ packets at a time. A single core can process tens of millions of packets per second for simple forwarding; DPDK applications drive 100 GbE and beyond at line rate.
The price: the kernel no longer sees that NIC. No tcpdump, no ss, no iptables, no kernel TCP. If you need TCP you bring your own user-space stack (F-Stack, Seastar, mTCP, VPP's host stack). Polling cores sit at 100% CPU even when idle. Operational tooling must be rebuilt around the application.
When bypass is worth it
| Situation | Recommendation |
|---|---|
| Web/API service, < 1 Mpps per host | Stay on the kernel stack. Tune RSS, backlogs, buffers |
| Latency-sensitive RPC with tight p99 | Busy polling, IRQ pinning, RFS; maybe io_uring |
| DDoS filtering, L4 load balancing, packet sampling | XDP — keeps the kernel, drops/forwards at tens of Mpps |
| Custom packet processing in user space, kernel coexistence | AF_XDP |
| Virtual switches, 5G user plane, NFV, line-rate 100G+ forwarding | DPDK |
| "We want faster HTTP" | Almost never bypass. Your bottleneck is application code, TLS, or serialization |
Rule of thumb
Kernel bypass saves microseconds per packet. If your request takes milliseconds of application work, bypass changes nothing measurable. Profile first: if %soft + %sys is less than ~30% of CPU, the network stack is not your problem.
Real Numbers
Rough figures for a modern x86 server (2020s CPU, 25–100 GbE NIC). Your hardware will differ; use them for order-of-magnitude reasoning, not capacity planning.
| Metric | Approximate value |
|---|---|
| Per-packet budget at 10 GbE, 64-byte frames | 67 ns |
| Per-packet budget at 100 GbE, 64-byte frames | 6.7 ns |
| Kernel stack, full TCP receive, per core | ~1–3 Mpps (small packets), several Gbit/s of bulk with GRO |
| iptables raw-table drop, per core | ~2–5 Mpps |
| XDP native drop, per core | ~20–25 Mpps |
| DPDK simple forwarding, per core | ~10–30+ Mpps |
| Single TCP flow throughput, one core, with GRO/TSO | ~20–40 Gbit/s |
| Loopback TCP request/response latency (netperf TCP_RR) | ~10–20 µs |
| Same-rack TCP round trip, kernel stack | ~20–50 µs |
| Same with busy polling | ~5–15 µs lower |
| skb metadata size | ~230 bytes |
Default accept queue cap (somaxconn, kernel 5.4+) | 4096 |
Default per-CPU backlog (netdev_max_backlog) | 1000 packets |
| Default softirq budget | 300 packets or 2 ms |
Measure your own with netperf/iperf3 for throughput, sockperf or netperf -t TCP_RR for latency, and pktgen or trex for packet rate.
Troubleshooting: Symptoms to Counters
The single most useful habit: when something is wrong, snapshot counters, wait, snapshot again, and look at what moved. nstat does this automatically — it prints deltas since its last run.
nstat -n # reset baseline
sleep 10
nstat # everything that changed in the last 10 s
| Symptom | Where it dropped | Counter / command |
|---|---|---|
| Packets "lost", nothing in kernel stats | NIC ring full, hardware drop | ethtool -S eth0 → rx_missed_errors, rx_no_buffer, per-queue drops. Fix: bigger ring, more queues, faster softirq |
One CPU at 100% %soft, others idle | Single RX queue or single elephant flow | mpstat -P ALL 1, grep eth0 /proc/interrupts. Fix: RSS/RPS, IRQ affinity, more connections |
Throughput plateaus, ksoftirqd busy | Softirq budget exhaustion | /proc/net/softnet_stat column 3 (time_squeeze) rising. Fix: spread load, cheaper per-packet path, raise budget |
| Drops with RPS enabled | Per-CPU backlog full | /proc/net/softnet_stat column 2. Fix: raise netdev_max_backlog, rebalance RPS |
| New connections time out, existing ones fine, Docker/K8s host | Conntrack table full | dmesg "table full", conntrack -S → drop, insert_failed. Fix: raise nf_conntrack_max, NOTRACK |
| Connect latency in 1 s / 3 s steps under load | Accept queue overflow | ss -ltn Recv-Q ≥ Send-Q, nstat TcpExtListenOverflows TcpExtListenDrops. Fix: accept faster, raise backlog and somaxconn |
| SYN cookies sent without an attack | SYN queue too small | TcpExtSyncookiesSent, TcpExtTCPReqQFullDrop. Fix: raise tcp_max_syn_backlog and listen backlog |
Large Recv-Q on established sockets | App not reading fast enough | ss -tn, check app CPU/GC/locks. Not a kernel problem |
Large Send-Q, low throughput | Peer slow, small window, or congestion | ss -tin → cwnd, rtt, retrans, rcv_space, snd_wnd. See TCP/IP Deep Dive |
| Throughput capped on long-RTT paths | Buffer smaller than BDP | ss -tim → skmem vs BDP. Fix: raise tcp_rmem/tcp_wmem max |
| Receive-side memory drops | Socket buffer or tcp_mem pressure | TcpExtTCPRcvQDrop, TcpExtPruneCalled, TcpExtRcvPruned, cat /proc/net/sockstat |
| Rising retransmits | Loss on path, or local qdisc drops | nstat TcpRetransSegs TcpExtTCPLostRetransmit, tc -s qdisc drops |
| UDP packets missing | Socket receive buffer full | nstat UdpRcvbufErrors UdpInErrors, ss -uam. Fix: larger SO_RCVBUF, read faster. See UDP |
| RST storms, "connection reset by peer" | Accept overflow with tcp_abort_on_overflow=1, closed listener, or middlebox | nstat TcpExtTCPAbortOnData TcpOutRsts, tcpdump 'tcp[tcpflags] & tcp-rst != 0' |
Finding exactly where the kernel freed a packet
Every drop in the kernel ends in kfree_skb, and since kernel 5.17 the tracepoint carries a drop reason (SKB_DROP_REASON_*: NETFILTER_DROP, TCP_CSUM, SOCKET_RCVBUFF, NO_SOCKET, and dozens more):
# Count drops by reason, system-wide
sudo bpftrace -e 'tracepoint:skb:kfree_skb { @[args->reason] = count(); }'
# Count drops by kernel stack (works on older kernels too)
sudo bpftrace -e 'tracepoint:skb:kfree_skb { @[kstack(8)] = count(); }'
# Same idea with perf
sudo perf record -e skb:kfree_skb -a -g -- sleep 10
sudo perf script | head -50
# Who is burning softirq time
sudo perf top -C 3 # CPU 3, where the busy queue lives
# Latency from packet arrival to app wakeup is harder; runqlat and
# softirqs from bcc-tools give the scheduling side
sudo /usr/share/bcc/tools/softirqs 1
sudo /usr/share/bcc/tools/tcpdrop # prints stack + tuple for each TCP drop
For a broader methodology covering packet captures and path debugging, see Network Debugging.
Decision Guide
flowchart TD
S["Network problem suspected"] --> A{"Is softirq + sys CPU high?"}
A -- "No" --> APP["Likely app, peer, or path. Check Recv-Q, Send-Q, ss -ti, retransmits"]
A -- "Yes" --> B{"Concentrated on one CPU?"}
B -- "Yes" --> C["Check RSS queues, IRQ affinity, single elephant flow"]
B -- "No" --> D{"Drops in ethtool -S or softnet_stat?"}
D -- "Ring drops" --> E["Grow RX ring, add queues"]
D -- "time_squeeze" --> F["Reduce per-packet cost: GRO on, fewer netfilter rules, conntrack exemptions"]
D -- "None, just high CPU" --> G{"Packet rate above ~1-2 Mpps per core?"}
G -- "Yes, filtering or forwarding" --> H["XDP or AF_XDP"]
G -- "Yes, full line-rate NFV" --> I["DPDK"]
G -- "No" --> J["Profile with perf, look for iptables, conntrack, copy costs"]
Defaults that are worth checking on every new fleet:
somaxconnand the backlog your framework actually passes tolisten().- RX/TX ring sizes versus hardware maximum.
- Number of combined queues equals the number of cores on the NIC's NUMA node you want doing network work.
irqbalancerunning (or a deliberate manual pinning policy — not both).- GRO, TSO, GSO, and checksum offloads on.
nf_conntrack_maxsized for peak flows on any NAT/container host.tcp_rmem/tcp_wmemmaxima covering your largest realistic BDP.- A dashboard of
ListenOverflows,softnet_statdrops and squeezes,rx_missed_errors, andTcpRetransSegs— they will explain most incidents before anyone opens a packet capture.
Interview Questions
Walk through what happens when a TCP packet arrives at a Linux server.
NIC validates and hashes the frame, DMAs it into an RX ring buffer slot, raises an MSI-X interrupt. The handler masks the queue's interrupt and schedules NAPI. The NET_RX softirq polls the driver, which builds an sk_buff around the buffer and refills the ring. GRO merges segments, netfilter and conntrack run, IP routes it for local delivery, TCP looks up the socket, validates sequence numbers, queues data, and sends an ACK. The socket's data-ready callback wakes epoll waiters; the app calls read(), which copies bytes into user space.
Why NAPI instead of one interrupt per packet? At 10 GbE line rate a packet arrives every 67 ns, and an interrupt costs over a microsecond. NAPI takes one interrupt, disables further ones, and polls the ring in batches until it's empty or the budget runs out — adaptive between low-latency interrupts at low load and efficient polling at high load.
Clients see 1-second and 3-second connect delays under load, the server's CPU is fine. What's going on?
Accept queue overflow. The final ACK of the handshake is dropped because the queue is full, so the server retransmits its SYN-ACK on its timeout schedule. Confirm with ss -ltn (Recv-Q at Send-Q) and TcpExtListenOverflows. Fix by accepting faster and raising both the listen() backlog and somaxconn.
What's the difference between RSS, RPS, and RFS? RSS is NIC hardware hashing flows to multiple RX queues with separate IRQs. RPS does the same distribution in software after the driver poll for NICs with too few queues. RFS steers each flow to the CPU where the consuming application thread runs, for cache locality.
When would you use XDP versus DPDK? XDP when you want to filter, drop, or forward packets at tens of Mpps while keeping the kernel in control of the NIC — DDoS mitigation, L4 load balancing. DPDK when you need the entire NIC in user space at line rate and are willing to give up kernel tooling and bring your own stack — virtual switches, telecom user planes.
Why can a single TCP connection not use a 10 Gbit/s link across an ocean?
Throughput is capped at window / RTT. With a 6 MB max receive buffer and 80 ms RTT, that's ~600 Mbit/s. The buffer must be at least the bandwidth-delay product (100 MB here), so raise tcp_rmem/tcp_wmem maxima — or use parallel streams.
Related
- Interrupts, DMA & Device Drivers — the hardware mechanics under the receive path
- I/O & Storage Internals — epoll and io_uring APIs
- Syscalls & Kernel Internals — what each
read()andwrite()costs - CPU Scheduling — how ksoftirqd competes with your threads
- Namespaces & cgroups — network namespaces and veth pairs add another hop to this path
- TCP/IP Deep Dive — the protocol the stack implements
- UDP — the simpler transport and its receive-buffer drops
- NAT — why conntrack exists
- Network Debugging — packet captures and path tools
- eBPF — the engine behind XDP and modern dataplanes