On this page

System Calls & Kernel Internals

Every I/O operation, every process creation, every memory mapping — every interaction between user code and the hardware — goes through a system call. Understanding syscall mechanics explains why context-switching is expensive, why async I/O exists, why Rust's tokio and Go's runtime avoid unnecessary syscalls, and why eBPF can add observability without modifying application code.

A system call is the only door between your program and the machine. Your process cannot open a file, send a packet, allocate a page, read the clock (well, almost — see vDSO), or even exit without asking the kernel. That makes the syscall boundary three things at once: a performance boundary (every crossing costs hundreds of nanoseconds, more since 2018), a security boundary (seccomp, containers and sandboxes all work by filtering it), and an observability boundary (strace, perf and eBPF can see everything a program does to the outside world by watching it).

This page covers the boundary from the CPU instruction up: how the switch to kernel mode happens, what it costs and why that changed with Meltdown, how the kernel is structured behind the door, how to watch syscalls in production, and how modern I/O interfaces avoid crossing the boundary at all.

Related reading

Processes & Threads for what the kernel is scheduling, Virtual Memory for page tables and the TLB (central to KPTI), and I/O & Storage for the I/O models that io_uring improves on.


The User/Kernel Boundary

The CPU operates in different privilege rings:

Ring 0 (kernel mode): full hardware access
Ring 1, 2: rarely used
Ring 3 (user mode): restricted — no direct hardware access, no arbitrary memory access

Modern x86-64 uses only Ring 0 and Ring 3.

User-space code runs in Ring 3. It cannot:

  • Access hardware directly (disk, NIC, GPU)
  • Read/write memory it doesn't own
  • Modify CPU control registers
  • Execute privileged instructions

To do any of these, it must request the kernel do it — via a system call.

How the CPU enforces it

The current privilege level (CPL) lives in the low two bits of the CS segment register. The hardware checks it in several places:

MechanismWhat it stops
Privileged instructions (mov cr3, wrmsr, hlt, in/out, lgdt)Fault with #GP if executed at CPL 3
Page-table U/S bitUser mode touching a supervisor page → page fault → SIGSEGV
SMEP (Supervisor Mode Execution Prevention)Kernel executing user-space pages — blocks the classic "jump to shellcode in user memory" exploit
SMAP (Supervisor Mode Access Prevention)Kernel reading/writing user pages except inside explicit stac/clac windows (copy_from_user)
IOMMUDevices DMA-ing to memory they weren't given

The other common "rings" you'll hear about are informal: ring −1 for the hypervisor (VMX root mode), ring −2 for System Management Mode firmware. ARM64 names them exception levels: EL0 (user), EL1 (kernel), EL2 (hypervisor), EL3 (secure monitor).

flowchart TB
    subgraph U["Ring 3 / EL0: user space"]
        APP["Application"]
        LIBC["libc wrappers, vDSO"]
    end
    subgraph K["Ring 0 / EL1: kernel"]
        ENTRY["Syscall entry (entry_SYSCALL_64)"]
        VFS["VFS and file systems"]
        NET["Network stack"]
        MM["Memory management"]
        SCHED["Scheduler"]
        DRV["Device drivers"]
    end
    HW["Hardware: CPU, RAM, NIC, NVMe"]
    APP --> LIBC
    LIBC -->|"SYSCALL instruction"| ENTRY
    ENTRY --> VFS
    ENTRY --> NET
    ENTRY --> MM
    ENTRY --> SCHED
    VFS --> DRV
    NET --> DRV
    DRV --> HW

The System Call Mechanism

x86-64 Linux system call path:

User code calls read(fd, buf, n)
  ↓
glibc wrapper:
  1. Load syscall number into %rax  (e.g., 0 for read)
  2. Load arguments into %rdi, %rsi, %rdx, %r10, %r8, %r9
  3. Execute SYSCALL instruction
  ↓
CPU switches to Ring 0:
  4. Saves user stack pointer (RSP) to kernel stack
  5. Loads kernel stack pointer from MSR_LSTAR
  6. Jumps to kernel entry point (entry_SYSCALL_64)
  ↓
Kernel handles the call:
  7. Saves user registers to kernel stack
  8. Looks up handler: sys_call_table[%rax]
  9. Executes sys_read()
  10. Returns value in %rax
  ↓
SYSRET instruction:
  11. Restores user registers
  12. Switches back to Ring 3
  13. Jumps back to user code (next instruction after SYSCALL)
  ↓
User code continues with return value

Precision on steps 4–6

The SYSCALL instruction itself does less than the list suggests. In hardware it only: saves the return address (RIP) into RCX, saves RFLAGS into R11, masks RFLAGS using the IA32_FMASK MSR (clearing the interrupt flag), loads CS/SS for ring 0 from IA32_STAR, and jumps to the address in IA32_LSTAR (MSR_LSTAR). It does not switch stacks. The first instructions of entry_SYSCALL_64 do that in software: swapgs to reach per-CPU data, stash the user RSP in a per-CPU slot, load the kernel stack pointer, and (under KPTI) switch CR3 to the kernel page tables. SYSRET reverses it: RIP ← RCX, RFLAGS ← R11, back to ring 3.

That's why the syscall calling convention uses %r10 instead of %rcx for the fourth argument (the normal C convention uses rcx): SYSCALL overwrites rcx and r11. You can make a syscall with no libc at all:

// raw_write.c — gcc -O2 raw_write.c -o raw_write
#include <sys/syscall.h>

static long raw_syscall3(long nr, long a1, long a2, long a3) {
    long ret;
    __asm__ volatile ("syscall"
                      : "=a"(ret)                         // return value in rax
                      : "a"(nr), "D"(a1), "S"(a2), "d"(a3) // rax, rdi, rsi, rdx
                      : "rcx", "r11", "memory");          // clobbered by SYSCALL
    return ret;
}

int main(void) {
    long n = raw_syscall3(SYS_write, 1, (long)"hello from ring 3\n", 18);
    return n == 18 ? 0 : 1;
}
sequenceDiagram
    participant App as "App (ring 3)"
    participant Libc as "glibc read()"
    participant CPU as "CPU"
    participant K as "Kernel (ring 0)"
    App->>Libc: read(fd, buf, 4096)
    Libc->>CPU: rax=0, rdi=fd, rsi=buf, rdx=4096, SYSCALL
    CPU->>K: save RIP to RCX and RFLAGS to R11, jump to LSTAR
    K->>K: swapgs, switch stack, switch CR3 (KPTI)
    K->>K: sys_call_table[0] = ksys_read
    K->>K: copy page cache data to buf (copy_to_user)
    K->>CPU: rax = bytes read or negative errno, SYSRET
    CPU->>Libc: back in ring 3
    Libc->>App: return n, or -1 and set errno

The SYSCALL/SYSRET pair (modern x86-64) is faster than the older INT 0x80 interrupt mechanism because it doesn't go through the interrupt descriptor table — it reads the kernel entry point from a Model-Specific Register (MSR).

Other entry mechanisms you'll meet:

MechanismWhereNotes
int 0x8032-bit x86 Linux (still works on x86-64 for 32-bit ABI)Goes through the IDT; slowest
sysenter/sysexit32-bit x86 (Intel)What the 32-bit vDSO used for speed
syscall/sysretx86-64The standard path
svc #0ARM64Syscall number in x8, args in x0–x5, return in x0
ecallRISC-VNumber in a7, args in a0–a5
FREDNewest Intel CPUs, Linux 6.9+Flexible Return and Event Delivery: replaces the legacy SYSCALL/IDT entry paths with a cleaner, faster event-delivery model

Return values, errno and interrupted calls

The kernel returns a single value in rax. Values from −4095 to −1 are errors (-EBADF, -EAGAIN …); glibc's wrapper turns them into -1 and sets the thread-local errno. Two consequences:

  • EINTR: a blocking syscall (read on a socket, nanosleep, futex wait) interrupted by a signal returns -EINTR unless the handler was installed with SA_RESTART — so robust code loops on EINTR. See Signals.
  • Short reads/writes are not errors: write(fd, buf, 1MB) on a socket may return 64KB. Every correct I/O loop handles partial completion.

A blocking syscall is also where your thread leaves the CPU: the kernel marks the task as sleeping, calls the scheduler, and another task runs until the I/O completes — the voluntary context switch counted in /proc/<pid>/status as voluntary_ctxt_switches. See CPU Scheduling.


Syscall Cost

A syscall takes 100–1000 ns depending on:

  • What the kernel does (simple check vs. disk I/O)
  • Cache state (kernel code and data may be cold)
  • Spectre mitigations (KPTI — Kernel Page Table Isolation)

Splitting that up: the bare SYSCALL→SYSRET round trip with an empty handler is on the order of 50–100 ns on a modern CPU with no mitigations. Mitigations for transient-execution vulnerabilities can add another 50–500 ns on affected CPUs. And the real work inside the handler — locking, copying data, walking the network stack — is often larger than either. Plus the hidden cost: the syscall evicts some of your hot data from L1/L2 caches, branch predictors and the TLB, so your user code runs slower for a while afterwards. Soares and Stumm's FlexSC paper (OSDI 2010) measured this indirect cost as larger than the direct cost for syscall-heavy server workloads.

Measure it yourself:

// syscost.c — gcc -O2 syscost.c -o syscost && ./syscost
#define _GNU_SOURCE
#include <stdio.h>
#include <time.h>
#include <unistd.h>
#include <sys/syscall.h>

static double now_ns(void) {
    struct timespec t;
    clock_gettime(CLOCK_MONOTONIC, &t);
    return t.tv_sec * 1e9 + t.tv_nsec;
}

int main(void) {
    enum { N = 10000000 };
    struct timespec ts;
    double t0 = now_ns();
    for (int i = 0; i < N; i++) syscall(SYS_getppid);            // real kernel entry
    double t1 = now_ns();
    for (int i = 0; i < N; i++) clock_gettime(CLOCK_MONOTONIC, &ts); // vDSO, no entry
    double t2 = now_ns();
    printf("getppid (syscall):     %6.1f ns\n", (t1 - t0) / N);
    printf("clock_gettime (vDSO):  %6.1f ns\n", (t2 - t1) / N);
    return 0;
}

Typical results (orders of magnitude; your CPU and kernel will differ):

Machinegetppid syscallclock_gettime via vDSO
Recent AMD/Intel, not Meltdown-vulnerable, default mitigations~60–120 ns~15–25 ns
2015–2018 Intel server with KPTI + PCID~150–300 ns~20 ns
Same, KPTI without PCID (old CPU or VM without PCID passthrough)~400–1000 ns~20 ns
VM whose clocksource can't be read from user space—falls back to a syscall: ~300+ ns

KPTI overhead: After the Meltdown vulnerability (2018), Linux separated kernel and user page tables. On every syscall entry, the CPU switches to the kernel's full page table; on return, it switches back. This requires a TLB flush on CPUs without PCID support, adding 10–30% overhead to syscall-heavy workloads.

Meltdown, Spectre and their mitigations

Meltdown (2018) let user code speculatively read kernel memory through the cache side channel, because the kernel was mapped (supervisor-only) in every process's page tables. KPTI removes almost the entire kernel from the page tables used in user mode, leaving only a tiny trampoline for entry/exit — hence the CR3 switch on every crossing. With PCID (see Virtual Memory), user and kernel page tables get separate TLB tags so the switch doesn't flush; without it, every syscall flushes the TLB.

KPTI is one of a family; each added a cost somewhere on the syscall path:

MitigationVulnerabilityWhere it costs
KPTI (page table isolation)Meltdown (2018)CR3 write on every entry and exit
Retpoline / IBRS / eIBRSSpectre v2, branch target injection (2018)Every indirect branch in the kernel
VERW buffer clearingMDS / ZombieLoad (2019)Every return to user space
Return thunks, IBPB on entry, RSB stuffingRetbleed (2022), AMD SRSO/Inception (2023)Function returns, context switches
BHI clearing sequenceBranch History Injection (2024)Syscall entry
grep . /sys/devices/system/cpu/vulnerabilities/*
# /sys/devices/system/cpu/vulnerabilities/meltdown:Not affected
# /sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Enhanced / Automatic IBRS; IBPB: conditional; ...
# /sys/devices/system/cpu/vulnerabilities/retbleed:Mitigation: untrained return thunk; SMT enabled with STIBP protection
dmesg | grep -i 'page table isolation'

Real-world impact depends almost entirely on syscall rate. Brendan Gregg's 2018 analysis of KPTI at Netflix showed overhead under ~1% for workloads below roughly 10,000 syscalls/second per CPU, rising steeply for workloads making hundreds of thousands per second per CPU — and dramatically larger without PCID. Database and proxy benchmarks of the time (PostgreSQL SELECT 1 loops, Redis with small values) showed 5–30% throughput loss on affected hardware. CPUs designed after these disclosures fix many of them in hardware (Intel since Cascade Lake/Ice Lake and AMD Zen are not affected by Meltdown, so KPTI is off on them), but newer issues like Retbleed and SRSO brought double-digit-percent costs back for some older CPUs in syscall-heavy benchmarks.

mitigations=off

The mitigations=off kernel parameter disables all of these and can win back 5–30% on syscall-heavy workloads on older CPUs. It is only defensible on single-tenant machines running only trusted code — never on multi-tenant hosts, CI runners, or anything that runs JavaScript or untrusted plugins. Reducing syscall count is the better lever.

Why this matters for system design:

  • An HTTP server doing 100K RPS with 4 syscalls per request = 400K syscalls/sec. At 500ns each = 200ms of CPU per second just on syscalls.
  • io_uring reduces this to near zero by batching submissions/completions through shared ring buffers.
  • Go's runtime and Rust's tokio are designed to minimize syscalls per operation.

(In practice a request needs more than 4 — epoll_wait, accept4, read, write, maybe setsockopt, close, upstream connect/sendmsg/recvmsg, logging writes — 10–20 is common, which is why syscall counting per request is a worthwhile profiling exercise.)

# Count syscalls for a program
strace -c ./app

# Same, but trace every syscall class explicitly (the default already traces all;
# -e trace=... is useful to narrow it down instead, e.g. -e trace=%network)
strace -c -e trace=all ./app

# Watch live syscalls
strace -p <pid>

# Count syscalls across all threads
strace -f -c ./multithreaded-app

vDSO: Virtual Dynamic Shared Object

For some syscalls, the cost of ring transition is unnecessary because the kernel data is safe to read from user space. The vDSO maps a small read-only kernel-owned page into every process's address space.

Functions that can be satisfied from vDSO:

  • clock_gettime() — reads from a kernel-maintained time struct
  • gettimeofday() — same
  • getcpu() — reads CPU and NUMA node without a syscall
  • time() — seconds since epoch

Linux 6.11 added a fifth: getrandom() in the vDSO (glibc 2.41+ uses it), so generating random numbers no longer needs a kernel entry per call.

// Looks like a syscall but executes entirely in user space via vDSO:
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
// No mode switch to ring 0, no TLB invalidation
// ~5ns instead of ~100ns
# View vDSO mapping in a process
cat /proc/<pid>/maps | grep vdso
# 7ffd...000 r-xp [vdso]

How it actually works

The vDSO is a real ELF shared library (linux-vdso.so.1 in ldd output) that the kernel maps into every process at execve, alongside a data page called [vvar]. The kernel's timekeeping code updates vvar on every timer tick with the current base time, the TSC value at that moment, and the multiplier to convert TSC cycles to nanoseconds. The vDSO's clock_gettime then:

  1. Reads a sequence counter from vvar (a seqlock)
  2. Reads the base time and conversion parameters
  3. Executes rdtsc (or rdtscp) — readable from ring 3
  4. Re-reads the sequence counter; if it changed (the kernel updated mid-read), retries
  5. Returns base + (tsc_now − tsc_base) × mult >> shift

No privilege change is needed because nothing privileged happens: it's reading memory the kernel made readable and a CPU counter user mode is allowed to read. glibc finds the vDSO's symbols through the auxiliary vector (AT_SYSINFO_EHDR) and calls them in place of the syscall.

It only works if the clocksource is readable from user space. With tsc (or arm_arch_timer on ARM) it is; with hpet, acpi_pm, or some hypervisor clocks it isn't, and the vDSO silently falls back to a real syscall.

cat /sys/devices/system/clocksource/clocksource0/current_clocksource      # tsc  ← good
cat /sys/devices/system/clocksource/clocksource0/available_clocksource    # tsc hpet acpi_pm

War Story (real, well documented)

In 2017 Packagecloud published "Two frequently used system calls are ~77% slower on AWS EC2." On Xen-based EC2 instances, the default clocksource was xen, which the vDSO couldn't read, so every gettimeofday/clock_gettime became a full syscall. Applications that timestamp heavily — logging, metrics, tracing, databases — paid a real kernel entry for every call; strace -c on such a process shows thousands of clock_gettime calls per second, which should never appear at all. The fix was switching the clocksource to tsc (echo tsc > .../current_clocksource, or clocksource=tsc on the kernel command line), which AWS later documented for its Xen instance types. Newer Nitro (KVM-based) instances use kvm-clock/tsc that the vDSO supports.

Why not vDSO everything? Most syscalls need kernel to act (write to disk, create a socket, etc.) — they can't be done in user space.


Kernel vs User Time

The time command reports wall time, user time, and sys time:

time dd if=/dev/zero of=/dev/null bs=1M count=1000

real    0m0.271s
user    0m0.008s
sys     0m0.262s
  • user: time spent executing user-space code (your application)
  • sys: time spent executing kernel code on behalf of your process (syscalls)
  • real: wall clock time (includes I/O wait, scheduling delays)

A high sys time relative to user time indicates the program is syscall-heavy — potential candidate for io_uring, batching, or vDSO-eligible calls.

(In the dd example, nearly all the time is sys because each 1MB read from /dev/zero is the kernel zeroing memory, and each write to /dev/null is a no-op handler: the program is pure syscalls.)

For a running service:

pidstat -u -p <pid> 1          # %usr vs %system per second
top -H -p <pid>                # per-thread; press 1 for per-CPU us/sy split
perf top -p <pid>              # kernel symbols (copy_user_enhanced_fast_string,
                               # _raw_spin_lock, tcp_sendmsg ...) show where sys time goes
cat /proc/<pid>/stat | awk '{print "utime="$14, "stime="$15}'   # clock ticks

sys time can also be high for reasons that aren't syscalls you chose: page faults (kernel handles them), memory reclaim and compaction, TLB shootdowns, spinlock contention inside the kernel (_raw_spin_lock at the top of perf top — see Kernel Synchronization), or softirq processing of network traffic.


Kernel Architecture: Monolithic, Micro, Hybrid

Behind the syscall door, how is the kernel itself organised? The main design question is how much code runs in ring 0.

flowchart LR
    subgraph MONO["Monolithic (Linux, FreeBSD)"]
        direction TB
        MU["User apps"] -->|"syscall"| MK["One kernel image in ring 0: scheduler, MM, VFS, file systems, TCP/IP, drivers"]
    end
    subgraph MICRO["Microkernel (seL4, QNX, MINIX 3)"]
        direction TB
        UU["User apps"] -->|"IPC"| SRV["User-space servers: file system, network, drivers"]
        SRV -->|"IPC"| UK["Tiny kernel in ring 0: address spaces, threads, IPC"]
    end
MonolithicMicrokernelHybrid
Ring 0 containsEverything: drivers, file systems, network stackOnly scheduling, address spaces, IPCCore plus selected services
ExamplesLinux, FreeBSD, OpenBSDseL4, QNX, MINIX 3, Fuchsia's ZirconWindows NT, macOS XNU (Mach + BSD)
PerformanceSubsystems call each other directly: fastEvery service request is IPC between address spaces: historically slower (L4/seL4 narrowed the gap greatly)In between
Fault isolationA driver bug can crash or compromise the whole kernelA crashing driver is restarted like a processPartial
Verification~40M lines (Linux) — impracticalseL4's ~10k lines are formally proven correct—
Where it winsGeneral-purpose servers, desktops, AndroidSafety-critical and embedded (QNX in cars, seL4 in defence/aerospace)Desktop OSes with legacy

The Torvalds–Tanenbaum debate (1992) argued exactly this. Linux stayed monolithic, and in practice it gets much of the microkernel benefit in other ways: user-space drivers and file systems where isolation matters more than speed (FUSE, VFIO/UIO, DPDK and SPDK for kernel-bypass networking and storage), and eBPF for safe kernel extension.

Loadable kernel modules

"Monolithic" doesn't mean "one static blob". Linux loads drivers, file systems, network protocols and more as modules at runtime — they run in ring 0 with full privileges, as part of the kernel, but are linked in on demand.

lsmod | head                       # loaded modules, size, users
modinfo nvme                       # description, parameters, signer, dependencies
sudo modprobe br_netfilter         # load with dependencies
sudo modprobe -r dummy             # unload
cat /proc/sys/kernel/tainted       # non-zero: proprietary/unsigned/out-of-tree modules loaded
cat /sys/module/nvme_core/parameters/io_timeout
// hello.c — a minimal out-of-tree module
#include <linux/init.h>
#include <linux/module.h>

static int __init hello_init(void) { pr_info("hello: loaded\n"); return 0; }
static void __exit hello_exit(void) { pr_info("hello: unloaded\n"); }

module_init(hello_init);
module_exit(hello_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Minimal example module");
# Makefile:  obj-m += hello.o
make -C /lib/modules/$(uname -r)/build M=$PWD modules
sudo insmod hello.ko && dmesg | tail -1     # hello: loaded
sudo rmmod hello

Modules are powerful and dangerous: a bug is a kernel crash, there is no stable in-kernel ABI (modules must be rebuilt for each kernel version, which is why DKMS exists), and Secure Boot systems require them to be signed. That's the gap eBPF fills for observability, networking and security use cases.


The Kernel System Call Table

Linux x86-64 has ~340 system calls. The kernel maps syscall numbers to handler functions in sys_call_table[].

(Updated count: as of 6.x kernels there are roughly 380 — numbers 0–335 plus 424 onward; 336–423 are skipped so new syscalls get the same number on every architecture. Recent additions include io_uring_setup (425), landlock_create_ruleset (444), cachestat (451), mseal (462).)

Key syscalls grouped by category:

// Files & I/O
0   sys_read          1   sys_write
2   sys_open          3   sys_close
8   sys_lseek       257   sys_openat       // modern glibc open() uses openat
217 sys_getdents64  137   sys_statfs
19  sys_readv        20   sys_writev
40  sys_sendfile    275   sys_splice

// Process
56  sys_clone (threads/processes)   59  sys_execve
62  sys_kill        63   sys_uname
39  sys_getpid     202   sys_futex       // the slow path of every mutex

// Memory
9   sys_mmap         11  sys_munmap
12  sys_brk          10  sys_mprotect
319 sys_memfd_create

// Network
41  sys_socket        49  sys_bind
43  sys_accept        42  sys_connect
44  sys_sendto        45  sys_recvfrom
291 sys_epoll_create1 232 sys_epoll_wait

// Async I/O, observability
425 sys_io_uring_setup  426 sys_io_uring_enter
321 sys_bpf

Correction

Earlier versions of this table listed read/write/open/close under "Process", and had several wrong numbers: bind is 49 (50 is listen), connect is 42 (45 is recvfrom), recvfrom is 45 (46 is sendmsg), getdents64 is 217 (78 is the older getdents), and memfd_create is 319 (258 is mkdirat). Always check against your system:

ausyscall --dump | head             # from the auditd package
grep -E '__NR_(bind|connect|memfd_create) ' /usr/include/x86_64-linux-gnu/asm/unistd_64.h

Numbers are per-architecture: read is 0 on x86-64, 63 on ARM64, 3 on 32-bit x86. This is why seccomp filters must check the architecture field before the syscall number.


Tracing System Calls

Because every interaction with the outside world is a syscall, watching syscalls is the fastest way to answer "what is this process actually doing?" — which file is it failing to open, which host is it connecting to, why is it stuck. There are four main tools, with very different overheads.

ToolMechanismOverheadBest for
straceptrace: the tracer is woken at every syscall entry and exitHigh: 2× to 100×+ slower for syscall-heavy programsDebugging one process, dev boxes, "why does this fail?"
ltraceBreakpoints on PLT entries of shared-library callsVery highLibrary calls (malloc, strlen, SSL_read)
perf traceKernel tracepoints via perf ring buffersLow–moderatestrace-like output with much less slowdown; system-wide
bpftrace / BCCeBPF programs on tracepoints/kprobes, aggregation in kernelLow (only aggregates leave the kernel)Production: histograms, counts, filtered events

strace

strace -f -tt -T -e trace=%network -p <pid>
# 12:04:31.218734 connect(7, {sa_family=AF_INET, sin_port=htons(5432), sin_addr=inet_addr("10.0.3.17")}, 16)
#                   = -1 EINPROGRESS (Operation now in progress) <0.000041>
# 12:04:31.218902 poll([{fd=7, events=POLLOUT}], 1, 5000) = 0 (Timeout) <5.005112>   ← 5s connect timeout

strace -e trace=%file -f ./app 2>&1 | grep ENOENT     # which config file is it looking for?
strace -y -e trace=read,write -p <pid>                 # -y: show paths/sockets for fds
strace -k -e trace=mmap -p <pid>                       # -k: stack trace for each call
strace -c -f -p <pid>                                  # Ctrl-C after a while for a summary
strace --seccomp-bpf -f -e trace=execve ./build.sh     # only stop on traced calls: much lower overhead
$ strace -c -f -p 2112        # summary from a busy web worker (example output)
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 41.20    1.204331          11    104212           epoll_wait
 22.87    0.668611           6    104187     12044 read
 18.03    0.527013           5    104201           write
  9.11    0.266210         128      2079           futex
  4.32    0.126301           4     28901           clock_gettime    ← vDSO not working?
  ...
------ ----------- ----------- --------- --------- ----------------
100.00    2.923211                 352112     12044 total

That last line is a red flag in its own right: clock_gettime should never appear in strace when the vDSO works.

strace in production

Because every syscall stops the process twice and context-switches to the tracer, strace can make a syscall-heavy service many times slower — and if strace is killed uncleanly the target can be left in an odd state. Prefer perf trace or bpftrace on production hosts, keep strace sessions short, and use --seccomp-bpf with a narrow -e trace= filter when you must.

ltrace

ltrace -c ./app                    # count library calls
ltrace -e malloc+free -p <pid>     # watch allocator calls
ltrace -S -e SSL_write ./client    # -S also shows syscalls interleaved

Note that ltrace only sees calls through the PLT (dynamically linked functions); statically linked binaries and most Go programs are invisible to it. uprobes via bpftrace are the lower-overhead alternative (bpftrace -e 'uprobe:/lib/x86_64-linux-gnu/libc.so.6:malloc { @[comm] = count(); }').

perf trace

perf trace -p <pid>                          # strace-like live output
perf trace -s -p <pid> -- sleep 10           # per-thread summary with min/avg/max latency per syscall
perf trace --duration 10 -p <pid>            # only syscalls that took longer than 10 ms
perf trace -e 'openat,connect' -a            # system-wide, selected syscalls
perf stat -e 'raw_syscalls:sys_enter' -a -- sleep 1   # total syscalls/sec on the box

bpftrace

# Trace open() syscalls in real time with bpftrace
bpftrace -e 'tracepoint:syscalls:sys_enter_open { printf("%s %s\n", comm, str(args->filename)); }'
# Modern glibc calls openat(), not open(), so this usually catches more:
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'

# Top syscalls by process, system-wide, for 10 seconds
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm, args->id] = count(); } interval:s:10 { exit(); }'

# Latency histogram of read() for one process (microseconds)
bpftrace -e '
tracepoint:syscalls:sys_enter_read /pid == $1/ { @start[tid] = nsecs; }
tracepoint:syscalls:sys_exit_read  /@start[tid]/ {
  @usecs = hist((nsecs - @start[tid]) / 1000); delete(@start[tid]);
}' <pid>

# Which processes fail syscalls, and with which error?
bpftrace -e 'tracepoint:raw_syscalls:sys_exit /args->ret < 0/ { @[comm, args->id, -args->ret] = count(); }'

The BCC toolkit ships ready-made versions: syscount (counts and latency by syscall), opensnoop, execsnoop, tcpconnect, killsnoop. For the full eBPF tooling story, see eBPF and eBPF in Production.


Batching and Avoiding Syscalls

The fastest syscall is the one you don't make. Four strategies, in roughly the order you should reach for them:

  1. Buffer in user space. stdio, Go's bufio, Rust's BufWriter, Java's BufferedOutputStream: turn 10,000 tiny writes into a few large ones. Unbuffered logging (write per line) is the most common accidental syscall storm.
  2. Do more per syscall — vectored I/O and multi-message calls.
  3. Move data inside the kernel — sendfile, splice, copy_file_range: zero-copy, and fewer crossings.
  4. Change the interface — epoll instead of polling each fd, then io_uring to batch submissions and completions through shared memory.

Vectored I/O: one syscall, many buffers

An HTTP response is a header buffer plus a body buffer. Copying them into one buffer costs a memcpy; writing them separately costs two syscalls (and, without TCP_CORK/MSG_MORE, possibly two packets). writev does both in one call with no copy:

#include <sys/uio.h>

struct iovec iov[3];
iov[0].iov_base = hdr;   iov[0].iov_len = hdr_len;
iov[1].iov_base = body;  iov[1].iov_len = body_len;
iov[2].iov_base = "\r\n"; iov[2].iov_len = 2;
ssize_t n = writev(sock, iov, 3);       // one syscall, three buffers (gather)
// readv() is the scatter counterpart; preadv2/pwritev2 add offsets and flags

For datagrams, sendmmsg/recvmmsg send or receive many messages per syscall — essential for UDP-heavy servers such as DNS resolvers and QUIC stacks, which can process dozens of packets per crossing. UDP_SEGMENT (GSO) goes further, handing the kernel one large buffer to split into packets.

Zero-copy: sendfile, splice, copy_file_range

Serving a static file with read + write copies every byte twice through user space:

read():   disk/page cache ──copy──▶ user buffer     (syscall 1)
write():  user buffer ──copy──▶ socket buffer ──DMA──▶ NIC   (syscall 2)

sendfile keeps the data in the kernel and, with a NIC that supports scatter-gather, avoids the CPU copy entirely by pointing the NIC's DMA at page-cache pages:

sendfile(): page cache ──(page refs, no CPU copy)──▶ socket ──DMA──▶ NIC   (one syscall)
#include <sys/sendfile.h>
off_t off = 0;
while (off < st.st_size) {
    ssize_t n = sendfile(sock, file_fd, &off, st.st_size - off);   // advances off
    if (n <= 0) { if (n < 0 && errno == EAGAIN) wait_writable(sock); else break; }
}
CallMoves data betweenTypical use
sendfile(out, in, …)file → socket (or any fd since 2.6.33)Static file servers: nginx sendfile on, Kafka consumers reading log segments
splice(in, out, …)any fd ↔ pipe, by moving page referencesProxies: socket → pipe → socket (HAProxy, socat)
teepipe → pipe without consumingDuplicating a stream
vmspliceuser memory → pipeFeeding pipes without a copy
copy_file_rangefile → file (can use reflinks / server-side copy on NFS, XFS, Btrfs)cp, backup tools
MSG_ZEROCOPY (send flag)user buffer → socket without copy; completion via error queueVery large sends (≥ ~10KB) where the copy dominates

Go does this for you: io.Copy(tcpConn, file) uses sendfile, and io.Copy(tcpConn, otherTCPConn) uses splice on Linux. Kafka's famously high consumer throughput comes largely from serving log segments to consumers with sendfile straight from the page cache. TLS complicates zero-copy because data must be encrypted; kTLS moves symmetric encryption into the kernel (or the NIC) so sendfile works over HTTPS — nginx and Netflix's FreeBSD CDN use it.

io_uring: shared-memory rings

epoll tells you which fds are ready, but you still make a read/write syscall per operation. io_uring (Linux 5.1, 2019, by Jens Axboe) replaces that with two ring buffers shared between your process and the kernel:

flowchart LR
    subgraph USER["User space"]
        APP["Application"]
    end
    subgraph SHARED["Shared memory (mmap)"]
        SQ["Submission Queue: SQEs (op, fd, buf, len, user_data)"]
        CQ["Completion Queue: CQEs (user_data, result)"]
    end
    subgraph KERN["Kernel"]
        IO["io_uring worker / SQPOLL thread"]
    end
    APP -->|"write SQEs, bump tail"| SQ
    SQ -->|"io_uring_enter once per batch, or none with SQPOLL"| IO
    IO -->|"post CQEs"| CQ
    CQ -->|"read CQEs, bump head"| APP
  • The application writes submission queue entries (read this fd into this buffer, accept on this socket, send this, fsync that…) into the SQ ring and makes one io_uring_enter call to submit a whole batch.
  • The kernel posts completion queue entries into the CQ ring; the application reads them from memory without a syscall.
  • With IORING_SETUP_SQPOLL, a kernel thread polls the SQ, so a busy application makes zero syscalls for I/O.
  • Registered files and buffers (io_uring_register) pre-pin fds and memory, skipping per-operation reference counting and page pinning.
  • It covers nearly everything: file and socket I/O, accept, connect, openat, statx, fsync, timeouts, linked operations (do B only if A succeeded), multishot accept/recv (one SQE, many completions), and zero-copy send/receive.
// Read N files with one submit syscall — gcc batch.c -luring
#include <liburing.h>

struct io_uring ring;
io_uring_queue_init(256, &ring, 0);

for (int i = 0; i < n; i++) {
    struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
    io_uring_prep_read(sqe, fds[i], bufs[i], 4096, 0);
    io_uring_sqe_set_data64(sqe, i);            // tag to match the completion
}
io_uring_submit(&ring);                          // ONE syscall submits n reads

for (int i = 0; i < n; i++) {
    struct io_uring_cqe *cqe;
    io_uring_wait_cqe(&ring, &cqe);              // no syscall if a CQE is already there
    handle(io_uring_cqe_get_data64(cqe), cqe->res);   // res = bytes or -errno
    io_uring_cqe_seen(&ring, cqe);
}
io_uring_queue_exit(&ring);

In Rust, tokio-uring, glommio and monoio build runtimes on it; in C/C++, liburing; databases and storage engines (ScyllaDB/Seastar, TigerBeetle, RocksDB's MultiGet, PostgreSQL 18's asynchronous I/O) use it for storage I/O. It's most compelling for storage I/O, where epoll never worked (regular files are always "ready"), and for very high-rate networking.

io_uring and security

io_uring is a large, fast-moving attack surface that bypasses syscall-level filtering: once a ring is set up, operations flow through shared memory, not through individual syscalls that seccomp could inspect. Google reported in 2023 that 60% of the kernel exploits submitted to its kCTF vulnerability reward program in 2022 targeted io_uring, and disabled it on ChromeOS, Android apps and its production servers. Newer Docker default seccomp profiles also block io_uring_setup, and kernel.io_uring_disabled (6.6+) lets you disable it system-wide or for unprivileged users. Check whether your platform allows it before designing around it.

Other ways syscalls disappear

  • futex fast path: an uncontended pthread_mutex_lock or Go sync.Mutex is a single atomic instruction in user space; only contention calls futex() to sleep. See Kernel Synchronization.
  • Runtime batching: Go's netpoller parks goroutines and calls epoll_wait once for all of them; tokio does the same per worker thread.
  • TCP_CORK / MSG_MORE: coalesce several writes into full packets.
  • accept4(SOCK_NONBLOCK | SOCK_CLOEXEC), pipe2, O_CLOEXEC on openat: set flags at creation instead of a follow-up fcntl.
  • Kernel bypass (DPDK, SPDK, AF_XDP): the NIC or NVMe queue is driven directly from user space with polling — zero syscalls, at the price of dedicated cores and losing the kernel's stack.
TechniqueSyscalls savedEffort
User-space buffering10–1000× fewer writesTrivial
writev / sendmmsg2–64×Low
sendfile / splice2× syscalls, plus the copyLow
epoll vs per-fd pollingO(fds) → O(ready events)Medium (event loop)
io_uringBatch of N → 1 (or 0 with SQPOLL)Medium–high
Kernel bypassAllHigh; dedicated hardware/cores

seccomp: Syscall Filtering

seccomp (secure computing) restricts which syscalls a process may make. If a process calls a forbidden syscall, the kernel either kills it (SECCOMP_RET_KILL) or returns an error.

// Minimal seccomp filter using BPF:
struct sock_filter filter[] = {
    BPF_STMT(BPF_LD | BPF_W | BPF_ABS, offsetof(struct seccomp_data, nr)),
    BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, __NR_read,  1, 0),
    BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, __NR_write, 0, 1),
    BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ALLOW),
    BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_KILL),  // Kill on any other syscall
};

That minimal filter is incomplete

A real filter must first check seccomp_data.arch (e.g. AUDIT_ARCH_X86_64) and kill on mismatch — otherwise a process can switch to the 32-bit int 0x80 ABI, where syscall numbers mean different things, and slip past a number-based allow list. It must also be installed after prctl(PR_SET_NO_NEW_PRIVS, 1) (or with CAP_SYS_ADMIN). In practice, use libseccomp rather than hand-writing BPF:

#include <seccomp.h>   // gcc sandbox.c -lseccomp

scmp_filter_ctx ctx = seccomp_init(SCMP_ACT_ERRNO(EPERM));   // default: fail with EPERM
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(read), 0);
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 0);
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(exit_group), 0);
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(futex), 0);
seccomp_load(ctx);     // libseccomp adds the arch check for you

Filters can return ALLOW, KILL_PROCESS, ERRNO(n), TRAP (deliver SIGSYS), LOG, TRACE (hand to a ptrace tracer), or USER_NOTIF (hand to a supervisor process that decides — used by container runtimes to emulate syscalls like mount safely). The filter runs on every syscall, but it's a short classic-BPF program, so the cost is typically tens of nanoseconds.

Docker's default seccomp profile blocks ~44 dangerous syscalls including:

  • ptrace (can read/write other processes' memory)
  • mount (can escape filesystem restrictions)
  • swapon (filesystem changes)
  • kexec_load (load a new kernel)
  • open_by_handle_at (bypass mount namespace restrictions)

Why seccomp matters for security: A container escape via a kernel vulnerability requires calling a dangerous syscall. Seccomp prevents most exploit chains by blocking the syscalls vulnerabilities need.

grep Seccomp /proc/<pid>/status          # 0 = off, 1 = strict, 2 = filter
docker run --security-opt seccomp=profile.json ...
# Kubernetes: securityContext.seccompProfile.type: RuntimeDefault

seccomp is one layer of a broader sandboxing toolkit — capabilities, namespaces, LSMs (SELinux, AppArmor, Landlock). The full treatment, including generating profiles from observed behaviour, is in Process Security.


eBPF: Extending the Kernel Safely

eBPF (extended Berkeley Packet Filter) allows running sandboxed programs inside the kernel without modifying kernel source or loading kernel modules. eBPF programs are verified before execution — the verifier rejects programs that could crash the kernel or loop forever.

eBPF program → verifier → JIT compiler → kernel execution

The verifier walks every possible path through the program, tracking the type and range of every register: it rejects out-of-bounds memory access, unbounded loops (bounded loops are allowed since 5.3), uninitialised reads, and calls to helpers the program type isn't allowed to use. Programs share data with user space through maps (hash tables, arrays, ring buffers, per-CPU variants), and BTF + CO-RE ("compile once, run everywhere") let one compiled program adapt to different kernels' struct layouts.

flowchart LR
    SRC["C / bpftrace script"] --> CLANG["clang to BPF bytecode"]
    CLANG -->|"bpf() syscall"| VER["Verifier"]
    VER -->|"safe"| JIT["JIT to native code"]
    VER -->|"unsafe"| REJ["Rejected with error log"]
    JIT --> HOOK["Attached to hook: tracepoint, kprobe, XDP, TC, LSM"]
    HOOK <-->|"maps and ring buffers"| USERSPACE["User-space agent"]

Where eBPF programs run (hook points):

  • Syscall entry/exit (tracepoints, kprobes)
  • Network packet processing (XDP, TC)
  • Function entry/exit (kprobes, uprobes)
  • Hardware performance counters (perf events)
  • Security hooks (BPF LSM) — enforce policy at the same points SELinux and AppArmor use
  • Scheduler policy (sched_ext, merged in 6.12) — whole CPU schedulers written as BPF programs

Use cases:

Use caseToolWhat it does
Network performanceXDPProcess packets before kernel network stack (line rate)
Observabilitybpftrace, BCCTrace syscalls, latency, function calls without restart
SecurityFalco, TetragonDetect suspicious syscalls in production
ProfilingBPF-based profilersCPU flamegraphs from production without instrumentation
Service meshCiliumL3/L4/L7 networking without sidecar proxies
# Trace open() syscalls in real time with bpftrace
bpftrace -e 'tracepoint:syscalls:sys_enter_open { printf("%s %s\n", comm, str(args->filename)); }'

# Count syscalls per second
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); } interval:s:1 { print(@); clear(@); }'

# Histogram of disk I/O request sizes (sectors) by read/write type
# (note: this measures SIZE, not latency — see the next one-liner for latency)
bpftrace -e 'tracepoint:block:block_rq_complete /args->nr_sector > 0 && args->error == 0/ { @[args->rwbs] = hist(args->nr_sector); }'

# Trace slow disk I/O (> 10ms): latency from issue to completion
bpftrace -e '
tracepoint:block:block_rq_issue { @start[args->dev, args->sector] = nsecs; }
tracepoint:block:block_rq_complete /@start[args->dev, args->sector]/ {
  $ms = (nsecs - @start[args->dev, args->sector]) / 1000000;
  if ($ms > 10) { printf("slow I/O: %d ms, %d sectors\n", $ms, args->nr_sector); }
  delete(@start[args->dev, args->sector]);
}'

eBPF vs kernel modules:

  • Kernel modules can crash the kernel; eBPF programs are verified
  • Kernel modules require kernel headers and version matching; eBPF programs are portable via BTF (BPF Type Format)
  • eBPF is hot-loadable; kernel modules require careful management

The trade-off runs the other way too: eBPF programs can only do what the verifier can prove safe and what the available helpers allow, so drivers and file systems are still modules. Loading BPF requires CAP_BPF (plus CAP_PERFMON/CAP_NET_ADMIN depending on the program type); unprivileged BPF is disabled by default on most distros (kernel.unprivileged_bpf_disabled). The deep dives are eBPF and eBPF in Production.


Kernel Space Memory

The kernel has its own memory regions. On x86-64 with a 48-bit virtual address space:

User space:   0x0000000000000000 – 0x00007fffffffffff  (128TB)
Kernel space: 0xffff800000000000 – 0xffffffffffffffff  (128TB)

The kernel address space is mapped into every process's page tables (but marked Ring 0 only — user code cannot access it). This "higher half" mapping allows the kernel to access its own data structures without a full context switch, but KPTI separates them for Meltdown mitigation.

Inside the kernel half there is a direct map of all physical RAM (so the kernel can reach any frame at PAGE_OFFSET + phys), the vmalloc area for virtually contiguous allocations, the kernel image and modules, and per-CPU areas — all randomised by KASLR at boot. Each thread also has a small fixed-size kernel stack (16KB on x86-64) used while it executes syscalls; deep call chains in the kernel are therefore a real constraint, and kernel stack overflow is caught by guard pages (CONFIG_VMAP_STACK).

When a syscall needs user data, the kernel never dereferences user pointers directly: it uses copy_from_user / copy_to_user, which check that the address range is in user space, open the SMAP window, and handle a fault gracefully (returning -EFAULT to the caller instead of crashing the kernel) if the user passed a bad pointer.


Real Numbers

QuantityValue
Bare SYSCALL/SYSRET round trip, no mitigations~50–100 ns
Simple syscall (getppid) with typical mitigations~100–300 ns; up to ~1 µs with KPTI and no PCID
clock_gettime via vDSO~15–25 ns
seccomp filter evaluationtens of ns per syscall
Context switch (blocking syscall → another thread)~1–5 µs direct, more with cache effects
strace slowdown on syscall-heavy code2× to 100×+
KPTI overhead< 1% below ~10K syscalls/s per CPU; tens of percent for syscall-bound workloads on affected CPUs
Linux x86-64 syscalls~380 (numbers 0–335 and 424+)
Docker default seccomp profileblocks ~44 syscalls
Kernel stack per thread (x86-64)16 KB
io_uring1 syscall per batch, 0 with SQPOLL

Decision Guide

SituationApproach
High sys time, many small read/writeBuffer in user space; writev; check per-request syscall count with strace -c
Serving static files or log segmentssendfile (io.Copy in Go, nginx sendfile on); kTLS for HTTPS
Proxying bytes between socketssplice through a pipe, or a runtime that does it for you
Lots of storage I/O, need concurrencyio_uring (or a thread pool if io_uring is blocked on your platform)
UDP at high packet ratesrecvmmsg/sendmmsg, UDP_SEGMENT/GRO; AF_XDP or DPDK at the extreme
clock_gettime visible in straceFix the clocksource (tsc)
Debug a failing process on a dev boxstrace -f -e trace=%file,%network
Investigate a production processperf trace -s, bpftrace, BCC syscount/opensnoop
Sandbox untrusted codeseccomp allow list via libseccomp, plus namespaces and capabilities (Process Security)
Extend kernel behavioureBPF if it fits the hooks and verifier; module only for drivers/file systems
Older CPUs, syscall-bound, considering mitigations=offReduce syscalls first; only disable on single-tenant trusted hosts

Advanced Topics

  • Syscall User Dispatch (5.11): redirect syscalls made from certain code regions to a SIGSYS handler — built for Wine to emulate Windows system calls made directly by games.
  • gVisor: a user-space kernel (written in Go) that intercepts a sandboxed application's syscalls and implements them itself, only making a small, filtered set of host syscalls. Stronger isolation than seccomp alone, at a syscall-performance cost. Used by Google Cloud Run and App Engine.
  • Unikernels (MirageOS, Unikraft): link the application with just the OS pieces it needs into one ring-0 image; a "syscall" becomes a function call.
  • FlexSC / exception-less syscalls: the research idea (OSDI 2010) of posting syscalls to shared pages and having kernel threads on other cores execute them — io_uring is its production descendant.
  • Landlock (5.13): unprivileged, self-imposed file-system and network access restrictions — a complement to seccomp that filters objects rather than syscall numbers.
  • Restartable sequences (rseq): a per-thread area the kernel updates on preemption, letting user space do per-CPU operations without atomics — the basis of tcmalloc's per-CPU caches and glibc's fast sched_getcpu.

Interview Questions

"Why are system calls expensive?" A syscall crosses the Ring 3 → Ring 0 boundary: the CPU executes SYSCALL instruction, saves user registers to the kernel stack, switches to kernel mode, executes the handler, then returns. The overhead is ~100–1000ns — partly the mode switch, partly TLB invalidation (KPTI), partly cache misses (kernel code/data may be cold). At 100K RPS with 5 syscalls each = 500K syscalls/sec; at 500ns each = 250ms CPU/sec just on syscalls.

"What is vDSO?" A small read-only shared library that the kernel maps into every process. It exposes certain syscalls (clock_gettime, gettimeofday, getcpu) in user space — they read from kernel-maintained memory without a mode switch. clock_gettime(CLOCK_MONOTONIC) is ~5ns via vDSO vs ~100ns as a real syscall.

"What is seccomp and why do containers use it?" seccomp filters restrict which syscalls a process may call using BPF programs. Docker's default profile blocks ~44 dangerous syscalls (ptrace, mount, kexec_load). This limits the damage from container escape exploits — even if an attacker controls user code, they can't call the syscalls needed to escalate to host root.

"What is eBPF?" eBPF lets you run sandboxed programs inside the kernel at hook points (syscall entry/exit, network packet processing, function calls) without modifying kernel source or loading modules. The verifier ensures safety. Used by Cilium for networking, Falco for security, bpftrace/BCC for observability. The key property: you can add instrumentation to a running production kernel without restarting anything.

"What exactly does the SYSCALL instruction do?" It saves RIP in RCX and RFLAGS in R11, masks flags, loads ring-0 code and stack segments from STAR, and jumps to the address in LSTAR. It doesn't switch stacks — the kernel's entry code does that (and switches CR3 under KPTI). That's why the fourth syscall argument goes in r10, not rcx.

"What did Meltdown change about syscall cost?" Meltdown let user code speculatively read kernel memory mapped in its page tables. KPTI fixes it by keeping two page-table sets and switching on every kernel entry and exit; without PCID that flushes the TLB each time. Syscall-heavy workloads on affected CPUs lost 5–30%, which renewed interest in batching interfaces like io_uring.

"How does sendfile achieve zero-copy?" Instead of copying file data into a user buffer and back into a socket buffer, it passes references to page-cache pages to the socket; with scatter-gather DMA, the NIC reads directly from those pages. One syscall, no CPU copies of the payload.

"How would you find out why a process is hanging?" cat /proc/<pid>/stack and /proc/<pid>/wchan for where the kernel has it blocked, strace -p (briefly) to see the syscall it's stuck in and with what arguments (-y for fd names), and perf trace --duration or bpftrace to find slow syscalls in production without the ptrace overhead.