System Calls & Kernel Internals
Every I/O operation, every process creation, every memory mapping — every interaction between user code and the hardware — goes through a system call. Understanding syscall mechanics explains why context-switching is expensive, why async I/O exists, why Rust's tokio and Go's runtime avoid unnecessary syscalls, and why eBPF can add observability without modifying application code.
A system call is the only door between your program and the machine. Your process cannot open a file, send a packet, allocate a page, read the clock (well, almost — see vDSO), or even exit without asking the kernel. That makes the syscall boundary three things at once: a performance boundary (every crossing costs hundreds of nanoseconds, more since 2018), a security boundary (seccomp, containers and sandboxes all work by filtering it), and an observability boundary (strace, perf and eBPF can see everything a program does to the outside world by watching it).
This page covers the boundary from the CPU instruction up: how the switch to kernel mode happens, what it costs and why that changed with Meltdown, how the kernel is structured behind the door, how to watch syscalls in production, and how modern I/O interfaces avoid crossing the boundary at all.
Related reading
Processes & Threads for what the kernel is scheduling, Virtual Memory for page tables and the TLB (central to KPTI), and I/O & Storage for the I/O models that io_uring improves on.
The User/Kernel Boundary
The CPU operates in different privilege rings:
Ring 0 (kernel mode): full hardware access
Ring 1, 2: rarely used
Ring 3 (user mode): restricted — no direct hardware access, no arbitrary memory access
Modern x86-64 uses only Ring 0 and Ring 3.
User-space code runs in Ring 3. It cannot:
- Access hardware directly (disk, NIC, GPU)
- Read/write memory it doesn't own
- Modify CPU control registers
- Execute privileged instructions
To do any of these, it must request the kernel do it — via a system call.
How the CPU enforces it
The current privilege level (CPL) lives in the low two bits of the CS segment register. The hardware checks it in several places:
| Mechanism | What it stops |
|---|---|
Privileged instructions (mov cr3, wrmsr, hlt, in/out, lgdt) | Fault with #GP if executed at CPL 3 |
| Page-table U/S bit | User mode touching a supervisor page → page fault → SIGSEGV |
| SMEP (Supervisor Mode Execution Prevention) | Kernel executing user-space pages — blocks the classic "jump to shellcode in user memory" exploit |
| SMAP (Supervisor Mode Access Prevention) | Kernel reading/writing user pages except inside explicit stac/clac windows (copy_from_user) |
| IOMMU | Devices DMA-ing to memory they weren't given |
The other common "rings" you'll hear about are informal: ring −1 for the hypervisor (VMX root mode), ring −2 for System Management Mode firmware. ARM64 names them exception levels: EL0 (user), EL1 (kernel), EL2 (hypervisor), EL3 (secure monitor).
flowchart TB
subgraph U["Ring 3 / EL0: user space"]
APP["Application"]
LIBC["libc wrappers, vDSO"]
end
subgraph K["Ring 0 / EL1: kernel"]
ENTRY["Syscall entry (entry_SYSCALL_64)"]
VFS["VFS and file systems"]
NET["Network stack"]
MM["Memory management"]
SCHED["Scheduler"]
DRV["Device drivers"]
end
HW["Hardware: CPU, RAM, NIC, NVMe"]
APP --> LIBC
LIBC -->|"SYSCALL instruction"| ENTRY
ENTRY --> VFS
ENTRY --> NET
ENTRY --> MM
ENTRY --> SCHED
VFS --> DRV
NET --> DRV
DRV --> HW
The System Call Mechanism
x86-64 Linux system call path:
User code calls read(fd, buf, n)
↓
glibc wrapper:
1. Load syscall number into %rax (e.g., 0 for read)
2. Load arguments into %rdi, %rsi, %rdx, %r10, %r8, %r9
3. Execute SYSCALL instruction
↓
CPU switches to Ring 0:
4. Saves user stack pointer (RSP) to kernel stack
5. Loads kernel stack pointer from MSR_LSTAR
6. Jumps to kernel entry point (entry_SYSCALL_64)
↓
Kernel handles the call:
7. Saves user registers to kernel stack
8. Looks up handler: sys_call_table[%rax]
9. Executes sys_read()
10. Returns value in %rax
↓
SYSRET instruction:
11. Restores user registers
12. Switches back to Ring 3
13. Jumps back to user code (next instruction after SYSCALL)
↓
User code continues with return value
Precision on steps 4–6
The SYSCALL instruction itself does less than the list suggests. In hardware it only: saves the return address (RIP) into RCX, saves RFLAGS into R11, masks RFLAGS using the IA32_FMASK MSR (clearing the interrupt flag), loads CS/SS for ring 0 from IA32_STAR, and jumps to the address in IA32_LSTAR (MSR_LSTAR). It does not switch stacks. The first instructions of entry_SYSCALL_64 do that in software: swapgs to reach per-CPU data, stash the user RSP in a per-CPU slot, load the kernel stack pointer, and (under KPTI) switch CR3 to the kernel page tables. SYSRET reverses it: RIP ← RCX, RFLAGS ← R11, back to ring 3.
That's why the syscall calling convention uses %r10 instead of %rcx for the fourth argument (the normal C convention uses rcx): SYSCALL overwrites rcx and r11. You can make a syscall with no libc at all:
// raw_write.c — gcc -O2 raw_write.c -o raw_write
#include <sys/syscall.h>
static long raw_syscall3(long nr, long a1, long a2, long a3) {
long ret;
__asm__ volatile ("syscall"
: "=a"(ret) // return value in rax
: "a"(nr), "D"(a1), "S"(a2), "d"(a3) // rax, rdi, rsi, rdx
: "rcx", "r11", "memory"); // clobbered by SYSCALL
return ret;
}
int main(void) {
long n = raw_syscall3(SYS_write, 1, (long)"hello from ring 3\n", 18);
return n == 18 ? 0 : 1;
}
sequenceDiagram
participant App as "App (ring 3)"
participant Libc as "glibc read()"
participant CPU as "CPU"
participant K as "Kernel (ring 0)"
App->>Libc: read(fd, buf, 4096)
Libc->>CPU: rax=0, rdi=fd, rsi=buf, rdx=4096, SYSCALL
CPU->>K: save RIP to RCX and RFLAGS to R11, jump to LSTAR
K->>K: swapgs, switch stack, switch CR3 (KPTI)
K->>K: sys_call_table[0] = ksys_read
K->>K: copy page cache data to buf (copy_to_user)
K->>CPU: rax = bytes read or negative errno, SYSRET
CPU->>Libc: back in ring 3
Libc->>App: return n, or -1 and set errno
The SYSCALL/SYSRET pair (modern x86-64) is faster than the older INT 0x80 interrupt mechanism because it doesn't go through the interrupt descriptor table — it reads the kernel entry point from a Model-Specific Register (MSR).
Other entry mechanisms you'll meet:
| Mechanism | Where | Notes |
|---|---|---|
int 0x80 | 32-bit x86 Linux (still works on x86-64 for 32-bit ABI) | Goes through the IDT; slowest |
sysenter/sysexit | 32-bit x86 (Intel) | What the 32-bit vDSO used for speed |
syscall/sysret | x86-64 | The standard path |
svc #0 | ARM64 | Syscall number in x8, args in x0–x5, return in x0 |
ecall | RISC-V | Number in a7, args in a0–a5 |
| FRED | Newest Intel CPUs, Linux 6.9+ | Flexible Return and Event Delivery: replaces the legacy SYSCALL/IDT entry paths with a cleaner, faster event-delivery model |
Return values, errno and interrupted calls
The kernel returns a single value in rax. Values from −4095 to −1 are errors (-EBADF, -EAGAIN …); glibc's wrapper turns them into -1 and sets the thread-local errno. Two consequences:
EINTR: a blocking syscall (readon a socket,nanosleep,futexwait) interrupted by a signal returns-EINTRunless the handler was installed withSA_RESTART— so robust code loops onEINTR. See Signals.- Short reads/writes are not errors:
write(fd, buf, 1MB)on a socket may return 64KB. Every correct I/O loop handles partial completion.
A blocking syscall is also where your thread leaves the CPU: the kernel marks the task as sleeping, calls the scheduler, and another task runs until the I/O completes — the voluntary context switch counted in /proc/<pid>/status as voluntary_ctxt_switches. See CPU Scheduling.
Syscall Cost
A syscall takes 100–1000 ns depending on:
- What the kernel does (simple check vs. disk I/O)
- Cache state (kernel code and data may be cold)
- Spectre mitigations (KPTI — Kernel Page Table Isolation)
Splitting that up: the bare SYSCALL→SYSRET round trip with an empty handler is on the order of 50–100 ns on a modern CPU with no mitigations. Mitigations for transient-execution vulnerabilities can add another 50–500 ns on affected CPUs. And the real work inside the handler — locking, copying data, walking the network stack — is often larger than either. Plus the hidden cost: the syscall evicts some of your hot data from L1/L2 caches, branch predictors and the TLB, so your user code runs slower for a while afterwards. Soares and Stumm's FlexSC paper (OSDI 2010) measured this indirect cost as larger than the direct cost for syscall-heavy server workloads.
Measure it yourself:
// syscost.c — gcc -O2 syscost.c -o syscost && ./syscost
#define _GNU_SOURCE
#include <stdio.h>
#include <time.h>
#include <unistd.h>
#include <sys/syscall.h>
static double now_ns(void) {
struct timespec t;
clock_gettime(CLOCK_MONOTONIC, &t);
return t.tv_sec * 1e9 + t.tv_nsec;
}
int main(void) {
enum { N = 10000000 };
struct timespec ts;
double t0 = now_ns();
for (int i = 0; i < N; i++) syscall(SYS_getppid); // real kernel entry
double t1 = now_ns();
for (int i = 0; i < N; i++) clock_gettime(CLOCK_MONOTONIC, &ts); // vDSO, no entry
double t2 = now_ns();
printf("getppid (syscall): %6.1f ns\n", (t1 - t0) / N);
printf("clock_gettime (vDSO): %6.1f ns\n", (t2 - t1) / N);
return 0;
}
Typical results (orders of magnitude; your CPU and kernel will differ):
| Machine | getppid syscall | clock_gettime via vDSO |
|---|---|---|
| Recent AMD/Intel, not Meltdown-vulnerable, default mitigations | ~60–120 ns | ~15–25 ns |
| 2015–2018 Intel server with KPTI + PCID | ~150–300 ns | ~20 ns |
| Same, KPTI without PCID (old CPU or VM without PCID passthrough) | ~400–1000 ns | ~20 ns |
| VM whose clocksource can't be read from user space | — | falls back to a syscall: ~300+ ns |
KPTI overhead: After the Meltdown vulnerability (2018), Linux separated kernel and user page tables. On every syscall entry, the CPU switches to the kernel's full page table; on return, it switches back. This requires a TLB flush on CPUs without PCID support, adding 10–30% overhead to syscall-heavy workloads.
Meltdown, Spectre and their mitigations
Meltdown (2018) let user code speculatively read kernel memory through the cache side channel, because the kernel was mapped (supervisor-only) in every process's page tables. KPTI removes almost the entire kernel from the page tables used in user mode, leaving only a tiny trampoline for entry/exit — hence the CR3 switch on every crossing. With PCID (see Virtual Memory), user and kernel page tables get separate TLB tags so the switch doesn't flush; without it, every syscall flushes the TLB.
KPTI is one of a family; each added a cost somewhere on the syscall path:
| Mitigation | Vulnerability | Where it costs |
|---|---|---|
| KPTI (page table isolation) | Meltdown (2018) | CR3 write on every entry and exit |
| Retpoline / IBRS / eIBRS | Spectre v2, branch target injection (2018) | Every indirect branch in the kernel |
| VERW buffer clearing | MDS / ZombieLoad (2019) | Every return to user space |
| Return thunks, IBPB on entry, RSB stuffing | Retbleed (2022), AMD SRSO/Inception (2023) | Function returns, context switches |
| BHI clearing sequence | Branch History Injection (2024) | Syscall entry |
grep . /sys/devices/system/cpu/vulnerabilities/*
# /sys/devices/system/cpu/vulnerabilities/meltdown:Not affected
# /sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Enhanced / Automatic IBRS; IBPB: conditional; ...
# /sys/devices/system/cpu/vulnerabilities/retbleed:Mitigation: untrained return thunk; SMT enabled with STIBP protection
dmesg | grep -i 'page table isolation'
Real-world impact depends almost entirely on syscall rate. Brendan Gregg's 2018 analysis of KPTI at Netflix showed overhead under ~1% for workloads below roughly 10,000 syscalls/second per CPU, rising steeply for workloads making hundreds of thousands per second per CPU — and dramatically larger without PCID. Database and proxy benchmarks of the time (PostgreSQL SELECT 1 loops, Redis with small values) showed 5–30% throughput loss on affected hardware. CPUs designed after these disclosures fix many of them in hardware (Intel since Cascade Lake/Ice Lake and AMD Zen are not affected by Meltdown, so KPTI is off on them), but newer issues like Retbleed and SRSO brought double-digit-percent costs back for some older CPUs in syscall-heavy benchmarks.
mitigations=off
The mitigations=off kernel parameter disables all of these and can win back 5–30% on syscall-heavy workloads on older CPUs. It is only defensible on single-tenant machines running only trusted code — never on multi-tenant hosts, CI runners, or anything that runs JavaScript or untrusted plugins. Reducing syscall count is the better lever.
Why this matters for system design:
- An HTTP server doing 100K RPS with 4 syscalls per request = 400K syscalls/sec. At 500ns each = 200ms of CPU per second just on syscalls.
- io_uring reduces this to near zero by batching submissions/completions through shared ring buffers.
- Go's runtime and Rust's tokio are designed to minimize syscalls per operation.
(In practice a request needs more than 4 — epoll_wait, accept4, read, write, maybe setsockopt, close, upstream connect/sendmsg/recvmsg, logging writes — 10–20 is common, which is why syscall counting per request is a worthwhile profiling exercise.)
# Count syscalls for a program
strace -c ./app
# Same, but trace every syscall class explicitly (the default already traces all;
# -e trace=... is useful to narrow it down instead, e.g. -e trace=%network)
strace -c -e trace=all ./app
# Watch live syscalls
strace -p <pid>
# Count syscalls across all threads
strace -f -c ./multithreaded-app
vDSO: Virtual Dynamic Shared Object
For some syscalls, the cost of ring transition is unnecessary because the kernel data is safe to read from user space. The vDSO maps a small read-only kernel-owned page into every process's address space.
Functions that can be satisfied from vDSO:
clock_gettime()— reads from a kernel-maintained time structgettimeofday()— samegetcpu()— reads CPU and NUMA node without a syscalltime()— seconds since epoch
Linux 6.11 added a fifth: getrandom() in the vDSO (glibc 2.41+ uses it), so generating random numbers no longer needs a kernel entry per call.
// Looks like a syscall but executes entirely in user space via vDSO:
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
// No mode switch to ring 0, no TLB invalidation
// ~5ns instead of ~100ns
# View vDSO mapping in a process
cat /proc/<pid>/maps | grep vdso
# 7ffd...000 r-xp [vdso]
How it actually works
The vDSO is a real ELF shared library (linux-vdso.so.1 in ldd output) that the kernel maps into every process at execve, alongside a data page called [vvar]. The kernel's timekeeping code updates vvar on every timer tick with the current base time, the TSC value at that moment, and the multiplier to convert TSC cycles to nanoseconds. The vDSO's clock_gettime then:
- Reads a sequence counter from
vvar(a seqlock) - Reads the base time and conversion parameters
- Executes
rdtsc(orrdtscp) — readable from ring 3 - Re-reads the sequence counter; if it changed (the kernel updated mid-read), retries
- Returns
base + (tsc_now − tsc_base) × mult >> shift
No privilege change is needed because nothing privileged happens: it's reading memory the kernel made readable and a CPU counter user mode is allowed to read. glibc finds the vDSO's symbols through the auxiliary vector (AT_SYSINFO_EHDR) and calls them in place of the syscall.
It only works if the clocksource is readable from user space. With tsc (or arm_arch_timer on ARM) it is; with hpet, acpi_pm, or some hypervisor clocks it isn't, and the vDSO silently falls back to a real syscall.
cat /sys/devices/system/clocksource/clocksource0/current_clocksource # tsc ← good
cat /sys/devices/system/clocksource/clocksource0/available_clocksource # tsc hpet acpi_pm
War Story (real, well documented)
In 2017 Packagecloud published "Two frequently used system calls are ~77% slower on AWS EC2." On Xen-based EC2 instances, the default clocksource was xen, which the vDSO couldn't read, so every gettimeofday/clock_gettime became a full syscall. Applications that timestamp heavily — logging, metrics, tracing, databases — paid a real kernel entry for every call; strace -c on such a process shows thousands of clock_gettime calls per second, which should never appear at all. The fix was switching the clocksource to tsc (echo tsc > .../current_clocksource, or clocksource=tsc on the kernel command line), which AWS later documented for its Xen instance types. Newer Nitro (KVM-based) instances use kvm-clock/tsc that the vDSO supports.
Why not vDSO everything? Most syscalls need kernel to act (write to disk, create a socket, etc.) — they can't be done in user space.
Kernel vs User Time
The time command reports wall time, user time, and sys time:
time dd if=/dev/zero of=/dev/null bs=1M count=1000
real 0m0.271s
user 0m0.008s
sys 0m0.262s
- user: time spent executing user-space code (your application)
- sys: time spent executing kernel code on behalf of your process (syscalls)
- real: wall clock time (includes I/O wait, scheduling delays)
A high sys time relative to user time indicates the program is syscall-heavy — potential candidate for io_uring, batching, or vDSO-eligible calls.
(In the dd example, nearly all the time is sys because each 1MB read from /dev/zero is the kernel zeroing memory, and each write to /dev/null is a no-op handler: the program is pure syscalls.)
For a running service:
pidstat -u -p <pid> 1 # %usr vs %system per second
top -H -p <pid> # per-thread; press 1 for per-CPU us/sy split
perf top -p <pid> # kernel symbols (copy_user_enhanced_fast_string,
# _raw_spin_lock, tcp_sendmsg ...) show where sys time goes
cat /proc/<pid>/stat | awk '{print "utime="$14, "stime="$15}' # clock ticks
sys time can also be high for reasons that aren't syscalls you chose: page faults (kernel handles them), memory reclaim and compaction, TLB shootdowns, spinlock contention inside the kernel (_raw_spin_lock at the top of perf top — see Kernel Synchronization), or softirq processing of network traffic.
Kernel Architecture: Monolithic, Micro, Hybrid
Behind the syscall door, how is the kernel itself organised? The main design question is how much code runs in ring 0.
flowchart LR
subgraph MONO["Monolithic (Linux, FreeBSD)"]
direction TB
MU["User apps"] -->|"syscall"| MK["One kernel image in ring 0: scheduler, MM, VFS, file systems, TCP/IP, drivers"]
end
subgraph MICRO["Microkernel (seL4, QNX, MINIX 3)"]
direction TB
UU["User apps"] -->|"IPC"| SRV["User-space servers: file system, network, drivers"]
SRV -->|"IPC"| UK["Tiny kernel in ring 0: address spaces, threads, IPC"]
end
| Monolithic | Microkernel | Hybrid | |
|---|---|---|---|
| Ring 0 contains | Everything: drivers, file systems, network stack | Only scheduling, address spaces, IPC | Core plus selected services |
| Examples | Linux, FreeBSD, OpenBSD | seL4, QNX, MINIX 3, Fuchsia's Zircon | Windows NT, macOS XNU (Mach + BSD) |
| Performance | Subsystems call each other directly: fast | Every service request is IPC between address spaces: historically slower (L4/seL4 narrowed the gap greatly) | In between |
| Fault isolation | A driver bug can crash or compromise the whole kernel | A crashing driver is restarted like a process | Partial |
| Verification | ~40M lines (Linux) — impractical | seL4's ~10k lines are formally proven correct | — |
| Where it wins | General-purpose servers, desktops, Android | Safety-critical and embedded (QNX in cars, seL4 in defence/aerospace) | Desktop OSes with legacy |
The Torvalds–Tanenbaum debate (1992) argued exactly this. Linux stayed monolithic, and in practice it gets much of the microkernel benefit in other ways: user-space drivers and file systems where isolation matters more than speed (FUSE, VFIO/UIO, DPDK and SPDK for kernel-bypass networking and storage), and eBPF for safe kernel extension.
Loadable kernel modules
"Monolithic" doesn't mean "one static blob". Linux loads drivers, file systems, network protocols and more as modules at runtime — they run in ring 0 with full privileges, as part of the kernel, but are linked in on demand.
lsmod | head # loaded modules, size, users
modinfo nvme # description, parameters, signer, dependencies
sudo modprobe br_netfilter # load with dependencies
sudo modprobe -r dummy # unload
cat /proc/sys/kernel/tainted # non-zero: proprietary/unsigned/out-of-tree modules loaded
cat /sys/module/nvme_core/parameters/io_timeout
// hello.c — a minimal out-of-tree module
#include <linux/init.h>
#include <linux/module.h>
static int __init hello_init(void) { pr_info("hello: loaded\n"); return 0; }
static void __exit hello_exit(void) { pr_info("hello: unloaded\n"); }
module_init(hello_init);
module_exit(hello_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Minimal example module");
# Makefile: obj-m += hello.o
make -C /lib/modules/$(uname -r)/build M=$PWD modules
sudo insmod hello.ko && dmesg | tail -1 # hello: loaded
sudo rmmod hello
Modules are powerful and dangerous: a bug is a kernel crash, there is no stable in-kernel ABI (modules must be rebuilt for each kernel version, which is why DKMS exists), and Secure Boot systems require them to be signed. That's the gap eBPF fills for observability, networking and security use cases.
The Kernel System Call Table
Linux x86-64 has ~340 system calls. The kernel maps syscall numbers to handler functions in sys_call_table[].
(Updated count: as of 6.x kernels there are roughly 380 — numbers 0–335 plus 424 onward; 336–423 are skipped so new syscalls get the same number on every architecture. Recent additions include io_uring_setup (425), landlock_create_ruleset (444), cachestat (451), mseal (462).)
Key syscalls grouped by category:
// Files & I/O
0 sys_read 1 sys_write
2 sys_open 3 sys_close
8 sys_lseek 257 sys_openat // modern glibc open() uses openat
217 sys_getdents64 137 sys_statfs
19 sys_readv 20 sys_writev
40 sys_sendfile 275 sys_splice
// Process
56 sys_clone (threads/processes) 59 sys_execve
62 sys_kill 63 sys_uname
39 sys_getpid 202 sys_futex // the slow path of every mutex
// Memory
9 sys_mmap 11 sys_munmap
12 sys_brk 10 sys_mprotect
319 sys_memfd_create
// Network
41 sys_socket 49 sys_bind
43 sys_accept 42 sys_connect
44 sys_sendto 45 sys_recvfrom
291 sys_epoll_create1 232 sys_epoll_wait
// Async I/O, observability
425 sys_io_uring_setup 426 sys_io_uring_enter
321 sys_bpf
Correction
Earlier versions of this table listed read/write/open/close under "Process", and had several wrong numbers: bind is 49 (50 is listen), connect is 42 (45 is recvfrom), recvfrom is 45 (46 is sendmsg), getdents64 is 217 (78 is the older getdents), and memfd_create is 319 (258 is mkdirat). Always check against your system:
ausyscall --dump | head # from the auditd package
grep -E '__NR_(bind|connect|memfd_create) ' /usr/include/x86_64-linux-gnu/asm/unistd_64.h
Numbers are per-architecture: read is 0 on x86-64, 63 on ARM64, 3 on 32-bit x86. This is why seccomp filters must check the architecture field before the syscall number.
Tracing System Calls
Because every interaction with the outside world is a syscall, watching syscalls is the fastest way to answer "what is this process actually doing?" — which file is it failing to open, which host is it connecting to, why is it stuck. There are four main tools, with very different overheads.
| Tool | Mechanism | Overhead | Best for |
|---|---|---|---|
strace | ptrace: the tracer is woken at every syscall entry and exit | High: 2× to 100×+ slower for syscall-heavy programs | Debugging one process, dev boxes, "why does this fail?" |
ltrace | Breakpoints on PLT entries of shared-library calls | Very high | Library calls (malloc, strlen, SSL_read) |
perf trace | Kernel tracepoints via perf ring buffers | Low–moderate | strace-like output with much less slowdown; system-wide |
bpftrace / BCC | eBPF programs on tracepoints/kprobes, aggregation in kernel | Low (only aggregates leave the kernel) | Production: histograms, counts, filtered events |
strace
strace -f -tt -T -e trace=%network -p <pid>
# 12:04:31.218734 connect(7, {sa_family=AF_INET, sin_port=htons(5432), sin_addr=inet_addr("10.0.3.17")}, 16)
# = -1 EINPROGRESS (Operation now in progress) <0.000041>
# 12:04:31.218902 poll([{fd=7, events=POLLOUT}], 1, 5000) = 0 (Timeout) <5.005112> ← 5s connect timeout
strace -e trace=%file -f ./app 2>&1 | grep ENOENT # which config file is it looking for?
strace -y -e trace=read,write -p <pid> # -y: show paths/sockets for fds
strace -k -e trace=mmap -p <pid> # -k: stack trace for each call
strace -c -f -p <pid> # Ctrl-C after a while for a summary
strace --seccomp-bpf -f -e trace=execve ./build.sh # only stop on traced calls: much lower overhead
$ strace -c -f -p 2112 # summary from a busy web worker (example output)
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
41.20 1.204331 11 104212 epoll_wait
22.87 0.668611 6 104187 12044 read
18.03 0.527013 5 104201 write
9.11 0.266210 128 2079 futex
4.32 0.126301 4 28901 clock_gettime ← vDSO not working?
...
------ ----------- ----------- --------- --------- ----------------
100.00 2.923211 352112 12044 total
That last line is a red flag in its own right: clock_gettime should never appear in strace when the vDSO works.
strace in production
Because every syscall stops the process twice and context-switches to the tracer, strace can make a syscall-heavy service many times slower — and if strace is killed uncleanly the target can be left in an odd state. Prefer perf trace or bpftrace on production hosts, keep strace sessions short, and use --seccomp-bpf with a narrow -e trace= filter when you must.
ltrace
ltrace -c ./app # count library calls
ltrace -e malloc+free -p <pid> # watch allocator calls
ltrace -S -e SSL_write ./client # -S also shows syscalls interleaved
Note that ltrace only sees calls through the PLT (dynamically linked functions); statically linked binaries and most Go programs are invisible to it. uprobes via bpftrace are the lower-overhead alternative (bpftrace -e 'uprobe:/lib/x86_64-linux-gnu/libc.so.6:malloc { @[comm] = count(); }').
perf trace
perf trace -p <pid> # strace-like live output
perf trace -s -p <pid> -- sleep 10 # per-thread summary with min/avg/max latency per syscall
perf trace --duration 10 -p <pid> # only syscalls that took longer than 10 ms
perf trace -e 'openat,connect' -a # system-wide, selected syscalls
perf stat -e 'raw_syscalls:sys_enter' -a -- sleep 1 # total syscalls/sec on the box
bpftrace
# Trace open() syscalls in real time with bpftrace
bpftrace -e 'tracepoint:syscalls:sys_enter_open { printf("%s %s\n", comm, str(args->filename)); }'
# Modern glibc calls openat(), not open(), so this usually catches more:
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'
# Top syscalls by process, system-wide, for 10 seconds
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm, args->id] = count(); } interval:s:10 { exit(); }'
# Latency histogram of read() for one process (microseconds)
bpftrace -e '
tracepoint:syscalls:sys_enter_read /pid == $1/ { @start[tid] = nsecs; }
tracepoint:syscalls:sys_exit_read /@start[tid]/ {
@usecs = hist((nsecs - @start[tid]) / 1000); delete(@start[tid]);
}' <pid>
# Which processes fail syscalls, and with which error?
bpftrace -e 'tracepoint:raw_syscalls:sys_exit /args->ret < 0/ { @[comm, args->id, -args->ret] = count(); }'
The BCC toolkit ships ready-made versions: syscount (counts and latency by syscall), opensnoop, execsnoop, tcpconnect, killsnoop. For the full eBPF tooling story, see eBPF and eBPF in Production.
Batching and Avoiding Syscalls
The fastest syscall is the one you don't make. Four strategies, in roughly the order you should reach for them:
- Buffer in user space.
stdio, Go'sbufio, Rust'sBufWriter, Java'sBufferedOutputStream: turn 10,000 tinywrites into a few large ones. Unbuffered logging (writeper line) is the most common accidental syscall storm. - Do more per syscall — vectored I/O and multi-message calls.
- Move data inside the kernel —
sendfile,splice,copy_file_range: zero-copy, and fewer crossings. - Change the interface —
epollinstead of polling each fd, thenio_uringto batch submissions and completions through shared memory.
Vectored I/O: one syscall, many buffers
An HTTP response is a header buffer plus a body buffer. Copying them into one buffer costs a memcpy; writing them separately costs two syscalls (and, without TCP_CORK/MSG_MORE, possibly two packets). writev does both in one call with no copy:
#include <sys/uio.h>
struct iovec iov[3];
iov[0].iov_base = hdr; iov[0].iov_len = hdr_len;
iov[1].iov_base = body; iov[1].iov_len = body_len;
iov[2].iov_base = "\r\n"; iov[2].iov_len = 2;
ssize_t n = writev(sock, iov, 3); // one syscall, three buffers (gather)
// readv() is the scatter counterpart; preadv2/pwritev2 add offsets and flags
For datagrams, sendmmsg/recvmmsg send or receive many messages per syscall — essential for UDP-heavy servers such as DNS resolvers and QUIC stacks, which can process dozens of packets per crossing. UDP_SEGMENT (GSO) goes further, handing the kernel one large buffer to split into packets.
Zero-copy: sendfile, splice, copy_file_range
Serving a static file with read + write copies every byte twice through user space:
read(): disk/page cache ──copy──▶ user buffer (syscall 1)
write(): user buffer ──copy──▶ socket buffer ──DMA──▶ NIC (syscall 2)
sendfile keeps the data in the kernel and, with a NIC that supports scatter-gather, avoids the CPU copy entirely by pointing the NIC's DMA at page-cache pages:
sendfile(): page cache ──(page refs, no CPU copy)──▶ socket ──DMA──▶ NIC (one syscall)
#include <sys/sendfile.h>
off_t off = 0;
while (off < st.st_size) {
ssize_t n = sendfile(sock, file_fd, &off, st.st_size - off); // advances off
if (n <= 0) { if (n < 0 && errno == EAGAIN) wait_writable(sock); else break; }
}
| Call | Moves data between | Typical use |
|---|---|---|
sendfile(out, in, …) | file → socket (or any fd since 2.6.33) | Static file servers: nginx sendfile on, Kafka consumers reading log segments |
splice(in, out, …) | any fd ↔ pipe, by moving page references | Proxies: socket → pipe → socket (HAProxy, socat) |
tee | pipe → pipe without consuming | Duplicating a stream |
vmsplice | user memory → pipe | Feeding pipes without a copy |
copy_file_range | file → file (can use reflinks / server-side copy on NFS, XFS, Btrfs) | cp, backup tools |
MSG_ZEROCOPY (send flag) | user buffer → socket without copy; completion via error queue | Very large sends (≥ ~10KB) where the copy dominates |
Go does this for you: io.Copy(tcpConn, file) uses sendfile, and io.Copy(tcpConn, otherTCPConn) uses splice on Linux. Kafka's famously high consumer throughput comes largely from serving log segments to consumers with sendfile straight from the page cache. TLS complicates zero-copy because data must be encrypted; kTLS moves symmetric encryption into the kernel (or the NIC) so sendfile works over HTTPS — nginx and Netflix's FreeBSD CDN use it.
io_uring: shared-memory rings
epoll tells you which fds are ready, but you still make a read/write syscall per operation. io_uring (Linux 5.1, 2019, by Jens Axboe) replaces that with two ring buffers shared between your process and the kernel:
flowchart LR
subgraph USER["User space"]
APP["Application"]
end
subgraph SHARED["Shared memory (mmap)"]
SQ["Submission Queue: SQEs (op, fd, buf, len, user_data)"]
CQ["Completion Queue: CQEs (user_data, result)"]
end
subgraph KERN["Kernel"]
IO["io_uring worker / SQPOLL thread"]
end
APP -->|"write SQEs, bump tail"| SQ
SQ -->|"io_uring_enter once per batch, or none with SQPOLL"| IO
IO -->|"post CQEs"| CQ
CQ -->|"read CQEs, bump head"| APP
- The application writes submission queue entries (read this fd into this buffer, accept on this socket, send this,
fsyncthat…) into the SQ ring and makes oneio_uring_entercall to submit a whole batch. - The kernel posts completion queue entries into the CQ ring; the application reads them from memory without a syscall.
- With
IORING_SETUP_SQPOLL, a kernel thread polls the SQ, so a busy application makes zero syscalls for I/O. - Registered files and buffers (
io_uring_register) pre-pin fds and memory, skipping per-operation reference counting and page pinning. - It covers nearly everything: file and socket I/O,
accept,connect,openat,statx,fsync, timeouts, linked operations (do B only if A succeeded), multishot accept/recv (one SQE, many completions), and zero-copy send/receive.
// Read N files with one submit syscall — gcc batch.c -luring
#include <liburing.h>
struct io_uring ring;
io_uring_queue_init(256, &ring, 0);
for (int i = 0; i < n; i++) {
struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
io_uring_prep_read(sqe, fds[i], bufs[i], 4096, 0);
io_uring_sqe_set_data64(sqe, i); // tag to match the completion
}
io_uring_submit(&ring); // ONE syscall submits n reads
for (int i = 0; i < n; i++) {
struct io_uring_cqe *cqe;
io_uring_wait_cqe(&ring, &cqe); // no syscall if a CQE is already there
handle(io_uring_cqe_get_data64(cqe), cqe->res); // res = bytes or -errno
io_uring_cqe_seen(&ring, cqe);
}
io_uring_queue_exit(&ring);
In Rust, tokio-uring, glommio and monoio build runtimes on it; in C/C++, liburing; databases and storage engines (ScyllaDB/Seastar, TigerBeetle, RocksDB's MultiGet, PostgreSQL 18's asynchronous I/O) use it for storage I/O. It's most compelling for storage I/O, where epoll never worked (regular files are always "ready"), and for very high-rate networking.
io_uring and security
io_uring is a large, fast-moving attack surface that bypasses syscall-level filtering: once a ring is set up, operations flow through shared memory, not through individual syscalls that seccomp could inspect. Google reported in 2023 that 60% of the kernel exploits submitted to its kCTF vulnerability reward program in 2022 targeted io_uring, and disabled it on ChromeOS, Android apps and its production servers. Newer Docker default seccomp profiles also block io_uring_setup, and kernel.io_uring_disabled (6.6+) lets you disable it system-wide or for unprivileged users. Check whether your platform allows it before designing around it.
Other ways syscalls disappear
- futex fast path: an uncontended
pthread_mutex_lockor Gosync.Mutexis a single atomic instruction in user space; only contention callsfutex()to sleep. See Kernel Synchronization. - Runtime batching: Go's netpoller parks goroutines and calls
epoll_waitonce for all of them; tokio does the same per worker thread. TCP_CORK/MSG_MORE: coalesce several writes into full packets.accept4(SOCK_NONBLOCK | SOCK_CLOEXEC),pipe2,O_CLOEXEConopenat: set flags at creation instead of a follow-upfcntl.- Kernel bypass (DPDK, SPDK, AF_XDP): the NIC or NVMe queue is driven directly from user space with polling — zero syscalls, at the price of dedicated cores and losing the kernel's stack.
| Technique | Syscalls saved | Effort |
|---|---|---|
| User-space buffering | 10–1000× fewer writes | Trivial |
writev / sendmmsg | 2–64× | Low |
sendfile / splice | 2× syscalls, plus the copy | Low |
epoll vs per-fd polling | O(fds) → O(ready events) | Medium (event loop) |
| io_uring | Batch of N → 1 (or 0 with SQPOLL) | Medium–high |
| Kernel bypass | All | High; dedicated hardware/cores |
seccomp: Syscall Filtering
seccomp (secure computing) restricts which syscalls a process may make. If a process calls a forbidden syscall, the kernel either kills it (SECCOMP_RET_KILL) or returns an error.
// Minimal seccomp filter using BPF:
struct sock_filter filter[] = {
BPF_STMT(BPF_LD | BPF_W | BPF_ABS, offsetof(struct seccomp_data, nr)),
BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, __NR_read, 1, 0),
BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, __NR_write, 0, 1),
BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ALLOW),
BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_KILL), // Kill on any other syscall
};
That minimal filter is incomplete
A real filter must first check seccomp_data.arch (e.g. AUDIT_ARCH_X86_64) and kill on mismatch — otherwise a process can switch to the 32-bit int 0x80 ABI, where syscall numbers mean different things, and slip past a number-based allow list. It must also be installed after prctl(PR_SET_NO_NEW_PRIVS, 1) (or with CAP_SYS_ADMIN). In practice, use libseccomp rather than hand-writing BPF:
#include <seccomp.h> // gcc sandbox.c -lseccomp
scmp_filter_ctx ctx = seccomp_init(SCMP_ACT_ERRNO(EPERM)); // default: fail with EPERM
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(read), 0);
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 0);
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(exit_group), 0);
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(futex), 0);
seccomp_load(ctx); // libseccomp adds the arch check for you
Filters can return ALLOW, KILL_PROCESS, ERRNO(n), TRAP (deliver SIGSYS), LOG, TRACE (hand to a ptrace tracer), or USER_NOTIF (hand to a supervisor process that decides — used by container runtimes to emulate syscalls like mount safely). The filter runs on every syscall, but it's a short classic-BPF program, so the cost is typically tens of nanoseconds.
Docker's default seccomp profile blocks ~44 dangerous syscalls including:
ptrace(can read/write other processes' memory)mount(can escape filesystem restrictions)swapon(filesystem changes)kexec_load(load a new kernel)open_by_handle_at(bypass mount namespace restrictions)
Why seccomp matters for security: A container escape via a kernel vulnerability requires calling a dangerous syscall. Seccomp prevents most exploit chains by blocking the syscalls vulnerabilities need.
grep Seccomp /proc/<pid>/status # 0 = off, 1 = strict, 2 = filter
docker run --security-opt seccomp=profile.json ...
# Kubernetes: securityContext.seccompProfile.type: RuntimeDefault
seccomp is one layer of a broader sandboxing toolkit — capabilities, namespaces, LSMs (SELinux, AppArmor, Landlock). The full treatment, including generating profiles from observed behaviour, is in Process Security.
eBPF: Extending the Kernel Safely
eBPF (extended Berkeley Packet Filter) allows running sandboxed programs inside the kernel without modifying kernel source or loading kernel modules. eBPF programs are verified before execution — the verifier rejects programs that could crash the kernel or loop forever.
eBPF program → verifier → JIT compiler → kernel execution
The verifier walks every possible path through the program, tracking the type and range of every register: it rejects out-of-bounds memory access, unbounded loops (bounded loops are allowed since 5.3), uninitialised reads, and calls to helpers the program type isn't allowed to use. Programs share data with user space through maps (hash tables, arrays, ring buffers, per-CPU variants), and BTF + CO-RE ("compile once, run everywhere") let one compiled program adapt to different kernels' struct layouts.
flowchart LR
SRC["C / bpftrace script"] --> CLANG["clang to BPF bytecode"]
CLANG -->|"bpf() syscall"| VER["Verifier"]
VER -->|"safe"| JIT["JIT to native code"]
VER -->|"unsafe"| REJ["Rejected with error log"]
JIT --> HOOK["Attached to hook: tracepoint, kprobe, XDP, TC, LSM"]
HOOK <-->|"maps and ring buffers"| USERSPACE["User-space agent"]
Where eBPF programs run (hook points):
- Syscall entry/exit (tracepoints, kprobes)
- Network packet processing (XDP, TC)
- Function entry/exit (kprobes, uprobes)
- Hardware performance counters (perf events)
- Security hooks (BPF LSM) — enforce policy at the same points SELinux and AppArmor use
- Scheduler policy (
sched_ext, merged in 6.12) — whole CPU schedulers written as BPF programs
Use cases:
| Use case | Tool | What it does |
|---|---|---|
| Network performance | XDP | Process packets before kernel network stack (line rate) |
| Observability | bpftrace, BCC | Trace syscalls, latency, function calls without restart |
| Security | Falco, Tetragon | Detect suspicious syscalls in production |
| Profiling | BPF-based profilers | CPU flamegraphs from production without instrumentation |
| Service mesh | Cilium | L3/L4/L7 networking without sidecar proxies |
# Trace open() syscalls in real time with bpftrace
bpftrace -e 'tracepoint:syscalls:sys_enter_open { printf("%s %s\n", comm, str(args->filename)); }'
# Count syscalls per second
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); } interval:s:1 { print(@); clear(@); }'
# Histogram of disk I/O request sizes (sectors) by read/write type
# (note: this measures SIZE, not latency — see the next one-liner for latency)
bpftrace -e 'tracepoint:block:block_rq_complete /args->nr_sector > 0 && args->error == 0/ { @[args->rwbs] = hist(args->nr_sector); }'
# Trace slow disk I/O (> 10ms): latency from issue to completion
bpftrace -e '
tracepoint:block:block_rq_issue { @start[args->dev, args->sector] = nsecs; }
tracepoint:block:block_rq_complete /@start[args->dev, args->sector]/ {
$ms = (nsecs - @start[args->dev, args->sector]) / 1000000;
if ($ms > 10) { printf("slow I/O: %d ms, %d sectors\n", $ms, args->nr_sector); }
delete(@start[args->dev, args->sector]);
}'
eBPF vs kernel modules:
- Kernel modules can crash the kernel; eBPF programs are verified
- Kernel modules require kernel headers and version matching; eBPF programs are portable via BTF (BPF Type Format)
- eBPF is hot-loadable; kernel modules require careful management
The trade-off runs the other way too: eBPF programs can only do what the verifier can prove safe and what the available helpers allow, so drivers and file systems are still modules. Loading BPF requires CAP_BPF (plus CAP_PERFMON/CAP_NET_ADMIN depending on the program type); unprivileged BPF is disabled by default on most distros (kernel.unprivileged_bpf_disabled). The deep dives are eBPF and eBPF in Production.
Kernel Space Memory
The kernel has its own memory regions. On x86-64 with a 48-bit virtual address space:
User space: 0x0000000000000000 – 0x00007fffffffffff (128TB)
Kernel space: 0xffff800000000000 – 0xffffffffffffffff (128TB)
The kernel address space is mapped into every process's page tables (but marked Ring 0 only — user code cannot access it). This "higher half" mapping allows the kernel to access its own data structures without a full context switch, but KPTI separates them for Meltdown mitigation.
Inside the kernel half there is a direct map of all physical RAM (so the kernel can reach any frame at PAGE_OFFSET + phys), the vmalloc area for virtually contiguous allocations, the kernel image and modules, and per-CPU areas — all randomised by KASLR at boot. Each thread also has a small fixed-size kernel stack (16KB on x86-64) used while it executes syscalls; deep call chains in the kernel are therefore a real constraint, and kernel stack overflow is caught by guard pages (CONFIG_VMAP_STACK).
When a syscall needs user data, the kernel never dereferences user pointers directly: it uses copy_from_user / copy_to_user, which check that the address range is in user space, open the SMAP window, and handle a fault gracefully (returning -EFAULT to the caller instead of crashing the kernel) if the user passed a bad pointer.
Real Numbers
| Quantity | Value |
|---|---|
Bare SYSCALL/SYSRET round trip, no mitigations | ~50–100 ns |
Simple syscall (getppid) with typical mitigations | ~100–300 ns; up to ~1 µs with KPTI and no PCID |
clock_gettime via vDSO | ~15–25 ns |
| seccomp filter evaluation | tens of ns per syscall |
| Context switch (blocking syscall → another thread) | ~1–5 µs direct, more with cache effects |
| strace slowdown on syscall-heavy code | 2× to 100×+ |
| KPTI overhead | < 1% below ~10K syscalls/s per CPU; tens of percent for syscall-bound workloads on affected CPUs |
| Linux x86-64 syscalls | ~380 (numbers 0–335 and 424+) |
| Docker default seccomp profile | blocks ~44 syscalls |
| Kernel stack per thread (x86-64) | 16 KB |
| io_uring | 1 syscall per batch, 0 with SQPOLL |
Decision Guide
| Situation | Approach |
|---|---|
High sys time, many small read/write | Buffer in user space; writev; check per-request syscall count with strace -c |
| Serving static files or log segments | sendfile (io.Copy in Go, nginx sendfile on); kTLS for HTTPS |
| Proxying bytes between sockets | splice through a pipe, or a runtime that does it for you |
| Lots of storage I/O, need concurrency | io_uring (or a thread pool if io_uring is blocked on your platform) |
| UDP at high packet rates | recvmmsg/sendmmsg, UDP_SEGMENT/GRO; AF_XDP or DPDK at the extreme |
clock_gettime visible in strace | Fix the clocksource (tsc) |
| Debug a failing process on a dev box | strace -f -e trace=%file,%network |
| Investigate a production process | perf trace -s, bpftrace, BCC syscount/opensnoop |
| Sandbox untrusted code | seccomp allow list via libseccomp, plus namespaces and capabilities (Process Security) |
| Extend kernel behaviour | eBPF if it fits the hooks and verifier; module only for drivers/file systems |
Older CPUs, syscall-bound, considering mitigations=off | Reduce syscalls first; only disable on single-tenant trusted hosts |
Advanced Topics
- Syscall User Dispatch (5.11): redirect syscalls made from certain code regions to a
SIGSYShandler — built for Wine to emulate Windows system calls made directly by games. - gVisor: a user-space kernel (written in Go) that intercepts a sandboxed application's syscalls and implements them itself, only making a small, filtered set of host syscalls. Stronger isolation than seccomp alone, at a syscall-performance cost. Used by Google Cloud Run and App Engine.
- Unikernels (MirageOS, Unikraft): link the application with just the OS pieces it needs into one ring-0 image; a "syscall" becomes a function call.
- FlexSC / exception-less syscalls: the research idea (OSDI 2010) of posting syscalls to shared pages and having kernel threads on other cores execute them — io_uring is its production descendant.
- Landlock (5.13): unprivileged, self-imposed file-system and network access restrictions — a complement to seccomp that filters objects rather than syscall numbers.
- Restartable sequences (
rseq): a per-thread area the kernel updates on preemption, letting user space do per-CPU operations without atomics — the basis of tcmalloc's per-CPU caches and glibc's fastsched_getcpu.
Interview Questions
"Why are system calls expensive?" A syscall crosses the Ring 3 → Ring 0 boundary: the CPU executes SYSCALL instruction, saves user registers to the kernel stack, switches to kernel mode, executes the handler, then returns. The overhead is ~100–1000ns — partly the mode switch, partly TLB invalidation (KPTI), partly cache misses (kernel code/data may be cold). At 100K RPS with 5 syscalls each = 500K syscalls/sec; at 500ns each = 250ms CPU/sec just on syscalls.
"What is vDSO?"
A small read-only shared library that the kernel maps into every process. It exposes certain syscalls (clock_gettime, gettimeofday, getcpu) in user space — they read from kernel-maintained memory without a mode switch. clock_gettime(CLOCK_MONOTONIC) is ~5ns via vDSO vs ~100ns as a real syscall.
"What is seccomp and why do containers use it?" seccomp filters restrict which syscalls a process may call using BPF programs. Docker's default profile blocks ~44 dangerous syscalls (ptrace, mount, kexec_load). This limits the damage from container escape exploits — even if an attacker controls user code, they can't call the syscalls needed to escalate to host root.
"What is eBPF?" eBPF lets you run sandboxed programs inside the kernel at hook points (syscall entry/exit, network packet processing, function calls) without modifying kernel source or loading modules. The verifier ensures safety. Used by Cilium for networking, Falco for security, bpftrace/BCC for observability. The key property: you can add instrumentation to a running production kernel without restarting anything.
"What exactly does the SYSCALL instruction do?"
It saves RIP in RCX and RFLAGS in R11, masks flags, loads ring-0 code and stack segments from STAR, and jumps to the address in LSTAR. It doesn't switch stacks — the kernel's entry code does that (and switches CR3 under KPTI). That's why the fourth syscall argument goes in r10, not rcx.
"What did Meltdown change about syscall cost?" Meltdown let user code speculatively read kernel memory mapped in its page tables. KPTI fixes it by keeping two page-table sets and switching on every kernel entry and exit; without PCID that flushes the TLB each time. Syscall-heavy workloads on affected CPUs lost 5–30%, which renewed interest in batching interfaces like io_uring.
"How does sendfile achieve zero-copy?" Instead of copying file data into a user buffer and back into a socket buffer, it passes references to page-cache pages to the socket; with scatter-gather DMA, the NIC reads directly from those pages. One syscall, no CPU copies of the payload.
"How would you find out why a process is hanging?"
cat /proc/<pid>/stack and /proc/<pid>/wchan for where the kernel has it blocked, strace -p (briefly) to see the syscall it's stuck in and with what arguments (-y for fd names), and perf trace --duration or bpftrace to find slow syscalls in production without the ptrace overhead.
Related Pages
- Virtual Memory — page tables, TLB, PCID (the mechanics behind KPTI)
- Memory Allocators — how
mallocavoidsbrk/mmapsyscalls - Process Security — seccomp, capabilities, LSMs in depth
- Kernel Synchronization — futexes, spinlocks, RCU
- I/O & Storage — blocking, non-blocking, epoll and async models
- Signals —
EINTR,SA_RESTART,SIGSYS - eBPF and eBPF in Production
- Namespaces & cgroups — the other half of container isolation