Linux Namespaces & cgroups
A container is not a VM. It's a process (or group of processes) running on the host kernel, isolated via namespaces and constrained via cgroups. There is no separate guest OS, no hypervisor, no emulated hardware. The container process calls the same kernel as the host. Namespaces make the process believe it's the only thing on the machine; cgroups make sure it can't consume more resources than its share.
Understanding this explains why containers start in milliseconds (not seconds), why they're cheaper than VMs, and why certain workloads (like those needing different kernel versions) genuinely need VMs.
Why These Mechanisms Exist
Before namespaces and cgroups, a Unix machine had exactly one of everything: one process tree, one set of network interfaces, one hostname, one filesystem tree, one user database. Running two services that both wanted port 80, or both wanted /etc/myapp.conf, or one of which must never starve the other of RAM, meant one machine per service — or a VM per service, with a whole guest kernel and seconds-to-minutes of boot time.
The pieces arrived one at a time:
| Year | Mechanism | What it added |
|---|---|---|
| 1979 | chroot (Version 7 Unix) | A different root directory — filesystem view only, trivially escapable by root |
| 2000 | FreeBSD jails | Filesystem + process + network isolation in one primitive |
| 2002 | Linux mount namespace (2.4.19) | First Linux namespace; CLONE_NEWNS ("new namespace" — nobody expected more) |
| 2004 | Solaris Zones | Full OS-level virtualization |
| 2006–2013 | UTS, IPC, PID, network, user namespaces | Linux 2.6.19 → 3.8 |
| 2008 | cgroups v1 (2.6.24) | Google's "process containers" — resource accounting and limits |
| 2008 | LXC | First complete Linux container manager built from these parts |
| 2013 | Docker | Packaging (image layers + registry) made containers usable by everyone |
| 2016 | cgroups v2 (4.5), cgroup namespace (4.6) | A single, consistent hierarchy |
| 2020 | Time namespace (5.6) | Per-container monotonic and boot-time clocks |
The key insight: isolation and resource control are separate problems, and Linux solved them with separate mechanisms. Namespaces answer "what can this process see?"; cgroups answer "how much can this process use?". A container runtime is the glue that sets up both, plus a root filesystem, plus security filters.
The Two Mechanisms
| Mechanism | What it does |
|---|---|
| Namespaces | Isolation — each namespace wraps a global resource and makes it appear private to the processes inside |
| cgroups | Accounting and limits — tracks and enforces how much CPU, memory, I/O, and number of processes a group of processes can use |
A container is: a set of namespaces (isolation) + a cgroup (limits) + a filesystem (union mount via overlay fs).
In practice a production container is also: + dropped capabilities + a seccomp filter + an LSM profile (AppArmor/SELinux). Namespaces alone are not a security boundary — see Security Implications.
flowchart TB
subgraph Host["Host kernel (shared)"]
direction TB
subgraph C1["Container A"]
P1["PID 1: nginx"]
end
subgraph C2["Container B"]
P2["PID 1: postgres"]
end
NS["Namespaces: pid, net, mnt, uts, ipc, user, cgroup, time"]
CG["cgroups: cpu, memory, io, pids"]
FS["Root filesystem: overlayfs layers"]
SEC["Security: capabilities, seccomp, AppArmor or SELinux"]
end
C1 -.-> NS
C1 -.-> CG
C1 -.-> FS
C1 -.-> SEC
C2 -.-> NS
C2 -.-> CG
Linux Namespaces
Linux has 8 namespace types. Each wraps one global resource.
| Namespace | clone flag | Kernel | Isolates | unshare flag |
|---|---|---|---|---|
| Mount | CLONE_NEWNS | 2.4.19 | Mount table — the filesystem tree | --mount / -m |
| UTS | CLONE_NEWUTS | 2.6.19 | Hostname, NIS domain name | --uts / -u |
| IPC | CLONE_NEWIPC | 2.6.19 | System V IPC, POSIX message queues | --ipc / -i |
| PID | CLONE_NEWPID | 2.6.24 | Process ID number space | --pid / -p (with --fork) |
| Network | CLONE_NEWNET | 2.6.29 | Interfaces, routes, iptables/nftables, sockets, /proc/net | --net / -n |
| User | CLONE_NEWUSER | 3.8 | UIDs/GIDs, capabilities | --user / -U |
| Cgroup | CLONE_NEWCGROUP | 4.6 | View of the cgroup hierarchy | --cgroup / -C |
| Time | CLONE_NEWTIME | 5.6 | CLOCK_MONOTONIC, CLOCK_BOOTTIME offsets | --time / -T |
Every process belongs to exactly one namespace of each type. Namespaces are reference-counted kernel objects exposed as magic symlinks:
$ ls -l /proc/self/ns/
lrwxrwxrwx cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx ipc -> 'ipc:[4026531839]'
lrwxrwxrwx mnt -> 'mnt:[4026531841]'
lrwxrwxrwx net -> 'net:[4026531840]'
lrwxrwxrwx pid -> 'pid:[4026531836]'
lrwxrwxrwx pid_for_children -> 'pid:[4026531836]'
lrwxrwxrwx time -> 'time:[4026531834]'
lrwxrwxrwx time_for_children -> 'time:[4026531834]'
lrwxrwxrwx user -> 'user:[4026531837]'
lrwxrwxrwx uts -> 'uts:[4026531838]'
$ lsns # every namespace on the system and its owning processes
NS TYPE NPROCS PID USER COMMAND
4026531840 net 312 1 root /sbin/init
4026532577 net 3 8812 root /pause
...
Two processes are in the same namespace if the inode numbers match. A namespace lives as long as some process is in it or something holds a reference — an open fd on the /proc/<pid>/ns/* file, or a bind mount of it (that's how ip netns add keeps a network namespace alive with no processes: it bind-mounts it at /run/netns/<name>).
The three syscalls:
clone(flags)— create a child in new namespaces.unshare(flags)— move the caller into new namespaces (PID is special: only the caller's future children go in).setns(fd, type)— join an existing namespace via its/proc/<pid>/ns/*fd. This is whatnsenter,docker execandkubectl execdo.
1. PID Namespace
Processes inside a PID namespace have their own PID space starting from 1. PID 1 inside the container is the init process for that namespace. From outside the container, the actual PID is visible (e.g., 5432). From inside, it appears as PID 1.
Host: PID 5432 (nginx) PID 5433 (nginx worker)
Container: PID 1 (nginx) PID 2 (nginx worker)
Why PID 1 matters: In Linux, PID 1 has special responsibilities — it must reap orphan processes (call wait()). If PID 1 doesn't handle SIGTERM, the whole container hangs on shutdown. This is why Docker containers should use exec form (["nginx", "-g", "daemon off;"]) not shell form (nginx -g daemon off), and why images commonly use tini or dumb-init as PID 1 (Docker's --init flag injects tini; Kubernetes itself does not add an init, but shareProcessNamespace: true makes the pod's pause container PID 1 and reaper). The kernel also ignores default-action signals sent to a namespace's PID 1 from inside — details in Signals.
Hands-on:
# New PID namespace. --fork is required: the unshare process itself stays in the
# old namespace; its child becomes PID 1. --mount-proc remounts /proc so ps works.
$ sudo unshare --pid --fork --mount-proc bash
# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 8600 5200 pts/1 S 10:01 0:00 bash
root 8 0.0 0.0 10200 3300 pts/1 R+ 10:01 0:00 ps aux
# From another host terminal - the same bash has an ordinary host PID:
$ ps -ef | grep 'unshare --pid' ; pstree -p $(pgrep -f 'unshare --pid')
unshare(9120)───bash(9121)
# Map between namespaces: NSpid shows the PID at every nesting level
$ grep NSpid /proc/9121/status
NSpid: 9121 1
PID namespaces nest: a process has a PID in its own namespace and in every ancestor namespace. The host sees everything; a container sees only its subtree. When PID 1 of a namespace exits, the kernel sends SIGKILL to every other process in it.
2. Network Namespace
Each network namespace has its own:
- Network interfaces (eth0, lo)
- Routing tables
- iptables rules
- Sockets (bound ports)
Plus its own /proc/net, /sys/class/net, conntrack table, and per-namespace sysctls under net.* (e.g. net.core.somaxconn is per-namespace; net.netfilter.nf_conntrack_max is mostly global).
Two processes in different network namespaces can both bind port 80 — they have separate port spaces.
# Create a new network namespace
ip netns add myns
# Run a command in the namespace
ip netns exec myns ip addr show # Only sees lo (loopback)
# Create a veth pair (virtual ethernet cable)
ip link add veth0 type veth peer name veth1
ip link set veth1 netns myns # Move one end into namespace
# Now myns has veth1; host has veth0 — they're connected
The commands above create the cable but nothing can talk yet. To get a working, internet-reachable namespace — exactly what Docker's default bridge network does:
# Addresses and links up
ip addr add 10.200.0.1/24 dev veth0
ip link set veth0 up
ip netns exec myns ip addr add 10.200.0.2/24 dev veth1
ip netns exec myns ip link set veth1 up
ip netns exec myns ip link set lo up
ip netns exec myns ip route add default via 10.200.0.1
# Host forwards and masquerades traffic from the namespace
sysctl -w net.ipv4.ip_forward=1
iptables -t nat -A POSTROUTING -s 10.200.0.0/24 ! -o veth0 -j MASQUERADE
ip netns exec myns ping -c1 1.1.1.1 # works
ip netns exec myns python3 -m http.server 80 &
curl 10.200.0.2 # host reaches the server inside
# Publish a port like docker -p 8080:80
iptables -t nat -A PREROUTING -p tcp --dport 8080 -j DNAT --to-destination 10.200.0.2:80
Docker creates a veth pair per container. One end in the container's network namespace (renamed to eth0), one end in the host namespace, connected to a bridge (docker0).
flowchart LR
subgraph NS1["netns: container A"]
E1["eth0 172.17.0.2"]
end
subgraph NS2["netns: container B"]
E2["eth0 172.17.0.3"]
end
subgraph Host["host netns"]
V1["vethA"] --- BR["docker0 bridge 172.17.0.1"]
V2["vethB"] --- BR
BR --> NAT["iptables MASQUERADE and DNAT"]
NAT --> ETH["eth0 (physical)"]
end
E1 === V1
E2 === V2
In Kubernetes, every pod gets one network namespace, created and held by the pause container; all containers in the pod join it (setns), which is why they share localhost and a port space. The CNI plugin (Calico, Cilium, Flannel, AWS VPC CNI) wires that namespace to the node — see CNI Networking and Linux Networking Stack.
# Enter a running container's network namespace with host tools (tcpdump, ss, ip)
PID=$(docker inspect -f '{{.State.Pid}}' web)
sudo nsenter -t "$PID" -n ss -tlnp
sudo nsenter -t "$PID" -n tcpdump -i eth0 -nn port 80
This "borrow the namespace, keep the host's binaries" trick is the single most useful debugging technique for distroless images that ship without a shell.
3. Mount Namespace
Each mount namespace has its own view of the filesystem hierarchy. Mounting a filesystem in one namespace doesn't affect others.
unshare --mount bash # New mount namespace
mount -t tmpfs tmpfs /mnt/test # Visible only in this namespace
ls /mnt/test # Visible
exit
ls /mnt/test # Not visible in host
Mount propagation complicates this. On systemd hosts / is mounted shared, so a naive new mount namespace starts as a peer and mounts can leak back. unshare --mount defaults to --propagation private to avoid that. The four modes:
| Propagation | Mounts flow from peer → me | Mounts flow from me → peer | Use |
|---|---|---|---|
shared | Yes | Yes | Host default under systemd |
slave | Yes | No | Container sees new host mounts (e.g. a USB drive), can't push its own |
private | No | No | Default for containers |
unbindable | No | No, and can't be bind-mounted | Prevent mount explosion in recursive binds |
Kubernetes exposes this as mountPropagation: HostToContainer (slave) or Bidirectional (shared, privileged only) on a volume mount — used by CSI drivers that mount volumes from inside a pod.
A container doesn't just get a new mount namespace; it changes its root with pivot_root() (not chroot(), which is escapable by root with a second chroot + chdir("..")). pivot_root swaps the root mount and lets the old root be unmounted entirely, so nothing of the host filesystem remains reachable.
OverlayFS: Containers use OverlayFS to layer an image's read-only layers with a writable container layer. Changes go to the writable layer; the image layers are shared across all containers using the same image.
Container filesystem:
Writable layer: /var/lib/docker/overlay2/<id>/diff (container-specific)
↓ merged view
Lower layer: ubuntu:22.04 image layers (read-only, shared)
Copy-up, whiteouts and the performance gotchas of overlay are covered in File Systems.
4. UTS Namespace
Isolates hostname and NIS domain name. Each container can have its own hostname without affecting the host.
unshare --uts bash
hostname mycontainer
hostname # → mycontainer (inside)
# host still shows original hostname
("UTS" is the struct utsname returned by uname() — "UNIX Time-sharing System".) In Kubernetes the pod's hostname is the pod name; containers in a pod share the UTS namespace.
5. IPC Namespace
Isolates System V IPC objects (shared memory segments, message queues, semaphores) and POSIX message queues. Prevents containers from communicating via shared memory accidentally.
$ ipcmk -M 1048576 # create a 1 MiB SysV shared memory segment on the host
Shared memory id: 32769
$ sudo unshare --ipc ipcs -m # new IPC namespace: the segment is invisible
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
POSIX shared memory (shm_open) lives in /dev/shm, which is a tmpfs — isolated by the mount namespace, not the IPC one. Docker gives each container a 64 MB /dev/shm by default (--shm-size), a classic cause of crashes in Chrome/Puppeteer and PyTorch DataLoader workers. In Kubernetes, mount an emptyDir with medium: Memory at /dev/shm to raise it. Containers in the same pod share the IPC namespace. More in IPC.
6. User Namespace
Maps user/group IDs. UID 0 (root) inside a user namespace can map to a non-root UID outside. Enables rootless containers — a process can appear as root inside the container but runs as an unprivileged user on the host.
# Map host UID 1000 to UID 0 inside the namespace
unshare --user --map-root-user bash
id # → uid=0(root) gid=0(root) — appears as root inside
# But host sees this process running as UID 1000
The mapping lives in /proc/<pid>/uid_map as inside-start outside-start count:
$ cat /proc/$$/uid_map # inside the namespace above
0 1000 1
# Rootless Podman / Docker: a range of subordinate IDs from /etc/subuid
$ grep alice /etc/subuid
alice:100000:65536
$ podman unshare cat /proc/self/uid_map
0 1000 1
1 100000 65536
So container UID 0 = host 1000, container UID 1–65536 = host 100000–165535. A file created by "root" in the container is owned by UID 1000 on the host; a container escape lands the attacker as an unprivileged user.
User namespaces are also the only namespace an unprivileged user can create, and creating one first grants a full capability set inside it — which then allows creating the other namespace types. That's what makes rootless containers possible, and also why user namespaces have been a large source of kernel privilege-escalation bugs: they expose code paths (e.g. netfilter, filesystem mounting) to unprivileged users. Some distributions restrict them (Ubuntu 24.04 gates unprivileged user namespaces behind AppArmor; kernel.unprivileged_userns_clone=0 on older Debian).
Kubernetes added pod-level user namespaces with hostUsers: false (beta, on by default since 1.33), relying on idmapped mounts so volumes still have sensible ownership.
7. Time Namespace
Time namespace (Linux 5.6): Isolates CLOCK_MONOTONIC and CLOCK_BOOTTIME. Allows containers to have different views of elapsed time (useful for migration and checkpointing).
It does not virtualize wall-clock time (CLOCK_REALTIME) — you cannot give a container a different date with it. Its real purpose is CRIU live migration: a checkpointed process restored on another machine expects CLOCK_MONOTONIC to continue from where it left off, not jump to the new host's uptime.
$ sudo unshare --time --fork --boottime 86400 uptime
10:12:01 up 1 day, ... # boot time shifted by one day inside
8. Cgroup Namespace
A process in a cgroup namespace sees its cgroup root as /, hiding the host's cgroup hierarchy. Prevents containers from seeing or escaping to parent cgroups.
# On the host
$ cat /proc/$(docker inspect -f '{{.State.Pid}}' web)/cgroup
0::/system.slice/docker-3f9a1c...scope
# Inside the container (cgroup namespace)
$ docker exec web cat /proc/self/cgroup
0::/
This also lets a container read its own limits from /sys/fs/cgroup/memory.max without knowing its host path — which is how the JVM, Go 1.25+ and .NET discover container limits.
The clone() System Call
Namespaces are created with the clone() syscall (or unshare() in an existing process). clone() is fork() on steroids:
// This is essentially what Docker/containerd does:
int flags = CLONE_NEWPID // New PID namespace
| CLONE_NEWNET // New network namespace
| CLONE_NEWNS // New mount namespace
| CLONE_NEWUTS // New UTS namespace
| CLONE_NEWIPC; // New IPC namespace
pid_t pid = clone(container_main, stack_top, flags | SIGCHLD, arg);
unshare() creates new namespaces for the calling process without forking.
A complete, compilable minimal example:
#define _GNU_SOURCE
#include <sched.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/mount.h>
#include <sys/wait.h>
#include <unistd.h>
static char child_stack[1024 * 1024];
static int child(void *arg) {
(void)arg;
sethostname("box", 3);
mount(NULL, "/", NULL, MS_REC | MS_PRIVATE, NULL); // stop propagation to host
mount("proc", "/proc", "proc", 0, NULL); // fresh /proc for our PID ns
printf("inside: pid=%d\n", getpid()); // prints 1
execlp("sh", "sh", NULL);
perror("exec");
return 1;
}
int main(void) {
int flags = CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWUTS | CLONE_NEWIPC | CLONE_NEWNET;
pid_t pid = clone(child, child_stack + sizeof child_stack, flags | SIGCHLD, NULL);
if (pid < 0) { perror("clone"); return 1; }
printf("outside: child pid=%d\n", pid); // a normal host PID
waitpid(pid, NULL, 0);
}
clone3() (Linux 5.3+) adds a struct-based interface and CLONE_INTO_CGROUP (5.7), so a child can be created directly inside a target cgroup — no window where it runs unconstrained. runc and systemd use it when available.
Build a Container by Hand
Everything Docker does at docker run can be reproduced with stock util-linux tools. This is the best way to internalize that "a container" is just a process with some kernel settings. (A longer walkthrough lives in Containers From Scratch.)
#!/usr/bin/env bash
# mini-container.sh - run as root on a cgroup v2 host
set -euo pipefail
ROOT=/tmp/mini/rootfs
mkdir -p "$ROOT"
# 1. A root filesystem: any image's contents will do
if [ ! -x "$ROOT/bin/sh" ]; then
curl -sL https://dl-cdn.alpinelinux.org/alpine/v3.20/releases/x86_64/alpine-minirootfs-3.20.3-x86_64.tar.gz \
| tar -xz -C "$ROOT"
# or: docker export "$(docker create alpine:3.20)" | tar -x -C "$ROOT"
fi
# 2. A cgroup with limits
CG=/sys/fs/cgroup/mini
mkdir -p "$CG"
echo "+cpu +memory +pids" > /sys/fs/cgroup/cgroup.subtree_control
echo "50000 100000" > "$CG/cpu.max" # 0.5 CPU
echo 256M > "$CG/memory.max"
echo 0 > "$CG/memory.swap.max"
echo 64 > "$CG/pids.max"
echo $$ > "$CG/cgroup.procs" # this shell (and its children) are now limited
# 3. New namespaces + pivot_root + exec
exec unshare --pid --fork --mount --uts --ipc --net --cgroup \
/bin/bash -c "
set -e
hostname mini
mount --make-rprivate /
mount --bind '$ROOT' '$ROOT' # pivot_root needs new root to be a mount point
cd '$ROOT'
mkdir -p .old_root
pivot_root . .old_root
cd /
mount -t proc proc /proc
mount -t sysfs sysfs /sys
mount -t tmpfs tmpfs /tmp
umount -l /.old_root && rmdir /.old_root
exec /bin/sh
"
Inside the result:
/ # ps # PID 1 is sh - PID namespace
PID USER TIME COMMAND
1 root 0:00 /bin/sh
2 root 0:00 ps
/ # hostname # mini - UTS namespace
/ # ip link # only lo, DOWN - network namespace
/ # cat /sys/fs/cgroup/memory.max 2>/dev/null || cat /proc/self/cgroup # 0::/ - cgroup namespace
/ # yes > /dev/null & # pinned at 50% CPU on the host (top shows it)
/ # :(){ :|:& };: # fork bomb stops at 64 processes: "can't fork"
What this toy is still missing compared with runc: a user namespace (it runs as real root), dropped capabilities, a seccomp filter, masked paths (/proc/kcore, /proc/sys read-only), device cgroup rules, networking (wire a veth in from the host using the network namespace steps above and the PID of the sh), and overlayfs instead of a plain directory. Each of those is a few more lines — which is roughly what runc's libcontainer is.
cgroups v1 vs v2
Control groups (cgroups) organize processes into a hierarchy and enforce resource limits. Every process belongs to exactly one cgroup (per hierarchy in v1; exactly one, period, in v2).
cgroups v2 (unified hierarchy, merged in Linux 4.5, production-ready for all major controllers by ~5.8) exposes the cgroup filesystem at /sys/fs/cgroup/. Distributions made it the default from around 2020–2021 (Fedora 31, Debian 11, Ubuntu 21.10, RHEL 9). Kubernetes support for v2 went GA in 1.25; v1 support entered maintenance mode in 1.31, and recent kubelets refuse to start on v1 hosts unless explicitly configured to.
$ stat -fc %T /sys/fs/cgroup
cgroup2fs # v2 (unified). "tmpfs" means v1 or hybrid.
$ cat /sys/fs/cgroup/cgroup.controllers
cpuset cpu io memory hugetlb pids rdma misc
| Aspect | cgroups v1 | cgroups v2 |
|---|---|---|
| Hierarchies | One per controller (/sys/fs/cgroup/memory/..., /sys/fs/cgroup/cpu/...) — a process can be in different places in each | One unified tree; controllers are enabled per subtree |
| Process placement | Processes can live in any node | "No internal processes" rule: a cgroup with controllers enabled for children can't hold processes itself (root excepted) |
| Memory limit | memory.limit_in_bytes, soft limit poorly defined | memory.max, memory.high, memory.low, memory.min — clear semantics |
| Writeback I/O attribution | Buffered writes charged to whoever flushes (often root) — I/O limits only worked for direct I/O | Page cache dirtied by a cgroup is charged to it; io.max works for buffered writes |
| CPU | cpu.shares (2–262144, default 1024), cpu.cfs_quota_us / cpu.cfs_period_us | cpu.weight (1–10000, default 100), cpu.max |
| Pressure metrics | None | PSI per cgroup (cpu.pressure, memory.pressure, io.pressure) |
| OOM grouping | Kill one process | memory.oom.group — kill the whole cgroup atomically |
| Delegation to unprivileged users | Unsafe | Designed in (rootless containers, systemd user slices) |
| Event notification | eventfd + cgroup.event_control | inotify/poll on *.events files |
| Thread granularity | tasks file per thread | cgroup.threads in "threaded" subtrees only |
flowchart TB
subgraph V1["cgroups v1: a tree per controller"]
M["/sys/fs/cgroup/memory"] --> MA["docker/abc"]
C["/sys/fs/cgroup/cpu"] --> CA["docker/abc"]
B["/sys/fs/cgroup/blkio"] --> BA["docker/abc"]
end
subgraph V2["cgroups v2: one tree, controllers per subtree"]
R["/sys/fs/cgroup (cgroup.subtree_control: cpu memory io pids)"] --> S["system.slice"]
R --> K["kubepods.slice"]
K --> KB["kubepods-burstable.slice"]
KB --> POD["pod-uid.slice"]
POD --> CT["cri-containerd-id.scope: cpu.max memory.max pids.max"]
end
Controllers are switched on for children by writing to the parent's cgroup.subtree_control:
cat /sys/fs/cgroup/cgroup.subtree_control # cpu io memory pids
mkdir -p /sys/fs/cgroup/app
echo "+cpu +memory +io +pids" > /sys/fs/cgroup/app/cgroup.subtree_control # for app/* children
mkdir /sys/fs/cgroup/app/web /sys/fs/cgroup/app/batch
On systemd machines, don't hand-create cgroups under /sys/fs/cgroup for long-lived services — systemd owns the tree. Use units and slices instead; systemd writes the same files:
systemd-run --scope -p MemoryMax=512M -p CPUQuota=50% -p TasksMax=100 ./batch-job
systemctl set-property nginx.service MemoryHigh=1G CPUWeight=200
systemd-cgls # tree of cgroups and their processes
systemd-cgtop # top, per cgroup
cgroups v2 Controllers
CPU Limits
# Create a cgroup
mkdir /sys/fs/cgroup/myapp
# Limit to 0.5 CPU cores (50ms out of every 100ms)
echo "50000 100000" > /sys/fs/cgroup/myapp/cpu.max
# ↑ quota ↑ period (microseconds)
# Move a process into the cgroup
echo $PID > /sys/fs/cgroup/myapp/cgroup.procs
CPU shares (weight-based): cpu.weight (default 100). A cgroup with weight 200 gets 2× the CPU of a cgroup with weight 100 when both are CPU-constrained.
Kubernetes CPU requests/limits map to:
- CPU request →
cpu.weight(scheduling priority) - CPU limit →
cpu.max(hard cap via CFS bandwidth throttling)
The two knobs are fundamentally different:
cpu.weight | cpu.max | |
|---|---|---|
| Kind | Relative share | Absolute ceiling |
| When does it matter? | Only when the CPU is contended | Always, even on an idle machine |
| Effect when exceeded | Nothing — you just get less when others want more | Throttled: runnable threads are descheduled until the next period |
| Work-conserving? | Yes — idle CPU is handed out | No — idle CPU is wasted |
How CFS bandwidth throttling actually works. Every period (default 100 ms), the cgroup's runtime pool is refilled to quota. Each CPU running a thread of the cgroup pulls slices (default 5 ms, sched_cfs_bandwidth_slice_us) from the pool. When the pool is empty, every thread of the cgroup on every CPU is throttled until the next period starts.
The quota is in CPU-time, not wall-time, and it is shared across all threads. That produces the most misunderstood behavior in container performance:
gantt
title "limit = 1 CPU (100ms quota per 100ms period), 4 busy threads"
dateFormat X
axisFormat %L
section Thread 1
"running" :a1, 0, 25
"throttled" :crit, a2, 25, 100
section Thread 2
"running" :b1, 0, 25
"throttled" :crit, b2, 25, 100
section Thread 3
"running" :c1, 0, 25
"throttled" :crit, c2, 25, 100
section Thread 4
"running" :d1, 0, 25
"throttled" :crit, d2, 25, 100
Four threads burn 100 ms of CPU-time in 25 ms of wall-clock, then the whole container is frozen for 75 ms. A request that needed 5 ms of CPU and arrived at t=26 ms waits until t=100 ms: a 75 ms tail-latency spike while the average CPU usage graph shows a calm "1 core". This is why multi-threaded runtimes (Go, JVM with many GC threads, Node with libuv workers) see p99 spikes under CPU limits even at low average utilization.
$ cat /sys/fs/cgroup/kubepods.slice/.../cpu.stat
usage_usec 912345678
user_usec 700000000
system_usec 212345678
nr_periods 86400 # periods in which the cgroup was runnable
nr_throttled 21600 # periods in which it hit the quota
throttled_usec 1620000000 # total time spent throttled (27 minutes!)
nr_bursts 0
burst_usec 0
Throttle ratio = nr_throttled / nr_periods. Above ~1–5 % for a latency-sensitive service is worth investigating; 25 % (as above) is a guaranteed tail-latency problem. In Prometheus: rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m]).
Mitigations:
- Match thread count to the limit. Go ≤ 1.24: set
GOMAXPROCSfrom the limit (go.uber.org/automaxprocs); Go 1.25+ does this automatically fromcpu.max. JVM:-XX:ActiveProcessorCount, or rely on container support (JDK 10+ / 8u191+) which derives it from the quota. cpu.max.burst(Linux 5.14+) lets a cgroup accumulate unused quota and spend it in bursts.- Remove CPU limits for latency-sensitive services and rely on requests (weights) for fairness — a widely used practice, at the cost of less predictable noisy-neighbor behavior. Keep limits for batch jobs and untrusted tenants.
- Static CPU manager policy in Kubernetes (
cpuManagerPolicy: static) gives Guaranteed pods with integer CPU requests exclusive cores viacpuset— no sharing, no throttling quirks.
cpuset.cpus / cpuset.mems pin a cgroup to specific CPUs and NUMA nodes — the right tool for DPDK, databases on big NUMA boxes, and latency-critical trading/audio workloads (see CPU Scheduling).
Memory Limits
# Limit to 512MB RSS
echo $((512 * 1024 * 1024)) > /sys/fs/cgroup/myapp/memory.max
# Limit swap usage
echo 0 > /sys/fs/cgroup/myapp/memory.swap.max # No swap
# Current usage
cat /sys/fs/cgroup/myapp/memory.current
Important: memory.max limits all memory charged to the cgroup, not just RSS — anonymous memory, page cache for files the cgroup read or wrote, tmpfs/shm, kernel memory for its sockets, dentries and inodes. A container that streams a 10 GB file through a 512 MB limit doesn't fail because the page cache is reclaimable — but a container that writes to a tmpfs emptyDir with medium: Memory is consuming unreclaimable charged memory.
OOM killer: When a cgroup exceeds memory.max and the kernel can't reclaim enough, the OOM killer selects a process in the cgroup to kill. With Kubernetes, this results in a pod OOMKilled exit. The process selected is the one with the highest oom_score — roughly its share of memory (RSS + swap + page tables) adjusted by oom_score_adj (−1000 to +1000; Kubernetes sets it per QoS class). With memory.oom.group=1 the whole cgroup is killed together instead — Kubernetes enables this for containers on cgroup v2 nodes since 1.28, so a multi-process container is killed as a unit (kubelet's singleProcessOOMKill restores the old behavior).
memory.low (soft limit): The kernel tries to protect this cgroup's memory from reclamation under global memory pressure — a "memory reservation."
The Four Memory Knobs: min, low, high, max
flowchart LR
A["0"] --> B["memory.min: hard protection, never reclaimed"]
B --> C["memory.low: best-effort protection"]
C --> D["memory.high: throttle point, heavy reclaim and allocation stalls"]
D --> E["memory.max: hard limit, OOM kill if reclaim fails"]
| Knob | Type | What happens at the boundary |
|---|---|---|
memory.min | Hard protection | Memory below this is never reclaimed for others, even if that means OOM elsewhere |
memory.low | Soft protection | Reclaimed only if no unprotected memory is left anywhere |
memory.high | Soft limit / throttle | Above it the kernel reclaims aggressively and puts allocating tasks to sleep (up to ~2 s per allocation) — the cgroup slows down but is never OOM-killed for crossing it |
memory.max | Hard limit | Allocation that can't be satisfied after reclaim triggers the cgroup OOM killer |
memory.high vs memory.max in practice: memory.max alone gives you a cliff — the app is fine until it's suddenly dead. Setting memory.high somewhat below memory.max (say 90 %) gives you a ramp: as the app approaches its budget it slows down, the page cache gets squeezed, and pressure metrics rise, giving monitoring and autoscalers time to react before an OOM kill. Facebook's production approach (and systemd-oomd's design) is built on memory.high + PSI rather than hard limits. Kubernetes' MemoryQoS feature gate (alpha) sets memory.high from limits and memory.min from requests; without it, memory requests are not enforced by cgroups at all — they are a scheduling and eviction concept only.
$ cat memory.events
low 0
high 18234 # times we crossed memory.high and got throttled
max 412 # times we hit memory.max
oom 3 # OOM conditions
oom_kill 3 # processes killed
oom_group_kill 1
$ grep -E '^(anon|file|kernel|sock|shmem|file_dirty|file_writeback) ' memory.stat
anon 402653184 # heap, stacks: the "real" usage
file 134217728 # page cache: reclaimable (mostly)
kernel 20971520
sock 1048576
shmem 0 # tmpfs and /dev/shm: NOT reclaimable without swap
file_dirty 4194304
file_writeback 0
Kubernetes' container_memory_working_set_bytes — what kubectl top and the eviction manager use — is memory.current − inactive_file. It includes active page cache, so a service that reads a lot of files can appear to "use" far more memory than its heap.
I/O Limits
# Limit I/O bandwidth (device 8:0 = /dev/sda)
echo "8:0 rbps=10485760 wbps=10485760" > /sys/fs/cgroup/myapp/io.max
# ↑ 10MB/s read ↑ 10MB/s write
# View I/O statistics
cat /sys/fs/cgroup/myapp/io.stat
io.max also accepts riops= and wiops=. Find device numbers with lsblk -o NAME,MAJ:MIN (NVMe is typically 259:N). Other I/O knobs:
| File | Model | Notes |
|---|---|---|
io.max | Absolute bandwidth/IOPS caps | Works for buffered writes in v2 (writeback attributed to the dirtying cgroup) |
io.weight | Proportional share (1–10000, default 100) | Needs the BFQ scheduler or io.cost to have effect |
io.latency | Target latency per device; protects a cgroup by throttling siblings | Good for "protect the database from the log shipper" |
io.cost.qos / io.cost.model | Cost-model-based proportional control (root only) | The most accurate work-conserving controller; used at Meta |
io.pressure | PSI for I/O | See below |
$ cat io.stat
259:0 rbytes=1843200000 wbytes=7340032000 rios=45000 wios=180000 dbytes=0 dios=0
Kubernetes does not expose I/O limits on pods; I/O isolation between pods on a node is one of the weakest parts of container isolation. Put noisy-I/O workloads on separate disks or nodes. See I/O & Storage for schedulers and devices.
Process Limits
# Limit number of processes (prevents fork bombs)
echo 100 > /sys/fs/cgroup/myapp/pids.max
pids.max counts tasks — threads included, because each Linux thread consumes a PID. A Java service with 400 threads needs pids.max above 400. pids.current and pids.events (max N — how many forks were refused) tell you whether you're hitting it. In Kubernetes, set podPidsLimit in the kubelet config; the node-wide kernel.pid_max (kernel default 32768; systemd 243+ raises it to 4,194,304 on 64-bit) is shared by all pods.
PSI: Pressure Stall Information
CPU utilization and "memory used" answer the wrong question. 100 % CPU utilization is fine if nothing is waiting; 60 % memory used is a crisis if the app is thrashing. PSI (Linux 4.20+) answers the right question: what fraction of time were tasks stalled waiting for a resource?
$ cat /proc/pressure/memory
some avg10=12.45 avg60=8.02 avg300=3.11 total=482931245
full avg10=4.20 avg60=2.10 avg300=0.80 total=120384002
$ cat /sys/fs/cgroup/kubepods.slice/.../cpu.pressure
some avg10=31.60 avg60=28.90 avg300=25.12 total=9812374123
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
some— percentage of wall time in which at least one task was stalled on the resource. Latency is being added.full— percentage of time in which all non-idle tasks were stalled simultaneously. Throughput is being lost; nothing productive happened.avg10/60/300are running averages over 10 s, 60 s, 5 min;totalis cumulative microseconds (use it for exact rates).- Files:
cpu,memory,io, andirq(6.1+, withCONFIG_IRQ_TIME_ACCOUNTING), system-wide in/proc/pressure/and per cgroup.
Interpreting them:
| Signal | Meaning | Typical action |
|---|---|---|
cpu.pressure some high in a container | Runnable threads waiting for CPU — contention or CFS throttling | Check cpu.stat nr_throttled; raise limit or reduce threads |
memory.pressure some rising | Reclaim and refaults are adding latency | Raise memory, find the leak, or shed load |
memory.pressure full > a few % | The cgroup is thrashing — near-livelock | Kill or restart it proactively (what systemd-oomd / oomd do) |
io.pressure full high | Everything waits on disk | See I/O & Storage |
PSI supports triggers: write some 150000 1000000 (150 ms of stall within any 1 s window) to a pressure file, then poll() it for POLLPRI — you get a kernel notification instead of polling metrics. Android's low-memory killer daemon, systemd-oomd and Meta's oomd all use this to kill or shed load before the kernel OOM killer would.
How Docker Uses Namespaces + cgroups
When you run docker run --memory=512m --cpus=0.5 nginx:
- Docker creates a new set of namespaces (PID, net, mount, UTS, IPC, cgroup; user optional)
- Creates a cgroup —
/sys/fs/cgroup/system.slice/docker-<container-id>.scope/with the systemd cgroup driver, or/sys/fs/cgroup/docker/<container-id>/with the cgroupfs driver - Writes
memory.max = 536870912(512MB) - Writes
cpu.max = 50000 100000(0.5 CPU) - Sets up OverlayFS mount for the container's rootfs
- Forks the container process into all the namespaces
- Writes the container's PID to
cgroup.procs - Sets up veth pair for networking
The nginx process runs on the host kernel, but sees its own PID 1, its own network interfaces, its own hostname, and cannot use more than 512MB or 0.5 CPU.
In reality "Docker" is several programs, and only the last one touches the kernel:
flowchart LR
CLI["docker CLI"] -->|"REST over unix socket"| D["dockerd"]
D -->|"gRPC"| CD["containerd"]
KL["kubelet"] -->|"CRI gRPC"| CD
CD --> SHIM["containerd-shim-runc-v2 (one per container, subreaper)"]
SHIM -->|"exec with config.json"| RUNC["runc (or crun, gVisor runsc, Kata)"]
RUNC -->|"clone3, setns, pivot_root, write cgroup files, seccomp, exec"| P["container process"]
RUNC -.->|"exits after start"| X["gone"]
runc reads an OCI runtime spec (config.json) that says exactly which namespaces, mounts, cgroup values, capabilities and seccomp rules to apply, sets them up, execs your entrypoint and exits. The shim stays as the container's parent to hold stdio, reap it and report its exit code. A trimmed spec:
{
"process": { "args": ["nginx", "-g", "daemon off;"], "user": { "uid": 0, "gid": 0 },
"capabilities": { "bounding": ["CAP_CHOWN", "CAP_NET_BIND_SERVICE", "CAP_SETUID", "CAP_SETGID"] } },
"root": { "path": "rootfs", "readonly": false },
"linux": {
"namespaces": [ {"type": "pid"}, {"type": "network"}, {"type": "mount"},
{"type": "uts"}, {"type": "ipc"}, {"type": "cgroup"} ],
"resources": {
"memory": { "limit": 536870912 },
"cpu": { "quota": 50000, "period": 100000, "shares": 512 },
"pids": { "limit": 4096 }
},
"maskedPaths": ["/proc/kcore", "/proc/keys", "/sys/firmware"],
"readonlyPaths": ["/proc/sys", "/proc/sysrq-trigger"],
"seccomp": { "defaultAction": "SCMP_ACT_ERRNO", "syscalls": [ "..." ] }
}
}
How Kubernetes Maps Requests and Limits to cgroups
resources:
requests: { cpu: "500m", memory: "512Mi" }
limits: { cpu: "2", memory: "1Gi" }
| Spec | cgroup v2 file | Enforced by | Effect |
|---|---|---|---|
requests.cpu | cpu.weight (converted from legacy shares: millicores × 1024 / 1000) | CFS scheduler | Share of CPU under contention; also the scheduler's bin-packing input |
limits.cpu | cpu.max = limit × 100000 100000 | CFS bandwidth control | Hard throttling ceiling |
requests.memory | (none by default); memory.min with MemoryQoS | kube-scheduler + kubelet eviction | Placement and eviction order only |
limits.memory | memory.max | Kernel memory controller | OOM kill at the limit |
| (none) | memory.high with MemoryQoS = a fraction of the limit | Kernel | Throttling before OOM |
podPidsLimit (kubelet) | pids.max | Kernel | Fork/thread creation fails |
| QoS class | oom_score_adj and cgroup parent slice | Kernel OOM killer | Who dies first under node pressure |
The classic shares → weight conversion is weight = 1 + ((shares − 2) × 9999) / 262142, which maps a 1-CPU request (1024 shares) to a weight of only 39, below the default 100 of system services — a long-standing oddity. Newer OCI runtime releases switched to a non-linear mapping that sends 1024 shares to weight 100; check your runtime version if relative priority between pods and system daemons matters.
QoS Classes and the Node's cgroup Tree
flowchart TB
R["/sys/fs/cgroup"] --> SYS["system.slice (kubelet, containerd, sshd)"]
R --> KP["kubepods.slice (node allocatable)"]
KP --> G["pod-uidA.slice: Guaranteed, requests equal limits"]
KP --> BU["kubepods-burstable.slice"]
KP --> BE["kubepods-besteffort.slice"]
BU --> PB["pod-uidB.slice"]
PB --> CB1["cri-containerd-app.scope"]
PB --> CB2["cri-containerd-sidecar.scope"]
BE --> PE["pod-uidC.slice"]
| QoS class | Rule | oom_score_adj | Under node memory pressure |
|---|---|---|---|
| Guaranteed | Every container has requests = limits for CPU and memory | −997 | Killed last |
| Burstable | At least one request or limit set, not Guaranteed | 2–999 (lower for bigger requests relative to node memory) | Middle |
| BestEffort | No requests or limits at all | 1000 | Killed first, evicted first |
Two different things can kill a pod for memory, and they look different:
| Container OOM kill | Node-pressure eviction | |
|---|---|---|
| Who acts | The kernel, inside the container's cgroup | The kubelet, based on node signals (memory.available < 100Mi default hard threshold) |
| Trigger | Container exceeded its own limits.memory | The node as a whole is running low |
| Signal | SIGKILL immediately | Graceful (respects grace period for soft thresholds; immediate for hard) |
| Shows as | lastState.terminated.reason: OOMKilled, exit 137, container restarts in place | Pod status.reason: Evicted, pod is deleted and rescheduled |
| Fix | Raise limit or fix the leak | Set accurate requests; reserve resources for the system (kube-reserved, system-reserved) |
More practical guidance: Resource Management and Pod Lifecycle.
Common Failures
OOMKilled
$ kubectl get pod api-7d9 -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'
{"exitCode":137,"reason":"OOMKilled","startedAt":"...","finishedAt":"..."}
$ dmesg -T | grep -iE 'oom|killed process'
[Sat Nov 8 03:12:44 2026] memory: usage 1048576kB, limit 1048576kB, failcnt 412
[Sat Nov 8 03:12:44 2026] oom-kill:constraint=CONSTRAINT_MEMCG,...,oom_memcg=/kubepods.slice/kubepods-burstable.slice/...
[Sat Nov 8 03:12:44 2026] Memory cgroup out of memory: Killed process 81234 (java) total-vm:4123456kB, anon-rss:1012345kB
CONSTRAINT_MEMCG means the container hit its own limit; CONSTRAINT_NONE means the whole node ran out. Common root causes:
- Runtime not container-aware or misconfigured. JVM heap set with
-Xmxequal to the container limit leaves nothing for metaspace, thread stacks (1 MB each), direct buffers, JIT code cache and GC structures. Use-XX:MaxRAMPercentage=70–75. Node.js:--max-old-space-sizebelow the limit. Go:GOMEMLIMITat ~90 % of the limit makes the GC work harder before the kernel kills you. - tmpfs /
emptyDir.medium: Memory//dev/shm— charged to the container and not reclaimable. - Page cache +
memory.maxwith nomemory.high— usually reclaimable, but dirty pages under heavy writes must be written back before reclaim, which can be too slow. See Page Cache & fsync. - Many processes in one container (a shell script forking workers), each counted.
- A genuine leak — see the Memory Leak playbook.
CPU Throttling
Symptoms: p99 latency spikes, timeouts and failing liveness probes while kubectl top shows the pod using less than its CPU limit. Confirm with the throttle ratio from cpu.stat (above). Fix by aligning thread counts to the limit, raising or removing CPU limits, or moving to exclusive cores.
War Story (illustrative)
A Go API with limits.cpu: 1 on 64-core nodes showed 200–400 ms p99 latency with average CPU at 40 % of the limit. Before Go 1.25, GOMAXPROCS defaulted to the host's 64 cores, so during each GC cycle dozens of threads ran in parallel and exhausted the 100 ms quota in a few milliseconds; the container was then throttled for the rest of the period. nr_throttled / nr_periods was 38 %. Setting GOMAXPROCS=1 (via automaxprocs) cut p99 to 20 ms with lower total CPU use.
PID Exhaustion
$ kubectl logs worker-5f8
fork: Resource temporarily unavailable # EAGAIN from clone()
runtime: failed to create new OS thread (have 4095 already; errno=11)
$ cat /sys/fs/cgroup/.../pids.current /sys/fs/cgroup/.../pids.max
4096
4096
$ cat /sys/fs/cgroup/.../pids.events
max 1842
Causes: leaking threads (a goroutine-per-blocking-syscall pattern, unbounded thread pools), zombie accumulation because PID 1 doesn't reap (see Signals), or a node-wide pid_max exhaustion where one pod's fork bomb takes down every pod on the node. Set podPidsLimit (a few thousand is typical) so one pod can't starve the node.
Other Container-Isolation Surprises
| Surprise | Why | Workaround |
|---|---|---|
free, top, /proc/meminfo, nproc show host values in a container | /proc isn't fully namespaced | Read /sys/fs/cgroup/memory.max and cpu.max; lxcfs virtualizes /proc files |
inotify watches exhausted on a node | fs.inotify.max_user_instances/max_user_watches are per host UID, shared across containers running as the same UID | Raise sysctls on the node |
Conntrack table full (nf_conntrack: table full, dropping packet) | Conntrack is effectively node-global | Raise nf_conntrack_max; reduce short-lived connections |
Kernel keyring, dmesg, kernel modules, sysctls under kernel.* | Not namespaced | Restrict with seccomp / kernel.dmesg_restrict |
| Clock changes affect all containers | CLOCK_REALTIME isn't namespaced | Never give containers CAP_SYS_TIME |
| Kernel version/features are the host's | No guest kernel | Use VMs or microVMs for kernel isolation |
Security Implications
Namespaces provide isolation, NOT security. A process running as root inside a container (with no user namespace) is root from the kernel's perspective — with default capabilities it cannot load kernel modules (Docker drops CAP_SYS_MODULE), but a privileged container can, and any kernel exploit reached through the shared syscall surface escapes all namespaces at once.
Hardening:
- Seccomp: filters the syscalls a container can make (Docker's default seccomp profile blocks ~44 dangerous syscalls)
- AppArmor/SELinux: MAC (mandatory access control) policies that restrict file access and capabilities
- Capabilities: break root's omnipotence into ~40 fine-grained capabilities. Drop unneeded ones (e.g.,
CAP_NET_ADMIN). - User namespaces + rootless containers: container's root maps to an unprivileged UID on the host
Privilege escalation attacks: runc CVE-2019-5736 allowed a container to overwrite the host's runc binary via /proc/self/exe. Fixed by having runc re-exec itself from a sealed in-memory copy (memfd) before entering the container. Container escapes require kernel vulns or misconfigurations — not as common as VM escapes but not impossible. CVE-2024-21626 ("Leaky Vessels") was another runc escape, via a leaked host fd used as the working directory.
The most common real-world escapes are not kernel 0-days but configuration: privileged: true, a mounted /var/run/docker.sock, hostPath: /, hostPID: true, or CAP_SYS_ADMIN. Each one hands the container a direct path to host root. Details in Process Security, Container Security and Container Escapes.
| Isolation option | Boundary | Startup | Overhead | Use when |
|---|---|---|---|---|
| Plain container (runc) | Shared kernel, namespaces + seccomp | ~100 ms | ~0 | Trusted, first-party workloads |
| Rootless / user-namespaced container | + UID remapping | ~100 ms | ~0 | Multi-tenant CI, defence in depth |
gVisor (runsc) | User-space kernel intercepts syscalls | ~150 ms | 10–30 %+ on syscall-heavy work | Untrusted code, moderate I/O |
| Kata Containers / Firecracker microVM | Hardware virtualization, own kernel | ~125 ms–1 s | Small CPU, some memory | Untrusted multi-tenant (Lambda, Fargate) |
| Full VM | Own kernel and virtual hardware | Seconds+ | Highest | Different OS/kernel, strongest isolation |
Observability
# List namespaces of a process
ls -la /proc/<pid>/ns/
# Check which cgroup a process is in
cat /proc/<pid>/cgroup
# Inspect a container's cgroup limits (Docker)
docker inspect <container> | jq '.[0].HostConfig.Memory'
# Enter a container's namespace (for debugging)
nsenter --target <pid> --pid --net --mount bash
# Kubernetes: cgroup v2 paths
cat /sys/fs/cgroup/kubepods/burstable/pod<id>/<container-id>/memory.current
With the systemd cgroup driver (the default for kubeadm and most managed clusters), that path is spelled with slices:
cat /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod<uid>.slice/cri-containerd-<id>.scope/memory.current
# Easier: find it from the container's PID
PID=$(crictl inspect <container-id> | jq .info.pid)
CG=/sys/fs/cgroup$(cut -d: -f3 /proc/$PID/cgroup)
cat $CG/memory.max $CG/memory.current $CG/cpu.max
cat $CG/cpu.stat $CG/memory.events $CG/pids.current
cat $CG/cpu.pressure $CG/memory.pressure $CG/io.pressure
# Live per-cgroup resource usage
systemd-cgtop -m # sort by memory
docker stats --no-stream
kubectl top pod --containers
# Debug a distroless pod with an ephemeral container sharing its namespaces
kubectl debug -it pod/api-7d9 --image=busybox --target=api
Key Prometheus metrics (cAdvisor, built into the kubelet):
| Metric | Use |
|---|---|
container_memory_working_set_bytes | What eviction and kubectl top see; compare to the limit |
container_oom_events_total | OOM kills per container |
container_cpu_cfs_throttled_periods_total / container_cpu_cfs_periods_total | Throttle ratio |
container_pressure_cpu_waiting_seconds_total (and memory/io) | PSI per container (newer cAdvisor) |
kube_pod_container_status_last_terminated_reason | Counts of OOMKilled (kube-state-metrics) |
Real Numbers
| Measurement | Typical value |
|---|---|
| Create a set of namespaces + exec (runc) | ~50–150 ms for docker run end to end; unshare alone < 1 ms |
| Create a new network namespace | ~1–5 ms (more with many interfaces/rules) |
| veth + bridge overhead vs host networking | ~2–10 µs added latency, a few % throughput at high packet rates |
| CFS default period / slice | 100 ms / 5 ms |
Default cpu.weight / legacy cpu.shares | 100 / 1024 |
Docker default /dev/shm | 64 MB |
| Docker default seccomp profile | ~44 syscalls blocked out of ~350+ |
| Linux capabilities | 41 (as of 6.x) |
| Kubelet default hard eviction | memory.available < 100Mi, nodefs.available < 10%, imagefs.available < 15% |
memory.high throttling penalty | Up to ~2 s sleep per over-limit allocation |
| PSI averaging windows | 10 s, 60 s, 300 s |
| Max user namespace nesting | 32 levels |
Decision Guide
- Setting CPU limits? For latency-sensitive services, prefer accurate requests and either no limit or a generous one, and make sure the runtime's thread count matches what you allow. Use limits for batch, untrusted or multi-tenant workloads where predictability beats latency.
- Setting memory? Always set a memory limit (memory is not compressible: overcommitting it ends in OOM). Set requests close to real working-set usage. For Guaranteed QoS, set requests = limits.
- Need graceful degradation before OOM? Use
memory.high(systemdMemoryHigh=, or KubernetesMemoryQoSwhere enabled) and alert onmemory.pressure. - Need stronger isolation than namespaces? Rootless/user namespaces first; gVisor or Kata/Firecracker for untrusted code.
- Debugging a container without tools?
nsenter -t <pid> -nfor network;kubectl debug --targetfor everything else. - Monitoring? Alert on throttle ratio, OOM events,
pids.currentnearpids.max, and PSI — not just utilization.
Advanced Topics
- cgroup BPF programs attach to a cgroup and filter/redirect its sockets, devices and sysctls — how Cilium enforces policy and how cgroup v2 implements the device controller (there is no
devices.allowfile in v2; it's a BPF program). See eBPF. - Delegation. A cgroup subtree can be
chowned to a user (Delegate=yesin systemd), letting rootless Podman or a nested container runtime manage its own children safely. - Idmapped mounts (5.12+) let a filesystem be mounted with a UID shift, so user-namespaced containers can share volumes without
chown -R. - CRIU (Checkpoint/Restore In Userspace) uses namespaces, the time namespace and the freezer to snapshot and restore a running container; Kubernetes has alpha "forensic container checkpointing".
cgroup.kill(5.14+) kills every process in a cgroup atomically — no race with forks, unlike looping overcgroup.procs.- Sysbox / nested containers run Docker or Kubernetes inside containers using user namespaces and syscall interception, without
privileged: true.
Interview Questions
"What is a container at the OS level?" A process (or process group) running on the host kernel, isolated via Linux namespaces (PID, network, mount, UTS, IPC) and constrained via cgroups. No separate OS, no hypervisor. The container shares the host kernel. Start time is milliseconds vs seconds for VMs because there's no OS to boot.
"Why can a container have PID 1 if there's already a PID 1 on the host?" PID namespaces create a separate PID number space. Inside the container's PID namespace, PIDs start at 1. From the host, the same process has a different PID (e.g., 5432). The kernel maps between them; they don't conflict.
"What happens when a container exceeds its memory limit?"
The kernel's OOM killer fires within the cgroup. It selects the process with the highest badness score (roughly: most memory used, adjusted by oom_score_adj) and sends it SIGKILL — or kills the whole cgroup if memory.oom.group is set. In Kubernetes, this manifests as a pod with OOMKilled exit reason. The fix is either to increase the memory limit or to reduce the application's memory usage.
"What's the difference between CPU requests and limits in Kubernetes?"
CPU request maps to cpu.weight — it's a scheduling hint that guarantees relative CPU priority when the node is saturated. CPU limit maps to cpu.max (CFS bandwidth throttling) — it's a hard cap. A pod will be throttled (not killed) if it exceeds its CPU limit. A pod WILL be killed if it exceeds its memory limit (OOM).
"Why does my pod show high latency while using only 40% of its CPU limit?"
CFS quota is CPU-time per 100 ms period shared across all threads. Many threads can burn the whole quota in a fraction of the period and then the container is throttled for the rest. Check nr_throttled / nr_periods in cpu.stat; align thread counts (GOMAXPROCS, JVM processors) with the limit or relax the limit.
"memory.high vs memory.max?"
memory.high is a throttle: crossing it triggers aggressive reclaim and allocation stalls but never an OOM kill. memory.max is a wall: if reclaim can't keep usage under it, the OOM killer runs. Use high below max to get a warning ramp instead of a cliff.
"What's the difference between cgroups v1 and v2?"
v1 had a separate hierarchy per controller and inconsistent semantics; v2 has one unified tree, consistent interfaces, correct attribution of buffered writeback I/O, PSI, memory.high/min, OOM groups and safe delegation.
"How would you build a container without Docker?"
Get a root filesystem, create a cgroup and write limits, unshare (or clone) into new PID/mount/UTS/IPC/net/cgroup namespaces, make mounts private, pivot_root into the rootfs, mount /proc, then exec the entrypoint. Add a user namespace, capability dropping and seccomp for security.
Related
- Processes & Threads —
fork,clone, process trees - Signals — PID 1 signal semantics, graceful shutdown in Kubernetes
- CPU Scheduling — CFS/EEVDF, the scheduler behind
cpu.weight - Virtual Memory — reclaim, page cache and the OOM killer
- File Systems — overlayfs and container image layers
- Process Security — capabilities, seccomp, LSMs
- Docker Internals and Containers From Scratch
- Kubernetes Resource Management