Process Isolation & Security: Users, Capabilities & seccomp
Every time a process opens a file, binds a socket, sends a signal or loads a kernel module, the kernel asks one question: is this process allowed to do this? The answer comes from several independent mechanisms stacked on top of each other — user and group IDs, file permission bits, capabilities, seccomp filters, Linux Security Modules (LSMs) and namespaces. Each one was added at a different point in Linux's history to close a gap the previous one left open.
This page is a defensive guide. It explains how each layer works, what class of risk it reduces, and how to configure it for the things engineers actually run: systemd services, Docker/containerd containers and Kubernetes pods. It deliberately does not cover how to break these mechanisms — for the attacker's view of containers, and why each misconfiguration matters, see Container Escape Techniques.
The goal is simple: when your service has a bug — and it will — the bug should give an attacker as little as possible.
The Mental Model: Many Gates, One Request
A single open("/etc/shadow", O_RDONLY) passes through a chain of checks. Any one of them can say no, and the request fails.
flowchart TD
A["Process calls open('/etc/shadow')"] --> B{"seccomp filter: is openat allowed?"}
B -- "no" --> X1["EPERM or process killed"]
B -- "yes" --> C{"Namespaces: is the path visible in this mount namespace?"}
C -- "no" --> X2["ENOENT"]
C -- "yes" --> D{"DAC: owner/group/other mode bits and ACLs vs fsuid/fsgid"}
D -- "denied" --> E{"Capability override? CAP_DAC_OVERRIDE / CAP_DAC_READ_SEARCH"}
E -- "no" --> X3["EACCES"]
E -- "yes" --> F
D -- "allowed" --> F{"LSM hook: SELinux / AppArmor / Landlock"}
F -- "denied" --> X4["EACCES + audit log entry"]
F -- "allowed" --> G["File descriptor returned"]
Three things to notice:
- The checks are independent. Passing the Unix permission check does not bypass SELinux; being root does not bypass seccomp. This is what makes defense in depth work.
- Classic Unix permissions (DAC — Discretionary Access Control) are owner-controlled. The owner of a file decides who may read it. LSMs add MAC (Mandatory Access Control) — a policy written by the administrator that even the file's owner, and even root, cannot override.
- "Root" is not one thing. Since Linux 2.2, root's power is a bundle of capabilities, and every one of them can be removed independently.
The Permission Model: UIDs, GIDs and File Modes
Users and groups are just numbers
The kernel knows nothing about usernames. /etc/passwd maps the name www-data to the number 33; the kernel only ever sees 33. A process carries several IDs in its credentials (struct cred):
| ID | Meaning | Used for |
|---|---|---|
Real UID/GID (ruid) | Who started the process | Accounting, who may send it signals |
Effective UID/GID (euid) | Whose privileges the process is using right now | Most permission checks |
Saved set-UID/GID (suid) | A stashed copy of a previous effective ID | Lets a program drop privilege temporarily and regain it later |
Filesystem UID/GID (fsuid) | Linux-specific; tracks euid almost always | File access checks |
| Supplementary groups | Extra group memberships | Group permission checks |
You can see them all in /proc:
grep -E '^(Uid|Gid|Groups)' /proc/self/status
# Uid: 1000 1000 1000 1000 <- real, effective, saved, fs
# Gid: 1000 1000 1000 1000
# Groups: 4 27 1000
Normally all four UIDs are equal. They diverge in two situations: when a setuid binary runs (the effective UID becomes the file owner's, the real UID stays yours), and when a privileged daemon drops privileges after startup.
Dropping privileges correctly
The classic daemon pattern is: start as root, do the one thing that needs root (bind port 443, open a log file in /var/log), then switch to an unprivileged user for the rest of its life.
// Order matters: groups first, then GID, then UID.
// Once the UID is no longer 0 you lose the right to change groups.
if (setgroups(0, NULL) != 0) die("setgroups");
if (setresgid(gid, gid, gid) != 0) die("setresgid");
if (setresuid(uid, uid, uid) != 0) die("setresuid");
// Verify: trying to get root back must fail.
if (setuid(0) == 0) die("privilege drop did not stick");
setresuid() sets real, effective and saved IDs at once. A common mistake is calling only seteuid(), which leaves the saved UID as 0 — so any later code path (or an attacker controlling the process) can switch back to root. Forgetting setgroups() is another: the process keeps root's supplementary groups.
TIP
In practice, let the supervisor drop privileges for you. User= in a systemd unit, USER in a Dockerfile and runAsUser in Kubernetes start the process as the unprivileged user from the very first instruction — there is no window where your code runs as root.
File modes: the nine bits (plus three)
Every inode has an owner UID, a group GID and a mode. The kernel's check is short and strictly ordered:
- If the process's
fsuidequals the file owner → use the owner bits. - Else, if the file's group is the process's
fsgidor one of its supplementary groups → use the group bits. - Else → use the other bits.
It picks one class and stops. A file with mode 0604 owned by you denies you write but grants everyone else read — the owner class matched first. (See File Systems for how inodes store this.)
ls -l /usr/bin/passwd /tmp /etc/shadow
# -rwsr-xr-x 1 root root ... /usr/bin/passwd <- 's' = setuid
# drwxrwxrwt 9 root root ... /tmp <- 't' = sticky bit
# -rw-r----- 1 root shadow ... /etc/shadow
| Bit | Octal | On a file | On a directory |
|---|---|---|---|
| setuid | 4000 | Run with the file owner's UID | (ignored on Linux) |
| setgid | 2000 | Run with the file's GID | New files inherit the directory's group |
| sticky | 1000 | (ignored) | Only a file's owner can delete or rename it — why /tmp is safe to share |
On directories, r means "list names", w means "create/delete entries", and x means "traverse" (look up a name inside). A directory with x but not r lets you open a file whose name you already know, but not discover names.
ACLs: when nine bits are not enough
POSIX ACLs add extra entries for specific users and groups without changing ownership — useful when a log directory must be readable by both the app user and a log shipper:
setfacl -m u:fluentbit:rx /var/log/myapp
setfacl -d -m u:fluentbit:r /var/log/myapp # default ACL: applies to new files
getfacl /var/log/myapp
# user::rwx
# user:fluentbit:r-x
# group::r-x
# mask::r-x <- the mask caps every named entry and the group entry
# other::---
A + at the end of ls -l output (drwxr-x---+) tells you an ACL is present. The mask entry is the most common surprise: a chmod g-r on a file with an ACL lowers the mask, silently removing access you granted to named users.
Why "run as root" is the default risk
UID 0 is special in two ways. Traditionally the kernel skipped permission checks entirely for it, and today root holds every capability by default. A process running as root that is compromised — through a deserialization bug, a path traversal, a vulnerable dependency — can typically:
- read and modify any file, including credentials, keys and other tenants' data;
- load kernel modules, change firewall rules and network routes;
- attach to and read the memory of any other process;
- in a container, use any gap in isolation to reach the host.
Most software does not need any of this. The rest of this page is about taking those powers away, starting with the cheapest and most effective step: do not run as root. Inside containers this matters just as much — without user namespaces, UID 0 in a container is UID 0 on the host kernel, constrained only by the other layers described below.
setuid and setgid Binaries
What they are for
A setuid binary runs with the privileges of the file's owner rather than the user who launched it. This is how an ordinary user can change their own password: passwd must write /etc/shadow, which only root can modify, so /usr/bin/passwd is owned by root with the setuid bit set. When you run it, its effective UID is 0, while its real UID still says who you are — so the program can check that you are only changing your password.
Historically, ping was setuid root because sending ICMP requires a raw socket. Modern distributions have replaced that with either a file capability (cap_net_raw) or unprivileged ICMP sockets (net.ipv4.ping_group_range), so ping no longer needs to be setuid at all. That migration is a good illustration of the general trend: setuid is being replaced by narrower mechanisms.
Why every setuid binary is attack surface
A setuid-root program is a door from "any local user" to "root", guarded only by that program's own code. It runs with full root privileges, but its inputs come from an untrusted user: command-line arguments, environment variables, file descriptors it inherits, the current directory, resource limits and signal dispositions. A bug in any input handling — or in a library it links — can let a local user act as root.
The kernel and the dynamic loader add protections (they ignore LD_PRELOAD and similar variables in "secure-execution" mode, for instance), but the history of local privilege escalation is dominated by setuid programs. The defensive conclusion is the same every time: have as few as possible, and none in places attackers can reach.
Finding and minimising them
# Every setuid file on the root filesystem (-xdev: don't cross into other mounts)
find / -xdev -perm -4000 -type f -exec ls -l {} + 2>/dev/null
# setgid files too
find / -xdev -perm -2000 -type f -exec ls -l {} + 2>/dev/null
# Files with capabilities attached (the modern alternative — audit these as well)
getcap -r / 2>/dev/null
A typical server has 10–25 setuid binaries: passwd, chsh, chfn, newgrp, gpasswd, su, sudo, mount, umount, pkexec, fusermount3 and a few helpers. Questions to ask about each:
| Question | Action if "no" |
|---|---|
| Do humans log into this machine? | Remove chsh, chfn, newgrp setuid bits; consider removing su |
| Do non-root users need to mount things? | Remove setuid from mount/umount/fusermount |
| Is polkit actually used? | Remove the package that ships pkexec |
| Is this a container image? | It should have zero setuid binaries — strip them at build time |
For container images, stripping is one line in the Dockerfile (distroless and scratch-based images start with none):
# Remove setuid/setgid bits from everything in the image
RUN find / -xdev -perm /6000 -type f -exec chmod a-s {} + || true
Track the list over time. A new setuid binary appearing on a server you did not change is worth an alert — file-integrity tools (AIDE, osquery, Falco) can watch for it.
The nosuid mount option
nosuid tells the kernel to ignore the setuid and setgid bits (and file capabilities) for every file on that mount. Anything a user or a service can write to should be mounted nosuid, so that a writable location can never be used to plant a privileged binary:
# /etc/fstab
tmpfs /tmp tmpfs defaults,nosuid,nodev,noexec,size=1g 0 0
/dev/vdb1 /srv/data ext4 defaults,nosuid,nodev 0 2
| Option | Effect |
|---|---|
nosuid | setuid/setgid bits and file capabilities have no effect |
nodev | Device files on this mount cannot be opened as devices |
noexec | Files cannot be executed directly (defense against drop-and-run; not a hard barrier, since interpreters can still read scripts) |
ro | Read-only — the strongest of all |
Container runtimes mount /proc, /sys, /dev/shm and tmpfs volumes with these flags by default, and systemd's ProtectSystem=, PrivateTmp= and NoNewPrivileges= achieve the same for services.
Linux Capabilities: Splitting Root
The idea
Before capabilities, the kernel asked "is euid == 0?" and either allowed everything or nothing. Capabilities split that single bit into about 40 separate privileges (41 on current kernels), each guarding a specific family of operations. The kernel now asks "does this process hold CAP_NET_BIND_SERVICE?" rather than "is it root?". A web server that needs to bind port 80 can hold exactly that one capability and nothing else.
capsh --print | head -3 # what the current shell holds
man 7 capabilities # the authoritative list
The ones you will actually meet
| Capability | Grants | Risk if granted unnecessarily |
|---|---|---|
CAP_NET_BIND_SERVICE | Bind TCP/UDP ports below 1024 | Low — the classic safe capability |
CAP_NET_RAW | Raw and packet sockets (ping, tcpdump) | Medium — can craft and sniff packets on the network namespace |
CAP_NET_ADMIN | Configure interfaces, routes, firewall rules, tc | High — can reroute or intercept traffic in its network namespace |
CAP_CHOWN, CAP_FOWNER, CAP_DAC_OVERRIDE | Change ownership; bypass file permission checks | High — defeats the DAC layer entirely |
CAP_SETUID, CAP_SETGID | Change to any UID/GID | High — become any user, including root |
CAP_KILL | Signal processes owned by other users | Medium |
CAP_SYS_PTRACE | Trace and read/write memory of other processes | High — read secrets from any process it can see |
CAP_SYS_MODULE | Load kernel modules | Critical — arbitrary kernel code |
CAP_SYS_ADMIN | Mount filesystems, namespaces, BPF (partly), quotas, dozens more | Critical — "the new root" |
CAP_BPF, CAP_PERFMON | Load BPF programs; use perf events (split out of CAP_SYS_ADMIN in 5.8) | High — kernel attack surface |
CAP_SYS_ADMIN deserves its own warning. When kernel developers added a privileged operation and could not decide which capability should guard it, they often chose CAP_SYS_ADMIN. The result is a capability covering hundreds of call sites — mounting filesystems, creating most namespace types, configuring many devices — so broad that holding it is close to holding root. Treat any request for CAP_SYS_ADMIN as a request for root and look for a narrower alternative first (CAP_BPF, CAP_PERFMON, CAP_CHECKPOINT_RESTORE and others were carved out precisely so tools could stop asking for it).
WARNING
--privileged in Docker and privileged: true in Kubernetes grant every capability, disable seccomp and the LSM profile, and expose host devices. It is not "a few more permissions" — it removes almost every layer this page describes. See Container Escape Techniques for why.
Process capability sets
Each thread carries five capability sets. You rarely need all of them, but you will see them in /proc/<pid>/status and in error messages:
| Set | Meaning |
|---|---|
Permitted (CapPrm) | The ceiling: capabilities the thread may make effective |
Effective (CapEff) | What the kernel actually checks right now |
Inheritable (CapInh) | Can pass across execve() — but only to programs whose file also allows them |
Bounding (CapBnd) | An absolute limit: nothing outside this set can ever be gained, not even via setuid-root |
Ambient (CapAmb) | Passed across execve() to ordinary (non-capability-aware) programs |
grep Cap /proc/$(pgrep -o nginx)/status
# CapInh: 0000000000000000
# CapPrm: 0000000000000400
# CapEff: 0000000000000400
# CapBnd: 0000000000000400
# CapAmb: 0000000000000000
capsh --decode=0000000000000400
# 0x0000000000000400=cap_net_bind_service
The bounding set is the important one for hardening: drop a capability from it and that process and all its descendants can never acquire it again — no setuid binary, no file capability, nothing. This is what container runtimes and systemd's CapabilityBoundingSet= manipulate.
File capabilities: the targeted setuid replacement
Capabilities can also be attached to an executable (stored in the security.capability extended attribute). When the file is executed, the process gains just those capabilities — a far narrower grant than setuid root.
# Give one binary permission to bind low ports, nothing else
sudo setcap 'cap_net_bind_service=+ep' /usr/local/bin/myserver
getcap /usr/local/bin/myserver
# /usr/local/bin/myserver cap_net_bind_service=ep
# Remove it
sudo setcap -r /usr/local/bin/myserver
e = effective, p = permitted, i = inheritable. Caveats: file capabilities are lost when the file is rewritten (so your deployment must re-apply them), they are ignored on nosuid mounts, and some copy/archive tools drop extended attributes. Like setuid binaries, every file with capabilities is something to inventory with getcap -r /.
Giving a web server port 80 without root
There are four common answers, from most to least preferred:
| Approach | How | Notes |
|---|---|---|
| Don't bind port 80 in the app | App listens on 8080; a load balancer, Kubernetes Service or reverse proxy maps 80 → 8080 | The usual answer in containers and Kubernetes — no capability needed |
| systemd socket activation | systemd (root) binds port 80 and passes the open socket to your unprivileged service | Clean; the service never holds any privilege |
| Ambient capability via systemd | AmbientCapabilities=CAP_NET_BIND_SERVICE with User=www | One capability, no file xattrs to manage |
| File capability | setcap cap_net_bind_service=+ep on the binary | Works anywhere, but must be re-applied on every upgrade |
(Linux also has net.ipv4.ip_unprivileged_port_start, a per-network-namespace sysctl that lowers the privileged-port boundary. Docker sets it to 0 inside containers on recent versions, which is why a non-root container can often bind port 80 with no capability at all.)
Capabilities in systemd
systemd exposes the capability sets directly in unit files:
# /etc/systemd/system/myserver.service
[Service]
User=www
Group=www
ExecStart=/usr/local/bin/myserver --listen :80
# Hand the unprivileged process exactly one capability...
AmbientCapabilities=CAP_NET_BIND_SERVICE
# ...and make it impossible for it or any child to ever gain more.
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
NoNewPrivileges=yes
CapabilityBoundingSet= with an empty value (CapabilityBoundingSet=) drops everything, which is what most services should use. systemd-analyze security myserver.service scores a unit's exposure and lists every option you have not set — a fast way to find the gaps.
Capabilities in containers
Docker and containerd start containers with a reduced default set of 14 capabilities (listed in Docker Security Hardening). That default is a compatibility compromise; for your own services, drop everything and add back only what you need:
docker run --cap-drop=ALL --cap-add=NET_BIND_SERVICE --user 10001:10001 myimage
To discover what a program needs, run it with everything dropped and add capabilities one at a time as EPERM errors appear — or trace it with tools such as capable from the BCC toolkit (see eBPF), which logs every capability check the kernel performs.
no_new_privs: Closing the Exec Escape Hatch
All the above has a loophole: a process with few privileges can still execve() a setuid-root binary or a binary with file capabilities and gain privileges. Dropping capabilities from the bounding set limits the capability part, but setuid still changes UIDs.
The no_new_privs flag closes this. Once set with prctl(PR_SET_NO_NEW_PRIVS, 1):
execve()will never grant privileges: setuid/setgid bits are ignored and file capabilities are not applied;- the flag is inherited by all children and cannot be unset;
- LSMs refuse transitions to more privileged domains on exec.
grep NoNewPrivs /proc/<pid>/status
# NoNewPrivs: 1
Why containers set it. A container may contain setuid binaries the image author forgot about. With no_new_privs, they are inert — if the container's user is 10001, nothing it executes can make it UID 0. This is what allowPrivilegeEscalation: false in Kubernetes and --security-opt no-new-privileges in Docker turn on.
Why seccomp needs it. An unprivileged process is only allowed to install a seccomp filter if no_new_privs is set (or if it holds CAP_SYS_ADMIN). Otherwise it could install a filter that makes a setuid program's privilege-dropping call silently "succeed" without doing anything, then exec that program — tricking a privileged binary into running in an unexpected state. Requiring no_new_privs guarantees that a filtered process can never exec into more privilege.
TIP
no_new_privs is nearly free and breaks almost nothing — the main casualty is sudo inside the process tree, which a service should never need. Turn it on everywhere: NoNewPrivileges=yes in systemd, allowPrivilegeEscalation: false in Kubernetes.
seccomp: Filtering System Calls
What it filters
Every interaction between a process and the kernel goes through a system call (see System Calls & Kernel Internals for the mechanism). Linux exposes over 450 of them on x86-64; a typical web service uses perhaps 60–100. Every unused syscall is kernel code an attacker could reach — and kernel vulnerabilities are usually in exactly those rarely-used, complicated corners (obscure socket families, filesystem features, keyring management).
seccomp lets a process install a filter that the kernel runs on every syscall before executing it. The filter sees the syscall number, the architecture and the raw argument values, and returns a verdict:
| Action | Effect |
|---|---|
SECCOMP_RET_ALLOW | Run the syscall normally |
SECCOMP_RET_ERRNO | Skip it and return an error (usually EPERM) |
SECCOMP_RET_KILL_PROCESS / KILL_THREAD | Terminate immediately |
SECCOMP_RET_TRAP | Deliver SIGSYS to the process (see Signals) |
SECCOMP_RET_LOG | Allow, but log — used for building profiles |
SECCOMP_RET_USER_NOTIF | Ask a supervisor process in user space to decide |
SECCOMP_RET_TRACE | Hand off to a ptrace-based tracer |
Filters can only be added, never removed, and are inherited across fork() and execve(). A process can tighten its own sandbox but never loosen it.
Strict mode vs filter mode
| Mode | Since | Allowed syscalls | Use |
|---|---|---|---|
Strict (SECCOMP_MODE_STRICT) | 2.6.12 (2005) | Only read, write, _exit, sigreturn | Pure computation on already-open file descriptors — rarely used today |
Filter (SECCOMP_MODE_FILTER, "seccomp-bpf") | 3.5 (2012) | Whatever your BPF program allows | Everything modern: Docker, Kubernetes, systemd, Chrome, Firefox, OpenSSH |
Filter programs are written in classic BPF and cannot dereference pointers — they can check that the first argument of socket() is AF_INET, but cannot inspect the contents of a path string. That is by design (checking user memory would be racy) and is why path-based restrictions belong in LSMs or Landlock, not seccomp. Nobody writes the BPF by hand; use libseccomp or a JSON profile.
The default container profile
Docker, containerd and CRI-O ship a default seccomp profile that is an allowlist of roughly 350 syscalls with everything else returning EPERM. In effect it blocks around 40–60 syscalls (the exact number shifts as new syscalls are added and argument conditions change). The blocked ones fall into a few families:
| Family | Examples | Why blocked |
|---|---|---|
| Kernel and system control | kexec_load, init_module, finit_module, delete_module, reboot | Containers must not change the host kernel |
| Clock and accounting | settimeofday, clock_settime, acct | Host-global state |
| Mounting and namespaces | mount, umount2, pivot_root, unshare and namespace-creating clone flags (unless CAP_SYS_ADMIN) | Reshaping the filesystem view is a common isolation gap |
| Kernel keyring | add_key, request_key, keyctl | Keyrings are not namespaced |
| Debugging other processes | ptrace (on older kernels), process_vm_readv/writev (without CAP_SYS_PTRACE) | Cross-process memory access |
| Rare and historically buggy | userfaultfd, perf_event_open, bpf, open_by_handle_at, swapon, obsolete syscalls | Large kernel attack surface for little legitimate use |
The profile is also capability-aware: if you add CAP_SYS_ADMIN, the syscalls it governs become allowed. That is one more reason dropping capabilities matters.
WARNING
For years, Kubernetes ran pods Unconfined — with no seccomp filter at all — unless you opted in. Since 1.27 the kubelet can default to RuntimeDefault with the --seccomp-default flag, but many clusters still do not enable it. Set seccompProfile: RuntimeDefault explicitly in every pod spec.
Generating a tight profile for your application
The default profile is a generic compromise. For high-value services, a custom allowlist can reduce reachable syscalls to the hundred or so your program actually uses. The workflow:
flowchart LR
A["Run app under representative load"] --> B["Record syscalls used: strace -f -c, SECCOMP_RET_LOG, or an eBPF/OCI hook"]
B --> C["Generate allowlist profile JSON"]
C --> D["Deploy in log or complain mode"]
D --> E{"Unexpected syscalls in audit log?"}
E -- "yes" --> C
E -- "no" --> F["Switch to enforce: default action ERRNO"]
# 1. See which syscalls the app actually makes during a realistic test run
strace -f -c -o /tmp/syscalls.txt ./myserver --config test.yaml
# (exercise every code path: startup, requests, TLS reload, graceful shutdown)
# 2. Tools that produce profiles directly
# - oci-seccomp-bpf-hook (Podman): podman run --annotation io.containers.trace-syscall=of:/tmp/profile.json ...
# - Security Profiles Operator (Kubernetes): records profiles from running pods into SeccompProfile CRDs
# - Inspektor Gadget: kubectl gadget advise seccomp-profile
A trimmed custom profile looks like this:
{
"defaultAction": "SCMP_ACT_ERRNO",
"defaultErrnoRet": 1,
"architectures": ["SCMP_ARCH_X86_64", "SCMP_ARCH_AARCH64"],
"syscalls": [
{
"names": ["accept4", "bind", "brk", "close", "epoll_ctl", "epoll_pwait",
"exit_group", "fstat", "futex", "getrandom", "listen", "mmap",
"munmap", "newfstatat", "openat", "read", "rt_sigaction",
"rt_sigprocmask", "rt_sigreturn", "sendto", "recvfrom",
"setsockopt", "socket", "write", "writev"],
"action": "SCMP_ACT_ALLOW"
}
]
}
Lessons teams learn the hard way:
- Language runtimes call syscalls you did not write. Go, the JVM and glibc use
clone3,rseq,membarrier,sched_getaffinityand others at startup or under load. Missing one often surfaces as a confusing crash minutes later, not an obviousEPERMat boot. - Base image and library upgrades change the set. A glibc upgrade that starts using
clone3instead ofclonewill break a profile that was correct yesterday. Regenerate profiles in CI and test them. - Prefer
ERRNOoverKILLas the default action for application profiles. An unexpected syscall returningEPERMis usually handled gracefully; a killed process is an outage. - The default profile already captures most of the value. Custom profiles are worth it for internet-facing parsers, multi-tenant workloads and anything processing untrusted files — not necessarily for every internal service.
seccomp in systemd
systemd uses predefined syscall groups so you do not have to maintain lists by hand:
# Allow the set normally needed by system services, then remove dangerous groups
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @resources @mount @debug @reboot @swap @module @obsolete
SystemCallErrorNumber=EPERM
SystemCallArchitectures=native
systemd-analyze syscall-filter @system-service prints what each group contains. SystemCallArchitectures=native blocks the 32-bit compatibility syscall table on 64-bit hosts — an entire second attack surface that most services never use.
Linux Security Modules: SELinux and AppArmor
Mandatory access control
Everything so far is either discretionary (file owners decide) or self-imposed (the process restricts itself). An LSM adds a policy that the administrator writes and the kernel enforces regardless of UID. Even a root process confined by SELinux or AppArmor can only touch what its policy allows. LSMs hook into the kernel at hundreds of points — file open, exec, socket connect, capability use, mount, ptrace — and get a final say after DAC has allowed an operation.
Several LSMs can be active at once ("stacking"): typically one major MAC module (SELinux or AppArmor) plus minor ones such as Yama (ptrace restrictions), Lockdown and Landlock. Check what is active with cat /sys/kernel/security/lsm.
SELinux vs AppArmor
| SELinux | AppArmor | |
|---|---|---|
| Model | Type enforcement on labels: every process and object has a security context, policy says which types may interact | Path-based profiles: each program has a profile listing which paths and operations it may use |
| Identity of a file | Its label (stored in an xattr) — survives renames and hard links | Its path — a file reachable by two paths may be treated differently |
| Default distros | RHEL, CentOS Stream, Fedora, Rocky/Alma, Amazon Linux 2023, Android | Ubuntu, Debian, SUSE |
| Coverage | Whole system confined by default ("targeted" policy for daemons) | Only programs with a loaded profile are confined |
| Learning curve | Steep — labels, types, booleans, MLS/MCS | Gentle — profiles read like a list of paths |
| Containers | Each container gets a unique MCS category pair (container_t:s0:c123,c456), so containers cannot read each other's files even as the same UID | Runtime loads the docker-default / cri-containerd.apparmor.d profile |
SELinux labels look like user:role:type:level. The type is what matters for most decisions:
ls -Z /var/www/html/index.html
# system_u:object_r:httpd_sys_content_t:s0 /var/www/html/index.html
ps -eZ | grep nginx
# system_u:system_r:httpd_t:s0 1234 ? 00:00:01 nginx
The policy contains rules like "processes of type httpd_t may read files of type httpd_sys_content_t". If nginx is compromised, it still cannot read /etc/shadow (shadow_t) or a database's files (postgresql_db_t) — no matter what Unix permissions say.
AppArmor profiles are text files in /etc/apparmor.d/:
# /etc/apparmor.d/usr.local.bin.myserver (simplified)
/usr/local/bin/myserver {
#include <abstractions/base>
network inet stream,
capability net_bind_service,
/etc/myserver/** r,
/var/lib/myserver/** rw,
/var/log/myserver/*.log w,
deny /etc/shadow r,
}
When you meet each
In practice, you meet the LSM when something that "should work" gets Permission denied despite correct Unix permissions:
- A service on RHEL cannot read files in a non-standard directory → SELinux label is wrong.
- A container on Fedora cannot read a bind-mounted host directory → the directory lacks the container label (the
:z/:Zvolume suffix relabels it). - A program on Ubuntu cannot write a new log location → its AppArmor profile does not list the path.
Reading a denial
SELinux logs AVC (Access Vector Cache) denials to the audit log:
type=AVC msg=audit(1731052800.123:456): avc: denied { read } for pid=1234
comm="nginx" name="index.html" dev="vda1" ino=98765
scontext=system_u:system_r:httpd_t:s0
tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0
Read it as: process (scontext, type httpd_t) was denied an operation (read) on an object (tcontext, type user_home_t, class file). The file was probably moved from a home directory and kept its home-directory label.
ausearch -m AVC -ts recent # recent denials
sealert -a /var/log/audit/audit.log # human-readable explanation and suggested fix
restorecon -Rv /var/www/html # reset labels to policy defaults — usually the real fix
getsebool -a | grep httpd # booleans toggle common policy options
AppArmor denials go to the kernel log:
audit: type=1400 apparmor="DENIED" operation="open" profile="/usr/local/bin/myserver"
name="/srv/uploads/a.png" pid=4321 comm="myserver" requested_mask="w" denied_mask="w"
journalctl -k | grep apparmor=\"DENIED\"
aa-status # loaded profiles and their mode
aa-complain /usr/local/bin/myserver # log-only while you fix the profile
aa-logprof # interactively add rules from logged denials
WARNING
The most common response to an LSM denial is setenforce 0 or disabling AppArmor — turning off the whole system's mandatory access control to fix one path. Fix the label (restorecon, semanage fcontext) or the profile instead. If you must unblock production, switch one domain or profile to permissive/complain mode (semanage permissive -a httpd_t, aa-complain) and fix it properly afterwards.
User Namespaces and Rootless Containers
Root inside is not root outside
A user namespace gives a process its own UID/GID mapping (see Namespaces & cgroups for the mechanics). UID 0 inside the namespace maps to an ordinary, unprivileged UID outside — for example 100000:
flowchart LR
subgraph Container["Container user namespace"]
A["UID 0 'root'"]
B["UID 1000 'app'"]
end
subgraph Host["Host (initial user namespace)"]
C["UID 100000 (no privileges)"]
D["UID 101000 (no privileges)"]
end
A -- "uid_map: 0 to 100000" --> C
B -- "uid_map: 1000 to 101000" --> D
cat /proc/<pid>/uid_map
# 0 100000 65536 <- inside-start outside-start count
Inside the namespace, the process has a full set of capabilities — but only over resources owned by that namespace. It can mount a tmpfs in its own mount namespace or configure its own network namespace; it cannot load kernel modules, change the host clock, or touch host files owned by real root, because the kernel checks capabilities against the namespace that owns the resource. If a process escapes the container's other boundaries, it lands on the host as UID 100000 — a nobody.
Rootless containers
Rootless goes one step further: the container runtime itself (Podman, rootless Docker, rootless containerd/nerdctl, BuildKit) runs as an ordinary user, using user namespaces plus helpers (newuidmap, slirp4netns/pasta for networking). A vulnerability in the runtime or daemon then yields an unprivileged user, not root.
| Rootful (default Docker) | Rootful + userns-remap | Rootless | |
|---|---|---|---|
| Daemon runs as | root | root | your user |
| Container UID 0 on the host | UID 0 | Unprivileged range | Unprivileged range |
| Impact of runtime/daemon bug | Host root | Host root (daemon) | One user account |
| Trade-offs | — | Volume ownership mapping | Some networking and storage limitations, no binding ports < 1024 without sysctl |
In Kubernetes, user namespaces for pods arrived via hostUsers: false in the pod spec (beta and enabled by default from 1.30, with runtime and filesystem support requirements). Each pod gets a distinct, non-overlapping UID range on the node.
INFO
User namespaces also expand the kernel attack surface: an unprivileged process can create one and then reach code paths (filesystem mounts, netfilter configuration) that previously only real root could reach. Several kernel vulnerabilities have been reachable this way, which is why some distributions restrict unprivileged user namespace creation (kernel.unprivileged_userns_clone, Ubuntu's AppArmor-based restriction) and why the default container seccomp profile blocks unshare for containers. Using user namespaces for containers is a clear win; letting arbitrary untrusted code create them is a separate decision.
Landlock: Unprivileged Self-Sandboxing
Every mechanism so far is configured from the outside — by root, systemd or the container runtime. Landlock (merged in Linux 5.13, 2021) lets an ordinary, unprivileged process sandbox itself, restricting its own filesystem access (and, since 6.7, TCP bind/connect ports) using a stackable LSM.
The use case is an application that knows exactly what it needs: a PDF converter that should read one input file and write one output directory, a build tool that should only touch the project tree, a CLI that parses untrusted archives. After startup, it declares a ruleset and enforces it on itself:
// Sketch: after this, the process can only read /usr and read/write /var/lib/myapp
int ruleset = landlock_create_ruleset(&attr, sizeof(attr), 0);
add_path_rule(ruleset, "/usr", LANDLOCK_ACCESS_FS_READ_FILE | LANDLOCK_ACCESS_FS_READ_DIR);
add_path_rule(ruleset, "/var/lib/myapp", READ | WRITE);
prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0); // required, as with seccomp
landlock_restrict_self(ruleset, 0); // irreversible; inherited by children
Key properties: it needs no privileges, restrictions can only be tightened, it composes with seccomp (seccomp limits which syscalls, Landlock limits which paths and ports those syscalls may touch), and libraries exist for Rust (landlock crate), Go (go-landlock) and C. systemd and several sandboxing tools use it, and it is a good fit for any program that processes untrusted input.
Kubernetes: Putting It Together in securityContext
Kubernetes exposes nearly every layer through the pod and container securityContext. This is a hardened baseline suitable for most stateless services:
apiVersion: v1
kind: Pod
metadata:
name: api
spec:
automountServiceAccountToken: false # only mount a token if the app calls the API
securityContext: # pod level: applies to all containers
runAsNonRoot: true # kubelet refuses to start a container that would run as UID 0
runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001 # group ownership for mounted volumes
seccompProfile:
type: RuntimeDefault # the container runtime's default filter
containers:
- name: api
image: registry.example.com/api@sha256:... # pin by digest
ports:
- containerPort: 8080 # unprivileged port; the Service maps 80/443
securityContext: # container level
allowPrivilegeEscalation: false # sets no_new_privs
readOnlyRootFilesystem: true
privileged: false
capabilities:
drop: ["ALL"]
# add: ["NET_BIND_SERVICE"] # only if the app truly must bind < 1024
volumeMounts:
- name: tmp
mountPath: /tmp
resources:
limits:
memory: "512Mi"
volumes:
- name: tmp
emptyDir:
medium: Memory
sizeLimit: 64Mi
What each field maps to in the kernel:
| Field | Kernel mechanism |
|---|---|
runAsNonRoot, runAsUser, runAsGroup | Process UID/GID |
allowPrivilegeEscalation: false | no_new_privs |
capabilities.drop: ["ALL"] | Empty bounding, permitted and effective sets |
seccompProfile: RuntimeDefault | seccomp filter |
readOnlyRootFilesystem: true | Root filesystem mounted ro |
appArmorProfile / seLinuxOptions | LSM profile or label |
hostUsers: false | User namespace |
privileged: true (avoid) | All of the above disabled |
Also keep hostNetwork, hostPID, hostIPC and hostPath volumes out of application pods — each one punches a hole in a namespace. Pair the pod hardening with tight RBAC for the service account and Network Policies for traffic.
Pod Security Standards
Kubernetes defines three standard profiles, enforced by the built-in Pod Security Admission controller per namespace:
| Level | Intent | Key restrictions |
|---|---|---|
| Privileged | Unrestricted — system components like CNI agents and storage drivers | None |
| Baseline | Blocks known privilege escalations while staying compatible with most images | No privileged, no host namespaces, no hostPath, capabilities limited to the default set, no unsafe sysctls, no Unconfined seccomp override |
| Restricted | Current hardening best practice | Baseline plus runAsNonRoot, allowPrivilegeEscalation: false, drop ALL (only NET_BIND_SERVICE may be added), seccomp RuntimeDefault or Localhost, restricted volume types |
# Enforce restricted; warn and audit at the same level so violations are visible
kubectl label namespace payments \
pod-security.kubernetes.io/enforce=restricted \
pod-security.kubernetes.io/warn=restricted \
pod-security.kubernetes.io/audit=restricted
# Preview what would break before enforcing
kubectl label --dry-run=server --overwrite namespace payments \
pod-security.kubernetes.io/enforce=restricted
Note that restricted does not require readOnlyRootFilesystem — add that via policy engines (Kyverno, Gatekeeper) or admission webhooks if you want it enforced. A practical rollout: label every namespace baseline (enforce) and restricted (warn), fix workloads as warnings appear, then raise enforcement namespace by namespace.
Defense in Depth
No single layer is sufficient. Each one assumes the others might fail, and each targets a different class of risk:
| Layer | What it removes | Class of risk it reduces |
|---|---|---|
| Non-root user | Root's identity and default capabilities | App compromise → full control of files and processes; most container-escape preconditions |
| Capabilities dropped | Specific privileged operations | Network reconfiguration, ptrace, module loading, mount-based escapes |
no_new_privs | Gaining privileges through exec | Abuse of leftover setuid binaries and file capabilities |
| seccomp | Reachable kernel syscalls | Kernel vulnerabilities in rarely-used syscalls; many escape techniques need a blocked call |
| LSM (SELinux/AppArmor) | Access the policy does not explicitly allow, even for root | Lateral movement to other files, containers or host resources after compromise |
| User namespaces | Host-level meaning of container UID 0 | Escapes landing as host root; runtime bugs yielding host root |
| Read-only root filesystem | Ability to modify the image at runtime | Persistence, dropped tools, binary tampering |
nosuid/nodev/noexec mounts | Privileged or executable files in writable locations | Planting privileged binaries or device nodes |
| Landlock | Filesystem and port access beyond declared needs | Exploited parser reaching unrelated files |
flowchart LR
A["Bug in application code"] --> B["Attacker runs code as app UID"]
B --> C{"Non-root + caps dropped?"}
C -- "yes" --> D["Cannot mount, load modules or ptrace"]
B --> E{"seccomp RuntimeDefault?"}
E -- "yes" --> F["Most kernel attack surface unreachable"]
B --> G{"LSM profile?"}
G -- "yes" --> H["Cannot read other containers or host files"]
B --> I{"User namespace?"}
I -- "yes" --> J["Any escape lands as unprivileged host UID"]
The value of layering is visible in the history of container-runtime vulnerabilities. CVE-2019-5736 in runc allowed a malicious container to overwrite the host's runc binary under certain conditions, gaining host root. At a high level, the published mitigations were the layers on this page: containers running as a non-root user were not affected in the common configuration, user namespaces prevented the container's root from having write access to the host binary, and SELinux policy on several distributions blocked the write in the default configuration. Teams that had already layered these defenses were protected before a patch existed. Later runtime vulnerabilities (such as the 2024 "Leaky Vessels" set in runc and BuildKit) followed the same pattern — non-root, user namespaces and LSMs narrowed or eliminated the impact. Details of these issues are discussed in Container Escape Techniques; the lesson here is that you cannot patch vulnerabilities you do not know about, but you can reduce what any of them yields.
War Story (illustrative)
A team ran an image-thumbnail service as root in a container, with Docker's default capabilities and no seccomp override in Kubernetes (so the pod was Unconfined). A vulnerability in the image-decoding library let a crafted upload execute code in the service. The attacker could read every secret mounted into the pod and, because the process was root with CAP_NET_RAW, sniff traffic in the pod's network namespace. The post-incident hardening took a single afternoon: runAsUser: 10001, capabilities.drop: ["ALL"], allowPrivilegeEscalation: false, seccompProfile: RuntimeDefault, readOnlyRootFilesystem: true, the namespace labelled restricted, and the decoder moved into a separate low-privilege sidecar with a Landlock policy limiting it to its input directory. Nothing broke except a debug script that had been writing to /app/logs — which moved to an emptyDir.
Production Hardening Checklist
Identity
-
Service runs as a dedicated, non-root UID (not
nobody, which is shared) — in containers, a fixed numeric UID ≥ 10000 - No login shell and no password for service accounts
-
Privilege drop, if done in code, uses
setgroups+setresgid+setresuidand verifies it cannot regain root
Filesystem
-
Root filesystem read-only; writable paths explicitly mounted (
emptyDir, tmpfs,StateDirectory=) -
Writable mounts are
nosuid,nodev(andnoexecwhere practical) -
Config and secrets readable only by the service user (
0600/0640) - No setuid/setgid binaries in container images; setuid inventory on hosts tracked and alerted on change
-
File capabilities inventoried with
getcap -r /
Privileges
- All capabilities dropped; only named, justified ones added back
-
No
CAP_SYS_ADMIN,CAP_SYS_PTRACE,CAP_SYS_MODULE,CAP_NET_ADMINwithout a documented reason -
no_new_privsset (NoNewPrivileges=yes,allowPrivilegeEscalation: false,no-new-privileges) -
No
--privileged/privileged: trueoutside a short, reviewed list of system components
Kernel surface
- seccomp enabled — at least the runtime default; custom profiles for internet-facing parsers
-
systemd services use
SystemCallFilter=@system-serviceandSystemCallArchitectures=native - SELinux enforcing or AppArmor profiles loaded — never globally disabled to fix a single denial
Isolation
-
No host namespaces (
hostNetwork,hostPID,hostIPC) orhostPathmounts in application pods - No container runtime socket mounted into workloads
-
User namespaces or rootless runtimes where supported (
hostUsers: false, Podman, rootless BuildKit for CI)
Verification
-
systemd-analyze securityscore reviewed for each service -
Pod Security Admission
restricted(warn/audit everywhere, enforce where possible) - CI checks manifests (Kyverno/Gatekeeper/kube-score) and images (see Docker Security Hardening)
Decision Guide
systemd service on a VM
User=myapp
Group=myapp
DynamicUser=yes # or: a static user; DynamicUser allocates a throwaway UID
NoNewPrivileges=yes
CapabilityBoundingSet= # none; add CAP_NET_BIND_SERVICE only if needed
AmbientCapabilities=
ProtectSystem=strict # whole filesystem read-only...
ReadWritePaths=/var/lib/myapp # ...except what you list
StateDirectory=myapp
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectClock=yes
RestrictNamespaces=yes
RestrictSUIDSGID=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes # remove for JIT runtimes (JVM, Node, .NET)
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @resources
SystemCallArchitectures=native
Then run systemd-analyze security myapp.service and aim for a score in the "OK" range or better.
Choosing per platform
| Situation | Start with | Add when |
|---|---|---|
| systemd service | Block above; systemd-analyze security | AppArmor/SELinux profile for internet-facing daemons; Landlock if the app supports it |
| Docker / containerd | --user 10001, --cap-drop=ALL, --security-opt no-new-privileges, --read-only, default seccomp and AppArmor/SELinux | Custom seccomp profile for high-risk services; rootless or userns-remap for multi-tenant hosts and CI builders |
| Kubernetes app pods | The securityContext above; namespace labelled restricted | hostUsers: false; Localhost seccomp profiles via Security Profiles Operator |
| Kubernetes system pods (CNI, CSI, monitoring agents) | Isolated namespace at privileged; grant only the specific capabilities and host mounts each agent documents | Review on every upgrade; never co-locate application workloads in that namespace |
| Untrusted or multi-tenant code | All of the above | A stronger boundary than namespaces: gVisor, Kata Containers or Firecracker microVMs (see Container Escape Techniques) |
flowchart TD
A["New workload"] --> B{"Runs third-party or tenant code?"}
B -- "yes" --> C["Sandboxed runtime: gVisor, Kata, microVM + full hardening"]
B -- "no" --> D{"Kubernetes?"}
D -- "yes" --> E["restricted PSS + hardened securityContext"]
D -- "no" --> F{"Container?"}
F -- "yes" --> G["Non-root, cap-drop ALL, no-new-privileges, read-only, default seccomp"]
F -- "no" --> H["systemd sandboxing options + systemd-analyze security"]
E --> I{"Parses untrusted input?"}
G --> I
H --> I
I -- "yes" --> J["Custom seccomp profile + LSM profile + Landlock where possible"]
I -- "no" --> K["Defaults are enough. Review yearly."]
Interview Questions
Q: What is the difference between real, effective and saved UIDs?
The real UID is who started the process; the effective UID is used for most permission checks; the saved UID stores a previous effective UID so a program can drop privilege temporarily and regain it. setuid binaries set the effective (and saved) UID to the file owner while leaving the real UID as the caller. To drop privileges permanently, all three must be changed (setresuid).
Q: Why is CAP_SYS_ADMIN called "the new root"?
It guards a huge, loosely related set of operations — mounting filesystems, creating namespaces, many device and kernel configuration interfaces — because it became the default choice for new privileged operations. Holding it is close to holding full root, so it should almost never be granted to applications.
Q: How do you let a non-root web server listen on port 80?
Best: don't — listen on 8080 and let a load balancer or Kubernetes Service map 80. Otherwise: systemd socket activation, AmbientCapabilities=CAP_NET_BIND_SERVICE in the unit, or setcap cap_net_bind_service=+ep on the binary. Inside containers, ip_unprivileged_port_start is often already 0.
Q: What does allowPrivilegeEscalation: false actually do?
It sets the kernel's no_new_privs flag on the container process, so execve() can never grant privileges — setuid bits and file capabilities are ignored. It is inherited and irreversible.
Q: What is the difference between seccomp and an LSM like AppArmor? seccomp filters which system calls a process can make, based on syscall number and raw argument values, and is usually self-applied or applied by the runtime. LSMs decide which objects (files, sockets, capabilities) a process can access, based on labels or paths, and are mandatory policy written by an administrator. seccomp cannot inspect file paths; LSMs do not reduce kernel syscall surface. They complement each other.
Q: If a container runs as root with user namespaces enabled, is that safe? Much safer: container UID 0 maps to an unprivileged host UID, so an escape lands with no host privileges, and capabilities only apply to resources the namespace owns. It is still worth running as non-root inside the container and keeping seccomp and LSM profiles, because user namespaces themselves expose more kernel code paths.
Q: What does the Kubernetes restricted Pod Security Standard require?
Non-root, allowPrivilegeEscalation: false, all capabilities dropped (only NET_BIND_SERVICE may be added back), seccomp RuntimeDefault or Localhost, plus everything in baseline: no privileged mode, no host namespaces, no hostPath volumes.
Related Pages
- Linux Namespaces & cgroups — the isolation and resource-limit primitives containers are built from
- System Calls & Kernel Internals — the user/kernel boundary that seccomp filters
- Processes & Threads — process credentials,
forkandexec - Signals & Process Control —
SIGSYSfrom seccomp traps - Containers from Scratch — building isolation by hand
- Docker Security Hardening — images, scanning, signing and daemon configuration
- Container Escape Techniques — why each misconfiguration matters, and stronger isolation runtimes
- Kubernetes RBAC — limiting what a compromised pod's service account can do
- Least Privilege — the principle behind every section of this page