On this page

Process Isolation & Security: Users, Capabilities & seccomp

Every time a process opens a file, binds a socket, sends a signal or loads a kernel module, the kernel asks one question: is this process allowed to do this? The answer comes from several independent mechanisms stacked on top of each other — user and group IDs, file permission bits, capabilities, seccomp filters, Linux Security Modules (LSMs) and namespaces. Each one was added at a different point in Linux's history to close a gap the previous one left open.

This page is a defensive guide. It explains how each layer works, what class of risk it reduces, and how to configure it for the things engineers actually run: systemd services, Docker/containerd containers and Kubernetes pods. It deliberately does not cover how to break these mechanisms — for the attacker's view of containers, and why each misconfiguration matters, see Container Escape Techniques.

The goal is simple: when your service has a bug — and it will — the bug should give an attacker as little as possible.


The Mental Model: Many Gates, One Request

A single open("/etc/shadow", O_RDONLY) passes through a chain of checks. Any one of them can say no, and the request fails.

flowchart TD
    A["Process calls open('/etc/shadow')"] --> B{"seccomp filter: is openat allowed?"}
    B -- "no" --> X1["EPERM or process killed"]
    B -- "yes" --> C{"Namespaces: is the path visible in this mount namespace?"}
    C -- "no" --> X2["ENOENT"]
    C -- "yes" --> D{"DAC: owner/group/other mode bits and ACLs vs fsuid/fsgid"}
    D -- "denied" --> E{"Capability override? CAP_DAC_OVERRIDE / CAP_DAC_READ_SEARCH"}
    E -- "no" --> X3["EACCES"]
    E -- "yes" --> F
    D -- "allowed" --> F{"LSM hook: SELinux / AppArmor / Landlock"}
    F -- "denied" --> X4["EACCES + audit log entry"]
    F -- "allowed" --> G["File descriptor returned"]

Three things to notice:

  1. The checks are independent. Passing the Unix permission check does not bypass SELinux; being root does not bypass seccomp. This is what makes defense in depth work.
  2. Classic Unix permissions (DAC — Discretionary Access Control) are owner-controlled. The owner of a file decides who may read it. LSMs add MAC (Mandatory Access Control) — a policy written by the administrator that even the file's owner, and even root, cannot override.
  3. "Root" is not one thing. Since Linux 2.2, root's power is a bundle of capabilities, and every one of them can be removed independently.

The Permission Model: UIDs, GIDs and File Modes

Users and groups are just numbers

The kernel knows nothing about usernames. /etc/passwd maps the name www-data to the number 33; the kernel only ever sees 33. A process carries several IDs in its credentials (struct cred):

IDMeaningUsed for
Real UID/GID (ruid)Who started the processAccounting, who may send it signals
Effective UID/GID (euid)Whose privileges the process is using right nowMost permission checks
Saved set-UID/GID (suid)A stashed copy of a previous effective IDLets a program drop privilege temporarily and regain it later
Filesystem UID/GID (fsuid)Linux-specific; tracks euid almost alwaysFile access checks
Supplementary groupsExtra group membershipsGroup permission checks

You can see them all in /proc:

grep -E '^(Uid|Gid|Groups)' /proc/self/status
# Uid:    1000    1000    1000    1000     <- real, effective, saved, fs
# Gid:    1000    1000    1000    1000
# Groups: 4 27 1000

Normally all four UIDs are equal. They diverge in two situations: when a setuid binary runs (the effective UID becomes the file owner's, the real UID stays yours), and when a privileged daemon drops privileges after startup.

Dropping privileges correctly

The classic daemon pattern is: start as root, do the one thing that needs root (bind port 443, open a log file in /var/log), then switch to an unprivileged user for the rest of its life.

// Order matters: groups first, then GID, then UID.
// Once the UID is no longer 0 you lose the right to change groups.
if (setgroups(0, NULL) != 0)        die("setgroups");
if (setresgid(gid, gid, gid) != 0)  die("setresgid");
if (setresuid(uid, uid, uid) != 0)  die("setresuid");

// Verify: trying to get root back must fail.
if (setuid(0) == 0)                 die("privilege drop did not stick");

setresuid() sets real, effective and saved IDs at once. A common mistake is calling only seteuid(), which leaves the saved UID as 0 — so any later code path (or an attacker controlling the process) can switch back to root. Forgetting setgroups() is another: the process keeps root's supplementary groups.

TIP

In practice, let the supervisor drop privileges for you. User= in a systemd unit, USER in a Dockerfile and runAsUser in Kubernetes start the process as the unprivileged user from the very first instruction — there is no window where your code runs as root.

File modes: the nine bits (plus three)

Every inode has an owner UID, a group GID and a mode. The kernel's check is short and strictly ordered:

  1. If the process's fsuid equals the file owner → use the owner bits.
  2. Else, if the file's group is the process's fsgid or one of its supplementary groups → use the group bits.
  3. Else → use the other bits.

It picks one class and stops. A file with mode 0604 owned by you denies you write but grants everyone else read — the owner class matched first. (See File Systems for how inodes store this.)

ls -l /usr/bin/passwd /tmp /etc/shadow
# -rwsr-xr-x 1 root root   ...  /usr/bin/passwd    <- 's' = setuid
# drwxrwxrwt 9 root root   ...  /tmp               <- 't' = sticky bit
# -rw-r----- 1 root shadow ...  /etc/shadow
BitOctalOn a fileOn a directory
setuid4000Run with the file owner's UID(ignored on Linux)
setgid2000Run with the file's GIDNew files inherit the directory's group
sticky1000(ignored)Only a file's owner can delete or rename it — why /tmp is safe to share

On directories, r means "list names", w means "create/delete entries", and x means "traverse" (look up a name inside). A directory with x but not r lets you open a file whose name you already know, but not discover names.

ACLs: when nine bits are not enough

POSIX ACLs add extra entries for specific users and groups without changing ownership — useful when a log directory must be readable by both the app user and a log shipper:

setfacl -m u:fluentbit:rx /var/log/myapp
setfacl -d -m u:fluentbit:r /var/log/myapp     # default ACL: applies to new files
getfacl /var/log/myapp
# user::rwx
# user:fluentbit:r-x
# group::r-x
# mask::r-x          <- the mask caps every named entry and the group entry
# other::---

A + at the end of ls -l output (drwxr-x---+) tells you an ACL is present. The mask entry is the most common surprise: a chmod g-r on a file with an ACL lowers the mask, silently removing access you granted to named users.

Why "run as root" is the default risk

UID 0 is special in two ways. Traditionally the kernel skipped permission checks entirely for it, and today root holds every capability by default. A process running as root that is compromised — through a deserialization bug, a path traversal, a vulnerable dependency — can typically:

  • read and modify any file, including credentials, keys and other tenants' data;
  • load kernel modules, change firewall rules and network routes;
  • attach to and read the memory of any other process;
  • in a container, use any gap in isolation to reach the host.

Most software does not need any of this. The rest of this page is about taking those powers away, starting with the cheapest and most effective step: do not run as root. Inside containers this matters just as much — without user namespaces, UID 0 in a container is UID 0 on the host kernel, constrained only by the other layers described below.


setuid and setgid Binaries

What they are for

A setuid binary runs with the privileges of the file's owner rather than the user who launched it. This is how an ordinary user can change their own password: passwd must write /etc/shadow, which only root can modify, so /usr/bin/passwd is owned by root with the setuid bit set. When you run it, its effective UID is 0, while its real UID still says who you are — so the program can check that you are only changing your password.

Historically, ping was setuid root because sending ICMP requires a raw socket. Modern distributions have replaced that with either a file capability (cap_net_raw) or unprivileged ICMP sockets (net.ipv4.ping_group_range), so ping no longer needs to be setuid at all. That migration is a good illustration of the general trend: setuid is being replaced by narrower mechanisms.

Why every setuid binary is attack surface

A setuid-root program is a door from "any local user" to "root", guarded only by that program's own code. It runs with full root privileges, but its inputs come from an untrusted user: command-line arguments, environment variables, file descriptors it inherits, the current directory, resource limits and signal dispositions. A bug in any input handling — or in a library it links — can let a local user act as root.

The kernel and the dynamic loader add protections (they ignore LD_PRELOAD and similar variables in "secure-execution" mode, for instance), but the history of local privilege escalation is dominated by setuid programs. The defensive conclusion is the same every time: have as few as possible, and none in places attackers can reach.

Finding and minimising them

# Every setuid file on the root filesystem (-xdev: don't cross into other mounts)
find / -xdev -perm -4000 -type f -exec ls -l {} + 2>/dev/null

# setgid files too
find / -xdev -perm -2000 -type f -exec ls -l {} + 2>/dev/null

# Files with capabilities attached (the modern alternative — audit these as well)
getcap -r / 2>/dev/null

A typical server has 10–25 setuid binaries: passwd, chsh, chfn, newgrp, gpasswd, su, sudo, mount, umount, pkexec, fusermount3 and a few helpers. Questions to ask about each:

QuestionAction if "no"
Do humans log into this machine?Remove chsh, chfn, newgrp setuid bits; consider removing su
Do non-root users need to mount things?Remove setuid from mount/umount/fusermount
Is polkit actually used?Remove the package that ships pkexec
Is this a container image?It should have zero setuid binaries — strip them at build time

For container images, stripping is one line in the Dockerfile (distroless and scratch-based images start with none):

# Remove setuid/setgid bits from everything in the image
RUN find / -xdev -perm /6000 -type f -exec chmod a-s {} + || true

Track the list over time. A new setuid binary appearing on a server you did not change is worth an alert — file-integrity tools (AIDE, osquery, Falco) can watch for it.

The nosuid mount option

nosuid tells the kernel to ignore the setuid and setgid bits (and file capabilities) for every file on that mount. Anything a user or a service can write to should be mounted nosuid, so that a writable location can never be used to plant a privileged binary:

# /etc/fstab
tmpfs      /tmp       tmpfs  defaults,nosuid,nodev,noexec,size=1g  0 0
/dev/vdb1  /srv/data  ext4   defaults,nosuid,nodev                  0 2
OptionEffect
nosuidsetuid/setgid bits and file capabilities have no effect
nodevDevice files on this mount cannot be opened as devices
noexecFiles cannot be executed directly (defense against drop-and-run; not a hard barrier, since interpreters can still read scripts)
roRead-only — the strongest of all

Container runtimes mount /proc, /sys, /dev/shm and tmpfs volumes with these flags by default, and systemd's ProtectSystem=, PrivateTmp= and NoNewPrivileges= achieve the same for services.


Linux Capabilities: Splitting Root

The idea

Before capabilities, the kernel asked "is euid == 0?" and either allowed everything or nothing. Capabilities split that single bit into about 40 separate privileges (41 on current kernels), each guarding a specific family of operations. The kernel now asks "does this process hold CAP_NET_BIND_SERVICE?" rather than "is it root?". A web server that needs to bind port 80 can hold exactly that one capability and nothing else.

capsh --print | head -3       # what the current shell holds
man 7 capabilities            # the authoritative list

The ones you will actually meet

CapabilityGrantsRisk if granted unnecessarily
CAP_NET_BIND_SERVICEBind TCP/UDP ports below 1024Low — the classic safe capability
CAP_NET_RAWRaw and packet sockets (ping, tcpdump)Medium — can craft and sniff packets on the network namespace
CAP_NET_ADMINConfigure interfaces, routes, firewall rules, tcHigh — can reroute or intercept traffic in its network namespace
CAP_CHOWN, CAP_FOWNER, CAP_DAC_OVERRIDEChange ownership; bypass file permission checksHigh — defeats the DAC layer entirely
CAP_SETUID, CAP_SETGIDChange to any UID/GIDHigh — become any user, including root
CAP_KILLSignal processes owned by other usersMedium
CAP_SYS_PTRACETrace and read/write memory of other processesHigh — read secrets from any process it can see
CAP_SYS_MODULELoad kernel modulesCritical — arbitrary kernel code
CAP_SYS_ADMINMount filesystems, namespaces, BPF (partly), quotas, dozens moreCritical — "the new root"
CAP_BPF, CAP_PERFMONLoad BPF programs; use perf events (split out of CAP_SYS_ADMIN in 5.8)High — kernel attack surface

CAP_SYS_ADMIN deserves its own warning. When kernel developers added a privileged operation and could not decide which capability should guard it, they often chose CAP_SYS_ADMIN. The result is a capability covering hundreds of call sites — mounting filesystems, creating most namespace types, configuring many devices — so broad that holding it is close to holding root. Treat any request for CAP_SYS_ADMIN as a request for root and look for a narrower alternative first (CAP_BPF, CAP_PERFMON, CAP_CHECKPOINT_RESTORE and others were carved out precisely so tools could stop asking for it).

WARNING

--privileged in Docker and privileged: true in Kubernetes grant every capability, disable seccomp and the LSM profile, and expose host devices. It is not "a few more permissions" — it removes almost every layer this page describes. See Container Escape Techniques for why.

Process capability sets

Each thread carries five capability sets. You rarely need all of them, but you will see them in /proc/<pid>/status and in error messages:

SetMeaning
Permitted (CapPrm)The ceiling: capabilities the thread may make effective
Effective (CapEff)What the kernel actually checks right now
Inheritable (CapInh)Can pass across execve() — but only to programs whose file also allows them
Bounding (CapBnd)An absolute limit: nothing outside this set can ever be gained, not even via setuid-root
Ambient (CapAmb)Passed across execve() to ordinary (non-capability-aware) programs
grep Cap /proc/$(pgrep -o nginx)/status
# CapInh: 0000000000000000
# CapPrm: 0000000000000400
# CapEff: 0000000000000400
# CapBnd: 0000000000000400
# CapAmb: 0000000000000000

capsh --decode=0000000000000400
# 0x0000000000000400=cap_net_bind_service

The bounding set is the important one for hardening: drop a capability from it and that process and all its descendants can never acquire it again — no setuid binary, no file capability, nothing. This is what container runtimes and systemd's CapabilityBoundingSet= manipulate.

File capabilities: the targeted setuid replacement

Capabilities can also be attached to an executable (stored in the security.capability extended attribute). When the file is executed, the process gains just those capabilities — a far narrower grant than setuid root.

# Give one binary permission to bind low ports, nothing else
sudo setcap 'cap_net_bind_service=+ep' /usr/local/bin/myserver
getcap /usr/local/bin/myserver
# /usr/local/bin/myserver cap_net_bind_service=ep

# Remove it
sudo setcap -r /usr/local/bin/myserver

e = effective, p = permitted, i = inheritable. Caveats: file capabilities are lost when the file is rewritten (so your deployment must re-apply them), they are ignored on nosuid mounts, and some copy/archive tools drop extended attributes. Like setuid binaries, every file with capabilities is something to inventory with getcap -r /.

Giving a web server port 80 without root

There are four common answers, from most to least preferred:

ApproachHowNotes
Don't bind port 80 in the appApp listens on 8080; a load balancer, Kubernetes Service or reverse proxy maps 80 → 8080The usual answer in containers and Kubernetes — no capability needed
systemd socket activationsystemd (root) binds port 80 and passes the open socket to your unprivileged serviceClean; the service never holds any privilege
Ambient capability via systemdAmbientCapabilities=CAP_NET_BIND_SERVICE with User=wwwOne capability, no file xattrs to manage
File capabilitysetcap cap_net_bind_service=+ep on the binaryWorks anywhere, but must be re-applied on every upgrade

(Linux also has net.ipv4.ip_unprivileged_port_start, a per-network-namespace sysctl that lowers the privileged-port boundary. Docker sets it to 0 inside containers on recent versions, which is why a non-root container can often bind port 80 with no capability at all.)

Capabilities in systemd

systemd exposes the capability sets directly in unit files:

# /etc/systemd/system/myserver.service
[Service]
User=www
Group=www
ExecStart=/usr/local/bin/myserver --listen :80

# Hand the unprivileged process exactly one capability...
AmbientCapabilities=CAP_NET_BIND_SERVICE
# ...and make it impossible for it or any child to ever gain more.
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
NoNewPrivileges=yes

CapabilityBoundingSet= with an empty value (CapabilityBoundingSet=) drops everything, which is what most services should use. systemd-analyze security myserver.service scores a unit's exposure and lists every option you have not set — a fast way to find the gaps.

Capabilities in containers

Docker and containerd start containers with a reduced default set of 14 capabilities (listed in Docker Security Hardening). That default is a compatibility compromise; for your own services, drop everything and add back only what you need:

docker run --cap-drop=ALL --cap-add=NET_BIND_SERVICE --user 10001:10001 myimage

To discover what a program needs, run it with everything dropped and add capabilities one at a time as EPERM errors appear — or trace it with tools such as capable from the BCC toolkit (see eBPF), which logs every capability check the kernel performs.


no_new_privs: Closing the Exec Escape Hatch

All the above has a loophole: a process with few privileges can still execve() a setuid-root binary or a binary with file capabilities and gain privileges. Dropping capabilities from the bounding set limits the capability part, but setuid still changes UIDs.

The no_new_privs flag closes this. Once set with prctl(PR_SET_NO_NEW_PRIVS, 1):

  • execve() will never grant privileges: setuid/setgid bits are ignored and file capabilities are not applied;
  • the flag is inherited by all children and cannot be unset;
  • LSMs refuse transitions to more privileged domains on exec.
grep NoNewPrivs /proc/<pid>/status
# NoNewPrivs:     1

Why containers set it. A container may contain setuid binaries the image author forgot about. With no_new_privs, they are inert — if the container's user is 10001, nothing it executes can make it UID 0. This is what allowPrivilegeEscalation: false in Kubernetes and --security-opt no-new-privileges in Docker turn on.

Why seccomp needs it. An unprivileged process is only allowed to install a seccomp filter if no_new_privs is set (or if it holds CAP_SYS_ADMIN). Otherwise it could install a filter that makes a setuid program's privilege-dropping call silently "succeed" without doing anything, then exec that program — tricking a privileged binary into running in an unexpected state. Requiring no_new_privs guarantees that a filtered process can never exec into more privilege.

TIP

no_new_privs is nearly free and breaks almost nothing — the main casualty is sudo inside the process tree, which a service should never need. Turn it on everywhere: NoNewPrivileges=yes in systemd, allowPrivilegeEscalation: false in Kubernetes.


seccomp: Filtering System Calls

What it filters

Every interaction between a process and the kernel goes through a system call (see System Calls & Kernel Internals for the mechanism). Linux exposes over 450 of them on x86-64; a typical web service uses perhaps 60–100. Every unused syscall is kernel code an attacker could reach — and kernel vulnerabilities are usually in exactly those rarely-used, complicated corners (obscure socket families, filesystem features, keyring management).

seccomp lets a process install a filter that the kernel runs on every syscall before executing it. The filter sees the syscall number, the architecture and the raw argument values, and returns a verdict:

ActionEffect
SECCOMP_RET_ALLOWRun the syscall normally
SECCOMP_RET_ERRNOSkip it and return an error (usually EPERM)
SECCOMP_RET_KILL_PROCESS / KILL_THREADTerminate immediately
SECCOMP_RET_TRAPDeliver SIGSYS to the process (see Signals)
SECCOMP_RET_LOGAllow, but log — used for building profiles
SECCOMP_RET_USER_NOTIFAsk a supervisor process in user space to decide
SECCOMP_RET_TRACEHand off to a ptrace-based tracer

Filters can only be added, never removed, and are inherited across fork() and execve(). A process can tighten its own sandbox but never loosen it.

Strict mode vs filter mode

ModeSinceAllowed syscallsUse
Strict (SECCOMP_MODE_STRICT)2.6.12 (2005)Only read, write, _exit, sigreturnPure computation on already-open file descriptors — rarely used today
Filter (SECCOMP_MODE_FILTER, "seccomp-bpf")3.5 (2012)Whatever your BPF program allowsEverything modern: Docker, Kubernetes, systemd, Chrome, Firefox, OpenSSH

Filter programs are written in classic BPF and cannot dereference pointers — they can check that the first argument of socket() is AF_INET, but cannot inspect the contents of a path string. That is by design (checking user memory would be racy) and is why path-based restrictions belong in LSMs or Landlock, not seccomp. Nobody writes the BPF by hand; use libseccomp or a JSON profile.

The default container profile

Docker, containerd and CRI-O ship a default seccomp profile that is an allowlist of roughly 350 syscalls with everything else returning EPERM. In effect it blocks around 40–60 syscalls (the exact number shifts as new syscalls are added and argument conditions change). The blocked ones fall into a few families:

FamilyExamplesWhy blocked
Kernel and system controlkexec_load, init_module, finit_module, delete_module, rebootContainers must not change the host kernel
Clock and accountingsettimeofday, clock_settime, acctHost-global state
Mounting and namespacesmount, umount2, pivot_root, unshare and namespace-creating clone flags (unless CAP_SYS_ADMIN)Reshaping the filesystem view is a common isolation gap
Kernel keyringadd_key, request_key, keyctlKeyrings are not namespaced
Debugging other processesptrace (on older kernels), process_vm_readv/writev (without CAP_SYS_PTRACE)Cross-process memory access
Rare and historically buggyuserfaultfd, perf_event_open, bpf, open_by_handle_at, swapon, obsolete syscallsLarge kernel attack surface for little legitimate use

The profile is also capability-aware: if you add CAP_SYS_ADMIN, the syscalls it governs become allowed. That is one more reason dropping capabilities matters.

WARNING

For years, Kubernetes ran pods Unconfined — with no seccomp filter at all — unless you opted in. Since 1.27 the kubelet can default to RuntimeDefault with the --seccomp-default flag, but many clusters still do not enable it. Set seccompProfile: RuntimeDefault explicitly in every pod spec.

Generating a tight profile for your application

The default profile is a generic compromise. For high-value services, a custom allowlist can reduce reachable syscalls to the hundred or so your program actually uses. The workflow:

flowchart LR
    A["Run app under representative load"] --> B["Record syscalls used: strace -f -c, SECCOMP_RET_LOG, or an eBPF/OCI hook"]
    B --> C["Generate allowlist profile JSON"]
    C --> D["Deploy in log or complain mode"]
    D --> E{"Unexpected syscalls in audit log?"}
    E -- "yes" --> C
    E -- "no" --> F["Switch to enforce: default action ERRNO"]
# 1. See which syscalls the app actually makes during a realistic test run
strace -f -c -o /tmp/syscalls.txt ./myserver --config test.yaml
# (exercise every code path: startup, requests, TLS reload, graceful shutdown)

# 2. Tools that produce profiles directly
#    - oci-seccomp-bpf-hook (Podman):  podman run --annotation io.containers.trace-syscall=of:/tmp/profile.json ...
#    - Security Profiles Operator (Kubernetes): records profiles from running pods into SeccompProfile CRDs
#    - Inspektor Gadget: kubectl gadget advise seccomp-profile

A trimmed custom profile looks like this:

{
  "defaultAction": "SCMP_ACT_ERRNO",
  "defaultErrnoRet": 1,
  "architectures": ["SCMP_ARCH_X86_64", "SCMP_ARCH_AARCH64"],
  "syscalls": [
    {
      "names": ["accept4", "bind", "brk", "close", "epoll_ctl", "epoll_pwait",
                "exit_group", "fstat", "futex", "getrandom", "listen", "mmap",
                "munmap", "newfstatat", "openat", "read", "rt_sigaction",
                "rt_sigprocmask", "rt_sigreturn", "sendto", "recvfrom",
                "setsockopt", "socket", "write", "writev"],
      "action": "SCMP_ACT_ALLOW"
    }
  ]
}

Lessons teams learn the hard way:

  • Language runtimes call syscalls you did not write. Go, the JVM and glibc use clone3, rseq, membarrier, sched_getaffinity and others at startup or under load. Missing one often surfaces as a confusing crash minutes later, not an obvious EPERM at boot.
  • Base image and library upgrades change the set. A glibc upgrade that starts using clone3 instead of clone will break a profile that was correct yesterday. Regenerate profiles in CI and test them.
  • Prefer ERRNO over KILL as the default action for application profiles. An unexpected syscall returning EPERM is usually handled gracefully; a killed process is an outage.
  • The default profile already captures most of the value. Custom profiles are worth it for internet-facing parsers, multi-tenant workloads and anything processing untrusted files — not necessarily for every internal service.

seccomp in systemd

systemd uses predefined syscall groups so you do not have to maintain lists by hand:

# Allow the set normally needed by system services, then remove dangerous groups
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @resources @mount @debug @reboot @swap @module @obsolete
SystemCallErrorNumber=EPERM
SystemCallArchitectures=native

systemd-analyze syscall-filter @system-service prints what each group contains. SystemCallArchitectures=native blocks the 32-bit compatibility syscall table on 64-bit hosts — an entire second attack surface that most services never use.


Linux Security Modules: SELinux and AppArmor

Mandatory access control

Everything so far is either discretionary (file owners decide) or self-imposed (the process restricts itself). An LSM adds a policy that the administrator writes and the kernel enforces regardless of UID. Even a root process confined by SELinux or AppArmor can only touch what its policy allows. LSMs hook into the kernel at hundreds of points — file open, exec, socket connect, capability use, mount, ptrace — and get a final say after DAC has allowed an operation.

Several LSMs can be active at once ("stacking"): typically one major MAC module (SELinux or AppArmor) plus minor ones such as Yama (ptrace restrictions), Lockdown and Landlock. Check what is active with cat /sys/kernel/security/lsm.

SELinux vs AppArmor

SELinuxAppArmor
ModelType enforcement on labels: every process and object has a security context, policy says which types may interactPath-based profiles: each program has a profile listing which paths and operations it may use
Identity of a fileIts label (stored in an xattr) — survives renames and hard linksIts path — a file reachable by two paths may be treated differently
Default distrosRHEL, CentOS Stream, Fedora, Rocky/Alma, Amazon Linux 2023, AndroidUbuntu, Debian, SUSE
CoverageWhole system confined by default ("targeted" policy for daemons)Only programs with a loaded profile are confined
Learning curveSteep — labels, types, booleans, MLS/MCSGentle — profiles read like a list of paths
ContainersEach container gets a unique MCS category pair (container_t:s0:c123,c456), so containers cannot read each other's files even as the same UIDRuntime loads the docker-default / cri-containerd.apparmor.d profile

SELinux labels look like user:role:type:level. The type is what matters for most decisions:

ls -Z /var/www/html/index.html
# system_u:object_r:httpd_sys_content_t:s0 /var/www/html/index.html

ps -eZ | grep nginx
# system_u:system_r:httpd_t:s0   1234 ?  00:00:01 nginx

The policy contains rules like "processes of type httpd_t may read files of type httpd_sys_content_t". If nginx is compromised, it still cannot read /etc/shadow (shadow_t) or a database's files (postgresql_db_t) — no matter what Unix permissions say.

AppArmor profiles are text files in /etc/apparmor.d/:

# /etc/apparmor.d/usr.local.bin.myserver  (simplified)
/usr/local/bin/myserver {
  #include <abstractions/base>
  network inet stream,
  capability net_bind_service,
  /etc/myserver/** r,
  /var/lib/myserver/** rw,
  /var/log/myserver/*.log w,
  deny /etc/shadow r,
}

When you meet each

In practice, you meet the LSM when something that "should work" gets Permission denied despite correct Unix permissions:

  • A service on RHEL cannot read files in a non-standard directory → SELinux label is wrong.
  • A container on Fedora cannot read a bind-mounted host directory → the directory lacks the container label (the :z/:Z volume suffix relabels it).
  • A program on Ubuntu cannot write a new log location → its AppArmor profile does not list the path.

Reading a denial

SELinux logs AVC (Access Vector Cache) denials to the audit log:

type=AVC msg=audit(1731052800.123:456): avc:  denied  { read } for  pid=1234
  comm="nginx" name="index.html" dev="vda1" ino=98765
  scontext=system_u:system_r:httpd_t:s0
  tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0

Read it as: process (scontext, type httpd_t) was denied an operation (read) on an object (tcontext, type user_home_t, class file). The file was probably moved from a home directory and kept its home-directory label.

ausearch -m AVC -ts recent              # recent denials
sealert -a /var/log/audit/audit.log     # human-readable explanation and suggested fix
restorecon -Rv /var/www/html            # reset labels to policy defaults — usually the real fix
getsebool -a | grep httpd               # booleans toggle common policy options

AppArmor denials go to the kernel log:

audit: type=1400 apparmor="DENIED" operation="open" profile="/usr/local/bin/myserver"
  name="/srv/uploads/a.png" pid=4321 comm="myserver" requested_mask="w" denied_mask="w"
journalctl -k | grep apparmor=\"DENIED\"
aa-status                       # loaded profiles and their mode
aa-complain /usr/local/bin/myserver   # log-only while you fix the profile
aa-logprof                      # interactively add rules from logged denials

WARNING

The most common response to an LSM denial is setenforce 0 or disabling AppArmor — turning off the whole system's mandatory access control to fix one path. Fix the label (restorecon, semanage fcontext) or the profile instead. If you must unblock production, switch one domain or profile to permissive/complain mode (semanage permissive -a httpd_t, aa-complain) and fix it properly afterwards.


User Namespaces and Rootless Containers

Root inside is not root outside

A user namespace gives a process its own UID/GID mapping (see Namespaces & cgroups for the mechanics). UID 0 inside the namespace maps to an ordinary, unprivileged UID outside — for example 100000:

flowchart LR
    subgraph Container["Container user namespace"]
        A["UID 0 'root'"]
        B["UID 1000 'app'"]
    end
    subgraph Host["Host (initial user namespace)"]
        C["UID 100000 (no privileges)"]
        D["UID 101000 (no privileges)"]
    end
    A -- "uid_map: 0 to 100000" --> C
    B -- "uid_map: 1000 to 101000" --> D
cat /proc/<pid>/uid_map
#          0     100000      65536     <- inside-start  outside-start  count

Inside the namespace, the process has a full set of capabilities — but only over resources owned by that namespace. It can mount a tmpfs in its own mount namespace or configure its own network namespace; it cannot load kernel modules, change the host clock, or touch host files owned by real root, because the kernel checks capabilities against the namespace that owns the resource. If a process escapes the container's other boundaries, it lands on the host as UID 100000 — a nobody.

Rootless containers

Rootless goes one step further: the container runtime itself (Podman, rootless Docker, rootless containerd/nerdctl, BuildKit) runs as an ordinary user, using user namespaces plus helpers (newuidmap, slirp4netns/pasta for networking). A vulnerability in the runtime or daemon then yields an unprivileged user, not root.

Rootful (default Docker)Rootful + userns-remapRootless
Daemon runs asrootrootyour user
Container UID 0 on the hostUID 0Unprivileged rangeUnprivileged range
Impact of runtime/daemon bugHost rootHost root (daemon)One user account
Trade-offs—Volume ownership mappingSome networking and storage limitations, no binding ports < 1024 without sysctl

In Kubernetes, user namespaces for pods arrived via hostUsers: false in the pod spec (beta and enabled by default from 1.30, with runtime and filesystem support requirements). Each pod gets a distinct, non-overlapping UID range on the node.

INFO

User namespaces also expand the kernel attack surface: an unprivileged process can create one and then reach code paths (filesystem mounts, netfilter configuration) that previously only real root could reach. Several kernel vulnerabilities have been reachable this way, which is why some distributions restrict unprivileged user namespace creation (kernel.unprivileged_userns_clone, Ubuntu's AppArmor-based restriction) and why the default container seccomp profile blocks unshare for containers. Using user namespaces for containers is a clear win; letting arbitrary untrusted code create them is a separate decision.


Landlock: Unprivileged Self-Sandboxing

Every mechanism so far is configured from the outside — by root, systemd or the container runtime. Landlock (merged in Linux 5.13, 2021) lets an ordinary, unprivileged process sandbox itself, restricting its own filesystem access (and, since 6.7, TCP bind/connect ports) using a stackable LSM.

The use case is an application that knows exactly what it needs: a PDF converter that should read one input file and write one output directory, a build tool that should only touch the project tree, a CLI that parses untrusted archives. After startup, it declares a ruleset and enforces it on itself:

// Sketch: after this, the process can only read /usr and read/write /var/lib/myapp
int ruleset = landlock_create_ruleset(&attr, sizeof(attr), 0);
add_path_rule(ruleset, "/usr",          LANDLOCK_ACCESS_FS_READ_FILE | LANDLOCK_ACCESS_FS_READ_DIR);
add_path_rule(ruleset, "/var/lib/myapp", READ | WRITE);
prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0);     // required, as with seccomp
landlock_restrict_self(ruleset, 0);          // irreversible; inherited by children

Key properties: it needs no privileges, restrictions can only be tightened, it composes with seccomp (seccomp limits which syscalls, Landlock limits which paths and ports those syscalls may touch), and libraries exist for Rust (landlock crate), Go (go-landlock) and C. systemd and several sandboxing tools use it, and it is a good fit for any program that processes untrusted input.


Kubernetes: Putting It Together in securityContext

Kubernetes exposes nearly every layer through the pod and container securityContext. This is a hardened baseline suitable for most stateless services:

apiVersion: v1
kind: Pod
metadata:
  name: api
spec:
  automountServiceAccountToken: false        # only mount a token if the app calls the API
  securityContext:                            # pod level: applies to all containers
    runAsNonRoot: true                        # kubelet refuses to start a container that would run as UID 0
    runAsUser: 10001
    runAsGroup: 10001
    fsGroup: 10001                            # group ownership for mounted volumes
    seccompProfile:
      type: RuntimeDefault                    # the container runtime's default filter
  containers:
    - name: api
      image: registry.example.com/api@sha256:...   # pin by digest
      ports:
        - containerPort: 8080                 # unprivileged port; the Service maps 80/443
      securityContext:                        # container level
        allowPrivilegeEscalation: false       # sets no_new_privs
        readOnlyRootFilesystem: true
        privileged: false
        capabilities:
          drop: ["ALL"]
          # add: ["NET_BIND_SERVICE"]         # only if the app truly must bind < 1024
      volumeMounts:
        - name: tmp
          mountPath: /tmp
      resources:
        limits:
          memory: "512Mi"
  volumes:
    - name: tmp
      emptyDir:
        medium: Memory
        sizeLimit: 64Mi

What each field maps to in the kernel:

FieldKernel mechanism
runAsNonRoot, runAsUser, runAsGroupProcess UID/GID
allowPrivilegeEscalation: falseno_new_privs
capabilities.drop: ["ALL"]Empty bounding, permitted and effective sets
seccompProfile: RuntimeDefaultseccomp filter
readOnlyRootFilesystem: trueRoot filesystem mounted ro
appArmorProfile / seLinuxOptionsLSM profile or label
hostUsers: falseUser namespace
privileged: true (avoid)All of the above disabled

Also keep hostNetwork, hostPID, hostIPC and hostPath volumes out of application pods — each one punches a hole in a namespace. Pair the pod hardening with tight RBAC for the service account and Network Policies for traffic.

Pod Security Standards

Kubernetes defines three standard profiles, enforced by the built-in Pod Security Admission controller per namespace:

LevelIntentKey restrictions
PrivilegedUnrestricted — system components like CNI agents and storage driversNone
BaselineBlocks known privilege escalations while staying compatible with most imagesNo privileged, no host namespaces, no hostPath, capabilities limited to the default set, no unsafe sysctls, no Unconfined seccomp override
RestrictedCurrent hardening best practiceBaseline plus runAsNonRoot, allowPrivilegeEscalation: false, drop ALL (only NET_BIND_SERVICE may be added), seccomp RuntimeDefault or Localhost, restricted volume types
# Enforce restricted; warn and audit at the same level so violations are visible
kubectl label namespace payments \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/audit=restricted

# Preview what would break before enforcing
kubectl label --dry-run=server --overwrite namespace payments \
  pod-security.kubernetes.io/enforce=restricted

Note that restricted does not require readOnlyRootFilesystem — add that via policy engines (Kyverno, Gatekeeper) or admission webhooks if you want it enforced. A practical rollout: label every namespace baseline (enforce) and restricted (warn), fix workloads as warnings appear, then raise enforcement namespace by namespace.


Defense in Depth

No single layer is sufficient. Each one assumes the others might fail, and each targets a different class of risk:

LayerWhat it removesClass of risk it reduces
Non-root userRoot's identity and default capabilitiesApp compromise → full control of files and processes; most container-escape preconditions
Capabilities droppedSpecific privileged operationsNetwork reconfiguration, ptrace, module loading, mount-based escapes
no_new_privsGaining privileges through execAbuse of leftover setuid binaries and file capabilities
seccompReachable kernel syscallsKernel vulnerabilities in rarely-used syscalls; many escape techniques need a blocked call
LSM (SELinux/AppArmor)Access the policy does not explicitly allow, even for rootLateral movement to other files, containers or host resources after compromise
User namespacesHost-level meaning of container UID 0Escapes landing as host root; runtime bugs yielding host root
Read-only root filesystemAbility to modify the image at runtimePersistence, dropped tools, binary tampering
nosuid/nodev/noexec mountsPrivileged or executable files in writable locationsPlanting privileged binaries or device nodes
LandlockFilesystem and port access beyond declared needsExploited parser reaching unrelated files
flowchart LR
    A["Bug in application code"] --> B["Attacker runs code as app UID"]
    B --> C{"Non-root + caps dropped?"}
    C -- "yes" --> D["Cannot mount, load modules or ptrace"]
    B --> E{"seccomp RuntimeDefault?"}
    E -- "yes" --> F["Most kernel attack surface unreachable"]
    B --> G{"LSM profile?"}
    G -- "yes" --> H["Cannot read other containers or host files"]
    B --> I{"User namespace?"}
    I -- "yes" --> J["Any escape lands as unprivileged host UID"]

The value of layering is visible in the history of container-runtime vulnerabilities. CVE-2019-5736 in runc allowed a malicious container to overwrite the host's runc binary under certain conditions, gaining host root. At a high level, the published mitigations were the layers on this page: containers running as a non-root user were not affected in the common configuration, user namespaces prevented the container's root from having write access to the host binary, and SELinux policy on several distributions blocked the write in the default configuration. Teams that had already layered these defenses were protected before a patch existed. Later runtime vulnerabilities (such as the 2024 "Leaky Vessels" set in runc and BuildKit) followed the same pattern — non-root, user namespaces and LSMs narrowed or eliminated the impact. Details of these issues are discussed in Container Escape Techniques; the lesson here is that you cannot patch vulnerabilities you do not know about, but you can reduce what any of them yields.

War Story (illustrative)

A team ran an image-thumbnail service as root in a container, with Docker's default capabilities and no seccomp override in Kubernetes (so the pod was Unconfined). A vulnerability in the image-decoding library let a crafted upload execute code in the service. The attacker could read every secret mounted into the pod and, because the process was root with CAP_NET_RAW, sniff traffic in the pod's network namespace. The post-incident hardening took a single afternoon: runAsUser: 10001, capabilities.drop: ["ALL"], allowPrivilegeEscalation: false, seccompProfile: RuntimeDefault, readOnlyRootFilesystem: true, the namespace labelled restricted, and the decoder moved into a separate low-privilege sidecar with a Landlock policy limiting it to its input directory. Nothing broke except a debug script that had been writing to /app/logs — which moved to an emptyDir.


Production Hardening Checklist

Identity

  • Service runs as a dedicated, non-root UID (not nobody, which is shared) — in containers, a fixed numeric UID ≥ 10000
  • No login shell and no password for service accounts
  • Privilege drop, if done in code, uses setgroups + setresgid + setresuid and verifies it cannot regain root

Filesystem

  • Root filesystem read-only; writable paths explicitly mounted (emptyDir, tmpfs, StateDirectory=)
  • Writable mounts are nosuid,nodev (and noexec where practical)
  • Config and secrets readable only by the service user (0600/0640)
  • No setuid/setgid binaries in container images; setuid inventory on hosts tracked and alerted on change
  • File capabilities inventoried with getcap -r /

Privileges

  • All capabilities dropped; only named, justified ones added back
  • No CAP_SYS_ADMIN, CAP_SYS_PTRACE, CAP_SYS_MODULE, CAP_NET_ADMIN without a documented reason
  • no_new_privs set (NoNewPrivileges=yes, allowPrivilegeEscalation: false, no-new-privileges)
  • No --privileged / privileged: true outside a short, reviewed list of system components

Kernel surface

  • seccomp enabled — at least the runtime default; custom profiles for internet-facing parsers
  • systemd services use SystemCallFilter=@system-service and SystemCallArchitectures=native
  • SELinux enforcing or AppArmor profiles loaded — never globally disabled to fix a single denial

Isolation

  • No host namespaces (hostNetwork, hostPID, hostIPC) or hostPath mounts in application pods
  • No container runtime socket mounted into workloads
  • User namespaces or rootless runtimes where supported (hostUsers: false, Podman, rootless BuildKit for CI)

Verification

  • systemd-analyze security score reviewed for each service
  • Pod Security Admission restricted (warn/audit everywhere, enforce where possible)
  • CI checks manifests (Kyverno/Gatekeeper/kube-score) and images (see Docker Security Hardening)

Decision Guide

systemd service on a VM

User=myapp
Group=myapp
DynamicUser=yes                  # or: a static user; DynamicUser allocates a throwaway UID
NoNewPrivileges=yes
CapabilityBoundingSet=           # none; add CAP_NET_BIND_SERVICE only if needed
AmbientCapabilities=
ProtectSystem=strict             # whole filesystem read-only...
ReadWritePaths=/var/lib/myapp    # ...except what you list
StateDirectory=myapp
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectClock=yes
RestrictNamespaces=yes
RestrictSUIDSGID=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes       # remove for JIT runtimes (JVM, Node, .NET)
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @resources
SystemCallArchitectures=native

Then run systemd-analyze security myapp.service and aim for a score in the "OK" range or better.

Choosing per platform

SituationStart withAdd when
systemd serviceBlock above; systemd-analyze securityAppArmor/SELinux profile for internet-facing daemons; Landlock if the app supports it
Docker / containerd--user 10001, --cap-drop=ALL, --security-opt no-new-privileges, --read-only, default seccomp and AppArmor/SELinuxCustom seccomp profile for high-risk services; rootless or userns-remap for multi-tenant hosts and CI builders
Kubernetes app podsThe securityContext above; namespace labelled restrictedhostUsers: false; Localhost seccomp profiles via Security Profiles Operator
Kubernetes system pods (CNI, CSI, monitoring agents)Isolated namespace at privileged; grant only the specific capabilities and host mounts each agent documentsReview on every upgrade; never co-locate application workloads in that namespace
Untrusted or multi-tenant codeAll of the aboveA stronger boundary than namespaces: gVisor, Kata Containers or Firecracker microVMs (see Container Escape Techniques)
flowchart TD
    A["New workload"] --> B{"Runs third-party or tenant code?"}
    B -- "yes" --> C["Sandboxed runtime: gVisor, Kata, microVM + full hardening"]
    B -- "no" --> D{"Kubernetes?"}
    D -- "yes" --> E["restricted PSS + hardened securityContext"]
    D -- "no" --> F{"Container?"}
    F -- "yes" --> G["Non-root, cap-drop ALL, no-new-privileges, read-only, default seccomp"]
    F -- "no" --> H["systemd sandboxing options + systemd-analyze security"]
    E --> I{"Parses untrusted input?"}
    G --> I
    H --> I
    I -- "yes" --> J["Custom seccomp profile + LSM profile + Landlock where possible"]
    I -- "no" --> K["Defaults are enough. Review yearly."]

Interview Questions

Q: What is the difference between real, effective and saved UIDs? The real UID is who started the process; the effective UID is used for most permission checks; the saved UID stores a previous effective UID so a program can drop privilege temporarily and regain it. setuid binaries set the effective (and saved) UID to the file owner while leaving the real UID as the caller. To drop privileges permanently, all three must be changed (setresuid).

Q: Why is CAP_SYS_ADMIN called "the new root"? It guards a huge, loosely related set of operations — mounting filesystems, creating namespaces, many device and kernel configuration interfaces — because it became the default choice for new privileged operations. Holding it is close to holding full root, so it should almost never be granted to applications.

Q: How do you let a non-root web server listen on port 80? Best: don't — listen on 8080 and let a load balancer or Kubernetes Service map 80. Otherwise: systemd socket activation, AmbientCapabilities=CAP_NET_BIND_SERVICE in the unit, or setcap cap_net_bind_service=+ep on the binary. Inside containers, ip_unprivileged_port_start is often already 0.

Q: What does allowPrivilegeEscalation: false actually do? It sets the kernel's no_new_privs flag on the container process, so execve() can never grant privileges — setuid bits and file capabilities are ignored. It is inherited and irreversible.

Q: What is the difference between seccomp and an LSM like AppArmor? seccomp filters which system calls a process can make, based on syscall number and raw argument values, and is usually self-applied or applied by the runtime. LSMs decide which objects (files, sockets, capabilities) a process can access, based on labels or paths, and are mandatory policy written by an administrator. seccomp cannot inspect file paths; LSMs do not reduce kernel syscall surface. They complement each other.

Q: If a container runs as root with user namespaces enabled, is that safe? Much safer: container UID 0 maps to an unprivileged host UID, so an escape lands with no host privileges, and capabilities only apply to resources the namespace owns. It is still worth running as non-root inside the container and keeping seccomp and LSM profiles, because user namespaces themselves expose more kernel code paths.

Q: What does the Kubernetes restricted Pod Security Standard require? Non-root, allowPrivilegeEscalation: false, all capabilities dropped (only NET_BIND_SERVICE may be added back), seccomp RuntimeDefault or Localhost, plus everything in baseline: no privileged mode, no host namespaces, no hostPath volumes.