On this page

Inter-Process Communication

Processes are isolated by design. Each has its own virtual address space (see Virtual Memory), its own file descriptor table, and no way to read another's memory. That isolation is what makes a crashing browser tab not take down the browser, and a compromised renderer not read your bank's cookies. But useful systems are built from cooperating processes, so the kernel provides deliberate, controlled holes in the wall: inter-process communication (IPC).

The Processes & Threads page introduces the basic IPC menu. This page goes deep: what each mechanism actually does inside the kernel, what it costs, how it fails, and how real systems — Chrome, PostgreSQL, Nginx, Redis, systemd, Docker — combine them.


Why Multiple Processes at All?

Threads share memory for free. Why pay for IPC?

ReasonExample
Fault isolationA segfault in one Chrome renderer kills one tab, not the browser
Security / privilege separationOpenSSH's unprivileged pre-auth child; Chrome's sandboxed renderers
Different languages / binariesA shell pipeline; a Python app talking to a Rust sidecar
Independent lifecycleRestart or upgrade a worker without dropping the master's listening socket (Nginx)
Copy-on-write snapshotsRedis fork()s to get a frozen point-in-time view of memory
Avoid shared-state bugsProcess-per-connection (PostgreSQL) — one backend's memory corruption can't silently scribble on another's private state

The cost: every byte that crosses a process boundary either goes through the kernel (a copy, a syscall, maybe a context switch) or through explicitly shared memory (which reintroduces all the synchronization problems of threads, across a fault boundary).


The Landscape

graph TD
    IPC["Same-host IPC"] --> BS["Byte streams / messages<br/>through the kernel"]
    IPC --> SM["Shared memory<br/>(no kernel on the data path)"]
    IPC --> NT["Notification only<br/>(no payload)"]

    BS --> P["Pipe / FIFO"]
    BS --> UDS["Unix domain socket<br/>stream, datagram, seqpacket"]
    BS --> MQ["POSIX / SysV message queue"]
    BS --> TCP["TCP over loopback"]

    SM --> SHM["shm_open + mmap<br/>SysV shmget"]
    SM --> MFD["memfd_create<br/>(sealable, passable fd)"]
    SM --> MMF["mmap of a shared file"]

    NT --> SIG["Signals"]
    NT --> EFD["eventfd"]
    NT --> FTX["Process-shared futex / semaphore"]

A useful mental split: kernel-mediated channels copy data through the kernel and give you synchronization for free (a read blocks until data arrives); shared memory avoids copies entirely but leaves synchronization to you; notification primitives carry no payload and are used to say "go look at the shared memory now."


Pipes

A pipe is a unidirectional, in-kernel byte buffer with a read end and a write end.

int fds[2];
pipe2(fds, O_CLOEXEC);          // fds[0] = read end, fds[1] = write end

if (fork() == 0) {               // child: writer
    close(fds[0]);
    const char msg[] = "hello from child\n";
    write(fds[1], msg, sizeof msg - 1);
    _exit(0);
}
close(fds[1]);                   // parent: reader. MUST close its copy of the write end
char buf[128];
ssize_t n;
while ((n = read(fds[0], buf, sizeof buf)) > 0)
    fwrite(buf, 1, n, stdout);   // read() returns 0 (EOF) only when ALL write ends are closed

Pipes are anonymous: they exist only as file descriptors, so they connect only related processes — a parent and its children, which inherit the fds across fork(). This is exactly what a shell does for ls | grep foo: pipe(), fork() twice, dup2() the ends onto stdout and stdin, exec().

Pipe Internals and Buffer Size

On Linux a pipe is a ring of pipe_buffer slots, each referencing a page. The default capacity is 16 pages = 65,536 bytes (since Linux 2.6.11). When the buffer is full, write() blocks (or returns EAGAIN if O_NONBLOCK); when empty, read() blocks.

# Check and change a pipe's capacity (Linux-specific fcntl)
$ python3 -c 'import os,fcntl; r,w=os.pipe(); print(fcntl.fcntl(w, 1032))'   # F_GETPIPE_SZ
65536

$ cat /proc/sys/fs/pipe-max-size          # ceiling for F_SETPIPE_SZ without CAP_SYS_RESOURCE
1048576
$ cat /proc/sys/fs/pipe-user-pages-soft   # per-user total; beyond this new pipes shrink to 2 pages
16384
fcntl(fds[1], F_SETPIPE_SZ, 1 << 20);   // 1 MB pipe — fewer wakeups for bulk streaming

The 64 KB default is why a producer that outruns its consumer stalls quickly — and why a child that writes a lot to stderr while the parent only reads stdout deadlocks: the child blocks on a full stderr pipe, the parent blocks waiting for stdout EOF. Every subprocess library (subprocess.communicate() in Python, cmd.Output() in Go) exists partly to read both streams concurrently.

Atomicity: PIPE_BUF

POSIX guarantees writes of ≤ PIPE_BUF bytes (4,096 on Linux) to a pipe are atomic — they will not be interleaved with other writers' data. Larger writes may be split and interleaved. This is why multiple processes can safely append short log lines to the same pipe, but a 10 KB JSON blob from two writers can come out spliced together.

SIGPIPE and EPIPE

Writing to a pipe (or socket) whose read end is closed delivers SIGPIPE, whose default action is to terminate the process silently. If SIGPIPE is ignored or blocked, write() instead fails with EPIPE.

$ yes | head -1
y
# 'yes' was killed by SIGPIPE when head exited — exactly the intended behaviour for CLI pipelines
$ yes | head -1; echo "${PIPESTATUS[@]}"
y
141 0          # 141 = 128 + 13 (SIGPIPE)

For CLI filters, dying on SIGPIPE is the right thing. For servers it is a classic outage: a client disconnects mid-response and the whole server dies. Long-running servers ignore it (signal(SIGPIPE, SIG_IGN)) or pass MSG_NOSIGNAL to send(). Go's runtime handles this for you: SIGPIPE on fd 1 or 2 kills the program like C, but on any other fd it is turned into an EPIPE error. See Signals.

Named Pipes (FIFOs)

A FIFO is a pipe with a name in the filesystem, so unrelated processes can rendezvous:

$ mkfifo /tmp/jobs
$ ls -l /tmp/jobs
prw-r--r-- 1 ayush ayush 0 Nov  8 10:12 /tmp/jobs      # 'p' = pipe, size always 0

# Terminal 1                        # Terminal 2
$ cat /tmp/jobs                     $ echo "build #42" > /tmp/jobs
build #42

open() on a FIFO blocks until the other side opens too (unless O_NONBLOCK). The data never touches disk; the inode is only a meeting point. FIFOs are still unidirectional and have no notion of separate clients — two writers' messages interleave in one stream — which is why anything beyond simple scripting moves to Unix domain sockets.


Unix Domain Sockets

A Unix domain socket (UDS, AF_UNIX) uses the sockets API — socket, bind, listen, accept, connect, send, recv — but stays entirely inside the kernel. No IP headers, no checksums, no TCP state machine, no congestion control, no routing. The kernel simply moves buffers from the sender's socket to the receiver's queue.

// Server
int s = socket(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0);
struct sockaddr_un addr = { .sun_family = AF_UNIX };
strncpy(addr.sun_path, "/run/myapp.sock", sizeof addr.sun_path - 1);   // max 108 bytes incl. NUL
unlink(addr.sun_path);                         // stale socket file from a previous run
bind(s, (struct sockaddr *)&addr, sizeof addr);
chmod(addr.sun_path, 0660);                    // filesystem permissions = access control
listen(s, 128);
int c = accept4(s, NULL, NULL, SOCK_CLOEXEC);

Three Socket Types

TypeSemanticsUse
SOCK_STREAMConnection-oriented byte stream, like TCPMost daemons: Docker, PostgreSQL local connections, MySQL, containerd, gRPC over UDS
SOCK_DGRAMConnectionless datagrams — reliable and ordered on Linux (unlike UDP), message boundaries preservedsyslog (/dev/log), systemd's journal and sd_notify socket
SOCK_SEQPACKETConnection-oriented, reliable, ordered, with message boundariesProtocols that want framed messages without writing a length-prefix parser

Naming: Paths and the Abstract Namespace

  • Filesystem path (/run/docker.sock) — access controlled by file permissions and directory permissions. The file must be unlink()ed before rebinding, or bind() fails with EADDRINUSE.
  • Abstract namespace (Linux only) — sun_path[0] = '\0' followed by a name. No file, disappears when the last socket closes, no stale-file cleanup. Scoped to the network namespace, not the filesystem — so it is not protected by file permissions, and containers sharing the host network namespace can reach host abstract sockets (a recurring container-escape vector).
  • Unnamed — socketpair(AF_UNIX, SOCK_STREAM, 0, sv) returns two connected sockets with no name at all; the bidirectional equivalent of pipe(), ideal for parent/child channels.
$ ss -xlp | head
Netid State  Recv-Q Send-Q Local Address:Port                       Peer Address:Port  Process
u_str LISTEN 0      4096   /run/containerd/containerd.sock 19804    * 0          users:(("containerd",pid=812,fd=8))
u_str LISTEN 0      4096   /var/run/docker.sock 23211               * 0          users:(("systemd",pid=1,fd=47))
u_dgr UNCONN 0      0      /run/systemd/notify 14012                * 0          users:(("systemd",pid=1,fd=33))
u_str LISTEN 0      4096   @/tmp/.X11-unix/X0 31877                 * 0          users:(("Xorg",pid=1503,fd=5))
#                          ^ '@' marks an abstract-namespace socket

Credentials: Who Is on the Other End?

Because both ends are on the same kernel, the kernel can tell you exactly who connected — something no network protocol can do without cryptography.

struct ucred cred;
socklen_t len = sizeof cred;
getsockopt(client_fd, SOL_SOCKET, SO_PEERCRED, &cred, &len);
printf("peer pid=%d uid=%d gid=%d\n", cred.pid, cred.uid, cred.gid);

PostgreSQL's peer authentication (pg_hba.conf: local all all peer) uses exactly this: the OS user connecting through /var/run/postgresql/.s.PGSQL.5432 is the database user, no password involved. polkit, D-Bus and systemd make authorization decisions the same way. Linux 6.5 added SO_PEERPIDFD, which returns a pidfd for the peer so the check cannot be fooled by PID reuse.

Passing File Descriptors: SCM_RIGHTS

The superpower of Unix domain sockets: you can send open file descriptors to another process. The kernel duplicates the underlying open file description into the receiver's fd table — same as dup(), but across processes. The receiver gets a new fd number pointing to the same open file, socket, pipe, memfd, or device.

sequenceDiagram
    participant A as Process A fd table
    participant K as Kernel
    participant B as Process B fd table

    Note over A: fd 7 refers to open TCP listening socket
    A->>K: sendmsg with SCM_RIGHTS control message listing fd 7
    K->>K: take a reference on the struct file, queue it with the message
    K->>B: recvmsg delivers the message
    Note over B: new fd 4 installed, same struct file as A fd 7
    A->>A: close fd 7, socket stays open because B holds a reference
// Send one fd over a connected AF_UNIX socket
int send_fd(int sock, int fd_to_send) {
    char dummy = 'F';
    struct iovec iov = { .iov_base = &dummy, .iov_len = 1 };   // must send >= 1 byte of data
    union { char buf[CMSG_SPACE(sizeof(int))]; struct cmsghdr align; } u;
    struct msghdr msg = {
        .msg_iov = &iov, .msg_iovlen = 1,
        .msg_control = u.buf, .msg_controllen = sizeof u.buf,
    };
    struct cmsghdr *cm = CMSG_FIRSTHDR(&msg);
    cm->cmsg_level = SOL_SOCKET;
    cm->cmsg_type  = SCM_RIGHTS;
    cm->cmsg_len   = CMSG_LEN(sizeof(int));
    memcpy(CMSG_DATA(cm), &fd_to_send, sizeof(int));
    return sendmsg(sock, &msg, 0);
}

int recv_fd(int sock) {
    char dummy;
    struct iovec iov = { .iov_base = &dummy, .iov_len = 1 };
    union { char buf[CMSG_SPACE(sizeof(int))]; struct cmsghdr align; } u;
    struct msghdr msg = {
        .msg_iov = &iov, .msg_iovlen = 1,
        .msg_control = u.buf, .msg_controllen = sizeof u.buf,
    };
    if (recvmsg(sock, &msg, MSG_CMSG_CLOEXEC) <= 0) return -1;
    struct cmsghdr *cm = CMSG_FIRSTHDR(&msg);
    if (!cm || cm->cmsg_type != SCM_RIGHTS) return -1;
    int fd;
    memcpy(&fd, CMSG_DATA(cm), sizeof(int));
    return fd;
}

Go wraps this in syscall.UnixRights / syscall.ParseUnixRights, used with (*net.UnixConn).WriteMsgUnix:

func sendListener(conn *net.UnixConn, ln *net.TCPListener) error {
    f, err := ln.File()            // dup of the listening socket's fd
    if err != nil { return err }
    defer f.Close()
    rights := syscall.UnixRights(int(f.Fd()))
    _, _, err = conn.WriteMsgUnix([]byte("L"), rights, nil)
    return err
}

Why this matters in practice:

  • Zero-downtime restarts. The old process hands its listening sockets to the new process, so the kernel's accept queue is never closed and no connection is refused. HAProxy's seamless reload (expose-fd listeners on the stats socket, since 1.8) and Envoy's hot restart both transfer listeners over a Unix socket with SCM_RIGHTS.
  • Privilege separation. A privileged helper opens a device, a raw socket, or a file the sandboxed process may not open itself, and passes back just that fd. Chrome's sandboxed renderers receive their fds this way; Linux's systemd-logind hands input and DRM device fds to unprivileged Wayland compositors (TakeDevice over D-Bus, which carries fds via SCM_RIGHTS).
  • Container runtimes. runc --console-socket receives the pseudo-terminal master fd for the container from the runtime process via SCM_RIGHTS; containerd's shim and Docker's docker exec -t path rely on this to wire your terminal into the container.
  • systemd's file-descriptor store. A service can push fds to PID 1 with sd_pid_notify_with_fds(..., "FDSTORE=1", fds, n) over the notify socket; systemd holds them across the service's restart and hands them back, so in-flight connections survive a crash or upgrade.

How systemd socket activation passes sockets

Socket activation itself uses plain fd inheritance, not SCM_RIGHTS: systemd binds and listens on the socket from the .socket unit, and when the first connection arrives it fork()s and exec()s the service with the listening sockets already open as fds 3, 4, ... and sets LISTEN_FDS=n and LISTEN_PID=pid in its environment. The service calls sd_listen_fds() (or just uses fd 3). Because systemd owns the socket, connections queue in the kernel's backlog while the service is (re)starting — the basis for both on-demand start and restarts that drop nothing. Docker's own docker.socket unit works this way: in the ss output above, PID 1 is the process listening on /var/run/docker.sock.

fd passing pitfalls

  • Always send at least one byte of real data with the control message; SOCK_STREAM will not deliver an ancillary-only message.
  • If the receiver's control buffer is too small, the fds are silently closed and MSG_CTRUNC is set — check it.
  • Received fds count against the receiver's RLIMIT_NOFILE; an untrusted peer can exhaust your fd table. Limit what you accept.
  • In-flight fds can form reference cycles (a socket sent over itself), which is why Linux has a dedicated garbage collector for AF_UNIX (net/unix/garbage.c) — historically a source of kernel CVEs.

Shared Memory

Every mechanism above copies data at least twice: sender's buffer → kernel → receiver's buffer. For large payloads (video frames, database pages, ML tensors) the copies dominate. Shared memory maps the same physical pages into multiple address spaces. After setup, communication is just memory loads and stores — no syscalls, no copies, no kernel involvement.

graph LR
    subgraph "Process A virtual memory"
        VA["0x7f3a2c000000<br/>mapping"]
    end
    subgraph "Physical RAM"
        PF["Frames 0x1a3f..0x1a4e<br/>(tmpfs pages of /dev/shm/ring)"]
    end
    subgraph "Process B virtual memory"
        VB["0x7f91e8000000<br/>mapping"]
    end
    VA --> PF
    VB --> PF

The two processes typically map the region at different virtual addresses, so you cannot store raw pointers in shared memory — store offsets from the base instead.

POSIX Shared Memory

// writer.c — gcc writer.c -o writer (add -lrt on glibc < 2.34)
#include <fcntl.h>
#include <sys/mman.h>
#include <unistd.h>
#include <string.h>

int main(void) {
    int fd = shm_open("/telemetry", O_CREAT | O_RDWR, 0600);   // creates /dev/shm/telemetry
    ftruncate(fd, 1 << 20);                                      // 1 MB
    char *p = mmap(NULL, 1 << 20, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
    close(fd);                                                   // mapping keeps it alive
    strcpy(p, "frame 1");
    // ... shm_unlink("/telemetry") when the segment is no longer needed
}
$ ls -l /dev/shm
-rw------- 1 ayush ayush 1048576 Nov  8 10:40 telemetry
$ df -h /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            16G  1.0M   16G   1% /dev/shm

On Linux, shm_open is just open() on the tmpfs mounted at /dev/shm. The pages are swap-backed RAM, counted as Shmem in /proc/meminfo, and they persist until shm_unlink() and every mapping is gone — a crashed program that never unlinks leaks RAM until reboot.

Docker's 64 MB /dev/shm

Containers get a private /dev/shm of 64 MB by default. PostgreSQL (dynamic shared memory for parallel query), Chrome/Puppeteer, and PyTorch DataLoader workers all use /dev/shm and fail with confusing errors — could not resize shared memory segment ... No space left on device, or a renderer crash — once it fills. Fix with docker run --shm-size=1g or, in Kubernetes, an emptyDir with medium: Memory mounted at /dev/shm.

System V Shared Memory

The older API from 1980s AT&T Unix: shmget(key, size, IPC_CREAT | 0600), shmat(), shmdt(), shmctl(IPC_RMID). Segments are named by integer keys (often from ftok()), live in a separate kernel namespace rather than the filesystem, and are inspected with ipcs:

$ ipcs -m
------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x00000000 32768      postgres   600        56         6          dest
0x0052e2c1 32769      oracle     640        12884901888 143

$ cat /proc/sys/kernel/shmmax   # max segment size
18446744073692774399
POSIX (shm_open)System V (shmget)
Naming/name — a file in /dev/shmInteger key
HandleFile descriptor (works with mmap, fstat, fchmod, fd passing)shmid integer
Inspect / clean upls /dev/shm, rmipcs -m, ipcrm -m
Limitstmpfs sizekernel.shmmax, shmall, shmmni sysctls
Huge pagesVia hugetlbfs mountSHM_HUGETLB flag
RecommendationUse for new codeLegacy software, or when huge-page SysV segments are needed

PostgreSQL is a good example of the migration: before 9.3 it allocated all of shared_buffers as one System V segment, which on many systems required raising kernel.shmmax before Postgres would start. Since 9.3 it keeps only a tiny SysV segment (used as an interlock to detect a still-running old postmaster — the 56-byte postgres segment above) and maps the bulk with anonymous shared mmap before forking backends.

Synchronizing Shared Memory

Shared memory gives you a data race across a process boundary. You need the same tools as threads — and they must live in the shared region:

struct shared {
    pthread_mutex_t lock;
    pthread_cond_t  ready;
    uint64_t        seq;
    char            payload[4096];
};

// Initialize once, in the creating process
pthread_mutexattr_t ma;
pthread_mutexattr_init(&ma);
pthread_mutexattr_setpshared(&ma, PTHREAD_PROCESS_SHARED);   // futex keyed on physical page
pthread_mutexattr_setrobust(&ma, PTHREAD_MUTEX_ROBUST);      // survive holder death
pthread_mutex_init(&s->lock, &ma);

// Any process
int r = pthread_mutex_lock(&s->lock);
if (r == EOWNERDEAD) {
    repair_invariants(s);                  // previous holder crashed mid-update
    pthread_mutex_consistent(&s->lock);
}

Options, from heaviest to lightest:

  1. Process-shared pthread mutex + condvar — easy, futex-backed, uncontended locking costs no syscalls. Make it robust, or one crashed process hangs everyone forever.
  2. POSIX named or unnamed semaphores (sem_open, or sem_init(&s, 1, 0) inside the shared region).
  3. Lock-free single-producer/single-consumer ring buffer — head and tail indexes as atomics with acquire/release ordering, on separate cache lines. This is how LMAX-style and high-frequency-trading IPC reaches ~100 ns latency. See Lock-Free Data Structures and Kernel Synchronization for the memory-ordering rules.
  4. Seqlock for a frequently-updated snapshot (a market data tick, a metrics block) read by many processes.

To avoid busy-polling a ring buffer, pair it with a notification primitive — an eventfd, a futex, or a byte on a pipe — that the consumer can sleep on when the ring is empty.


Message Queues

POSIX Message Queues

A kernel-managed queue of discrete, prioritized messages:

#include <mqueue.h>

struct mq_attr attr = { .mq_maxmsg = 10, .mq_msgsize = 512 };
mqd_t q = mq_open("/orders", O_CREAT | O_RDWR, 0600, &attr);

mq_send(q, "BUY 100 ACME", 13, /*priority*/ 5);

char buf[512];
unsigned prio;
ssize_t n = mq_receive(q, buf, sizeof buf, &prio);   // highest priority first, FIFO within a priority
$ sudo mount -t mqueue none /dev/mqueue     # usually already mounted
$ cat /dev/mqueue/orders
QSIZE:13         NOTIFY:0     SIGNO:0     NOTIFY_PID:0

$ cat /proc/sys/fs/mqueue/msg_max /proc/sys/fs/mqueue/msgsize_max /proc/sys/fs/mqueue/queues_max
10
8192
256

Features: message boundaries, priorities, mq_notify() (signal or thread on arrival), and on Linux the descriptor is an fd you can put in epoll. Limitations: tiny default limits (10 messages of 8 KB, adjustable by sysctl and RLIMIT_MSGQUEUE), Linux/POSIX-only, no fan-out, no persistence — the queue is gone on reboot. They are popular in embedded and real-time systems, rare in server software.

System V Message Queues

msgget / msgsnd / msgrcv, with messages tagged by a long type that receivers can filter on. Visible with ipcs -q. Effectively legacy.

Not the same as Kafka or RabbitMQ

Kernel message queues are a same-host, in-memory, non-durable transport. They share a name with distributed message brokers but solve a different problem. If you need durability, multiple hosts, consumer groups or replay, you want a broker — see Message Queues.


Signals as IPC

Signals can carry a notification between processes (kill(pid, SIGUSR1)), and many daemons use them as a control plane: SIGHUP to reload configuration, SIGUSR1 to reopen log files, SIGTERM to shut down gracefully. But as a data channel they are poor:

LimitationDetail
No payload (standard signals)Just the signal number
Not queuedStandard signals are a pending bit. Send SIGUSR1 five times while it's blocked → delivered once
Async handler restrictionsHandlers may only call async-signal-safe functions
Interrupts syscallsEINTR everywhere unless SA_RESTART
Real-time signals help a littleSIGRTMIN..SIGRTMAX are queued and sigqueue() attaches one int or pointer-sized value — but the queue is bounded by RLIMIT_SIGPENDING

Use signals for lifecycle control, not data. Modern code converts them into ordinary events with signalfd (an fd readable in epoll) or the self-pipe trick. All the details are in Signals & Process Control.


Modern Linux Primitives: memfd and eventfd

memfd_create

memfd_create() (Linux 3.17) creates an anonymous, RAM-backed file and returns an fd. It is like shm_open without a name — nothing in /dev/shm to leak or collide on, and it disappears when the last fd and mapping close. You share it by passing the fd over a Unix socket.

Its key feature is sealing. A receiver of shared memory from an untrusted sender has a problem: the sender can ftruncate() the file smaller while the receiver is reading it, causing SIGBUS in the receiver, or modify contents after the receiver validated them. Seals make the file immutable in specific ways, enforced by the kernel:

int fd = memfd_create("frame", MFD_CLOEXEC | MFD_ALLOW_SEALING);
ftruncate(fd, size);
void *p = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
render_frame(p);
munmap(p, size);

// Lock it down before handing to the other process
fcntl(fd, F_ADD_SEALS, F_SEAL_SHRINK | F_SEAL_GROW | F_SEAL_WRITE | F_SEAL_SEAL);
send_fd(sock, fd);

// Receiver verifies seals before trusting the buffer
int seals = fcntl(fd, F_GET_SEALS);
if ((seals & (F_SEAL_SHRINK | F_SEAL_WRITE)) != (F_SEAL_SHRINK | F_SEAL_WRITE))
    reject();

Wayland clients share pixel buffers with the compositor this way (wl_shm pools are typically memfds), and systemd uses sealed memfds to pass large journal payloads. memfd_create + fexecve is also how some tools run a binary without writing it to disk — which makes memfd: entries in /proc/PID/maps an interesting thing for security monitoring to watch.

eventfd

eventfd() (Linux 2.6.22) is a kernel object holding a 64-bit counter, exposed as an fd. write() adds to the counter; read() returns the counter and resets it to zero (or decrements by one with EFD_SEMAPHORE), blocking if it is zero. It is the cheapest possible "wake up" between threads or processes, and it plugs into epoll.

int efd = eventfd(0, EFD_CLOEXEC | EFD_NONBLOCK);

// Producer (after writing into a shared ring)
uint64_t one = 1;
write(efd, &one, sizeof one);

// Consumer event loop
struct epoll_event ev = { .events = EPOLLIN, .data.fd = efd };
epoll_ctl(ep, EPOLL_CTL_ADD, efd, &ev);
// ... epoll_wait returns, then:
uint64_t n;
read(efd, &n, sizeof n);     // n = number of notifications coalesced since last read
drain_ring();

Compared to the old self-pipe trick it uses one fd instead of two, 8 bytes of kernel state instead of a pipe buffer, and coalesces naturally. KVM uses eventfds (ioeventfd/irqfd) to signal between guests and device emulators like vhost; io_uring can post completion notifications to an eventfd; libuv and Tokio use it (or a pipe on non-Linux) to wake their event loops from other threads.


Performance: Real Numbers

IPC performance has two separate axes: latency for small messages (dominated by syscalls and wakeups) and throughput for bulk data (dominated by copies).

Small-Message Round Trip

Approximate one-way-and-back latency for a 64-byte ping-pong between two processes pinned to different cores on the same socket, modern x86 server, Linux 6.x. These are representative ranges from benchmarks such as ipc-bench and published comparisons — illustrative, measure on your own hardware:

MechanismRound-trip latencyWhy
Shared memory, busy-polling SPSC ring~0.1–0.3 µsTwo cache-line transfers, no syscalls
Shared memory + futex/eventfd wakeup~2–5 µsAdds sleep/wake (context switch) on each side
Pipe (pair of pipes)~4–8 µs2 syscalls + wakeup each way
Unix domain socket (stream)~4–8 µsSimilar to pipe; slightly more socket bookkeeping
TCP over loopback~8–15 µsFull TCP/IP stack traversal, both directions
gRPC over UDS / loopback~30–100 µsAdds HTTP/2 framing, protobuf, user-space scheduling

The gap between the first two rows is the central lesson: once you go through the kernel to wake someone, the wakeup costs far more than the data transfer. Low-latency systems avoid sleeping at all (busy-poll a dedicated core) or batch many messages per wakeup.

Bulk Throughput

MechanismThroughput (single stream, large messages, approx.)
Shared memoryMemory bandwidth: tens of GB/s
Unix domain socket~5–15 GB/s
Pipe (default 64 KB)~3–8 GB/s; larger with F_SETPIPE_SZ; vmsplice/splice can avoid copies
TCP loopback~3–10 GB/s (big MTU on lo, 64 KB, helps)

UDS vs Loopback TCP

The comparison people actually face: a sidecar, database or cache on the same host — TCP to 127.0.0.1 or a Unix socket? The Redis benchmark documentation reports that Unix domain sockets achieve roughly 50% more throughput than TCP loopback on Linux for redis-benchmark. Typical findings for request/response workloads:

TCP 127.0.0.1Unix domain socket
Per-message workIP + TCP headers, checksum (offloaded on lo), ACKs, congestion & flow control, socket lookup via TCP hash tablesEnqueue an skb on the peer's receive queue
LatencyBaseline~20–50% lower
Small-message throughputBaseline~1.3–2× higher
Access controlFirewall, port binding (anyone local can connect)File permissions + SO_PEERCRED
PortabilityWorks across hosts and container network namespacesSame host; needs a shared filesystem path or volume
# Try it yourself with Redis
$ redis-server --port 6379 --unixsocket /tmp/redis.sock --unixsocketperm 700 &
$ redis-benchmark -q -n 1000000 -t get,set -P 1 -h 127.0.0.1
SET: 128534.71 requests per second
GET: 131406.05 requests per second
$ redis-benchmark -q -n 1000000 -t get,set -P 1 -s /tmp/redis.sock
SET: 189393.94 requests per second
GET: 196078.42 requests per second
# (illustrative output — absolute numbers depend entirely on hardware)

The difference matters most for chatty, small-request protocols (Redis, PostgreSQL with many short queries, a local metrics agent) and fades when each request does real work. Pipelining (-P 16) shrinks the gap because syscall and wakeup costs are amortized over many commands.


How Real Systems Use IPC

Chrome: Multi-Process with Mojo

graph TD
    B["Browser process<br/>(UI, privileged, owns profile)"]
    G["GPU process"]
    N["Network service"]
    R1["Renderer: tab 1<br/>(sandboxed)"]
    R2["Renderer: tab 2<br/>(sandboxed)"]
    U["Utility processes<br/>(audio, storage, decoding)"]

    B <-->|"Mojo message pipes<br/>(UDS on Linux)"| R1
    B <-->|"Mojo"| R2
    B <-->|"Mojo"| N
    B <-->|"Mojo"| G
    B <-->|"Mojo"| U
    R1 <-->|"shared memory<br/>command buffers"| G
    R2 <-->|"shared memory<br/>command buffers"| G

Chrome runs the browser UI, each site's renderer (site isolation), the GPU, networking and various utilities as separate processes. They communicate through Mojo, an IPC system built on a per-platform transport — Unix domain sockets on Linux and ChromeOS, Mach ports on macOS, named pipes on Windows. Mojo messages can carry handles (other message pipes, shared memory regions, platform files), which on Linux means SCM_RIGHTS underneath. Bulk data — rendered tiles, GPU command buffers, decoded media — moves through shared memory regions, with Mojo messages acting as the control channel. Sandboxed renderers cannot open files or sockets themselves; everything they need is brokered by the browser process and passed in as a handle.

PostgreSQL: Shared Buffers and Process-per-Connection

The postmaster allocates the main shared memory region — shared_buffers (the buffer pool), the WAL buffers, the lock table, ProcArray (transaction visibility), the commit log cache — before forking. Each client connection gets a backend process that inherits the mapping. Every backend reads and modifies the same 8 KB pages in the same buffer pool, synchronized by Postgres's own spinlocks and LWLocks (lightweight locks) built on atomics in that shared region.

Other IPC in Postgres:

  • Signals for control: SIGUSR1 multiplexed into "procsignal" reasons (cancel query, catch-up on invalidations, barrier), SIGTERM/SIGINT for backend termination.
  • Latches — a "wake this process" primitive implemented with signals plus a self-pipe, or signalfd/epoll on Linux.
  • Dynamic shared memory (/dev/shm by default) for parallel query workers — which is why the Docker 64 MB limit bites.
  • Unix domain socket at /var/run/postgresql/.s.PGSQL.5432 for local clients, with peer auth via SO_PEERCRED.

The process model is why each Postgres connection costs megabytes and why connection poolers exist — see Connection Pooling and PostgreSQL Internals.

Nginx: Master and Workers

  1. The master process (often root) reads the config, binds listening sockets on ports 80/443, and forks N worker processes, which inherit the listening sockets and drop privileges.
  2. Workers accept and serve connections independently. Either they share one listening socket (with accept_mutex to avoid thundering herd on older setups) or each has its own via listen ... reuseport (SO_REUSEPORT), letting the kernel load-balance.
  3. The master controls workers with signals and a socketpair channel per worker:
    • nginx -s reload → SIGHUP to master → master re-reads config, forks new workers (which inherit the same listening sockets — no port is ever closed), tells old workers to finish in-flight requests and exit gracefully.
    • SIGUSR1 → reopen log files (log rotation).
    • SIGUSR2 → binary upgrade: the master exec()s the new binary, passing the listening sockets' fd numbers in the NGINX environment variable.
  4. Workers share state through shared memory zones: limit_req_zone, proxy_cache_path keys_zone, upstream zone, SSL session cache — protected by Nginx's own spinlock/mutex in shared memory.

Redis: fork() as a Snapshot Mechanism

Redis is single-threaded for command execution. To write an RDB snapshot (BGSAVE) or rewrite the AOF without blocking clients, it fork()s:

sequenceDiagram
    participant P as Redis parent
    participant K as Kernel
    participant C as Child

    P->>K: fork
    K-->>C: same memory, all pages shared copy-on-write
    Note over C: iterate the dataset, write temp RDB file, fsync, rename
    P->>P: keep serving writes
    P->>K: write to a shared page
    K->>K: page fault, copy just that page for the parent
    C->>P: report COW stats over a pipe
    C->>K: exit
    K-->>P: SIGCHLD, parent reaps child and logs result

The child sees a frozen point-in-time image of the entire dataset for free — the kernel's copy-on-write is the snapshot mechanism. The IPC here is minimal: a pipe from child to parent for progress and copy-on-write statistics, and SIGCHLD/waitpid for completion. The costs: fork() itself must copy the page tables (tens of milliseconds for a 25 GB instance — Redis reports it as latest_fork_usec in INFO), and a write-heavy workload during the snapshot can duplicate a large fraction of memory. Transparent huge pages make this much worse (one write copies 2 MB instead of 4 KB), which is why Redis warns at startup if THP is enabled.

systemd and Docker

  • systemd is IPC-heavy by design: D-Bus (over UDS) for its API, /run/systemd/notify (SOCK_DGRAM UDS) for sd_notify("READY=1") readiness and watchdog pings, socket activation via inherited fds, the fd store via SCM_RIGHTS, and /run/systemd/journal/socket for logs.
  • Docker: the docker CLI is an HTTP client speaking to dockerd over /var/run/docker.sock; dockerd talks gRPC to containerd over /run/containerd/containerd.sock; the per-container shim and runc exchange the console pty fd over a UDS with SCM_RIGHTS. Because access to docker.sock lets you start a privileged container that mounts the host filesystem, membership in the docker group is effectively root — a direct consequence of "UDS access control is file permissions."

Failure Modes

FailureSymptomCauseFix
Reader never sees EOFProcess hangs forever on read() from a pipeSome process (often the reader itself, or a forked sibling) still holds a write-end fdClose unused ends immediately after fork(); use O_CLOEXEC everywhere
Pipe deadlock with subprocessParent and child both blockedChild fills the 64 KB stderr pipe while parent reads only stdoutRead stdout and stderr concurrently (communicate()), or merge them
Server dies on client disconnectProcess exits with no log, status 141Unhandled SIGPIPESIG_IGN SIGPIPE, or MSG_NOSIGNAL
bind: Address already in use on a UDSDaemon fails to restartStale socket file from crashed rununlink() before bind() (after checking no live process owns it), or use abstract namespace / socket activation
Interleaved garbage from multiple writersCorrupted lines in a shared pipe/FIFOWrites > PIPE_BUF (4 KB) are not atomicKeep messages ≤ 4 KB, or use datagram/seqpacket sockets
All processes hang on a shared-memory lockTotal freeze after one worker crashedNon-robust mutex held by a dead processPTHREAD_MUTEX_ROBUST and handle EOWNERDEAD
SIGBUS reading shared memoryCrash on access to a mapped regionPeer truncated the file; or /dev/shm full when touching pagesmemfd seals; size /dev/shm properly
RAM slowly disappearsShmem in /proc/meminfo grows, no process owns itLeaked POSIX/SysV segmentsshm_unlink early (after all parties have mapped), ipcs/ipcrm, memfd instead
Lost notificationsHandler ran once for several eventsStandard signals coalesceUse eventfd/pipe/UDS for events; signals only as "wake up and check"
fd exhaustionEMFILE / Too many open filesLeaked or untrusted SCM_RIGHTS fdsValidate senders, close received fds you don't need, raise RLIMIT_NOFILE deliberately

War Story (illustrative)

A team moved a log-shipping agent from a VM into a container and saw it hang every few hours. The application wrote logs to a FIFO the agent read from. During agent restarts, nobody had the FIFO open for reading — and open(O_WRONLY) on a FIFO with no reader blocks. The app's logging thread stalled inside open() while holding the logger's mutex, so every request thread that tried to log stalled behind it. The fix had two parts: open the FIFO with O_WRONLY | O_NONBLOCK (which fails fast with ENXIO when there is no reader) and drop logs on the floor rather than block; and switch to a SOCK_DGRAM Unix socket, where a missing reader produces an immediate ECONNREFUSED instead of a hang. The general lesson: any IPC endpoint you block on is a dependency on the other process's liveness.


Decision Guide

NeedUseWhy
Parent ↔ child byte stream (shell-style)pipe2(O_CLOEXEC)Simplest; EOF and backpressure built in
Bidirectional parent ↔ child channelsocketpair(AF_UNIX, SOCK_STREAM)Two-way, can carry fds
Local client/server protocol (daemon API)Unix domain socket, SOCK_STREAMPermissions + SO_PEERCRED auth, faster than loopback TCP
Framed messages without writing a framerSOCK_SEQPACKET UDSReliable, ordered, boundaries kept
Fire-and-forget local events (logs, readiness)SOCK_DGRAM UDSReliable on Linux, no connection state
Hand an open file/socket/device to another processUDS + SCM_RIGHTSThe only general mechanism
Zero-downtime restart keeping listenersSocket activation or SCM_RIGHTS hand-offKernel accept queue never closes
Large data, highest throughputShared memory (memfd or shm_open + mmap)No copies
Lowest latency (sub-microsecond)Shared-memory SPSC ring, busy-poll on a pinned coreNo syscalls, no wakeups
Shared memory from an untrusted peerSealed memfd passed via SCM_RIGHTSKernel-enforced immutability
Wake another process / thread / event loopeventfdOne fd, coalescing, epoll-able
Prioritized small messages, embedded/RTPOSIX message queuePriorities, boundaries, mq_notify
Lifecycle control (reload, shutdown, rotate logs)Signals (SIGHUP, SIGTERM, SIGUSR1)Universal convention
Might move to another host laterTCP (or gRPC) — over UDS today if you want, same protocolSame code path either way
Durable, multi-consumer, multi-hostA message broker, not kernel IPCKernel IPC is in-memory and same-host
graph TD
    A["Two processes need to communicate"] --> B{"Same host forever?"}
    B -->|no| Z1["TCP / gRPC / message broker"]
    B -->|yes| C{"Need to pass open fds?"}
    C -->|yes| Z2["Unix domain socket + SCM_RIGHTS"]
    C -->|no| D{"Payload large or latency below 1µs?"}
    D -->|yes| Z3["Shared memory ring<br/>+ eventfd or futex for wakeups"]
    D -->|no| E{"Related by fork?"}
    E -->|"yes, one-way"| Z4["pipe"]
    E -->|"yes, two-way"| Z5["socketpair"]
    E -->|no| Z6["Unix domain socket<br/>(stream, dgram or seqpacket)"]

The default answer

If you are unsure, use a Unix domain stream socket with a simple length-prefixed protocol (or gRPC/HTTP over UDS). It is fast, debuggable with socat and ss -x, authenticated by the kernel, can pass fds when you need it, and upgrades to TCP with a one-line change if the processes ever move apart. Reach for shared memory only when profiles show copies or syscalls dominate.


Interview Questions

"What happens when you run cat big.log | grep ERROR and grep is slower than cat?" The pipe's 64 KB buffer fills, cat's write() blocks, and cat sleeps until grep drains data — natural backpressure. If grep exits early, cat's next write gets SIGPIPE and it dies quietly.

"Why are Unix domain sockets faster than TCP on localhost?" They skip the entire TCP/IP stack: no headers, checksums, sequence numbers, ACKs, congestion or flow control, or routing/port lookup. The kernel simply queues the buffer on the peer socket. Typical results are 20–50% lower latency and up to ~2× throughput for small messages.

"How can one process give another an open file descriptor?" Send it over a Unix domain socket in an SCM_RIGHTS control message with sendmsg(). The kernel installs a new descriptor in the receiver referring to the same open file description. Used for zero-downtime restarts, privilege separation and container consoles.

"What are the risks of shared memory?" No kernel-provided synchronization (you must put locks or atomics in the region), pointers aren't portable between processes (use offsets), a crashed process can leave locks held (use robust mutexes) and segments leaked (shm_unlink), and an untrusted peer can truncate or mutate the region under you (use sealed memfds).

"Why does Redis fork for BGSAVE instead of using threads?" fork() gives the child a consistent point-in-time view of the whole dataset via copy-on-write, with no locking in the hot path. The parent keeps serving requests; only pages it modifies get copied.

"Why is being in the docker group equivalent to root?" The Docker API is exposed on /var/run/docker.sock, whose only access control is file permissions. Anyone who can connect can ask dockerd (running as root) to start a privileged container with the host's root filesystem mounted.