On this page

Signals & Process Control

A signal is a software interrupt delivered to a process by the kernel (or by another process). Signals are how the OS tells a process "something happened" — a timer expired, a child exited, the user pressed Ctrl+C, or another process asked you to terminate.

Signals seem simple but have subtle behavior that causes real production bugs: dropped signals, uninterruptible syscalls, unsafe handler code, and graceful shutdown failures.


Why Signals Exist

Unix needed a way to notify a process of an event without the process asking. Almost everything else in Unix is pull-based: you read() a file, you wait() for a child, you poll() a socket. But some events cannot wait for the process to ask:

  • The hardware faulted on your behalf. You dereferenced a NULL pointer; the CPU raised a page fault the kernel cannot satisfy. The kernel must tell you — right now, at that instruction.
  • A human wants you to stop. Ctrl+C in a terminal must interrupt a program stuck in a loop that never calls read().
  • The system wants you to stop. A shutdown, an orchestrator draining a node, an OOM condition.
  • A related process changed state. A child exited; a pipe reader went away.

Signals are the 1970s answer: a tiny, fixed vocabulary of numbered events (originally 15, now 64 on Linux) that the kernel can inject into any process at almost any instruction. That power is exactly what makes them dangerous — "at almost any instruction" includes the middle of malloc().

Mental model

Think of a signal as a hardware interrupt, but for processes. A hardware interrupt makes the CPU stop what it is doing and jump to a handler; a signal makes a thread stop what it is doing and jump to a handler. Everything that is hard about interrupt handlers in kernels (reentrancy, locking, doing minimal work) is equally hard about signal handlers in user space. See Interrupts, DMA & Drivers for the hardware side.


The Signal Model

When a signal is sent to a process:

  1. The kernel sets a flag in the process's PCB (on Linux, a bit in a pending set on the task_struct or on the shared signal_struct)
  2. At the next opportunity — specifically, when a target thread is about to return from kernel mode to user mode (after a syscall, an interrupt, or being scheduled back in) — the kernel delivers the signal
  3. The process executes the registered handler (or the default action)

Important: Signal delivery is asynchronous — a signal sent by kill(pid, SIGTERM) may not be delivered immediately. The kernel queues it and delivers it when the process next runs.

Pending vs Blocked: the Two Bitmasks

Every thread has, conceptually, two sets of signals:

SetMeaningWhere to see it
PendingSignals that have been generated but not yet deliveredSigPnd (thread) and ShdPnd (process-wide) in /proc/<pid>/status
Blocked (the signal mask)Signals this thread refuses to accept right now; they stay pendingSigBlk
IgnoredDisposition set to SIG_IGN; the signal is discarded on generationSigIgn
CaughtA handler is installedSigCgt
$ grep -E 'Sig|ShdPnd' /proc/$(pgrep -n nginx)/status
SigQ:   0/63446                # queued signals / RLIMIT_SIGPENDING
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000001000       # bit 12 set → signal 13 (SIGPIPE) ignored
SigCgt: 0000000180014a07       # handlers for HUP, INT, QUIT, USR1, USR2, TERM, CHLD, WINCH, ...

Each hex mask is a 64-bit bitmap where bit n−1 represents signal n. 0x1000 is bit 12 → signal 13 → SIGPIPE. Decoding these masks is often the fastest way to answer "why didn't my process react to SIGTERM?" — if SigCgt doesn't contain bit 14 (signal 15), nothing is handling it, and if SigBlk does, it is being blocked.

Decode masks in one line

mask=0000000180014a07; for i in $(seq 1 64); do (( (0x$mask >> (i-1)) & 1 )) && echo -n "$(kill -l $i) "; done; echo
# HUP INT QUIT USR1 USR2 TERM CHLD WINCH ...

Standard signals do not queue. The pending set for signals 1–31 is a bitmap, not a counter. If three SIGCHLDs arrive while the signal is blocked or before the handler runs, the bit is set once and you get one delivery. This is the root cause of the classic "lost children" bug (see SIGCHLD and Reaping). Real-time signals (34–64) do queue — see Real-Time Signals.

The Delivery Path, Step by Step

sequenceDiagram
    participant S as "Sender (kill)"
    participant K as Kernel
    participant T as "Target thread"
    S->>K: "kill(pid, SIGTERM)"
    K->>K: "permission check, set bit in shared pending set"
    K->>T: "pick a thread that does not block SIGTERM, set TIF_SIGPENDING, wake it if sleeping interruptibly"
    Note over T: "thread is running user code or sleeping in a syscall"
    T->>K: "syscall returns or interrupt ends, exit_to_user_mode checks TIF_SIGPENDING"
    K->>K: "get_signal dequeues SIGTERM and looks up disposition"
    alt handler installed
        K->>T: "build signal frame on user stack, set PC to handler"
        T->>T: "handler runs"
        T->>K: "rt_sigreturn restores saved registers and mask"
        K->>T: "resume where it was interrupted"
    else default action is terminate
        K->>K: "do_group_exit kills all threads"
    end

Key consequences of this design:

  1. Signals are only acted on at the user/kernel boundary. A thread spinning in pure user code (while(1);) still gets the signal promptly, because the timer interrupt fires every few ms and the return-from-interrupt path checks TIF_SIGPENDING.
  2. A thread asleep in the kernel must be woken. If it is in TASK_INTERRUPTIBLE sleep (e.g. blocked in read() on a socket), the kernel wakes it; the syscall returns -EINTR (or is restarted). If it is in TASK_UNINTERRUPTIBLE (state D in ps) — typically waiting on disk I/O or an NFS server — the signal stays pending until the sleep ends. This is why kill -9 sometimes "doesn't work": the process is stuck in D state. Since Linux 2.6.25 there is also TASK_KILLABLE, which ignores everything except fatal signals; much NFS and page-fault code uses it.
  3. The handler runs on the interrupted thread's own stack (unless you configure an alternate stack with sigaltstack()), with a saved copy of registers. rt_sigreturn is a real syscall that the kernel arranges to be called when the handler returns — you can see it in strace.
  4. While a handler runs, its own signal is blocked (plus anything in sa_mask), unless SA_NODEFER is set. That prevents a handler from being re-entered by its own signal.
$ strace -f -e trace=signal,read ./server
read(5, 0x7ffd..., 4096)                = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=4312, si_uid=1000} ---
rt_sigreturn({mask=[]})                 = 0
read(5, ...

The si_pid and si_uid fields tell you who sent the signal — invaluable when something keeps killing your process and you don't know what. Use SA_SIGINFO to read them in your own handler.


Common Signals

SignalNumberDefault ActionDescription
SIGHUP1TerminateHangup (terminal disconnected); often used to reload config
SIGINT2TerminateKeyboard interrupt (Ctrl+C)
SIGQUIT3Core dumpQuit with core dump (Ctrl+\)
SIGILL4Core dumpIllegal instruction (bad JIT output, wrong CPU feature like AVX-512 on an older host)
SIGTRAP5Core dumpBreakpoint / trace trap (debuggers)
SIGABRT6Core dumpabort() — failed assert(), glibc heap corruption detection ("double free or corruption")
SIGBUS7Core dumpBus error — bad memory access to an existing mapping (e.g. truncated mmap'd file)
SIGFPE8Core dumpArithmetic error (integer divide by zero)
SIGKILL9TerminateImmediate kill — cannot be caught or ignored
SIGUSR110TerminateUser-defined signal 1
SIGSEGV11Core dumpSegmentation fault
SIGUSR212TerminateUser-defined signal 2
SIGPIPE13TerminateWrite to a broken pipe (common in network code)
SIGALRM14TerminateTimer expired (alarm())
SIGTERM15TerminatePolite termination request — can be caught
SIGCHLD17IgnoreChild process changed state
SIGCONT18ContinueResume a stopped process
SIGSTOP19StopPause process — cannot be caught or ignored
SIGTSTP20StopTerminal stop (Ctrl+Z) — can be caught
SIGTTIN / SIGTTOU21 / 22StopBackground process tried to read from / write to the terminal
SIGXCPU24Core dumpCPU time limit (RLIMIT_CPU) exceeded
SIGXFSZ25Core dumpFile size limit (RLIMIT_FSIZE) exceeded
SIGWINCH28IgnoreTerminal window resized
SIGSYS31Core dumpBad syscall — also what a seccomp filter sends with SECCOMP_RET_KILL/TRAP
SIGRTMIN–SIGRTMAX34–64TerminateReal-time signals — queued, ordered, carry data

Numbers above are for Linux on x86-64 and arm64. They differ on other architectures (e.g. on MIPS, SIGCHLD is 18; on macOS, SIGUSR1 is 30) — always use names, never numbers, in portable code and scripts. kill -l prints the local mapping.

`SIGRTMIN` is not 32

glibc reserves signals 32 and 33 for its own threading implementation (thread cancellation and setxid broadcasting), so SIGRTMIN is 34 for applications. Always compute real-time signals as SIGRTMIN + n.

Where Signals Come From

SourceExamplessi_code
Another processkill(), tgkill(), sigqueue(), pidfd_send_signal()SI_USER, SI_QUEUE
The terminal driverCtrl+C → SIGINT, Ctrl+\ → SIGQUIT, Ctrl+Z → SIGTSTP, hangup → SIGHUPSI_KERNEL
A CPU exceptionBad pointer → SIGSEGV, divide by zero → SIGFPE, ud2 → SIGILLSEGV_MAPERR, SEGV_ACCERR, FPE_INTDIV
Kernel eventsChild exit → SIGCHLD, broken pipe → SIGPIPE, rlimit → SIGXCPUCLD_EXITED, CLD_KILLED
Timersalarm(), setitimer(), timer_create()SI_TIMER
The OOM killerSIGKILLSI_KERNEL
The process itselfraise(), abort()SI_TKILL

Synchronous vs asynchronous: signals caused by the thread's own instruction (SIGSEGV, SIGFPE, SIGILL, SIGBUS from a fault) are synchronous — they are delivered to the faulting thread, immediately. Everything else is asynchronous — delivered to whichever thread the kernel picks.


SIGTERM vs SIGKILL

This distinction matters for every server process and is core to Kubernetes pod lifecycle management.

SIGTERM (15): Polite request to terminate. The process can catch this signal, finish in-flight requests, flush buffers, close database connections, and then exit cleanly. This is what kill <pid> sends by default.

SIGKILL (9): Unconditional, immediate termination. The kernel kills the process immediately — no handler runs, no cleanup, no buffered writes flushed. kill -9 <pid>. Cannot be caught, blocked, or ignored.

SIGSTOP / SIGCONT: Pause and resume. Used by debuggers, shells (Ctrl+Z), and cgroup freezers.

What SIGKILL does not lose and what it does:

StateSurvives SIGKILL?
Data already passed to write() (in the kernel page cache)Yes — the kernel owns it now and will flush it
Data sitting in a user-space buffer (stdio, a Go bufio.Writer, a logger's ring buffer)No
Data that was fsync()edYes
Open TCP connectionsClosed by the kernel — peers see a FIN (or RST if unread data was pending)
Advisory file locks (flock, fcntl)Released by the kernel
Temp files, PID files, Unix socket filesLeft behind
Distributed locks / leases (etcd, Redis, ZooKeeper)Held until TTL expiry
In-flight HTTP requestsClients see connection reset / 502
Consumer group membership (Kafka)Held until session timeout → delayed rebalance

Kubernetes Pod Termination

Kubernetes pod termination sequence:

1. kubectl delete pod → API server sets pod.deletionTimestamp
2. kubelet sends SIGTERM to PID 1 in the container
3. Container has `terminationGracePeriodSeconds` (default: 30s) to exit cleanly
4. If not exited after grace period: kubelet sends SIGKILL
5. Container terminates immediately

If your server doesn't handle SIGTERM, it gets SIGKILL'd after 30 seconds — in-flight requests are dropped, connections are abruptly closed, write buffers are lost.

The simplified sequence above hides two details that cause most real-world shutdown errors: the preStop hook and the endpoint-removal race.

sequenceDiagram
    participant API as "API server"
    participant KL as Kubelet
    participant C as "Container PID 1"
    participant EP as "EndpointSlice controller"
    participant KP as "kube-proxy and ingress"
    API->>KL: "pod marked Terminating (deletionTimestamp set)"
    API->>EP: "pod marked Terminating"
    par in parallel, no ordering guarantee
        KL->>C: "run preStop hook (exec or HTTP), e.g. sleep 10"
        Note over C: "grace period clock is already running"
        KL->>C: "after preStop returns, send STOPSIGNAL (default SIGTERM)"
        C->>C: "stop accepting, drain in-flight, exit 0"
    and
        EP->>KP: "remove pod IP from endpoints"
        Note over KP: "takes 1 to 10+ seconds to propagate to every node and LB"
    end
    KL->>C: "SIGKILL if still running at terminationGracePeriodSeconds"

The race: the kubelet sending SIGTERM and the load balancers removing the pod from rotation happen concurrently. If your app exits immediately on SIGTERM, iptables rules on other nodes and external load balancers may still route new connections to it for a few seconds — those get connection refused and surface as 502/503s during every deploy.

The fix is to keep serving for a few seconds after SIGTERM before closing the listener:

spec:
  terminationGracePeriodSeconds: 45     # must cover preStop + drain
  containers:
  - name: api
    lifecycle:
      preStop:
        sleep:
          seconds: 10                   # native sleep action (K8s 1.30+ beta, GA in 1.33)
        # older clusters: exec: { command: ["sleep", "10"] }  (needs a sleep binary in the image)

The grace period includes the preStop hook

terminationGracePeriodSeconds is a single budget that starts when the pod is marked for deletion. If preStop sleeps 10s and your drain takes 30s, a 30s grace period kills you mid-drain. Budget: preStop + max request duration + cleanup + margin.

More Kubernetes signal details worth knowing:

  • STOPSIGNAL in the Dockerfile changes which signal is sent first (e.g. STOPSIGNAL SIGQUIT for nginx's graceful shutdown, SIGWINCH for Apache httpd's graceful-stop). Kubernetes honours the image's stop signal; a lifecycle.stopSignal field in the pod spec arrived as alpha in 1.33.
  • Only PID 1 gets the signal. The kubelet (via the container runtime) signals the container's init process. If PID 1 is /bin/sh -c "node server.js", the shell receives SIGTERM — and most shells don't forward it. Node never hears about it and is SIGKILLed 30 seconds later. See PID 1 Is Special.
  • Exit code 137 = 128 + 9 (killed by SIGKILL — grace period expired or OOM-killed; check reason). Exit code 143 = 128 + 15 (terminated by SIGTERM with no handler). A clean graceful shutdown should exit 0.
  • Jobs and restartPolicy: Never pods also get SIGTERM on deletion, and on node drain (kubectl drain) through the Eviction API, which respects PodDisruptionBudgets.

PID 1 Is Special

The kernel treats the init process of a PID namespace differently:

  1. No default actions. A signal whose disposition is SIG_DFL is ignored for PID 1 if sent from inside its own namespace. If your app is PID 1 and has no SIGTERM handler, kill -TERM 1 from inside the container does nothing. (From the parent namespace, the kernel does honour SIGKILL and SIGSTOP — that is how docker kill and the kubelet's final kill work.)
  2. It must reap orphans. Any process whose parent dies is re-parented to PID 1 of its namespace. If PID 1 never calls wait(), those orphans become zombies forever (see SIGCHLD and Reaping).
  3. When PID 1 exits, the kernel SIGKILLs everything else in the namespace. No grace for children.

Solutions:

ApproachHowWhen
Exec form in DockerfileCMD ["node", "server.js"] not CMD node server.jsAlways — makes your app PID 1 directly
exec in wrapper scriptsexec node server.js as the last line of entrypoint.shWhen you need a shell to set env vars first
A tiny initdocker run --init (injects tini), or ENTRYPOINT ["tini", "--"], or dumb-initApp spawns children or doesn't handle SIGTERM
Shared PID namespaceshareProcessNamespace: true in the pod spec → the pause container becomes PID 1 and reapsDebug sidecars, multi-process pods
# BAD: shell form — /bin/sh is PID 1 and swallows SIGTERM
CMD node server.js

# GOOD: exec form — node is PID 1 and receives SIGTERM directly
CMD ["node", "server.js"]

# GOOD: tini forwards signals and reaps zombies
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["node", "server.js"]

War Story (illustrative)

A team's deploys produced a 30-second pause and a burst of 502s on every rollout. kubectl get pod -o yaml showed exitCode: 137 for every terminated pod. The Dockerfile used CMD npm start; npm was PID 1 under /bin/sh, and neither forwarded SIGTERM to the Node process. Every pod was SIGKILLed at the grace-period deadline with requests in flight. Switching to CMD ["node", "dist/server.js"], adding a SIGTERM handler that calls server.close(), and a 5-second preStop sleep turned exit codes into 0 and eliminated the 502s.


Registering Signal Handlers

#include <signal.h>

// Old API (avoid in new code):
signal(SIGTERM, handler_func);  // Behavior varies across implementations

// POSIX API (preferred):
struct sigaction sa = {0};
sa.sa_handler = handler_func;
sa.sa_flags = SA_RESTART;  // Restart syscalls interrupted by this signal
sigemptyset(&sa.sa_mask);

sigaction(SIGTERM, &sa, NULL);

Why signal() is discouraged: historically (System V semantics) it reset the disposition to SIG_DFL after the first delivery, so a second Ctrl+C killed the process; BSD semantics did not. glibc's signal() uses BSD semantics today, but other libcs and compile flags differ. sigaction() is fully specified.

SA_RESTART: When a signal interrupts a blocking syscall (like read(), write(), accept()), the syscall can return -1 with errno = EINTR. With SA_RESTART, the kernel automatically restarts the syscall after the handler returns. Without it, your code must check for EINTR and retry manually.

// Without SA_RESTART — must handle EINTR manually:
ssize_t n;
do {
    n = read(fd, buf, size);
} while (n == -1 && errno == EINTR);

`SA_RESTART` does not cover every syscall

Some calls are never restarted regardless of SA_RESTART: select(), poll(), epoll_wait(), nanosleep(), sigsuspend(), socket calls with a timeout set via SO_RCVTIMEO/SO_SNDTIMEO, and several others (man 7 signal, section "Interruption of system calls"). Event loops must always handle EINTR from their wait call. Sometimes that's a feature: an interrupted epoll_wait() is a natural place to check a "shutdown requested" flag.

sigaction Flags You Will Actually Use

FlagEffect
SA_RESTARTAuto-restart interruptible syscalls
SA_SIGINFOUse sa_sigaction(int, siginfo_t *, void *) — get sender PID/UID, fault address, child exit status, queued value
SA_ONSTACKRun the handler on the alternate stack from sigaltstack() — required to handle stack-overflow SIGSEGV
SA_NOCLDSTOPDon't send SIGCHLD when children stop, only when they exit
SA_NOCLDWAITChildren don't become zombies (auto-reaped)
SA_RESETHANDReset to SIG_DFL after one delivery — handy for "clean up, then re-raise to die with the right status"
SA_NODEFERDon't block the signal during its own handler (rarely what you want)

A handler with SA_SIGINFO that logs who sent the signal (using only safe functions):

#include <errno.h>
#include <signal.h>
#include <unistd.h>
#include <string.h>

static void fmt_int(char *buf, long v) {        // tiny async-signal-safe itoa
    char tmp[24]; int i = 0, j = 0;
    if (v == 0) tmp[i++] = '0';
    while (v > 0) { tmp[i++] = '0' + v % 10; v /= 10; }
    while (i) buf[j++] = tmp[--i];
    buf[j] = '\0';
}

static void on_term(int sig, siginfo_t *si, void *ctx) {
    (void)sig; (void)ctx;
    int saved = errno;                            // handlers must preserve errno
    char msg[64] = "SIGTERM from pid ";
    fmt_int(msg + strlen(msg), si->si_pid);       // strlen is async-signal-safe
    strcat(msg, "\n");                            // strcat too (POSIX.1-2008 TC2+)
    write(STDERR_FILENO, msg, strlen(msg));
    errno = saved;
}

int main(void) {
    struct sigaction sa = {0};
    sa.sa_sigaction = on_term;
    sa.sa_flags = SA_SIGINFO | SA_RESTART;
    sigemptyset(&sa.sa_mask);
    sigaction(SIGTERM, &sa, NULL);
    for (;;) pause();
}

Dispositions Across fork() and exec()

After fork()After execve()
HandlersInherited (same code exists in child)Reset to SIG_DFL (handler code is gone)
SIG_IGNInheritedInherited
Signal maskInheritedInherited
Pending signalsCleared in childPreserved

The "SIG_IGN and mask survive exec" rows bite people: if a parent ignores SIGPIPE or blocks SIGTERM and then execs a child, the child starts with that state. Python historically set SIGPIPE to SIG_IGN, and subprocesses that expected to die on a closed pipe (yes | head) would instead spin on EPIPE; subprocess now restores defaults with restore_signals=True. Well-behaved supervisors reset dispositions and unblock all signals between fork() and exec().


Signal Masks

Each thread has a signal mask — a set of signals that are blocked (held pending, not delivered). Blocked signals are not ignored; they queue up and are delivered when unblocked.

sigset_t set;
sigemptyset(&set);
sigaddset(&set, SIGTERM);

// Block SIGTERM in this thread
sigprocmask(SIG_BLOCK, &set, NULL);
// ... critical section ...
sigprocmask(SIG_UNBLOCK, &set, NULL);  // SIGTERM delivered now if pending

Important: In a multi-threaded process, signals are delivered to any thread that has not blocked them. SIGTERM sent to a process can be handled by any thread. The self-pipe trick or signalfd + signal masking in all threads ensures signals are handled in a dedicated thread.

sigprocmask() is unspecified in multithreaded programs; use pthread_sigmask() there (on Linux they behave the same per-thread).

Signals and Threads

Signal dispositions (handlers) are per process; signal masks are per thread; pending sets exist at both levels.

flowchart TD
    A["Signal generated"] --> B{"Directed at a thread? (tgkill, pthread_kill, or a fault)"}
    B -->|yes| C["Thread pending set"]
    B -->|no, process-directed via kill| D["Shared process pending set"]
    C --> E["That exact thread handles it when it unblocks"]
    D --> F{"Any thread with the signal unblocked?"}
    F -->|"yes"| G["Kernel picks one (prefers the main thread if eligible)"]
    F -->|"no"| H["Stays pending until some thread unblocks it or calls sigwait"]

The robust multithreaded pattern: block all asynchronous signals in main() before creating any threads (threads inherit the mask), then dedicate one thread to sigwait():

#include <pthread.h>
#include <signal.h>
#include <stdio.h>

static void *signal_thread(void *arg) {
    sigset_t *set = arg;
    for (;;) {
        int sig;
        sigwait(set, &sig);                 // synchronous: normal code, any function is safe
        if (sig == SIGTERM || sig == SIGINT) {
            printf("shutting down\n");      // printf is fine here - not a handler
            request_shutdown();             // e.g. set atomic flag, close listener
            return NULL;
        }
        if (sig == SIGHUP) reload_config();
    }
}

int main(void) {
    static sigset_t set;
    sigemptyset(&set);
    sigaddset(&set, SIGTERM); sigaddset(&set, SIGINT); sigaddset(&set, SIGHUP);
    pthread_sigmask(SIG_BLOCK, &set, NULL); // before any pthread_create

    pthread_t t;
    pthread_create(&t, NULL, signal_thread, &set);
    start_worker_threads();                 // all inherit the blocked mask
    pthread_join(t, NULL);
}

Synchronous fault signals (SIGSEGV etc.) can't be redirected this way — they always go to the faulting thread, and blocking them while a fault occurs gets the process killed anyway.


Async-Signal Safety

Signal handlers run asynchronously — they can interrupt any point in your program's execution, including the middle of a malloc() call or an fprintf() call.

Problem: If your signal handler calls malloc() and your main code was in the middle of malloc() when the signal arrived, you have a re-entrant lock — the allocator's internal state is corrupted.

Rule: signal handlers must only call async-signal-safe functions.

Async-signal-safe functions are guaranteed to be safe even if called from a signal handler that interrupted the same function. The POSIX list includes:

  • write() (syscall) — yes
  • printf() — NO (calls malloc internally)
  • malloc() / free() — NO
  • exit() — NO (flushes stdio buffers)
  • _exit() — yes (no cleanup)
  • signal() / sigaction() — yes
  • Most syscalls — yes

Why Most Functions Aren't Safe

A function is unsafe in a handler if it does any of these:

  1. Takes a lock. malloc() takes an arena lock; printf() takes the FILE lock on stdout. If the main thread holds that lock when the signal arrives on the same thread, the handler blocks forever waiting for a lock its own thread holds — a self-deadlock. (A normal mutex is not reentrant; even a recursive one would let the handler walk into half-updated data.)
  2. Touches global mutable state non-atomically. The free list in malloc, the buffer pointer in a FILE, the static buffer in strtok() or localtime(). The handler sees the data mid-update.
  3. Modifies errno without restoring it. Even write() can set errno; the interrupted code might be between a failed syscall and reading errno.
  4. Allocates memory. Almost everything high-level does: syslog(), printf() with some format specifiers, C++ std::string, logging libraries, anything that throws.
sequenceDiagram
    participant M as "Main code (same thread)"
    participant L as "malloc arena lock"
    participant H as "Signal handler"
    M->>L: "malloc() acquires lock"
    Note over M: "halfway through unlinking a free-list chunk"
    H-->>M: "SIGALRM arrives, handler starts on this thread"
    H->>L: "handler calls printf, which calls malloc, which tries to take the lock"
    Note over H,L: "lock is held by this very thread, which is now suspended inside the handler: deadlock"

This bug is rare and nondeterministic: the signal has to land in the few hundred nanoseconds the lock is held. Programs with an unsafe handler can run for months before hanging, and the stack trace (gdb -p, pstack) shows a thread blocked in __lll_lock_wait inside a signal handler inside malloc — that signature is diagnostic.

What a handler may safely do:

SafeNot safe
Set a volatile sig_atomic_t flag or a C11 lock-free atomic_intCall printf, syslog, malloc, free, new, logging frameworks
write() a byte to a pipe or eventfdAcquire a pthread_mutex_t
_exit()exit() (runs atexit handlers, flushes stdio)
kill(), raise(), sigaction()pthread_cond_signal() (not on the list)
read, open, close, fsync, waitpidAnything in C++ that might throw or allocate
memcpy, strlen, strcpy (POSIX.1-2008 TC2 and later)localtime, strtok, getenv (non-reentrant)

The full list is in man 7 signal-safety.

The only portable flag type

volatile sig_atomic_t is the one type the C standard guarantees can be written from a handler and read in main code without tearing. volatile stops the compiler from caching it in a register across loop iterations; sig_atomic_t guarantees the store is a single instruction. It does not give inter-thread ordering — for multithreaded code use _Atomic / std::atomic that is lock-free (atomic_is_lock_free).

The self-pipe trick: A common pattern for safely handling signals in event loops. The signal handler writes a byte to a pipe; the event loop (epoll/select) watches the read end and processes the signal notification in the main thread where all functions are safe.

int signal_pipe[2];
pipe(signal_pipe);

void signal_handler(int sig) {
    // write() is async-signal-safe
    write(signal_pipe[1], &sig, sizeof(sig));
}

// In event loop:
// Watch signal_pipe[0] for reads
// When readable: read the signal number, handle it safely

A production-grade version needs three more details: make both ends non-blocking (if the pipe fills during a signal storm, a blocking write() in the handler would hang the process), set O_CLOEXEC so children don't inherit it, and save/restore errno:

#include <fcntl.h>
#include <errno.h>
#include <signal.h>
#include <sys/epoll.h>
#include <unistd.h>

static int sigpipe_fds[2];

static void on_signal(int sig) {
    int saved = errno;
    unsigned char b = (unsigned char)sig;
    (void)write(sigpipe_fds[1], &b, 1);  // EAGAIN when full is fine: a wakeup is already pending
    errno = saved;
}

int setup_signals(int epfd) {
    if (pipe2(sigpipe_fds, O_NONBLOCK | O_CLOEXEC) < 0) return -1;
    struct sigaction sa = { .sa_handler = on_signal, .sa_flags = SA_RESTART };
    sigemptyset(&sa.sa_mask);
    sigaction(SIGTERM, &sa, NULL);
    sigaction(SIGINT,  &sa, NULL);
    sigaction(SIGHUP,  &sa, NULL);
    struct epoll_event ev = { .events = EPOLLIN, .data.fd = sigpipe_fds[0] };
    return epoll_ctl(epfd, EPOLL_CTL_ADD, sigpipe_fds[0], &ev);
}

void drain_signals(void) {               // called from the event loop when readable
    unsigned char b;
    while (read(sigpipe_fds[0], &b, 1) == 1) {
        if (b == SIGHUP) reload_config();        // any function is safe here
        else             begin_graceful_shutdown();
    }
}

This trick was popularized by D. J. Bernstein and is how libuv (Node.js), libevent, and Python's signal.set_wakeup_fd() work on platforms without signalfd.

signalfd (Linux): A modern alternative — converts signals to file descriptor events, which can be watched with epoll. No signal handler needed, no async-signal-safety concerns.

sigset_t mask;
sigemptyset(&mask);
sigaddset(&mask, SIGTERM);
sigaddset(&mask, SIGUSR1);

// Block these signals from normal delivery
sigprocmask(SIG_BLOCK, &mask, NULL);

// Create a file descriptor that receives these signals
int sfd = signalfd(-1, &mask, SFD_NONBLOCK);

// Now watch sfd with epoll — signal delivery becomes an event

Reading from it returns struct signalfd_siginfo records:

#include <sys/signalfd.h>

struct signalfd_siginfo fdsi;
while (read(sfd, &fdsi, sizeof fdsi) == sizeof fdsi) {
    switch (fdsi.ssi_signo) {
    case SIGTERM: printf("TERM from pid %u\n", fdsi.ssi_pid); begin_graceful_shutdown(); break;
    case SIGUSR1: dump_stats(); break;
    }
}

signalfd pitfalls

  • You must block the signals in every thread, or the kernel may deliver them the normal way to an unblocked thread and the fd never sees them. Block before creating threads.
  • Children inherit the blocked mask across exec(). A child spawned by a signalfd-using program starts with SIGTERM blocked and will ignore your kill. Unblock in the child between fork() and exec() (or use posix_spawnattr_setsigmask).
  • Standard signals still coalesce — a signalfd doesn't make three SIGCHLDs into three reads. Always reap children in a waitpid(-1, ..., WNOHANG) loop.
TechniqueHandler runs?Works with epollPortabilityTypical users
Flag in handler, check in loopYes (minimal)Needs EINTR wakeupPOSIXSimple CLIs
Self-pipe / eventfdYes (one write)YesPOSIX (eventfd: Linux)libuv, libevent, Python
signalfdNoYesLinux onlysystemd, modern C daemons
Dedicated sigwait() threadNoN/APOSIXMultithreaded C/C++ servers
Runtime-managed channelNo (runtime does it)N/APer-languageGo signal.Notify, Rust signal-hook, Java sun.misc.Signal

SIGCHLD and Reaping

When a child exits, the kernel keeps a small record — exit status, resource usage, PID — until the parent collects it with wait()/waitpid(). A child that has exited but not been collected is a zombie (state Z, shown as <defunct> in ps). Zombies use no memory or CPU, but they hold a PID and a process-table slot. Enough of them and fork() fails with EAGAIN — see the PID exhaustion discussion in Namespaces & cgroups.

stateDiagram-v2
    [*] --> Running: "fork()"
    Running --> Zombie: "exit() - SIGCHLD sent to parent"
    Zombie --> [*]: "parent calls waitpid()"
    Running --> Orphan: "parent dies first"
    Orphan --> Running: "re-parented to PID 1 or subreaper"

SIGCHLD and zombie prevention:

// Automatically reap children without calling wait()
struct sigaction sa = {.sa_handler = SIG_DFL, .sa_flags = SA_NOCLDWAIT};
sigaction(SIGCHLD, &sa, NULL);

Or install a handler that calls waitpid(-1, NULL, WNOHANG) in a loop.

Setting SIGCHLD to SIG_IGN explicitly has the same auto-reap effect on Linux. Both approaches mean you can't learn exit statuses — if you need them (a supervisor, a job runner), you must reap.

The loop matters. Because standard signals coalesce, one SIGCHLD delivery may represent several dead children:

static void on_sigchld(int sig) {
    (void)sig;
    int saved = errno;
    // WRONG: if (waitpid(-1, NULL, WNOHANG) > 0) ...   reaps only one child
    while (waitpid(-1, NULL, WNOHANG) > 0)
        ;                          // reap every child that has exited
    errno = saved;
}

Why WNOHANG: without it, the last iteration blocks inside the handler waiting for a child that hasn't exited yet — freezing the whole thread.

Related mechanisms:

  • prctl(PR_SET_CHILD_SUBREAPER, 1) makes a process adopt orphaned descendants instead of PID 1. Used by systemd --user, tini -s, and container shims (containerd-shim is a subreaper for the container's processes).
  • prctl(PR_SET_PDEATHSIG, SIGTERM) asks the kernel to signal the child when its parent thread dies — a cheap way to avoid orphaned workers. (Caveat: it fires when the parent thread that forked exits, not just the process.)
  • pidfd_open() (Linux 5.3+) gives a file descriptor for a process; it becomes readable when the process exits, so you can watch children with epoll and avoid SIGCHLD entirely. pidfd_send_signal() also removes the PID-reuse race in kill() — you can't accidentally signal an unrelated process that recycled the PID.
// Go: os/exec reaps for you - Wait() calls wait4() on that specific PID.
cmd := exec.Command("worker", "--batch", "42")
if err := cmd.Start(); err != nil { log.Fatal(err) }
err := cmd.Wait()                        // forgetting Wait() leaks a zombie per child
var ee *exec.ExitError
if errors.As(err, &ee) {
    if ws, ok := ee.Sys().(syscall.WaitStatus); ok && ws.Signaled() {
        log.Printf("worker killed by %v", ws.Signal())
    }
}

SIGPIPE: The Silent Killer

When you write to a socket or pipe whose read end is closed, the kernel sends SIGPIPE to your process. Default action: terminate. This silently kills network servers if not handled.

// Option 1: ignore SIGPIPE globally
signal(SIGPIPE, SIG_IGN);
// write() will then return -1 with errno = EPIPE instead of killing the process

// Option 2: use MSG_NOSIGNAL per-send
send(sockfd, buf, len, MSG_NOSIGNAL);
// Returns -1/EPIPE without generating SIGPIPE

Most long-running servers ignore SIGPIPE globally and check write/send return codes.

Why does the default kill you at all? Because it makes shell pipelines work: in cat huge.log | head -1, once head exits, cat's next write raises SIGPIPE and cat dies instantly instead of reading the rest of a 50 GB file. For a server, a client disconnecting mid-response is routine and must never kill the process.

Language runtimes handle this for you to varying degrees:

RuntimeDefault behavior
GoIgnores SIGPIPE on sockets and non-stdout fds (you get EPIPE). A write to a broken stdout/stderr still kills the program, matching shell expectations.
PythonSets SIGPIPE to SIG_IGN at startup → BrokenPipeError exceptions
Node.jsIgnores SIGPIPE → EPIPE error events
Ruststd sets SIGPIPE to SIG_IGN before main for binaries → ErrorKind::BrokenPipe
JavaJVM ignores it → IOException: Broken pipe
C / C++Default action — process dies. You must ignore it yourself or use MSG_NOSIGNAL. On macOS/BSD use setsockopt(SO_NOSIGPIPE) since MSG_NOSIGNAL is not universal.

War Story (illustrative)

A C++ service backed by a third-party client library crashed a few times a day with no core dump, no log line and exit status 141 (128 + 13). The library wrote to a pooled TCP connection that the remote load balancer had idle-closed. The service had set SIGPIPE to SIG_IGN — but in a worker thread after startup, while the library's background thread had already been created. Dispositions are process-wide, so that wasn't the problem; the real cause was an embedded scripting runtime that re-installed SIG_DFL during its own initialization. Grepping /proc/<pid>/status for SigIgn before and after init found it in minutes.


SIGSEGV and SIGBUS Handlers

Fault signals are synchronous and land on the thread that executed the bad instruction.

SignalTypical causesi_code
SIGSEGVAddress not mapped at all (NULL deref, use-after-munmap)SEGV_MAPERR
SIGSEGVMapped but wrong permission (write to read-only page, execute NX page, guard page)SEGV_ACCERR
SIGBUSAccess to an mmap'd file beyond its current end (file was truncated)BUS_ADRERR
SIGBUSHardware memory error reported by the machine-check handlerBUS_MCEERR_AR
SIGBUSUnaligned access on architectures that enforce alignmentBUS_ADRALN

When is it legitimate to handle SIGSEGV?

  1. To print a crash report and die. Install a handler with SA_ONSTACK | SA_RESETHAND | SA_SIGINFO, write a stack trace with backtrace_symbols_fd() (writes directly to an fd — no malloc), then raise(sig) to die with the correct status and core dump. SA_ONSTACK with a sigaltstack() is mandatory: a stack overflow fault happens precisely because the stack is exhausted, so the handler cannot run on it.
  2. As a deliberate memory-management technique. The JVM uses SIGSEGV for implicit null checks and safepoint polling (it reads from a page that it makes unreadable when it wants threads to stop). Garbage collectors and userspace page-in schemes mprotect() pages and catch the fault. Modern Linux code should prefer userfaultfd for that.
  3. To survive a truncated mmap. Databases and search engines that mmap files (LMDB, Lucene/Elasticsearch, early MongoDB) can hit SIGBUS if another process truncates the file or the disk fills on a sparse file. Some handle it with sigsetjmp/siglongjmp to turn it into an I/O error.
#include <execinfo.h>
#include <signal.h>
#include <stdlib.h>
#include <unistd.h>

static void crash_handler(int sig, siginfo_t *si, void *uc) {
    (void)uc;
    static const char hdr[] = "FATAL: fault, backtrace follows\n";
    write(STDERR_FILENO, hdr, sizeof hdr - 1);
    void *frames[64];
    int n = backtrace(frames, 64);
    backtrace_symbols_fd(frames, n, STDERR_FILENO);
    (void)si;                       // si->si_addr is the faulting address
    raise(sig);                     // SA_RESETHAND restored SIG_DFL: die and core dump
}

void install_crash_handler(void) {
    static char altstack[64 * 1024];
    stack_t ss = { .ss_sp = altstack, .ss_size = sizeof altstack };
    sigaltstack(&ss, NULL);

    struct sigaction sa = {0};
    sa.sa_sigaction = crash_handler;
    sa.sa_flags = SA_SIGINFO | SA_ONSTACK | SA_RESETHAND;
    sigemptyset(&sa.sa_mask);
    sigaction(SIGSEGV, &sa, NULL);
    sigaction(SIGBUS,  &sa, NULL);
    sigaction(SIGFPE,  &sa, NULL);
    sigaction(SIGILL,  &sa, NULL);
    sigaction(SIGABRT, &sa, NULL);
}

Never "recover" from a real segfault and keep going

Returning from a SIGSEGV handler re-executes the faulting instruction, which faults again — an infinite loop. siglongjmp-ing past it leaves whatever lock or data structure the crashing code was holding in an undefined state. Log, dump, die, and let the supervisor restart you.

Call backtrace() once at startup before you need it: its first call may dlopen libgcc, which allocates — not something to do for the first time inside a crash handler.


Real-Time Signals

POSIX.1b added signals SIGRTMIN through SIGRTMAX (34–64 on Linux applications) with stronger semantics than standard signals:

PropertyStandard signals (1–31)Real-time signals (34–64)
Multiple instancesCoalesce — one pending bitQueue — each instance delivered
Delivery orderUnspecified (Linux delivers lowest number first)Lowest number first; same signal FIFO
PayloadNoneAn int or pointer via sigqueue() (si_value)
Predefined meaningYesNone — application-defined
Queue limitN/ARLIMIT_SIGPENDING per user (ulimit -i, often ~60k)
// Sender: queue a signal carrying a job ID
union sigval v = { .sival_int = 4217 };
sigqueue(target_pid, SIGRTMIN + 1, v);

// Receiver (SA_SIGINFO handler or signalfd)
static void on_rt(int sig, siginfo_t *si, void *ctx) {
    int job_id = si->si_value.sival_int;   // 4217
    /* ... record job_id in a lock-free queue ... */
}

In practice, real-time signals are mostly used as plumbing: glibc threading internals, POSIX timers (timer_create with SIGEV_SIGNAL), legacy POSIX AIO completions, and Go's runtime, which uses SIGURG (a standard signal) for asynchronous goroutine preemption since Go 1.14. For application-level messaging between processes, prefer a real IPC channel — see IPC.


Job Control

Job control is the terminal subsystem's use of signals to manage foreground and background process groups.

  • A session (one per login / terminal) contains process groups (one per pipeline/job), which contain processes.
  • The terminal has exactly one foreground process group. Keystrokes generate signals for every process in that group: Ctrl+C → SIGINT, Ctrl+\ → SIGQUIT, Ctrl+Z → SIGTSTP.
  • A background job that tries to read the terminal gets SIGTTIN (and is stopped); writing may get SIGTTOU if stty tostop is set.
  • When the terminal hangs up (SSH drops), the session leader gets SIGHUP, and the shell forwards SIGHUP to its jobs — which is why a long make dies when your SSH connection does.
$ sleep 1000 | cat &          # one job, one process group, two processes
[1] 4410
$ ps -o pid,pgid,sid,stat,cmd -t pts/0
  PID  PGID   SID STAT CMD
 4102  4102  4102 Ss   -bash
 4409  4409  4102 S    sleep 1000
 4410  4409  4102 S    cat
$ kill -TERM -4409            # negative PID = signal the whole process group
$ fg                          # SIGCONT + make it the foreground group
$ nohup ./long_task &         # ignore SIGHUP; output to nohup.out
$ setsid ./daemon             # new session, no controlling terminal at all
$ disown -h %1                # bash: don't forward SIGHUP to job 1

Process groups matter outside terminals too. kill(-pgid, sig) signals a whole tree that shares a group. Supervisors (and timeout(1)) put a child in its own group so they can kill it and all its descendants — otherwise killing a shell script leaves its children running as orphans. In Go:

cmd := exec.Command("bash", "-c", "./build.sh")
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}   // child leads a new group
_ = cmd.Start()
// later: signal the entire group, not just bash
_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGTERM)

Graceful Shutdown Pattern

Every production server should handle SIGTERM for graceful shutdown:

import signal
import sys

shutdown_requested = False

def handle_sigterm(signum, frame):
    global shutdown_requested
    shutdown_requested = True

signal.signal(signal.SIGTERM, handle_sigterm)
signal.signal(signal.SIGINT, handle_sigterm)  # Also handle Ctrl+C

# Main server loop
while not shutdown_requested:
    request = accept_connection()
    handle(request)

# Cleanup
flush_pending_writes()
close_database_connections()
sys.exit(0)

Note: CPython runs Python-level signal handlers only in the main thread, between bytecodes. The C-level handler just sets a flag. That makes Python handlers safe to write in normal Python — but it also means a main thread blocked in a long C call (accept() without a timeout, a big NumPy op) won't run the handler until that call returns or is interrupted with EINTR.

// Go graceful shutdown
server := &http.Server{Addr: ":8080"}
go server.ListenAndServe()

quit := make(chan os.Signal, 1)
signal.Notify(quit, syscall.SIGTERM, syscall.SIGINT)
<-quit  // Block until signal received

ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()

// Graceful shutdown: stop accepting new connections, wait for in-flight to finish
if err := server.Shutdown(ctx); err != nil {
    log.Fatal("Server forced to shutdown:", err)
}

signal.Notify channels must be buffered: the Go runtime does a non-blocking send, so an unbuffered channel can drop the signal if your goroutine isn't already waiting.

A Production-Grade Go Shutdown

The minimal version above has three gaps: it doesn't wait for the load balancer to stop sending traffic, it doesn't fail readiness first, and it doesn't shut down background workers. A fuller version:

package main

import (
	"context"
	"errors"
	"log"
	"net/http"
	"os"
	"os/signal"
	"sync/atomic"
	"syscall"
	"time"
)

func main() {
	var draining atomic.Bool

	mux := http.NewServeMux()
	mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
		if draining.Load() {
			http.Error(w, "draining", http.StatusServiceUnavailable)
			return
		}
		w.WriteHeader(http.StatusOK)
	})
	mux.HandleFunc("/", handler)

	srv := &http.Server{Addr: ":8080", Handler: mux, ReadHeaderTimeout: 5 * time.Second}

	// ctx is cancelled on the first SIGTERM/SIGINT.
	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)
	defer stop()

	go func() {
		if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
			log.Fatalf("listen: %v", err)
		}
	}()
	workersDone := startBackgroundWorkers(ctx) // workers watch ctx.Done()

	<-ctx.Done()
	stop() // restore default behavior: a second Ctrl+C now kills immediately
	log.Println("shutdown: signal received")

	// 1. Fail readiness so endpoints controllers and LBs stop routing to us.
	draining.Store(true)
	// 2. Keep serving while that propagates (skip if a preStop sleep already did this).
	time.Sleep(5 * time.Second)

	// 3. Stop accepting, wait for in-flight requests (bounded).
	shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
	defer cancel()
	if err := srv.Shutdown(shutdownCtx); err != nil {
		log.Printf("shutdown: forced close: %v", err)
		_ = srv.Close()
	}

	// 4. Drain workers / flush producers / close DB pools.
	select {
	case <-workersDone:
	case <-shutdownCtx.Done():
		log.Println("shutdown: workers did not finish in time")
	}
	log.Println("shutdown: complete")
	os.Exit(0)
}
flowchart LR
    A["SIGTERM"] --> B["Fail readiness probe"]
    B --> C["Keep serving 5 to 10s while endpoints update"]
    C --> D["Close listener, stop accepting"]
    D --> E["Wait for in-flight requests (bounded by deadline)"]
    E --> F["Stop consumers, commit offsets, flush buffers"]
    F --> G["Close DB pools, release locks and leases"]
    G --> H["exit 0"]
    E -->|"deadline exceeded"| I["Force-close connections, exit non-zero"]

Shutdown checklist by workload:

WorkloadOn SIGTERM
HTTP APIFail readiness → short delay → Shutdown() → close pools
gRPC serverGracefulStop() with a timeout fallback to Stop()
WebSocket / long-pollSend a close frame or "reconnect" hint, then close; don't wait for clients to leave voluntarily
Kafka consumerStop polling → finish current batch → commit offsets → Close() (leaves the group cleanly, triggers immediate rebalance)
Queue worker (SQS, RabbitMQ)Stop fetching → finish or nack/requeue in-flight messages → exit
Batch jobCheckpoint progress so a restart resumes, or exit non-zero so it's retried
Leader-elected serviceRelease the lease/lock explicitly so a peer takes over immediately instead of after TTL

kill Command and Signal Sending

kill <pid>          # Send SIGTERM (polite request to terminate)
kill -15 <pid>      # Same — SIGTERM = 15
kill -TERM <pid>    # Same, by name
kill -9 <pid>       # SIGKILL — immediate, no cleanup
kill -HUP <pid>     # SIGHUP — often used to reload config (nginx, sshd)
kill -USR1 <pid>    # SIGUSR1 — application-defined (e.g., dump stats, rotate logs)

killall nginx       # Send SIGTERM to all processes named "nginx"
pkill -f "python app.py"  # Match by full command line

# Check whether a process exists (no signal is actually sent):
kill -0 <pid>       # No signal sent; returns 0 if process exists, EPERM if no permission

More tools you will use when debugging signal behavior:

kill -l                         # list signal names and numbers on this system
kill -TERM -- -<pgid>           # signal an entire process group
timeout -s TERM -k 10 60 ./job  # SIGTERM after 60s, SIGKILL 10s later if still alive
docker kill --signal=SIGHUP web # signal a container's PID 1
docker stop -t 30 web           # SIGTERM (or STOPSIGNAL), then SIGKILL after 30s
systemctl kill -s HUP nginx     # signal a systemd unit's processes
kubectl exec pod -- kill -USR1 1  # signal PID 1 inside a pod (needs a kill binary in the image)

# Who is sending signals to my process? (bpftrace, needs root)
bpftrace -e 'tracepoint:signal:signal_generate /args->pid == 1234/ {
  printf("%s(%d) -> sig %d\n", comm, pid, args->sig); }'

# Trace only signal-related activity
strace -f -e trace=none -e signal=all -p <pid>

Permission rules: an unprivileged process may signal another if its real or effective UID matches the target's real or saved UID. CAP_KILL bypasses this. Inside a user namespace, you can only signal processes in that namespace or its descendants. SIGCONT is also allowed to any process in the same session.

Conventional signal meanings (not enforced by the kernel — every daemon defines its own):

DaemonSIGHUPSIGUSR1SIGUSR2SIGQUIT
nginxReload config, start new workersReopen log filesUpgrade binary on the flyGraceful shutdown
PostgreSQLReload config——Immediate shutdown (SIGTERM = smart, SIGINT = fast)
HAProxy—Soft stopReload (master-worker)—
Go runtimeDefault (terminate)——Dump all goroutine stacks and exit
JVM———Thread dump to stdout (doesn't exit)

Failure Modes

SymptomLikely causeHow to confirmFix
Pod always exits 137 after exactly the grace periodPID 1 doesn't handle/forward SIGTERMkubectl describe pod → Exit Code: 137, Reason: Error; ps in container shows sh -c as PID 1Exec-form CMD, exec in scripts, or tini
502s during every rolloutApp exits before LBs stop routingErrors cluster within seconds of pod terminationpreStop sleep or post-SIGTERM delay; fail readiness first
kill -9 does nothingProcess in D (uninterruptible) stateps -o stat,wchan:32 -p <pid>, cat /proc/<pid>/stackFix the underlying I/O (dead NFS, failing disk); reboot as last resort
Process hangs forever, occasionallyNon-async-signal-safe handler deadlockedgdb -p shows handler frame above malloc/__lll_lock_waitHandler only sets flag / writes to pipe
Child processes accumulate as <defunct>Parent never reapsps -eo stat,ppid,cmd | grep ^Zwaitpid loop, SA_NOCLDWAIT, or tini as PID 1
Process dies silently, exit 141Unhandled SIGPIPE128 + 13 = 141; strace shows --- SIGPIPE ---SIG_IGN + check EPIPE, or MSG_NOSIGNAL
Random EINTR errors in logsSignals interrupting syscalls without SA_RESTARTstrace shows EINTR/ERESTARTSYSSA_RESTART, retry loops, or block signals in worker threads
Child spawned by service ignores killInherited blocked mask or SIG_IGN across exec/proc/<child>/status SigBlk/SigIgnReset mask and dispositions before exec
Second Ctrl+C doesn't kill a stuck shutdownHandler still installed during shutdown—Restore default after first signal (signal.NotifyContext stop(), SA_RESETHAND)
Wrong process killedPID reuse between lookup and kill()Rare; hard to provepidfd_open + pidfd_send_signal

Exit Code Cheat Sheet

Exit codeMeaning
0Clean exit
1–125Application-defined error
126 / 127Command not executable / not found (shell)
128 + NKilled by signal N (as reported by shells, Docker, Kubernetes)
130SIGINT (Ctrl+C)
134SIGABRT (assert, glibc heap check)
137SIGKILL (OOM killer or grace period expired)
139SIGSEGV
141SIGPIPE
143SIGTERM with default action

Real Numbers

MeasurementTypical valueNotes
kill() → handler start (target idle on same machine)~1–5 µsDominated by wakeup + context switch
Signal delivery + rt_sigreturn overhead per signal~1–2 µsWhy signals are a bad high-rate IPC mechanism
Max standard-signal "queue" depth1Pending bitmap, coalesces
Real-time signal queue limitRLIMIT_SIGPENDING, often ~60k per userulimit -i
Kubernetes default terminationGracePeriodSeconds30 sPer pod; includes preStop
Docker docker stop default timeout10 s-t to change
systemd TimeoutStopSec default90 sDefaultTimeoutStopSec in system.conf
Endpoint removal propagation in Kubernetes~1–10 sDepends on cluster size, kube-proxy mode, external LB
Go async preemption signalSIGURG, sent when a goroutine runs > ~10 msVisible in strace as frequent SIGURG

Decision Guide

flowchart TD
    A["Need to react to a signal"] --> B{"Single-threaded C event loop?"}
    B -->|"yes, Linux only"| C["signalfd in epoll"]
    B -->|"yes, portable"| D["Self-pipe or eventfd"]
    B -->|no| E{"Multithreaded C or C++?"}
    E -->|yes| F["Block in main, dedicated sigwait thread"]
    E -->|no| G{"Managed runtime?"}
    G -->|"Go"| H["signal.NotifyContext"]
    G -->|"Python"| I["signal.signal sets flag, main thread acts"]
    G -->|"Node, Java, Rust"| J["process.on SIGTERM, Runtime shutdown hook, signal-hook crate"]
    A --> K{"Crash reporting for SIGSEGV?"}
    K -->|yes| L["SA_ONSTACK plus sigaltstack, write backtrace, re-raise"]

Rules of thumb:

  • Handlers do one thing: set a flag or write a byte. All real work happens in normal code.
  • Use SIGTERM for graceful stop, SIGHUP for reload, SIGUSR1/SIGUSR2 for diagnostics. Don't invent meanings for SIGINT beyond "stop".
  • Never use signals as a general IPC channel — no payload (except RT), coalescing, and expensive. Use a socket, pipe, or eventfd (IPC).
  • Always budget the shutdown: every graceful path needs a deadline and a forced path.
  • In containers, make sure your app is PID 1 or behind a real init.
  • Ignore SIGPIPE in C/C++ servers on day one.

Advanced Topics

  • sigsuspend() and the lost-wakeup race. Checking a flag and then calling pause() has a race: the signal can arrive between the check and the pause(), and you sleep forever. sigsuspend(&oldmask) atomically unblocks and sleeps. Modern equivalents: ppoll()/pselect()/epoll_pwait() take a signal mask applied atomically for the duration of the wait.
  • Signals and vfork/posix_spawn. posix_spawn lets you set the child's signal mask (posix_spawnattr_setsigmask) and reset handled signals to default (posix_spawnattr_setsigdefault) — the clean way to avoid inherited-mask bugs.
  • Seccomp and SIGSYS. A seccomp filter can return SECCOMP_RET_TRAP, making forbidden syscalls deliver SIGSYS with the syscall number in si_syscall — handy for emulating or logging syscalls in sandboxes. See Process Security.
  • Core dumps. Signals with "core" default action write a core file subject to ulimit -c, /proc/sys/kernel/core_pattern (often piped to systemd-coredump or apport), and container settings. coredumpctl list / coredumpctl gdb to inspect.
  • The cgroup freezer (cgroup.freeze in cgroups v2) stops a whole group of processes without sending them SIGSTOP, so they can't observe or race with it. docker pause and Kubernetes checkpointing use it.
  • SIGKILL from the OOM killer is indistinguishable from the kubelet's grace-period kill by exit code alone — check kubectl get pod -o jsonpath='{.status.containerStatuses[*].lastState}' for reason: OOMKilled and the kernel log (dmesg | grep -i oom).

Interview Questions

"What's the difference between SIGTERM and SIGKILL?" SIGTERM is a request — the process can catch it, do cleanup (flush buffers, finish requests, close connections), and exit gracefully. SIGKILL is an order — the kernel kills the process immediately, no handler runs, no cleanup happens. Kubernetes sends SIGTERM first, waits terminationGracePeriodSeconds, then sends SIGKILL.

"Why can't you call printf() in a signal handler?" printf calls malloc internally, which uses locks and shared state. If the signal interrupts a malloc call in the main thread, and the handler also calls malloc, the allocator's state is corrupted. Signal handlers must only call async-signal-safe functions (mostly raw syscalls). Use the self-pipe trick or signalfd to handle signals safely in the main event loop.

"How do you implement graceful shutdown?" Register a handler for SIGTERM (and SIGINT). Handler sets a flag or writes to a pipe. Main loop checks the flag after completing each request/batch. When set: stop accepting new work, finish in-flight requests within a deadline, flush buffers, close connections, then exit(0). In Go: signal.Notify + http.Server.Shutdown(ctx) with a timeout.

"What is SA_RESTART?" A flag on sigaction that tells the kernel to automatically restart blocking syscalls (read, write, accept, etc.) that were interrupted by signal delivery. Without it, those syscalls return -1/EINTR and you must retry them manually. Most servers want SA_RESTART to avoid sprinkling EINTR checks everywhere.

"When exactly is a signal delivered?" When a thread that doesn't block it next transitions from kernel mode back to user mode — returning from a syscall, an interrupt, or being scheduled. The kernel checks the pending set, builds a signal frame on the user stack, and redirects execution to the handler. A thread in uninterruptible sleep (D state) won't see it until the sleep ends.

"Why do I sometimes lose SIGCHLD signals?" You don't lose the information, you lose the count: standard signals are a pending bitmap, so multiple children exiting close together produce one delivery. Always reap with while (waitpid(-1, &st, WNOHANG) > 0).

"Why does my containerized app ignore SIGTERM?" Either a shell is PID 1 and doesn't forward it (shell-form CMD), or your app is PID 1 without a handler — and the kernel ignores default-action signals for a namespace's init process. Use exec form, exec in entrypoint scripts, or tini.

"Your pods return 502s during deploys even though you handle SIGTERM. Why?" SIGTERM and endpoint removal race. The app stops accepting connections before kube-proxy and the load balancers have removed the pod. Add a preStop sleep (or delay after SIGTERM while failing readiness), make the grace period long enough for sleep + drain.