Signals & Process Control
A signal is a software interrupt delivered to a process by the kernel (or by another process). Signals are how the OS tells a process "something happened" — a timer expired, a child exited, the user pressed Ctrl+C, or another process asked you to terminate.
Signals seem simple but have subtle behavior that causes real production bugs: dropped signals, uninterruptible syscalls, unsafe handler code, and graceful shutdown failures.
Why Signals Exist
Unix needed a way to notify a process of an event without the process asking. Almost everything else in Unix is pull-based: you read() a file, you wait() for a child, you poll() a socket. But some events cannot wait for the process to ask:
- The hardware faulted on your behalf. You dereferenced a NULL pointer; the CPU raised a page fault the kernel cannot satisfy. The kernel must tell you — right now, at that instruction.
- A human wants you to stop. Ctrl+C in a terminal must interrupt a program stuck in a loop that never calls
read(). - The system wants you to stop. A shutdown, an orchestrator draining a node, an OOM condition.
- A related process changed state. A child exited; a pipe reader went away.
Signals are the 1970s answer: a tiny, fixed vocabulary of numbered events (originally 15, now 64 on Linux) that the kernel can inject into any process at almost any instruction. That power is exactly what makes them dangerous — "at almost any instruction" includes the middle of malloc().
Mental model
Think of a signal as a hardware interrupt, but for processes. A hardware interrupt makes the CPU stop what it is doing and jump to a handler; a signal makes a thread stop what it is doing and jump to a handler. Everything that is hard about interrupt handlers in kernels (reentrancy, locking, doing minimal work) is equally hard about signal handlers in user space. See Interrupts, DMA & Drivers for the hardware side.
The Signal Model
When a signal is sent to a process:
- The kernel sets a flag in the process's PCB (on Linux, a bit in a pending set on the
task_structor on the sharedsignal_struct) - At the next opportunity — specifically, when a target thread is about to return from kernel mode to user mode (after a syscall, an interrupt, or being scheduled back in) — the kernel delivers the signal
- The process executes the registered handler (or the default action)
Important: Signal delivery is asynchronous — a signal sent by kill(pid, SIGTERM) may not be delivered immediately. The kernel queues it and delivers it when the process next runs.
Pending vs Blocked: the Two Bitmasks
Every thread has, conceptually, two sets of signals:
| Set | Meaning | Where to see it |
|---|---|---|
| Pending | Signals that have been generated but not yet delivered | SigPnd (thread) and ShdPnd (process-wide) in /proc/<pid>/status |
| Blocked (the signal mask) | Signals this thread refuses to accept right now; they stay pending | SigBlk |
| Ignored | Disposition set to SIG_IGN; the signal is discarded on generation | SigIgn |
| Caught | A handler is installed | SigCgt |
$ grep -E 'Sig|ShdPnd' /proc/$(pgrep -n nginx)/status
SigQ: 0/63446 # queued signals / RLIMIT_SIGPENDING
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000001000 # bit 12 set → signal 13 (SIGPIPE) ignored
SigCgt: 0000000180014a07 # handlers for HUP, INT, QUIT, USR1, USR2, TERM, CHLD, WINCH, ...
Each hex mask is a 64-bit bitmap where bit n−1 represents signal n. 0x1000 is bit 12 → signal 13 → SIGPIPE. Decoding these masks is often the fastest way to answer "why didn't my process react to SIGTERM?" — if SigCgt doesn't contain bit 14 (signal 15), nothing is handling it, and if SigBlk does, it is being blocked.
Decode masks in one line
mask=0000000180014a07; for i in $(seq 1 64); do (( (0x$mask >> (i-1)) & 1 )) && echo -n "$(kill -l $i) "; done; echo
# HUP INT QUIT USR1 USR2 TERM CHLD WINCH ...
Standard signals do not queue. The pending set for signals 1–31 is a bitmap, not a counter. If three SIGCHLDs arrive while the signal is blocked or before the handler runs, the bit is set once and you get one delivery. This is the root cause of the classic "lost children" bug (see SIGCHLD and Reaping). Real-time signals (34–64) do queue — see Real-Time Signals.
The Delivery Path, Step by Step
sequenceDiagram
participant S as "Sender (kill)"
participant K as Kernel
participant T as "Target thread"
S->>K: "kill(pid, SIGTERM)"
K->>K: "permission check, set bit in shared pending set"
K->>T: "pick a thread that does not block SIGTERM, set TIF_SIGPENDING, wake it if sleeping interruptibly"
Note over T: "thread is running user code or sleeping in a syscall"
T->>K: "syscall returns or interrupt ends, exit_to_user_mode checks TIF_SIGPENDING"
K->>K: "get_signal dequeues SIGTERM and looks up disposition"
alt handler installed
K->>T: "build signal frame on user stack, set PC to handler"
T->>T: "handler runs"
T->>K: "rt_sigreturn restores saved registers and mask"
K->>T: "resume where it was interrupted"
else default action is terminate
K->>K: "do_group_exit kills all threads"
end
Key consequences of this design:
- Signals are only acted on at the user/kernel boundary. A thread spinning in pure user code (
while(1);) still gets the signal promptly, because the timer interrupt fires every few ms and the return-from-interrupt path checksTIF_SIGPENDING. - A thread asleep in the kernel must be woken. If it is in
TASK_INTERRUPTIBLEsleep (e.g. blocked inread()on a socket), the kernel wakes it; the syscall returns-EINTR(or is restarted). If it is inTASK_UNINTERRUPTIBLE(stateDinps) — typically waiting on disk I/O or an NFS server — the signal stays pending until the sleep ends. This is whykill -9sometimes "doesn't work": the process is stuck inDstate. Since Linux 2.6.25 there is alsoTASK_KILLABLE, which ignores everything except fatal signals; much NFS and page-fault code uses it. - The handler runs on the interrupted thread's own stack (unless you configure an alternate stack with
sigaltstack()), with a saved copy of registers.rt_sigreturnis a real syscall that the kernel arranges to be called when the handler returns — you can see it instrace. - While a handler runs, its own signal is blocked (plus anything in
sa_mask), unlessSA_NODEFERis set. That prevents a handler from being re-entered by its own signal.
$ strace -f -e trace=signal,read ./server
read(5, 0x7ffd..., 4096) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=4312, si_uid=1000} ---
rt_sigreturn({mask=[]}) = 0
read(5, ...
The si_pid and si_uid fields tell you who sent the signal — invaluable when something keeps killing your process and you don't know what. Use SA_SIGINFO to read them in your own handler.
Common Signals
| Signal | Number | Default Action | Description |
|---|---|---|---|
SIGHUP | 1 | Terminate | Hangup (terminal disconnected); often used to reload config |
SIGINT | 2 | Terminate | Keyboard interrupt (Ctrl+C) |
SIGQUIT | 3 | Core dump | Quit with core dump (Ctrl+\) |
SIGILL | 4 | Core dump | Illegal instruction (bad JIT output, wrong CPU feature like AVX-512 on an older host) |
SIGTRAP | 5 | Core dump | Breakpoint / trace trap (debuggers) |
SIGABRT | 6 | Core dump | abort() — failed assert(), glibc heap corruption detection ("double free or corruption") |
SIGBUS | 7 | Core dump | Bus error — bad memory access to an existing mapping (e.g. truncated mmap'd file) |
SIGFPE | 8 | Core dump | Arithmetic error (integer divide by zero) |
SIGKILL | 9 | Terminate | Immediate kill — cannot be caught or ignored |
SIGUSR1 | 10 | Terminate | User-defined signal 1 |
SIGSEGV | 11 | Core dump | Segmentation fault |
SIGUSR2 | 12 | Terminate | User-defined signal 2 |
SIGPIPE | 13 | Terminate | Write to a broken pipe (common in network code) |
SIGALRM | 14 | Terminate | Timer expired (alarm()) |
SIGTERM | 15 | Terminate | Polite termination request — can be caught |
SIGCHLD | 17 | Ignore | Child process changed state |
SIGCONT | 18 | Continue | Resume a stopped process |
SIGSTOP | 19 | Stop | Pause process — cannot be caught or ignored |
SIGTSTP | 20 | Stop | Terminal stop (Ctrl+Z) — can be caught |
SIGTTIN / SIGTTOU | 21 / 22 | Stop | Background process tried to read from / write to the terminal |
SIGXCPU | 24 | Core dump | CPU time limit (RLIMIT_CPU) exceeded |
SIGXFSZ | 25 | Core dump | File size limit (RLIMIT_FSIZE) exceeded |
SIGWINCH | 28 | Ignore | Terminal window resized |
SIGSYS | 31 | Core dump | Bad syscall — also what a seccomp filter sends with SECCOMP_RET_KILL/TRAP |
SIGRTMIN–SIGRTMAX | 34–64 | Terminate | Real-time signals — queued, ordered, carry data |
Numbers above are for Linux on x86-64 and arm64. They differ on other architectures (e.g. on MIPS, SIGCHLD is 18; on macOS, SIGUSR1 is 30) — always use names, never numbers, in portable code and scripts. kill -l prints the local mapping.
`SIGRTMIN` is not 32
glibc reserves signals 32 and 33 for its own threading implementation (thread cancellation and setxid broadcasting), so SIGRTMIN is 34 for applications. Always compute real-time signals as SIGRTMIN + n.
Where Signals Come From
| Source | Examples | si_code |
|---|---|---|
| Another process | kill(), tgkill(), sigqueue(), pidfd_send_signal() | SI_USER, SI_QUEUE |
| The terminal driver | Ctrl+C → SIGINT, Ctrl+\ → SIGQUIT, Ctrl+Z → SIGTSTP, hangup → SIGHUP | SI_KERNEL |
| A CPU exception | Bad pointer → SIGSEGV, divide by zero → SIGFPE, ud2 → SIGILL | SEGV_MAPERR, SEGV_ACCERR, FPE_INTDIV |
| Kernel events | Child exit → SIGCHLD, broken pipe → SIGPIPE, rlimit → SIGXCPU | CLD_EXITED, CLD_KILLED |
| Timers | alarm(), setitimer(), timer_create() | SI_TIMER |
| The OOM killer | SIGKILL | SI_KERNEL |
| The process itself | raise(), abort() | SI_TKILL |
Synchronous vs asynchronous: signals caused by the thread's own instruction (SIGSEGV, SIGFPE, SIGILL, SIGBUS from a fault) are synchronous — they are delivered to the faulting thread, immediately. Everything else is asynchronous — delivered to whichever thread the kernel picks.
SIGTERM vs SIGKILL
This distinction matters for every server process and is core to Kubernetes pod lifecycle management.
SIGTERM (15): Polite request to terminate. The process can catch this signal, finish in-flight requests, flush buffers, close database connections, and then exit cleanly. This is what kill <pid> sends by default.
SIGKILL (9): Unconditional, immediate termination. The kernel kills the process immediately — no handler runs, no cleanup, no buffered writes flushed. kill -9 <pid>. Cannot be caught, blocked, or ignored.
SIGSTOP / SIGCONT: Pause and resume. Used by debuggers, shells (Ctrl+Z), and cgroup freezers.
What SIGKILL does not lose and what it does:
| State | Survives SIGKILL? |
|---|---|
Data already passed to write() (in the kernel page cache) | Yes — the kernel owns it now and will flush it |
Data sitting in a user-space buffer (stdio, a Go bufio.Writer, a logger's ring buffer) | No |
Data that was fsync()ed | Yes |
| Open TCP connections | Closed by the kernel — peers see a FIN (or RST if unread data was pending) |
Advisory file locks (flock, fcntl) | Released by the kernel |
| Temp files, PID files, Unix socket files | Left behind |
| Distributed locks / leases (etcd, Redis, ZooKeeper) | Held until TTL expiry |
| In-flight HTTP requests | Clients see connection reset / 502 |
| Consumer group membership (Kafka) | Held until session timeout → delayed rebalance |
Kubernetes Pod Termination
Kubernetes pod termination sequence:
1. kubectl delete pod → API server sets pod.deletionTimestamp
2. kubelet sends SIGTERM to PID 1 in the container
3. Container has `terminationGracePeriodSeconds` (default: 30s) to exit cleanly
4. If not exited after grace period: kubelet sends SIGKILL
5. Container terminates immediately
If your server doesn't handle SIGTERM, it gets SIGKILL'd after 30 seconds — in-flight requests are dropped, connections are abruptly closed, write buffers are lost.
The simplified sequence above hides two details that cause most real-world shutdown errors: the preStop hook and the endpoint-removal race.
sequenceDiagram
participant API as "API server"
participant KL as Kubelet
participant C as "Container PID 1"
participant EP as "EndpointSlice controller"
participant KP as "kube-proxy and ingress"
API->>KL: "pod marked Terminating (deletionTimestamp set)"
API->>EP: "pod marked Terminating"
par in parallel, no ordering guarantee
KL->>C: "run preStop hook (exec or HTTP), e.g. sleep 10"
Note over C: "grace period clock is already running"
KL->>C: "after preStop returns, send STOPSIGNAL (default SIGTERM)"
C->>C: "stop accepting, drain in-flight, exit 0"
and
EP->>KP: "remove pod IP from endpoints"
Note over KP: "takes 1 to 10+ seconds to propagate to every node and LB"
end
KL->>C: "SIGKILL if still running at terminationGracePeriodSeconds"
The race: the kubelet sending SIGTERM and the load balancers removing the pod from rotation happen concurrently. If your app exits immediately on SIGTERM, iptables rules on other nodes and external load balancers may still route new connections to it for a few seconds — those get connection refused and surface as 502/503s during every deploy.
The fix is to keep serving for a few seconds after SIGTERM before closing the listener:
spec:
terminationGracePeriodSeconds: 45 # must cover preStop + drain
containers:
- name: api
lifecycle:
preStop:
sleep:
seconds: 10 # native sleep action (K8s 1.30+ beta, GA in 1.33)
# older clusters: exec: { command: ["sleep", "10"] } (needs a sleep binary in the image)
The grace period includes the preStop hook
terminationGracePeriodSeconds is a single budget that starts when the pod is marked for deletion. If preStop sleeps 10s and your drain takes 30s, a 30s grace period kills you mid-drain. Budget: preStop + max request duration + cleanup + margin.
More Kubernetes signal details worth knowing:
STOPSIGNALin the Dockerfile changes which signal is sent first (e.g.STOPSIGNAL SIGQUITfor nginx's graceful shutdown,SIGWINCHfor Apache httpd's graceful-stop). Kubernetes honours the image's stop signal; alifecycle.stopSignalfield in the pod spec arrived as alpha in 1.33.- Only PID 1 gets the signal. The kubelet (via the container runtime) signals the container's init process. If PID 1 is
/bin/sh -c "node server.js", the shell receives SIGTERM — and most shells don't forward it. Node never hears about it and is SIGKILLed 30 seconds later. See PID 1 Is Special. - Exit code 137 = 128 + 9 (killed by SIGKILL — grace period expired or OOM-killed; check
reason). Exit code 143 = 128 + 15 (terminated by SIGTERM with no handler). A clean graceful shutdown should exit 0. - Jobs and
restartPolicy: Neverpods also get SIGTERM on deletion, and on node drain (kubectl drain) through the Eviction API, which respects PodDisruptionBudgets.
PID 1 Is Special
The kernel treats the init process of a PID namespace differently:
- No default actions. A signal whose disposition is
SIG_DFLis ignored for PID 1 if sent from inside its own namespace. If your app is PID 1 and has no SIGTERM handler,kill -TERM 1from inside the container does nothing. (From the parent namespace, the kernel does honourSIGKILLandSIGSTOP— that is howdocker killand the kubelet's final kill work.) - It must reap orphans. Any process whose parent dies is re-parented to PID 1 of its namespace. If PID 1 never calls
wait(), those orphans become zombies forever (see SIGCHLD and Reaping). - When PID 1 exits, the kernel SIGKILLs everything else in the namespace. No grace for children.
Solutions:
| Approach | How | When |
|---|---|---|
| Exec form in Dockerfile | CMD ["node", "server.js"] not CMD node server.js | Always — makes your app PID 1 directly |
exec in wrapper scripts | exec node server.js as the last line of entrypoint.sh | When you need a shell to set env vars first |
| A tiny init | docker run --init (injects tini), or ENTRYPOINT ["tini", "--"], or dumb-init | App spawns children or doesn't handle SIGTERM |
| Shared PID namespace | shareProcessNamespace: true in the pod spec → the pause container becomes PID 1 and reaps | Debug sidecars, multi-process pods |
# BAD: shell form — /bin/sh is PID 1 and swallows SIGTERM
CMD node server.js
# GOOD: exec form — node is PID 1 and receives SIGTERM directly
CMD ["node", "server.js"]
# GOOD: tini forwards signals and reaps zombies
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["node", "server.js"]
War Story (illustrative)
A team's deploys produced a 30-second pause and a burst of 502s on every rollout. kubectl get pod -o yaml showed exitCode: 137 for every terminated pod. The Dockerfile used CMD npm start; npm was PID 1 under /bin/sh, and neither forwarded SIGTERM to the Node process. Every pod was SIGKILLed at the grace-period deadline with requests in flight. Switching to CMD ["node", "dist/server.js"], adding a SIGTERM handler that calls server.close(), and a 5-second preStop sleep turned exit codes into 0 and eliminated the 502s.
Registering Signal Handlers
#include <signal.h>
// Old API (avoid in new code):
signal(SIGTERM, handler_func); // Behavior varies across implementations
// POSIX API (preferred):
struct sigaction sa = {0};
sa.sa_handler = handler_func;
sa.sa_flags = SA_RESTART; // Restart syscalls interrupted by this signal
sigemptyset(&sa.sa_mask);
sigaction(SIGTERM, &sa, NULL);
Why signal() is discouraged: historically (System V semantics) it reset the disposition to SIG_DFL after the first delivery, so a second Ctrl+C killed the process; BSD semantics did not. glibc's signal() uses BSD semantics today, but other libcs and compile flags differ. sigaction() is fully specified.
SA_RESTART: When a signal interrupts a blocking syscall (like read(), write(), accept()), the syscall can return -1 with errno = EINTR. With SA_RESTART, the kernel automatically restarts the syscall after the handler returns. Without it, your code must check for EINTR and retry manually.
// Without SA_RESTART — must handle EINTR manually:
ssize_t n;
do {
n = read(fd, buf, size);
} while (n == -1 && errno == EINTR);
`SA_RESTART` does not cover every syscall
Some calls are never restarted regardless of SA_RESTART: select(), poll(), epoll_wait(), nanosleep(), sigsuspend(), socket calls with a timeout set via SO_RCVTIMEO/SO_SNDTIMEO, and several others (man 7 signal, section "Interruption of system calls"). Event loops must always handle EINTR from their wait call. Sometimes that's a feature: an interrupted epoll_wait() is a natural place to check a "shutdown requested" flag.
sigaction Flags You Will Actually Use
| Flag | Effect |
|---|---|
SA_RESTART | Auto-restart interruptible syscalls |
SA_SIGINFO | Use sa_sigaction(int, siginfo_t *, void *) — get sender PID/UID, fault address, child exit status, queued value |
SA_ONSTACK | Run the handler on the alternate stack from sigaltstack() — required to handle stack-overflow SIGSEGV |
SA_NOCLDSTOP | Don't send SIGCHLD when children stop, only when they exit |
SA_NOCLDWAIT | Children don't become zombies (auto-reaped) |
SA_RESETHAND | Reset to SIG_DFL after one delivery — handy for "clean up, then re-raise to die with the right status" |
SA_NODEFER | Don't block the signal during its own handler (rarely what you want) |
A handler with SA_SIGINFO that logs who sent the signal (using only safe functions):
#include <errno.h>
#include <signal.h>
#include <unistd.h>
#include <string.h>
static void fmt_int(char *buf, long v) { // tiny async-signal-safe itoa
char tmp[24]; int i = 0, j = 0;
if (v == 0) tmp[i++] = '0';
while (v > 0) { tmp[i++] = '0' + v % 10; v /= 10; }
while (i) buf[j++] = tmp[--i];
buf[j] = '\0';
}
static void on_term(int sig, siginfo_t *si, void *ctx) {
(void)sig; (void)ctx;
int saved = errno; // handlers must preserve errno
char msg[64] = "SIGTERM from pid ";
fmt_int(msg + strlen(msg), si->si_pid); // strlen is async-signal-safe
strcat(msg, "\n"); // strcat too (POSIX.1-2008 TC2+)
write(STDERR_FILENO, msg, strlen(msg));
errno = saved;
}
int main(void) {
struct sigaction sa = {0};
sa.sa_sigaction = on_term;
sa.sa_flags = SA_SIGINFO | SA_RESTART;
sigemptyset(&sa.sa_mask);
sigaction(SIGTERM, &sa, NULL);
for (;;) pause();
}
Dispositions Across fork() and exec()
After fork() | After execve() | |
|---|---|---|
| Handlers | Inherited (same code exists in child) | Reset to SIG_DFL (handler code is gone) |
SIG_IGN | Inherited | Inherited |
| Signal mask | Inherited | Inherited |
| Pending signals | Cleared in child | Preserved |
The "SIG_IGN and mask survive exec" rows bite people: if a parent ignores SIGPIPE or blocks SIGTERM and then execs a child, the child starts with that state. Python historically set SIGPIPE to SIG_IGN, and subprocesses that expected to die on a closed pipe (yes | head) would instead spin on EPIPE; subprocess now restores defaults with restore_signals=True. Well-behaved supervisors reset dispositions and unblock all signals between fork() and exec().
Signal Masks
Each thread has a signal mask — a set of signals that are blocked (held pending, not delivered). Blocked signals are not ignored; they queue up and are delivered when unblocked.
sigset_t set;
sigemptyset(&set);
sigaddset(&set, SIGTERM);
// Block SIGTERM in this thread
sigprocmask(SIG_BLOCK, &set, NULL);
// ... critical section ...
sigprocmask(SIG_UNBLOCK, &set, NULL); // SIGTERM delivered now if pending
Important: In a multi-threaded process, signals are delivered to any thread that has not blocked them. SIGTERM sent to a process can be handled by any thread. The self-pipe trick or signalfd + signal masking in all threads ensures signals are handled in a dedicated thread.
sigprocmask() is unspecified in multithreaded programs; use pthread_sigmask() there (on Linux they behave the same per-thread).
Signals and Threads
Signal dispositions (handlers) are per process; signal masks are per thread; pending sets exist at both levels.
flowchart TD
A["Signal generated"] --> B{"Directed at a thread? (tgkill, pthread_kill, or a fault)"}
B -->|yes| C["Thread pending set"]
B -->|no, process-directed via kill| D["Shared process pending set"]
C --> E["That exact thread handles it when it unblocks"]
D --> F{"Any thread with the signal unblocked?"}
F -->|"yes"| G["Kernel picks one (prefers the main thread if eligible)"]
F -->|"no"| H["Stays pending until some thread unblocks it or calls sigwait"]
The robust multithreaded pattern: block all asynchronous signals in main() before creating any threads (threads inherit the mask), then dedicate one thread to sigwait():
#include <pthread.h>
#include <signal.h>
#include <stdio.h>
static void *signal_thread(void *arg) {
sigset_t *set = arg;
for (;;) {
int sig;
sigwait(set, &sig); // synchronous: normal code, any function is safe
if (sig == SIGTERM || sig == SIGINT) {
printf("shutting down\n"); // printf is fine here - not a handler
request_shutdown(); // e.g. set atomic flag, close listener
return NULL;
}
if (sig == SIGHUP) reload_config();
}
}
int main(void) {
static sigset_t set;
sigemptyset(&set);
sigaddset(&set, SIGTERM); sigaddset(&set, SIGINT); sigaddset(&set, SIGHUP);
pthread_sigmask(SIG_BLOCK, &set, NULL); // before any pthread_create
pthread_t t;
pthread_create(&t, NULL, signal_thread, &set);
start_worker_threads(); // all inherit the blocked mask
pthread_join(t, NULL);
}
Synchronous fault signals (SIGSEGV etc.) can't be redirected this way — they always go to the faulting thread, and blocking them while a fault occurs gets the process killed anyway.
Async-Signal Safety
Signal handlers run asynchronously — they can interrupt any point in your program's execution, including the middle of a malloc() call or an fprintf() call.
Problem: If your signal handler calls malloc() and your main code was in the middle of malloc() when the signal arrived, you have a re-entrant lock — the allocator's internal state is corrupted.
Rule: signal handlers must only call async-signal-safe functions.
Async-signal-safe functions are guaranteed to be safe even if called from a signal handler that interrupted the same function. The POSIX list includes:
write()(syscall) — yesprintf()— NO (calls malloc internally)malloc()/free()— NOexit()— NO (flushes stdio buffers)_exit()— yes (no cleanup)signal()/sigaction()— yes- Most syscalls — yes
Why Most Functions Aren't Safe
A function is unsafe in a handler if it does any of these:
- Takes a lock.
malloc()takes an arena lock;printf()takes theFILElock onstdout. If the main thread holds that lock when the signal arrives on the same thread, the handler blocks forever waiting for a lock its own thread holds — a self-deadlock. (A normal mutex is not reentrant; even a recursive one would let the handler walk into half-updated data.) - Touches global mutable state non-atomically. The free list in
malloc, the buffer pointer in aFILE, the static buffer instrtok()orlocaltime(). The handler sees the data mid-update. - Modifies
errnowithout restoring it. Evenwrite()can seterrno; the interrupted code might be between a failed syscall and readingerrno. - Allocates memory. Almost everything high-level does:
syslog(),printf()with some format specifiers, C++std::string, logging libraries, anything that throws.
sequenceDiagram
participant M as "Main code (same thread)"
participant L as "malloc arena lock"
participant H as "Signal handler"
M->>L: "malloc() acquires lock"
Note over M: "halfway through unlinking a free-list chunk"
H-->>M: "SIGALRM arrives, handler starts on this thread"
H->>L: "handler calls printf, which calls malloc, which tries to take the lock"
Note over H,L: "lock is held by this very thread, which is now suspended inside the handler: deadlock"
This bug is rare and nondeterministic: the signal has to land in the few hundred nanoseconds the lock is held. Programs with an unsafe handler can run for months before hanging, and the stack trace (gdb -p, pstack) shows a thread blocked in __lll_lock_wait inside a signal handler inside malloc — that signature is diagnostic.
What a handler may safely do:
| Safe | Not safe |
|---|---|
Set a volatile sig_atomic_t flag or a C11 lock-free atomic_int | Call printf, syslog, malloc, free, new, logging frameworks |
write() a byte to a pipe or eventfd | Acquire a pthread_mutex_t |
_exit() | exit() (runs atexit handlers, flushes stdio) |
kill(), raise(), sigaction() | pthread_cond_signal() (not on the list) |
read, open, close, fsync, waitpid | Anything in C++ that might throw or allocate |
memcpy, strlen, strcpy (POSIX.1-2008 TC2 and later) | localtime, strtok, getenv (non-reentrant) |
The full list is in man 7 signal-safety.
The only portable flag type
volatile sig_atomic_t is the one type the C standard guarantees can be written from a handler and read in main code without tearing. volatile stops the compiler from caching it in a register across loop iterations; sig_atomic_t guarantees the store is a single instruction. It does not give inter-thread ordering — for multithreaded code use _Atomic / std::atomic that is lock-free (atomic_is_lock_free).
The self-pipe trick: A common pattern for safely handling signals in event loops. The signal handler writes a byte to a pipe; the event loop (epoll/select) watches the read end and processes the signal notification in the main thread where all functions are safe.
int signal_pipe[2];
pipe(signal_pipe);
void signal_handler(int sig) {
// write() is async-signal-safe
write(signal_pipe[1], &sig, sizeof(sig));
}
// In event loop:
// Watch signal_pipe[0] for reads
// When readable: read the signal number, handle it safely
A production-grade version needs three more details: make both ends non-blocking (if the pipe fills during a signal storm, a blocking write() in the handler would hang the process), set O_CLOEXEC so children don't inherit it, and save/restore errno:
#include <fcntl.h>
#include <errno.h>
#include <signal.h>
#include <sys/epoll.h>
#include <unistd.h>
static int sigpipe_fds[2];
static void on_signal(int sig) {
int saved = errno;
unsigned char b = (unsigned char)sig;
(void)write(sigpipe_fds[1], &b, 1); // EAGAIN when full is fine: a wakeup is already pending
errno = saved;
}
int setup_signals(int epfd) {
if (pipe2(sigpipe_fds, O_NONBLOCK | O_CLOEXEC) < 0) return -1;
struct sigaction sa = { .sa_handler = on_signal, .sa_flags = SA_RESTART };
sigemptyset(&sa.sa_mask);
sigaction(SIGTERM, &sa, NULL);
sigaction(SIGINT, &sa, NULL);
sigaction(SIGHUP, &sa, NULL);
struct epoll_event ev = { .events = EPOLLIN, .data.fd = sigpipe_fds[0] };
return epoll_ctl(epfd, EPOLL_CTL_ADD, sigpipe_fds[0], &ev);
}
void drain_signals(void) { // called from the event loop when readable
unsigned char b;
while (read(sigpipe_fds[0], &b, 1) == 1) {
if (b == SIGHUP) reload_config(); // any function is safe here
else begin_graceful_shutdown();
}
}
This trick was popularized by D. J. Bernstein and is how libuv (Node.js), libevent, and Python's signal.set_wakeup_fd() work on platforms without signalfd.
signalfd (Linux): A modern alternative — converts signals to file descriptor events, which can be watched with epoll. No signal handler needed, no async-signal-safety concerns.
sigset_t mask;
sigemptyset(&mask);
sigaddset(&mask, SIGTERM);
sigaddset(&mask, SIGUSR1);
// Block these signals from normal delivery
sigprocmask(SIG_BLOCK, &mask, NULL);
// Create a file descriptor that receives these signals
int sfd = signalfd(-1, &mask, SFD_NONBLOCK);
// Now watch sfd with epoll — signal delivery becomes an event
Reading from it returns struct signalfd_siginfo records:
#include <sys/signalfd.h>
struct signalfd_siginfo fdsi;
while (read(sfd, &fdsi, sizeof fdsi) == sizeof fdsi) {
switch (fdsi.ssi_signo) {
case SIGTERM: printf("TERM from pid %u\n", fdsi.ssi_pid); begin_graceful_shutdown(); break;
case SIGUSR1: dump_stats(); break;
}
}
signalfd pitfalls
- You must block the signals in every thread, or the kernel may deliver them the normal way to an unblocked thread and the fd never sees them. Block before creating threads.
- Children inherit the blocked mask across
exec(). A child spawned by a signalfd-using program starts with SIGTERM blocked and will ignore yourkill. Unblock in the child betweenfork()andexec()(or useposix_spawnattr_setsigmask). - Standard signals still coalesce — a signalfd doesn't make three
SIGCHLDs into three reads. Always reap children in awaitpid(-1, ..., WNOHANG)loop.
| Technique | Handler runs? | Works with epoll | Portability | Typical users |
|---|---|---|---|---|
| Flag in handler, check in loop | Yes (minimal) | Needs EINTR wakeup | POSIX | Simple CLIs |
Self-pipe / eventfd | Yes (one write) | Yes | POSIX (eventfd: Linux) | libuv, libevent, Python |
signalfd | No | Yes | Linux only | systemd, modern C daemons |
Dedicated sigwait() thread | No | N/A | POSIX | Multithreaded C/C++ servers |
| Runtime-managed channel | No (runtime does it) | N/A | Per-language | Go signal.Notify, Rust signal-hook, Java sun.misc.Signal |
SIGCHLD and Reaping
When a child exits, the kernel keeps a small record — exit status, resource usage, PID — until the parent collects it with wait()/waitpid(). A child that has exited but not been collected is a zombie (state Z, shown as <defunct> in ps). Zombies use no memory or CPU, but they hold a PID and a process-table slot. Enough of them and fork() fails with EAGAIN — see the PID exhaustion discussion in Namespaces & cgroups.
stateDiagram-v2
[*] --> Running: "fork()"
Running --> Zombie: "exit() - SIGCHLD sent to parent"
Zombie --> [*]: "parent calls waitpid()"
Running --> Orphan: "parent dies first"
Orphan --> Running: "re-parented to PID 1 or subreaper"
SIGCHLD and zombie prevention:
// Automatically reap children without calling wait()
struct sigaction sa = {.sa_handler = SIG_DFL, .sa_flags = SA_NOCLDWAIT};
sigaction(SIGCHLD, &sa, NULL);
Or install a handler that calls waitpid(-1, NULL, WNOHANG) in a loop.
Setting SIGCHLD to SIG_IGN explicitly has the same auto-reap effect on Linux. Both approaches mean you can't learn exit statuses — if you need them (a supervisor, a job runner), you must reap.
The loop matters. Because standard signals coalesce, one SIGCHLD delivery may represent several dead children:
static void on_sigchld(int sig) {
(void)sig;
int saved = errno;
// WRONG: if (waitpid(-1, NULL, WNOHANG) > 0) ... reaps only one child
while (waitpid(-1, NULL, WNOHANG) > 0)
; // reap every child that has exited
errno = saved;
}
Why WNOHANG: without it, the last iteration blocks inside the handler waiting for a child that hasn't exited yet — freezing the whole thread.
Related mechanisms:
prctl(PR_SET_CHILD_SUBREAPER, 1)makes a process adopt orphaned descendants instead of PID 1. Used bysystemd --user,tini -s, and container shims (containerd-shim is a subreaper for the container's processes).prctl(PR_SET_PDEATHSIG, SIGTERM)asks the kernel to signal the child when its parent thread dies — a cheap way to avoid orphaned workers. (Caveat: it fires when the parent thread that forked exits, not just the process.)pidfd_open()(Linux 5.3+) gives a file descriptor for a process; it becomes readable when the process exits, so you can watch children with epoll and avoidSIGCHLDentirely.pidfd_send_signal()also removes the PID-reuse race inkill()— you can't accidentally signal an unrelated process that recycled the PID.
// Go: os/exec reaps for you - Wait() calls wait4() on that specific PID.
cmd := exec.Command("worker", "--batch", "42")
if err := cmd.Start(); err != nil { log.Fatal(err) }
err := cmd.Wait() // forgetting Wait() leaks a zombie per child
var ee *exec.ExitError
if errors.As(err, &ee) {
if ws, ok := ee.Sys().(syscall.WaitStatus); ok && ws.Signaled() {
log.Printf("worker killed by %v", ws.Signal())
}
}
SIGPIPE: The Silent Killer
When you write to a socket or pipe whose read end is closed, the kernel sends SIGPIPE to your process. Default action: terminate. This silently kills network servers if not handled.
// Option 1: ignore SIGPIPE globally
signal(SIGPIPE, SIG_IGN);
// write() will then return -1 with errno = EPIPE instead of killing the process
// Option 2: use MSG_NOSIGNAL per-send
send(sockfd, buf, len, MSG_NOSIGNAL);
// Returns -1/EPIPE without generating SIGPIPE
Most long-running servers ignore SIGPIPE globally and check write/send return codes.
Why does the default kill you at all? Because it makes shell pipelines work: in cat huge.log | head -1, once head exits, cat's next write raises SIGPIPE and cat dies instantly instead of reading the rest of a 50 GB file. For a server, a client disconnecting mid-response is routine and must never kill the process.
Language runtimes handle this for you to varying degrees:
| Runtime | Default behavior |
|---|---|
| Go | Ignores SIGPIPE on sockets and non-stdout fds (you get EPIPE). A write to a broken stdout/stderr still kills the program, matching shell expectations. |
| Python | Sets SIGPIPE to SIG_IGN at startup → BrokenPipeError exceptions |
| Node.js | Ignores SIGPIPE → EPIPE error events |
| Rust | std sets SIGPIPE to SIG_IGN before main for binaries → ErrorKind::BrokenPipe |
| Java | JVM ignores it → IOException: Broken pipe |
| C / C++ | Default action — process dies. You must ignore it yourself or use MSG_NOSIGNAL. On macOS/BSD use setsockopt(SO_NOSIGPIPE) since MSG_NOSIGNAL is not universal. |
War Story (illustrative)
A C++ service backed by a third-party client library crashed a few times a day with no core dump, no log line and exit status 141 (128 + 13). The library wrote to a pooled TCP connection that the remote load balancer had idle-closed. The service had set SIGPIPE to SIG_IGN — but in a worker thread after startup, while the library's background thread had already been created. Dispositions are process-wide, so that wasn't the problem; the real cause was an embedded scripting runtime that re-installed SIG_DFL during its own initialization. Grepping /proc/<pid>/status for SigIgn before and after init found it in minutes.
SIGSEGV and SIGBUS Handlers
Fault signals are synchronous and land on the thread that executed the bad instruction.
| Signal | Typical cause | si_code |
|---|---|---|
SIGSEGV | Address not mapped at all (NULL deref, use-after-munmap) | SEGV_MAPERR |
SIGSEGV | Mapped but wrong permission (write to read-only page, execute NX page, guard page) | SEGV_ACCERR |
SIGBUS | Access to an mmap'd file beyond its current end (file was truncated) | BUS_ADRERR |
SIGBUS | Hardware memory error reported by the machine-check handler | BUS_MCEERR_AR |
SIGBUS | Unaligned access on architectures that enforce alignment | BUS_ADRALN |
When is it legitimate to handle SIGSEGV?
- To print a crash report and die. Install a handler with
SA_ONSTACK | SA_RESETHAND | SA_SIGINFO, write a stack trace withbacktrace_symbols_fd()(writes directly to an fd — nomalloc), thenraise(sig)to die with the correct status and core dump.SA_ONSTACKwith asigaltstack()is mandatory: a stack overflow fault happens precisely because the stack is exhausted, so the handler cannot run on it. - As a deliberate memory-management technique. The JVM uses
SIGSEGVfor implicit null checks and safepoint polling (it reads from a page that it makes unreadable when it wants threads to stop). Garbage collectors and userspace page-in schemesmprotect()pages and catch the fault. Modern Linux code should preferuserfaultfdfor that. - To survive a truncated mmap. Databases and search engines that mmap files (LMDB, Lucene/Elasticsearch, early MongoDB) can hit
SIGBUSif another process truncates the file or the disk fills on a sparse file. Some handle it withsigsetjmp/siglongjmpto turn it into an I/O error.
#include <execinfo.h>
#include <signal.h>
#include <stdlib.h>
#include <unistd.h>
static void crash_handler(int sig, siginfo_t *si, void *uc) {
(void)uc;
static const char hdr[] = "FATAL: fault, backtrace follows\n";
write(STDERR_FILENO, hdr, sizeof hdr - 1);
void *frames[64];
int n = backtrace(frames, 64);
backtrace_symbols_fd(frames, n, STDERR_FILENO);
(void)si; // si->si_addr is the faulting address
raise(sig); // SA_RESETHAND restored SIG_DFL: die and core dump
}
void install_crash_handler(void) {
static char altstack[64 * 1024];
stack_t ss = { .ss_sp = altstack, .ss_size = sizeof altstack };
sigaltstack(&ss, NULL);
struct sigaction sa = {0};
sa.sa_sigaction = crash_handler;
sa.sa_flags = SA_SIGINFO | SA_ONSTACK | SA_RESETHAND;
sigemptyset(&sa.sa_mask);
sigaction(SIGSEGV, &sa, NULL);
sigaction(SIGBUS, &sa, NULL);
sigaction(SIGFPE, &sa, NULL);
sigaction(SIGILL, &sa, NULL);
sigaction(SIGABRT, &sa, NULL);
}
Never "recover" from a real segfault and keep going
Returning from a SIGSEGV handler re-executes the faulting instruction, which faults again — an infinite loop. siglongjmp-ing past it leaves whatever lock or data structure the crashing code was holding in an undefined state. Log, dump, die, and let the supervisor restart you.
Call backtrace() once at startup before you need it: its first call may dlopen libgcc, which allocates — not something to do for the first time inside a crash handler.
Real-Time Signals
POSIX.1b added signals SIGRTMIN through SIGRTMAX (34–64 on Linux applications) with stronger semantics than standard signals:
| Property | Standard signals (1–31) | Real-time signals (34–64) |
|---|---|---|
| Multiple instances | Coalesce — one pending bit | Queue — each instance delivered |
| Delivery order | Unspecified (Linux delivers lowest number first) | Lowest number first; same signal FIFO |
| Payload | None | An int or pointer via sigqueue() (si_value) |
| Predefined meaning | Yes | None — application-defined |
| Queue limit | N/A | RLIMIT_SIGPENDING per user (ulimit -i, often ~60k) |
// Sender: queue a signal carrying a job ID
union sigval v = { .sival_int = 4217 };
sigqueue(target_pid, SIGRTMIN + 1, v);
// Receiver (SA_SIGINFO handler or signalfd)
static void on_rt(int sig, siginfo_t *si, void *ctx) {
int job_id = si->si_value.sival_int; // 4217
/* ... record job_id in a lock-free queue ... */
}
In practice, real-time signals are mostly used as plumbing: glibc threading internals, POSIX timers (timer_create with SIGEV_SIGNAL), legacy POSIX AIO completions, and Go's runtime, which uses SIGURG (a standard signal) for asynchronous goroutine preemption since Go 1.14. For application-level messaging between processes, prefer a real IPC channel — see IPC.
Job Control
Job control is the terminal subsystem's use of signals to manage foreground and background process groups.
- A session (one per login / terminal) contains process groups (one per pipeline/job), which contain processes.
- The terminal has exactly one foreground process group. Keystrokes generate signals for every process in that group: Ctrl+C →
SIGINT, Ctrl+\ →SIGQUIT, Ctrl+Z →SIGTSTP. - A background job that tries to read the terminal gets
SIGTTIN(and is stopped); writing may getSIGTTOUifstty tostopis set. - When the terminal hangs up (SSH drops), the session leader gets
SIGHUP, and the shell forwardsSIGHUPto its jobs — which is why a longmakedies when your SSH connection does.
$ sleep 1000 | cat & # one job, one process group, two processes
[1] 4410
$ ps -o pid,pgid,sid,stat,cmd -t pts/0
PID PGID SID STAT CMD
4102 4102 4102 Ss -bash
4409 4409 4102 S sleep 1000
4410 4409 4102 S cat
$ kill -TERM -4409 # negative PID = signal the whole process group
$ fg # SIGCONT + make it the foreground group
$ nohup ./long_task & # ignore SIGHUP; output to nohup.out
$ setsid ./daemon # new session, no controlling terminal at all
$ disown -h %1 # bash: don't forward SIGHUP to job 1
Process groups matter outside terminals too. kill(-pgid, sig) signals a whole tree that shares a group. Supervisors (and timeout(1)) put a child in its own group so they can kill it and all its descendants — otherwise killing a shell script leaves its children running as orphans. In Go:
cmd := exec.Command("bash", "-c", "./build.sh")
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} // child leads a new group
_ = cmd.Start()
// later: signal the entire group, not just bash
_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGTERM)
Graceful Shutdown Pattern
Every production server should handle SIGTERM for graceful shutdown:
import signal
import sys
shutdown_requested = False
def handle_sigterm(signum, frame):
global shutdown_requested
shutdown_requested = True
signal.signal(signal.SIGTERM, handle_sigterm)
signal.signal(signal.SIGINT, handle_sigterm) # Also handle Ctrl+C
# Main server loop
while not shutdown_requested:
request = accept_connection()
handle(request)
# Cleanup
flush_pending_writes()
close_database_connections()
sys.exit(0)
Note: CPython runs Python-level signal handlers only in the main thread, between bytecodes. The C-level handler just sets a flag. That makes Python handlers safe to write in normal Python — but it also means a main thread blocked in a long C call (accept() without a timeout, a big NumPy op) won't run the handler until that call returns or is interrupted with EINTR.
// Go graceful shutdown
server := &http.Server{Addr: ":8080"}
go server.ListenAndServe()
quit := make(chan os.Signal, 1)
signal.Notify(quit, syscall.SIGTERM, syscall.SIGINT)
<-quit // Block until signal received
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
// Graceful shutdown: stop accepting new connections, wait for in-flight to finish
if err := server.Shutdown(ctx); err != nil {
log.Fatal("Server forced to shutdown:", err)
}
signal.Notify channels must be buffered: the Go runtime does a non-blocking send, so an unbuffered channel can drop the signal if your goroutine isn't already waiting.
A Production-Grade Go Shutdown
The minimal version above has three gaps: it doesn't wait for the load balancer to stop sending traffic, it doesn't fail readiness first, and it doesn't shut down background workers. A fuller version:
package main
import (
"context"
"errors"
"log"
"net/http"
"os"
"os/signal"
"sync/atomic"
"syscall"
"time"
)
func main() {
var draining atomic.Bool
mux := http.NewServeMux()
mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
if draining.Load() {
http.Error(w, "draining", http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusOK)
})
mux.HandleFunc("/", handler)
srv := &http.Server{Addr: ":8080", Handler: mux, ReadHeaderTimeout: 5 * time.Second}
// ctx is cancelled on the first SIGTERM/SIGINT.
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)
defer stop()
go func() {
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
log.Fatalf("listen: %v", err)
}
}()
workersDone := startBackgroundWorkers(ctx) // workers watch ctx.Done()
<-ctx.Done()
stop() // restore default behavior: a second Ctrl+C now kills immediately
log.Println("shutdown: signal received")
// 1. Fail readiness so endpoints controllers and LBs stop routing to us.
draining.Store(true)
// 2. Keep serving while that propagates (skip if a preStop sleep already did this).
time.Sleep(5 * time.Second)
// 3. Stop accepting, wait for in-flight requests (bounded).
shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
if err := srv.Shutdown(shutdownCtx); err != nil {
log.Printf("shutdown: forced close: %v", err)
_ = srv.Close()
}
// 4. Drain workers / flush producers / close DB pools.
select {
case <-workersDone:
case <-shutdownCtx.Done():
log.Println("shutdown: workers did not finish in time")
}
log.Println("shutdown: complete")
os.Exit(0)
}
flowchart LR
A["SIGTERM"] --> B["Fail readiness probe"]
B --> C["Keep serving 5 to 10s while endpoints update"]
C --> D["Close listener, stop accepting"]
D --> E["Wait for in-flight requests (bounded by deadline)"]
E --> F["Stop consumers, commit offsets, flush buffers"]
F --> G["Close DB pools, release locks and leases"]
G --> H["exit 0"]
E -->|"deadline exceeded"| I["Force-close connections, exit non-zero"]
Shutdown checklist by workload:
| Workload | On SIGTERM |
|---|---|
| HTTP API | Fail readiness → short delay → Shutdown() → close pools |
| gRPC server | GracefulStop() with a timeout fallback to Stop() |
| WebSocket / long-poll | Send a close frame or "reconnect" hint, then close; don't wait for clients to leave voluntarily |
| Kafka consumer | Stop polling → finish current batch → commit offsets → Close() (leaves the group cleanly, triggers immediate rebalance) |
| Queue worker (SQS, RabbitMQ) | Stop fetching → finish or nack/requeue in-flight messages → exit |
| Batch job | Checkpoint progress so a restart resumes, or exit non-zero so it's retried |
| Leader-elected service | Release the lease/lock explicitly so a peer takes over immediately instead of after TTL |
kill Command and Signal Sending
kill <pid> # Send SIGTERM (polite request to terminate)
kill -15 <pid> # Same — SIGTERM = 15
kill -TERM <pid> # Same, by name
kill -9 <pid> # SIGKILL — immediate, no cleanup
kill -HUP <pid> # SIGHUP — often used to reload config (nginx, sshd)
kill -USR1 <pid> # SIGUSR1 — application-defined (e.g., dump stats, rotate logs)
killall nginx # Send SIGTERM to all processes named "nginx"
pkill -f "python app.py" # Match by full command line
# Check whether a process exists (no signal is actually sent):
kill -0 <pid> # No signal sent; returns 0 if process exists, EPERM if no permission
More tools you will use when debugging signal behavior:
kill -l # list signal names and numbers on this system
kill -TERM -- -<pgid> # signal an entire process group
timeout -s TERM -k 10 60 ./job # SIGTERM after 60s, SIGKILL 10s later if still alive
docker kill --signal=SIGHUP web # signal a container's PID 1
docker stop -t 30 web # SIGTERM (or STOPSIGNAL), then SIGKILL after 30s
systemctl kill -s HUP nginx # signal a systemd unit's processes
kubectl exec pod -- kill -USR1 1 # signal PID 1 inside a pod (needs a kill binary in the image)
# Who is sending signals to my process? (bpftrace, needs root)
bpftrace -e 'tracepoint:signal:signal_generate /args->pid == 1234/ {
printf("%s(%d) -> sig %d\n", comm, pid, args->sig); }'
# Trace only signal-related activity
strace -f -e trace=none -e signal=all -p <pid>
Permission rules: an unprivileged process may signal another if its real or effective UID matches the target's real or saved UID. CAP_KILL bypasses this. Inside a user namespace, you can only signal processes in that namespace or its descendants. SIGCONT is also allowed to any process in the same session.
Conventional signal meanings (not enforced by the kernel — every daemon defines its own):
| Daemon | SIGHUP | SIGUSR1 | SIGUSR2 | SIGQUIT |
|---|---|---|---|---|
| nginx | Reload config, start new workers | Reopen log files | Upgrade binary on the fly | Graceful shutdown |
| PostgreSQL | Reload config | — | — | Immediate shutdown (SIGTERM = smart, SIGINT = fast) |
| HAProxy | — | Soft stop | Reload (master-worker) | — |
| Go runtime | Default (terminate) | — | — | Dump all goroutine stacks and exit |
| JVM | — | — | — | Thread dump to stdout (doesn't exit) |
Failure Modes
| Symptom | Likely cause | How to confirm | Fix |
|---|---|---|---|
| Pod always exits 137 after exactly the grace period | PID 1 doesn't handle/forward SIGTERM | kubectl describe pod → Exit Code: 137, Reason: Error; ps in container shows sh -c as PID 1 | Exec-form CMD, exec in scripts, or tini |
| 502s during every rollout | App exits before LBs stop routing | Errors cluster within seconds of pod termination | preStop sleep or post-SIGTERM delay; fail readiness first |
kill -9 does nothing | Process in D (uninterruptible) state | ps -o stat,wchan:32 -p <pid>, cat /proc/<pid>/stack | Fix the underlying I/O (dead NFS, failing disk); reboot as last resort |
| Process hangs forever, occasionally | Non-async-signal-safe handler deadlocked | gdb -p shows handler frame above malloc/__lll_lock_wait | Handler only sets flag / writes to pipe |
Child processes accumulate as <defunct> | Parent never reaps | ps -eo stat,ppid,cmd | grep ^Z | waitpid loop, SA_NOCLDWAIT, or tini as PID 1 |
| Process dies silently, exit 141 | Unhandled SIGPIPE | 128 + 13 = 141; strace shows --- SIGPIPE --- | SIG_IGN + check EPIPE, or MSG_NOSIGNAL |
Random EINTR errors in logs | Signals interrupting syscalls without SA_RESTART | strace shows EINTR/ERESTARTSYS | SA_RESTART, retry loops, or block signals in worker threads |
Child spawned by service ignores kill | Inherited blocked mask or SIG_IGN across exec | /proc/<child>/status SigBlk/SigIgn | Reset mask and dispositions before exec |
| Second Ctrl+C doesn't kill a stuck shutdown | Handler still installed during shutdown | — | Restore default after first signal (signal.NotifyContext stop(), SA_RESETHAND) |
| Wrong process killed | PID reuse between lookup and kill() | Rare; hard to prove | pidfd_open + pidfd_send_signal |
Exit Code Cheat Sheet
| Exit code | Meaning |
|---|---|
| 0 | Clean exit |
| 1–125 | Application-defined error |
| 126 / 127 | Command not executable / not found (shell) |
| 128 + N | Killed by signal N (as reported by shells, Docker, Kubernetes) |
| 130 | SIGINT (Ctrl+C) |
| 134 | SIGABRT (assert, glibc heap check) |
| 137 | SIGKILL (OOM killer or grace period expired) |
| 139 | SIGSEGV |
| 141 | SIGPIPE |
| 143 | SIGTERM with default action |
Real Numbers
| Measurement | Typical value | Notes |
|---|---|---|
kill() → handler start (target idle on same machine) | ~1–5 µs | Dominated by wakeup + context switch |
Signal delivery + rt_sigreturn overhead per signal | ~1–2 µs | Why signals are a bad high-rate IPC mechanism |
| Max standard-signal "queue" depth | 1 | Pending bitmap, coalesces |
| Real-time signal queue limit | RLIMIT_SIGPENDING, often ~60k per user | ulimit -i |
Kubernetes default terminationGracePeriodSeconds | 30 s | Per pod; includes preStop |
Docker docker stop default timeout | 10 s | -t to change |
systemd TimeoutStopSec default | 90 s | DefaultTimeoutStopSec in system.conf |
| Endpoint removal propagation in Kubernetes | ~1–10 s | Depends on cluster size, kube-proxy mode, external LB |
| Go async preemption signal | SIGURG, sent when a goroutine runs > ~10 ms | Visible in strace as frequent SIGURG |
Decision Guide
flowchart TD
A["Need to react to a signal"] --> B{"Single-threaded C event loop?"}
B -->|"yes, Linux only"| C["signalfd in epoll"]
B -->|"yes, portable"| D["Self-pipe or eventfd"]
B -->|no| E{"Multithreaded C or C++?"}
E -->|yes| F["Block in main, dedicated sigwait thread"]
E -->|no| G{"Managed runtime?"}
G -->|"Go"| H["signal.NotifyContext"]
G -->|"Python"| I["signal.signal sets flag, main thread acts"]
G -->|"Node, Java, Rust"| J["process.on SIGTERM, Runtime shutdown hook, signal-hook crate"]
A --> K{"Crash reporting for SIGSEGV?"}
K -->|yes| L["SA_ONSTACK plus sigaltstack, write backtrace, re-raise"]
Rules of thumb:
- Handlers do one thing: set a flag or write a byte. All real work happens in normal code.
- Use
SIGTERMfor graceful stop,SIGHUPfor reload,SIGUSR1/SIGUSR2for diagnostics. Don't invent meanings forSIGINTbeyond "stop". - Never use signals as a general IPC channel — no payload (except RT), coalescing, and expensive. Use a socket, pipe, or eventfd (IPC).
- Always budget the shutdown: every graceful path needs a deadline and a forced path.
- In containers, make sure your app is PID 1 or behind a real init.
- Ignore
SIGPIPEin C/C++ servers on day one.
Advanced Topics
sigsuspend()and the lost-wakeup race. Checking a flag and then callingpause()has a race: the signal can arrive between the check and thepause(), and you sleep forever.sigsuspend(&oldmask)atomically unblocks and sleeps. Modern equivalents:ppoll()/pselect()/epoll_pwait()take a signal mask applied atomically for the duration of the wait.- Signals and
vfork/posix_spawn.posix_spawnlets you set the child's signal mask (posix_spawnattr_setsigmask) and reset handled signals to default (posix_spawnattr_setsigdefault) — the clean way to avoid inherited-mask bugs. - Seccomp and
SIGSYS. A seccomp filter can returnSECCOMP_RET_TRAP, making forbidden syscalls deliverSIGSYSwith the syscall number insi_syscall— handy for emulating or logging syscalls in sandboxes. See Process Security. - Core dumps. Signals with "core" default action write a core file subject to
ulimit -c,/proc/sys/kernel/core_pattern(often piped tosystemd-coredumporapport), and container settings.coredumpctl list/coredumpctl gdbto inspect. - The cgroup freezer (
cgroup.freezein cgroups v2) stops a whole group of processes without sending themSIGSTOP, so they can't observe or race with it.docker pauseand Kubernetes checkpointing use it. SIGKILLfrom the OOM killer is indistinguishable from the kubelet's grace-period kill by exit code alone — checkkubectl get pod -o jsonpath='{.status.containerStatuses[*].lastState}'forreason: OOMKilledand the kernel log (dmesg | grep -i oom).
Interview Questions
"What's the difference between SIGTERM and SIGKILL?"
SIGTERM is a request — the process can catch it, do cleanup (flush buffers, finish requests, close connections), and exit gracefully. SIGKILL is an order — the kernel kills the process immediately, no handler runs, no cleanup happens. Kubernetes sends SIGTERM first, waits terminationGracePeriodSeconds, then sends SIGKILL.
"Why can't you call printf() in a signal handler?"
printf calls malloc internally, which uses locks and shared state. If the signal interrupts a malloc call in the main thread, and the handler also calls malloc, the allocator's state is corrupted. Signal handlers must only call async-signal-safe functions (mostly raw syscalls). Use the self-pipe trick or signalfd to handle signals safely in the main event loop.
"How do you implement graceful shutdown?"
Register a handler for SIGTERM (and SIGINT). Handler sets a flag or writes to a pipe. Main loop checks the flag after completing each request/batch. When set: stop accepting new work, finish in-flight requests within a deadline, flush buffers, close connections, then exit(0). In Go: signal.Notify + http.Server.Shutdown(ctx) with a timeout.
"What is SA_RESTART?"
A flag on sigaction that tells the kernel to automatically restart blocking syscalls (read, write, accept, etc.) that were interrupted by signal delivery. Without it, those syscalls return -1/EINTR and you must retry them manually. Most servers want SA_RESTART to avoid sprinkling EINTR checks everywhere.
"When exactly is a signal delivered?"
When a thread that doesn't block it next transitions from kernel mode back to user mode — returning from a syscall, an interrupt, or being scheduled. The kernel checks the pending set, builds a signal frame on the user stack, and redirects execution to the handler. A thread in uninterruptible sleep (D state) won't see it until the sleep ends.
"Why do I sometimes lose SIGCHLD signals?"
You don't lose the information, you lose the count: standard signals are a pending bitmap, so multiple children exiting close together produce one delivery. Always reap with while (waitpid(-1, &st, WNOHANG) > 0).
"Why does my containerized app ignore SIGTERM?"
Either a shell is PID 1 and doesn't forward it (shell-form CMD), or your app is PID 1 without a handler — and the kernel ignores default-action signals for a namespace's init process. Use exec form, exec in entrypoint scripts, or tini.
"Your pods return 502s during deploys even though you handle SIGTERM. Why?"
SIGTERM and endpoint removal race. The app stops accepting connections before kube-proxy and the load balancers have removed the pod. Add a preStop sleep (or delay after SIGTERM while failing readiness), make the grace period long enough for sleep + drain.
Related
- Processes & Threads — process lifecycle,
fork/exec, zombies - System Calls & the Kernel — the user/kernel boundary where signals are delivered
- Namespaces & cgroups — PID namespaces and why PID 1 is special
- IPC — better channels than signals for data
- Process Security — seccomp, capabilities,
SIGSYS - Pod Lifecycle — probes, hooks and termination in Kubernetes
- Pods Restarting playbook — diagnosing exit codes 137 and 143