On this page

The Boot Process: Firmware to systemd

Booting is the problem of getting from "a CPU with nothing in RAM" to "a full operating system running your services." Every layer in between exists because the layer below it is too dumb to load the layer above it directly. Firmware can't parse ext4. A bootloader can't drive your RAID controller. The kernel can't mount an encrypted LVM volume without userspace tools. And none of them knows which of your 80 services needs the network up first.

Most engineers never think about boot until a machine won't come up, a cloud instance takes 90 seconds to accept SSH, or a container ignores SIGTERM and gets killed after 30 seconds. All three are boot problems, and all three are easy once you know which stage owns which job.

This page walks the chain end to end, then looks at how cloud VMs, containers, microVMs, and unikernels shortcut or skip parts of it.


The Whole Chain at a Glance

flowchart TD
    PWR["Power on / reset"] --> FW["Firmware: BIOS or UEFI"]
    FW --> POST["POST + hardware init"]
    POST --> BL["Bootloader: GRUB2 / systemd-boot"]
    BL --> K["Kernel image (vmlinuz) decompresses itself"]
    K --> KI["start_kernel(): memory, scheduler, drivers"]
    KI --> IR["initramfs /init: find and mount the real root"]
    IR --> SR["switch_root to real root filesystem"]
    SR --> P1["PID 1: systemd"]
    P1 --> T["default.target: services in parallel"]
    T --> LOGIN["getty / sshd / your app"]

Each hop hands off a small amount of state to the next: the firmware hands the bootloader a memory map and (on UEFI) a table of services; the bootloader hands the kernel a command line and an initramfs; the kernel hands PID 1 a root filesystem and a set of mounted pseudo-filesystems; systemd hands your service a set of file descriptors, environment variables, and a cgroup.

StageRuns fromOwnsTypical time (physical server)Typical time (desktop/laptop)
Firmware + POSTSPI flash on motherboardCPU/RAM init, device enumeration, picking a boot device20 s – 3 min (memory training, RAID/NIC option ROMs)2 – 8 s
BootloaderEFI System Partition or MBRChoosing a kernel, loading it + initramfs0.5 – 5 s (plus menu timeout)0.5 – 3 s
KernelRAMHardware drivers, memory management, mounting initramfs0.5 – 3 s0.5 – 2 s
initramfsRAM (tmpfs)Storage drivers, LVM/LUKS/RAID, finding root1 – 10 s1 – 5 s
Userspace (systemd)Real root fsEverything else5 – 60 s3 – 20 s

The striking number is firmware on servers: a two-socket machine with 1 TB of RAM can spend minutes training DIMMs and initialising option ROMs before a single byte of your code runs. That's why "reboot to fix it" is so much more expensive on bare metal than it looks.


Stage 1: Firmware

What the CPU does at power-on

When an x86 CPU comes out of reset, it starts executing at a fixed physical address — the reset vector, 0xFFFFFFF0, 16 bytes below the 4 GB boundary. The chipset maps that address to the SPI flash chip holding the firmware. At this point there is no usable RAM (the memory controller hasn't been configured), so early firmware runs from flash and uses the CPU cache as temporary RAM ("cache-as-RAM").

The firmware's first job is POST (Power-On Self-Test) and platform initialisation:

  1. Initialise the memory controller and train the DIMMs (timing calibration — the slow part on servers)
  2. Enumerate PCIe buses and assign resources
  3. Run option ROMs — firmware blobs on add-in cards (RAID controllers, NICs with PXE boot, GPUs)
  4. Build tables describing the hardware for the OS: ACPI tables (power management, CPU topology, interrupt routing) and SMBIOS (model, serial numbers, DIMM layout)
  5. Pick a boot device and hand off

BIOS vs UEFI

Legacy BIOS dates from the 1981 IBM PC. It leaves the CPU in 16-bit real mode, reads the first 512-byte sector of the boot disk — the Master Boot Record — to physical address 0x7C00, checks that it ends with the signature 0x55AA, and jumps to it.

MBR layout (512 bytes)
┌──────────────────────────────┬──────────────────────┬────────┐
│ Bootstrap code (446 bytes)   │ Partition table      │ 0x55AA │
│                              │ 4 entries x 16 bytes │        │
└──────────────────────────────┴──────────────────────┴────────┘

446 bytes is not enough to do anything useful, so BIOS bootloaders are chained: a tiny stage in the MBR loads a bigger stage from the gap after the MBR, which loads the real bootloader from a filesystem. MBR partition tables also cap disks at 2 TiB (32-bit sector addresses × 512 bytes).

UEFI (Unified Extensible Firmware Interface) replaced this with a small operating system of its own:

  • Runs in 32- or 64-bit protected mode, with drivers for storage, network, USB, and graphics
  • Understands FAT32 natively and reads bootloaders as files from the EFI System Partition (ESP), typically mounted at /boot/efi or /efi
  • Bootloaders are PE/COFF executables (the Windows .exe format), e.g. \EFI\ubuntu\shimx64.efi, with a fallback path \EFI\BOOT\BOOTX64.EFI
  • Boot order lives in NVRAM variables, not in disk sector positions
  • Uses GPT partitioning (64-bit LBAs, 128 partitions by default, backup header at end of disk)
  • Exposes runtime services (variables, clock, reset) the OS can call after boot
AspectLegacy BIOSUEFI
CPU mode at handoff16-bit real mode32/64-bit protected/long mode
Boot entryFirst sector of a disk (MBR)A file on the ESP, chosen via NVRAM entry
PartitioningMBR, 4 primary, 2 TiB maxGPT, 128+ partitions, ~8 ZiB max
Filesystem awarenessNoneFAT32 (more via drivers)
Secure BootNoYes
Network bootPXE option ROMPXE and HTTP(S) boot built-in
Typical in 2026VMs with SeaBIOS, old hardwareAll new hardware, most cloud VMs

You can inspect and change UEFI boot entries from a running Linux system:

# Are we booted in UEFI mode?
[ -d /sys/firmware/efi ] && echo UEFI || echo BIOS

# List NVRAM boot entries and order
efibootmgr -v
# BootCurrent: 0001
# BootOrder: 0001,0000,0002
# Boot0001* ubuntu  HD(1,GPT,...)/File(\EFI\ubuntu\shimx64.efi)

# Boot entry 0002 once on next boot (e.g. a recovery image)
sudo efibootmgr --bootnext 0002

# Firmware-reported boot time on a systemd system
systemd-analyze
# Startup finished in 6.211s (firmware) + 2.004s (loader) + 1.402s (kernel) + 3.118s (initrd) + 9.870s (userspace) = 22.607s

Secure Boot: the chain of trust

Secure Boot answers one question: was every piece of code that ran before the kernel signed by someone I trust? Without it, anyone with brief physical access (or root, which can write the ESP) can install a bootkit that loads before the kernel and is invisible to it.

The trust anchors live in UEFI variables:

VariableContentsWho controls it
PK (Platform Key)One key; signs updates to KEKThe hardware vendor (OEM)
KEK (Key Exchange Keys)Keys allowed to update db/dbxOEM, Microsoft
db (signature database)Certs/hashes allowed to bootTypically Microsoft's Windows CA and Microsoft UEFI CA (third-party)
dbx (forbidden database)Revoked certs/hashesUpdated by OS vendors when a bootloader is found vulnerable

Linux distributions don't get every GRUB build signed by Microsoft. Instead they use shim: a small first-stage loader that Microsoft signs once, which embeds the distro's own certificate and verifies everything after it.

sequenceDiagram
    participant FW as UEFI Firmware
    participant SH as shim (signed by Microsoft UEFI CA)
    participant GR as GRUB2 (signed by distro)
    participant K as Kernel (signed by distro)
    participant M as Kernel modules
    FW->>FW: Verify shim against db, reject if in dbx
    FW->>SH: Execute shim
    SH->>SH: Load embedded distro cert plus MOK list
    SH->>GR: Verify and execute grubx64.efi
    GR->>K: Verify vmlinuz signature via shim protocol
    K->>K: Enable lockdown mode
    K->>M: Only load modules with valid signatures
    Note over K,M: An unsigned module or kexec of an unsigned kernel is refused

MOK (Machine Owner Key) is the escape hatch: if you build your own kernel or need an out-of-tree driver (NVIDIA, VirtualBox, ZFS), you enrol your own key with mokutil --import, confirm it at the next boot in the MokManager blue screen, and sign your modules with it.

Secure Boot on its own only enforces — it stops unsigned code. Measured boot records: each stage hashes the next into TPM Platform Configuration Registers (PCRs) before executing it. Firmware measures into PCRs 0–7, GRUB measures its config and commands into PCRs 8–9, and systemd-stub measures unified kernel images into PCR 11. Those PCR values can then gate secrets — systemd-cryptenroll --tpm2-pcrs=7 unlocks a LUKS disk only if the Secure Boot state is unchanged — or be sent to a remote attestation server.

Secure Boot is only as strong as its revocation list

Real-world Secure Boot bypasses have targeted signed-but-vulnerable code rather than the cryptography. BootHole (CVE-2020-10713) was a buffer overflow in GRUB2's grub.cfg parser: since grub.cfg isn't signed, an attacker who could write the ESP could get code execution inside a fully signed GRUB. BlackLotus, the first publicly analysed in-the-wild UEFI bootkit (2023), exploited CVE-2022-21894 in the Windows boot manager — old, vulnerable, validly signed binaries can be loaded until their hashes land in dbx. Fixing both required pushing large dbx updates, which themselves bricked some machines with small NVRAM. Keep firmware and dbx updated, and treat write access to the ESP as equivalent to root.


Stage 2: The Bootloader

The bootloader's job is narrow: find a kernel and an initramfs, load them into memory, give the kernel its command line, and jump. It also gives humans a menu to pick an older kernel when the new one doesn't boot — which is the single most valuable feature in practice.

GRUB2

GRUB2 is the default on most distributions. It is effectively a small OS with its own filesystem drivers (ext4, XFS, Btrfs, LVM, LUKS1/2), a scripting language, and a module loader.

On BIOS systems it boots in stages: boot.img (446 bytes, in the MBR) loads core.img from the post-MBR gap or a BIOS boot partition; core.img has just enough drivers to read /boot/grub, where it loads modules and grub.cfg. On UEFI it's a single grubx64.efi on the ESP.

You almost never edit grub.cfg directly — it's generated:

# /etc/default/grub — the human-edited source
GRUB_DEFAULT=saved
GRUB_TIMEOUT=3
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200n8"

# Regenerate grub.cfg after editing
sudo update-grub                                 # Debian/Ubuntu
sudo grub2-mkconfig -o /boot/grub2/grub.cfg      # RHEL/Fedora (BIOS layout)

# RHEL/Fedora: change kernel args for all installed kernels
sudo grubby --update-kernel=ALL --args="mitigations=auto,nosmt"

A generated menu entry is just a few commands:

menuentry 'Ubuntu, with Linux 6.8.0-45-generic' {
    insmod gzio
    insmod part_gpt
    insmod ext2
    search --no-floppy --fs-uuid --set=root 3e1c...b7a2
    linux   /boot/vmlinuz-6.8.0-45-generic root=UUID=9f02...e1 ro quiet
    initrd  /boot/initrd.img-6.8.0-45-generic
}

systemd-boot and Unified Kernel Images

systemd-boot (formerly gummiboot) is UEFI-only and deliberately simple: no filesystem drivers beyond what the firmware provides, no scripting. It reads small entry files from the ESP following the Boot Loader Specification:

# /boot/loader/entries/arch.conf
title   Arch Linux
linux   /vmlinuz-linux
initrd  /intel-ucode.img
initrd  /initramfs-linux.img
options root=UUID=9f02...e1 rw quiet

The modern trend is the Unified Kernel Image (UKI): kernel, initramfs, command line, and OS release info bundled into one PE binary by systemd-stub (via ukify or dracut --uefi). Because it's one file, it's signed once — which closes the gap where the initramfs and command line weren't covered by Secure Boot at all. Firmware can even boot a UKI directly with no bootloader.

GRUB2systemd-bootDirect UKI / EFI stub
BIOS supportYesNoNo
Reads kernel fromAlmost any FS, LVM, LUKSESP (FAT) onlyESP only
ConfigGenerated scriptStatic entry filesNVRAM entry
Complexity / attack surfaceHighLowMinimal
Best forMulti-boot, BIOS, complex storageModern UEFI laptops/serversAppliances, immutable OSes

The kernel command line

Whatever the bootloader, it passes a single string to the kernel. This is your most powerful debugging lever, because you can edit it at the boot menu (press e in GRUB) without touching the disk.

ParameterEffect
root=UUID=...Which device holds the real root filesystem
ro / rwMount root read-only (so fsck can run) or read-write
quiet, loglevel=7Suppress or maximise kernel console messages
console=ttyS0,115200Send console to serial — essential for cloud VMs and headless servers
init=/bin/bashRun bash as PID 1 instead of systemd (last-resort recovery)
systemd.unit=rescue.targetBoot to single-user rescue mode
systemd.unit=emergency.targetBoot to the most minimal possible shell
rd.break(dracut) Drop to a shell inside the initramfs, before switch_root
systemd.log_level=debugVerbose systemd logging during boot
panic=10Reboot 10 s after a kernel panic instead of hanging forever
mitigations=offDisable CPU vulnerability mitigations (benchmarks only!)
# What did this boot actually use?
cat /proc/cmdline

Stage 3: The Kernel Wakes Up

Decompression

The file in /boot named vmlinuz-* is a bzImage: a small uncompressed setup stub plus a compressed kernel. The "z" is for compression. On x86 the stub (in arch/x86/boot/) collects information from firmware, switches the CPU into 64-bit long mode if needed, then decompresses the real kernel in place and jumps to it.

Compression is a trade-off between reading bytes from disk and burning CPU:

AlgorithmKernel configRelative sizeDecompression speed
gzipCONFIG_KERNEL_GZIPBaselineModerate
xzCONFIG_KERNEL_XZ~25–30% smallerSlowest
lz4CONFIG_KERNEL_LZ4~10–20% largerFastest
zstdCONFIG_KERNEL_ZSTD (5.9+)Close to xzClose to lz4

On fast NVMe the disk read is nearly free, so lz4 or zstd wins. On slow SD cards or network boot, smaller wins.

start_kernel() and the birth of PID 1

After decompression, architecture-specific assembly sets up early page tables and calls start_kernel() in init/main.c, the C entry point. It initialises subsystems in a carefully ordered sequence:

flowchart TD
    SK["start_kernel()"] --> SA["setup_arch(): parse memory map, ACPI, CPU features"]
    SA --> MM["Memory init: page allocator, slab"]
    MM --> SCH["sched_init(): scheduler ready"]
    SCH --> IRQ["Interrupts, timers, console"]
    IRQ --> RI["rest_init()"]
    RI --> KT1["kernel_thread: kernel_init becomes PID 1"]
    RI --> KT2["kernel_thread: kthreadd becomes PID 2"]
    RI --> IDLE["Boot CPU becomes the idle task, PID 0"]
    KT1 --> IC["do_initcalls(): built-in drivers probe hardware"]
    IC --> UNPACK["Unpack initramfs into rootfs"]
    UNPACK --> EXEC["execve /init (or /sbin/init)"]

Three things are worth internalising:

  1. PID 1 starts life as a kernel thread. kernel_init runs in kernel mode, does the remaining setup, then calls execve() on a userspace binary. From that moment PID 1 is a normal userspace process — the same fork/exec model from Processes & Threads, applied to the very first process.
  2. PID 2 (kthreadd) is the parent of every kernel thread — the [kworker/...], [ksoftirqd/0], [kswapd0] entries in ps.
  3. If no init can be executed, the kernel panics with Kernel panic - not syncing: No working init found. The kernel tries, in order, the init= parameter, then /sbin/init, /etc/init, /bin/init, /bin/sh.
# The kernel's own boot log, from this boot
journalctl -k -b
dmesg --level=err,warn

# Which initcalls are slow? Add initcall_debug to the cmdline, then:
dmesg | grep 'initcall' | sort -t' ' -k6 -n | tail
# [    1.932012] initcall ahci_pci_driver_init+0x0/0x30 returned 0 after 412331 usecs

Stage 4: initramfs — Why a Second Tiny OS Exists

The chicken-and-egg problem

The kernel needs to mount the root filesystem. But reaching root might require:

  • A storage driver that's built as a module (NVMe, a specific RAID HBA, virtio-scsi) — and modules live on the root filesystem
  • Assembling software RAID (mdadm) or activating LVM volume groups (lvm)
  • Prompting for a passphrase and unlocking LUKS (cryptsetup), or unsealing a key from the TPM
  • Bringing up networking for NFS, iSCSI, or NVMe-over-TCP root
  • Waiting for slow USB or SAN devices to appear

You could compile everything into the kernel, but distributions ship one kernel for millions of different machines. Building every driver in makes the kernel huge, and none of the userspace tools (LVM, cryptsetup) can be "built in" anyway.

The answer is the initramfs: a compressed cpio archive the bootloader loads into RAM alongside the kernel. The kernel unpacks it into a tmpfs-backed root and runs /init from it. That /init (on most distros, a cut-down systemd) loads the right modules, does the RAID/LVM/LUKS dance, mounts the real root at /sysroot, and calls switch_root:

flowchart LR
    BL["Bootloader loads kernel + initramfs into RAM"] --> UNP["Kernel unpacks cpio into tmpfs"]
    UNP --> INIT["/init runs from RAM"]
    INIT --> UDEV["udev loads drivers for detected hardware"]
    UDEV --> STOR["Assemble RAID, activate LVM, unlock LUKS"]
    STOR --> MNT["Mount real root at /sysroot"]
    MNT --> SW["switch_root: delete initramfs contents, chroot, exec /sbin/init"]
    SW --> SYSD["systemd on the real root, still PID 1"]

switch_root frees the initramfs memory, makes /sysroot the new /, and execs the real init — so PID 1 stays PID 1 across the transition.

initrd vs initramfs

The older initrd was a block-device RAM disk containing a real filesystem image (ext2) that the kernel had to mount, which needed a filesystem driver built in and wasted memory on the page cache. initramfs (Linux 2.6+) is a cpio archive extracted directly into the kernel's rootfs. The bootloader keyword is still initrd and the files are often still named initrd.img, but they've been initramfs for nearly two decades.

Building and inspecting it

Distro familyToolRegenerateInspect
RHEL, Fedora, SUSEdracutdracut -flsinitrd
Archmkinitcpiomkinitcpio -Plsinitcpio
Debian, Ubuntuinitramfs-toolsupdate-initramfs -ulsinitramfs
# What's inside the current initramfs?
lsinitrd /boot/initramfs-$(uname -r).img | head -40

# Is the NVMe driver there?
lsinitrd | grep nvme

# Build a host-only image: only drivers for THIS machine (smaller, faster)
sudo dracut --force --hostonly

# Build a generic image with every driver (bigger, but survives a disk controller swap)
sudo dracut --force --no-hostonly

# Arch: hooks decide what goes in; order matters
grep ^HOOKS /etc/mkinitcpio.conf
# HOOKS=(base systemd autodetect microcode modconf kms keyboard sd-vconsole block sd-encrypt filesystems fsck)

Host-only images are typically 15–40 MB; generic images can be 80–150 MB. On a laptop, that difference is a second or two of decompression on every boot.

The most common "won't boot after an update" cause

You moved a disk to a new controller, changed the root filesystem, added LUKS, or converted a VM from IDE to virtio — and the host-only initramfs doesn't contain the driver for the new setup. The kernel boots, the initramfs can't find root, and after a timeout you land in the dracut emergency shell with Could not boot. /dev/disk/by-uuid/... does not exist. Fix: boot an older kernel or a rescue ISO, chroot in, regenerate the initramfs with --no-hostonly or --add-drivers, and reboot.


Stage 5: PID 1 and systemd

Once the real root is mounted, PID 1 has three jobs that never end:

  1. Bring the system to a target state — start services in the right order
  2. Supervise — restart crashed services, track which processes belong to which service
  3. Reap orphans — any process whose parent dies is reparented to PID 1, which must wait() on it or it becomes a zombie forever

PID 1 is also special to the kernel: if it exits, the kernel panics (Attempted to kill init!), and signals sent to it are ignored unless it has installed a handler for them — even SIGKILL from within its own PID namespace. Both facts come back to bite containers, below.

Why systemd replaced SysV init

SysV init ran shell scripts from /etc/rc3.d/ in lexical order: S10network, S20sshd, S55postgres. Strictly serial, ordering encoded in filenames, no supervision (a crashed daemon stayed dead), and services tracked by PID files that went stale. Upstart (Ubuntu, 2006) added events; systemd (2010) went further:

Problem with SysVsystemd's answer
Serial startupDependency graph, parallel job execution
Ordering by filenameExplicit After=/Before= and Requires=/Wants=
Daemons double-fork and escape trackingEvery service lives in its own cgroup — no process can escape
Crashed services stay deadRestart=on-failure, watchdogs, rate limits
Shell-script boilerplateDeclarative unit files
Logs scattered across filesJournal captures stdout/stderr of every unit
Everything starts at bootSocket, path, timer, and D-Bus activation start things on demand

The cgroup point is subtle but huge: a daemon that forks, double-forks, and reparents itself to PID 1 can't hide from systemd, because cgroup membership is inherited and only privileged processes can move out. systemctl stop kills everything in the unit's cgroup. See Namespaces & cgroups for the mechanism.

Units and targets

Everything systemd manages is a unit, identified by suffix:

Unit typeRepresentsExample
.serviceA process or group of processesnginx.service
.socketA listening socket, for activationsshd.socket
.targetA synchronisation point / group of unitsmulti-user.target
.mount / .automountA filesystem mount (generated from /etc/fstab)home.mount
.timerA cron replacementlogrotate.timer
.pathStart a unit when a file changescups.path
.deviceA kernel device exposed by udevdev-nvme0n1.device
.slice / .scopecgroup hierarchy nodesuser-1000.slice

Targets replaced runlevels. They do nothing themselves; they're named milestones other units hang off. The kernel cmdline or systemctl get-default picks the goal:

Old runlevelTargetMeaning
0poweroff.targetShut down
1 / Srescue.targetSingle-user: local filesystems, root shell
—emergency.targetEven less: root mounted read-only, nothing else
3multi-user.targetFull system, no GUI — what servers use
5graphical.targetmulti-user + display manager
6reboot.targetReboot

Under the default target sits a standard chain of milestones:

flowchart BT
    LFS["local-fs.target"] --> SI["sysinit.target"]
    SW["swap.target"] --> SI
    CS["cryptsetup.target"] --> SI
    UDEV["systemd-udevd.service"] --> SI
    SI --> BASIC["basic.target"]
    SOCK["sockets.target"] --> BASIC
    TIM["timers.target"] --> BASIC
    BASIC --> MU["multi-user.target"]
    NET["network-online.target"] --> APP["myapp.service"]
    APP --> MU
    SSHD["sshd.service"] --> MU
    MU --> GR["graphical.target"]

Dependencies vs ordering — the distinction everyone gets wrong

systemd keeps two independent axes:

  • Requirement — "if I start, also start X": Wants= (weak — keep going if X fails), Requires= (strong — fail if X fails to start), BindsTo= (stop me if X stops), PartOf= (propagate restarts/stops)
  • Ordering — "if we're both starting, X goes first": After=, Before=

Requires=postgresql.service without After=postgresql.service starts both in parallel. Your app races the database and loses. You almost always want both.

# /etc/systemd/system/myapp.service
[Unit]
Description=My API server
Wants=network-online.target
After=network-online.target postgresql.service
Requires=postgresql.service

[Service]
Type=notify                     # app calls sd_notify("READY=1") when actually ready
User=myapp
Group=myapp
ExecStart=/usr/local/bin/myapp --config /etc/myapp/config.yaml
Restart=on-failure
RestartSec=2s
TimeoutStopSec=30s              # SIGTERM, then SIGKILL after 30s
WatchdogSec=30s                 # app must ping sd_notify("WATCHDOG=1") or get restarted
LimitNOFILE=65536

# Sandboxing (see the process security page)
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadWritePaths=/var/lib/myapp

[Install]
WantedBy=multi-user.target      # "systemctl enable" adds a Wants= from multi-user.target

Type= tells systemd when a service counts as "started", which is what After= waits for:

Type="Started" meansUse for
simple (default)fork() succeeded — before exec evenRarely what you want if anything depends on you
execexecve() succeededBetter default for plain binaries
forkingThe parent process exited (classic daemon)Legacy daemons; pair with PIDFile=
notifyThe process sent READY=1 over the notify socketAnything others depend on
oneshotThe process exited successfullySetup scripts, migrations

Socket activation

Socket activation is systemd's most elegant trick, borrowed from macOS launchd and the old inetd. Instead of starting a service and waiting for it to bind its port, systemd creates the listening socket itself at boot, then starts the service on the first connection (or immediately, but in parallel) and hands over the already-bound file descriptor.

sequenceDiagram
    participant S as systemd (PID 1)
    participant K as Kernel
    participant C as Client
    participant D as myapp.service
    S->>K: socket, bind port 8080, listen
    Note over S: Boot continues. Nothing waits for myapp
    C->>K: connect to port 8080
    K-->>C: Accepted into the listen backlog
    K->>S: Socket readable
    S->>D: fork and exec, socket passed as fd 3, LISTEN_FDS=1
    D->>K: accept on fd 3
    K-->>D: Client connection, nothing was dropped

Why this is powerful:

  • Parallelism without races. Every service's socket exists from very early boot, so services that talk to each other can all start simultaneously; the kernel's listen backlog buffers connections until the server is ready.
  • Zero-downtime restarts. Restart the service and the socket stays open in PID 1 — clients queue rather than get ECONNREFUSED.
  • On-demand services. Rarely used daemons (CUPS, sshd on some distros) don't consume memory until needed.
  • Privileged ports without root. systemd binds port 443 as root; the service runs as an unprivileged user and just receives the fd.
# /etc/systemd/system/myapp.socket
[Socket]
ListenStream=8080
Backlog=4096

[Install]
WantedBy=sockets.target

The service side reads the inherited fds. File descriptors start at 3 (SD_LISTEN_FDS_START), and LISTEN_PID must match the process's own PID:

#include <systemd/sd-daemon.h>
#include <sys/socket.h>
#include <stdio.h>
#include <stdlib.h>

int main(void) {
    int n = sd_listen_fds(0);          /* reads LISTEN_FDS / LISTEN_PID */
    if (n != 1) {
        fprintf(stderr, "expected 1 socket from systemd, got %d\n", n);
        exit(1);
    }
    int listen_fd = SD_LISTEN_FDS_START;   /* == 3 */

    sd_notify(0, "READY=1");           /* for Type=notify */

    for (;;) {
        int c = accept(listen_fd, NULL, NULL);
        if (c < 0) continue;
        /* ... handle connection ... */
    }
}

Go, Rust, Python, and Node all have small libraries (coreos/go-systemd/activation, listenfd, systemd-python) that do the same in a few lines.

Measuring and debugging boot with systemd-analyze

# Total time per stage
systemd-analyze
# Startup finished in 3.4s (firmware) + 1.1s (loader) + 1.2s (kernel) + 2.6s (initrd) + 14.9s (userspace) = 23.2s
# multi-user.target reached after 14.7s in userspace.

# Which units took longest to start? (NOT the same as "what blocked boot")
systemd-analyze blame | head
#  12.004s NetworkManager-wait-online.service
#   2.310s snapd.service
#   1.120s dev-nvme0n1p2.device
#    880ms systemd-journal-flush.service

# The actual critical path: what the default target was waiting on
systemd-analyze critical-chain
# multi-user.target @14.7s
# └─myapp.service @14.1s +600ms
#   └─network-online.target @14.0s
#     └─NetworkManager-wait-online.service @2.0s +12.0s
#       └─NetworkManager.service @1.8s +190ms
#         └─basic.target @1.7s

# Critical chain for a specific unit
systemd-analyze critical-chain myapp.service

# Full SVG timeline of every unit
systemd-analyze plot > boot.svg

# Lint a unit file before shipping it
systemd-analyze verify /etc/systemd/system/myapp.service

# Score a service's sandboxing (0 = locked down, 10 = wide open)
systemd-analyze security myapp.service

blame lies by omission: a service that takes 10 s but nothing waits on doesn't delay boot at all, because it runs in parallel. Always read critical-chain before optimising anything.

War Story (illustrative)

A fleet of on-prem API servers took 2 minutes 10 seconds to come back after every kernel patch. systemd-analyze blame pointed at a slow snapd, which the team disabled — no change. critical-chain told the truth: systemd-networkd-wait-online.service was waiting for all configured links, including a second NIC that was cabled to nothing, and timing out at its 2-minute default. One line — ExecStart=/usr/lib/systemd/systemd-networkd-wait-online --any in a drop-in — brought boot to 19 seconds. The lesson: network-online.target means whatever wait-online decides "online" means, and its default is often "every interface."


Stage 6: Login and Services

At the end of the chain, multi-user.target pulls in getty@tty1.service (the text login prompt), sshd.service, and your own services. A login goes getty → login (PAM authentication) → your shell, and systemd-logind creates a session scope and a user-UID.slice cgroup so a user's processes are tracked and limited as a unit. A systemd --user instance starts per user, which is how desktop sessions and loginctl enable-linger user services work.

For a server, "boot complete" is better defined as "the health check passes" — something only your service and your load balancer know. That's why Type=notify plus a readiness signal (and in Kubernetes, a readiness probe) matters more than the moment systemd reaches its target.


Shutdown Is Boot in Reverse

Shutdown walks the dependency graph backwards: units ordered After=X stop before X. Each service gets KillSignal= (default SIGTERM), then SIGKILL after TimeoutStopSec= (default 90 s). A service that ignores SIGTERM therefore adds 90 seconds to every reboot — the classic "A stop job is running for..." message. The same signal mechanics from Signals & Process Control apply: catch SIGTERM, drain, exit.


Boot in the Cloud

A cloud VM still runs firmware, a bootloader, a kernel, an initramfs, and systemd — but the firmware is virtual and the machine doesn't know who it is until it asks.

Virtual firmware

Hypervisors emulate firmware: SeaBIOS (legacy BIOS) or OVMF (UEFI, from the EDK II project) on KVM/QEMU, and equivalent firmware in AWS Nitro, GCP, and Azure. There's no DIMM training, so the firmware stage drops to a second or two. Most providers now support UEFI with Secure Boot and a virtual TPM (Shielded VMs on GCP, Trusted Launch on Azure, UEFI Secure Boot on EC2).

Because there's no monitor, the serial console is the only window into early boot. Images set console=ttyS0 (or ttyAMA0 on ARM) so you can read kernel and systemd output via aws ec2 get-console-output or the provider's serial console when SSH never comes up.

cloud-init and the metadata service

A single AMI or image boots thousands of different instances. Hostname, SSH keys, network config, and bootstrap scripts are injected at first boot by cloud-init, which runs as several systemd units at different points in the boot:

flowchart TD
    GEN["systemd generator: is this a cloud? enable cloud-init"] --> LOCAL["cloud-init-local.service: before networking, find datasource, write network config"]
    LOCAL --> NETUP["Network comes up"]
    NETUP --> NET["cloud-init.service: fetch metadata + user-data, set hostname, SSH keys, users"]
    NET --> CFG["cloud-config.service: packages, mounts, write_files"]
    CFG --> FINAL["cloud-final.service: runcmd, user scripts, 'boot finished'"]

The data comes from the instance metadata service (IMDS), a link-local HTTP endpoint at 169.254.169.254 reachable only from inside the instance:

# AWS IMDSv2: get a session token first (PUT, not GET), then use it
TOKEN=$(curl -sX PUT http://169.254.169.254/latest/api/token \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/instance-id
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/user-data

# How long did cloud-init take, and what did it do?
cloud-init analyze blame
cloud-init status --long
sudo cat /var/log/cloud-init-output.log

A typical user-data payload:

#cloud-config
hostname: api-1
users:
  - name: deploy
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3Nz... deploy@ci
    sudo: ALL=(ALL) NOPASSWD:ALL
package_update: true
packages: [nginx, chrony]
write_files:
  - path: /etc/myapp/config.yaml
    content: |
      listen: 0.0.0.0:8080
runcmd:
  - systemctl enable --now nginx

The metadata service is a credential vending machine

IMDS also serves the instance's IAM role credentials. In the 2019 Capital One breach, an SSRF flaw in a misconfigured web application firewall let an attacker make the server request 169.254.169.254 on their behalf, retrieve role credentials, and read S3 data. IMDSv2's session token — obtained via a PUT with a custom header, and with a default hop limit of 1 so it doesn't reach containers behind an extra network hop — defeats most SSRF of this kind. Require IMDSv2 on every instance (HttpTokens=required) and never bake secrets into user-data, which any process on the box can read.

Cloud boot-to-SSH for a stock Linux image is typically 15–45 seconds, and most of that is userspace: cloud-init fetching metadata, regenerating SSH host keys, growing the root partition, and installing packages. Baking packages into the image ("golden AMI") instead of packages: in user-data is usually the biggest single win for autoscaling speed.


Containers: There Is No Boot

A container doesn't boot. There's no firmware, no bootloader, no kernel, no initramfs — the host kernel is already running. "Starting a container" is the runtime (runc, crun) creating namespaces and a cgroup, pivoting into the image's root filesystem, and calling execve() on your entrypoint. That's why containers start in tens to hundreds of milliseconds.

flowchart LR
    subgraph VM["Virtual machine"]
        V1["Firmware"] --> V2["Bootloader"] --> V3["Guest kernel"] --> V4["initramfs"] --> V5["systemd"] --> V6["App"]
    end
    subgraph CT["Container"]
        C1["runc: clone with new namespaces, join cgroup"] --> C2["pivot_root into image rootfs"] --> C3["execve entrypoint: becomes PID 1"]
    end

The consequence: your application becomes PID 1 of its PID namespace, with all of PID 1's special duties — and most applications weren't written for them.

Problem 1: Signals are ignored by default

The kernel protects init from being killed accidentally: a signal sent to PID 1 from inside its namespace is only delivered if PID 1 has a handler installed. The default action ("terminate") does not apply. So a Node, Python, or Java app that never registered a SIGTERM handler simply ignores docker stop and kubectl delete pod. After the grace period (10 s in Docker, 30 s terminationGracePeriodSeconds in Kubernetes) the runtime sends SIGKILL from the parent namespace, which always works — so every shutdown takes the full timeout and in-flight requests are dropped.

Shell-form commands make it worse:

# Shell form: PID 1 is /bin/sh -c, which does NOT forward SIGTERM to node
CMD node server.js

# Exec form: node is PID 1 and receives SIGTERM directly
CMD ["node", "server.js"]

# Entrypoint script? Use exec so the script is replaced by the app
# entrypoint.sh:
#   #!/bin/sh
#   ./migrate
#   exec node server.js

Problem 2: Nobody reaps the zombies

When a process exits, it stays a zombie until its parent calls wait(). If the parent dies first, the orphan is reparented to PID 1 — in a container, your app. If your app spawns subprocesses (shelling out to git, ffmpeg, headless Chrome, health-check scripts) and doesn't reap reparented grandchildren, zombies accumulate. Each holds a PID; when the cgroup's pids.max or the kernel's pid_max runs out, fork() fails with EAGAIN and the container falls over.

The fix: a tiny init

tini and dumb-init are minimal init processes (a few hundred lines of C) that do exactly two things: forward signals to the child and reap zombies.

sequenceDiagram
    participant RT as Container runtime
    participant T as tini (PID 1)
    participant A as node server.js (PID 7)
    participant Z as orphaned grandchild
    RT->>T: SIGTERM
    T->>A: Forward SIGTERM
    A->>A: Drain connections, exit 0
    Z->>T: Reparented on parent death, then exits
    T->>T: waitpid reaps zombie
    T->>RT: Exit with child exit code
FROM node:22-slim
RUN apt-get update && apt-get install -y --no-install-recommends tini \
    && rm -rf /var/lib/apt/lists/*
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["node", "server.js"]

Or without changing the image:

# Docker injects its bundled tini (docker-init) as PID 1
docker run --init myimage

# Verify what PID 1 is inside a running container
docker exec mycontainer cat /proc/1/cmdline | tr '\0' ' '

In Kubernetes, shareProcessNamespace: true makes the pod's pause container PID 1 for all containers in the pod; pause reaps zombies, but signal delivery then goes to each container's main process, which still needs a handler. Processes can also opt in to reaping their descendants without being PID 1 via prctl(PR_SET_CHILD_SUBREAPER, 1) — which is how tini works when it isn't PID 1, and how systemd's per-user manager and containerd-shim track their children.

OptionReaps zombiesForwards signalsNotes
App as PID 1, no handlerNoN/A — signals ignoredSlow, unclean shutdowns
App as PID 1 with SIGTERM handlerOnly its own childrenHandles it itselfFine if it never spawns subprocesses
tini / dumb-initYesYesThe default recommendation
docker run --initYesYesNo image change; Docker only
systemd inside the containerYesYesHeavy; needs cgroup and tmpfs setup; for "system containers" only

Graceful shutdown checklist for containers

Exec-form CMD, a real init (tini) if you spawn children, a SIGTERM handler that stops accepting work and drains in-flight requests, and a drain time shorter than the grace period. In Kubernetes, add a short preStop sleep so the endpoint is removed from load balancers before the process stops listening.


Firecracker and microVMs: Booting a Kernel in ~125 ms

Containers start fast but share the host kernel, which is a weak boundary for running untrusted, multi-tenant code (see Namespaces & cgroups and Container Escapes). Full VMs are strong boundaries but historically took seconds to boot. Firecracker, built by AWS for Lambda and Fargate, closes the gap: the Firecracker paper (NSDI 2020) reports booting to application code in as little as ~125 ms, with under 5 MiB of memory overhead per microVM and creation rates of up to 150 microVMs per second per host.

It gets there by deleting most of the boot chain:

Normal VMFirecracker
Emulated BIOS/UEFI firmware, POSTNo firmware. The VMM loads an uncompressed vmlinux straight into guest memory and jumps to its 64-bit entry point
BootloaderNone — kernel and cmdline are passed through the API
Hundreds of emulated devices (PCI, USB, VGA, ...)Five: virtio-net, virtio-block, vsock, a serial console, and a minimal keyboard controller used only to signal reset
Generic distro kernel + large initramfsMinimal custom kernel config with drivers built in; often no initramfs
systemd and dozens of servicesA small init, or the function runtime directly
# Configure and boot a microVM over Firecracker's REST API (Unix socket)
curl --unix-socket /tmp/fc.sock -X PUT http://localhost/boot-source \
  -H 'Content-Type: application/json' \
  -d '{"kernel_image_path": "./vmlinux",
       "boot_args": "console=ttyS0 reboot=k panic=1 pci=off quiet"}'

curl --unix-socket /tmp/fc.sock -X PUT http://localhost/drives/rootfs \
  -H 'Content-Type: application/json' \
  -d '{"drive_id": "rootfs", "path_on_host": "./rootfs.ext4",
       "is_root_device": true, "is_read_only": false}'

curl --unix-socket /tmp/fc.sock -X PUT http://localhost/actions \
  -H 'Content-Type: application/json' -d '{"action_type": "InstanceStart"}'

Note pci=off — there is no PCI bus to probe, which removes a large chunk of kernel init time. Lambda goes further with snapshots: boot a microVM once, initialise the runtime, snapshot its memory, then restore new instances from the snapshot (Lambda SnapStart). Restoring is faster than booting because there is no boot.

Similar designs: Cloud Hypervisor, QEMU's microvm machine type, and Kata Containers, which runs each Kubernetes pod inside a lightweight VM behind a normal container runtime interface.


Unikernels: No Operating System at All

A unikernel takes the idea to its end: compile the application together with just the OS pieces it needs (network stack, a filesystem driver, a scheduler) into a single image that runs directly on a hypervisor. One address space, one process, no users, no shell, no fork().

  • Boot time: single-digit to tens of milliseconds — there's barely anything to initialise. Unikraft reports boots in the low milliseconds on Firecracker.
  • Image size: often a few megabytes.
  • Attack surface: no shell to drop into, no unused syscalls to exploit, often no dynamic loading.
  • Projects: MirageOS (OCaml), Unikraft, OSv, IncludeOS, Nanos.

The costs are why they remain niche: no strace, top, or ssh to debug a misbehaving instance; applications that assume fork() or multiple processes need porting; and every library-OS bug is in your binary. They make sense for narrow, high-density workloads (network functions, edge functions) — rarely for a general web service.

Comparison

Bare metalCloud VMFirecracker microVMContainerUnikernel
Firmware/POST20 s – 3 min1 – 3 sNoneNoneNone (on Firecracker)
Own kernelYesYesYes (minimal)No — shares hostYes (library OS)
Typical start to app1 – 5 min15 – 60 s~125 ms – 1 s50 – 500 ms~5 – 50 ms
Isolation boundaryPhysicalHypervisorHypervisor (tiny device model)Kernel namespacesHypervisor
Memory overhead—100s of MB< 5 MiB VMMNear zeroA few MB
Who is PID 1systemdsystemdSmall init or appYour entrypointNo processes

Optimising Boot Time

Optimise in order of where the time actually goes. Measure with systemd-analyze first.

Firmware (bare metal)

  • Enable "fast boot" in firmware settings, disable unused option ROMs (PXE on NICs you don't network-boot, legacy RAID BIOS)
  • Disable memory re-training on every boot where the vendor allows it
  • Prefer kexec for kernel updates: kexec -l loads the new kernel and systemctl kexec jumps to it without going through firmware. This turns a 3-minute server reboot into seconds. Note: Secure Boot lockdown only allows kexec of signed kernels.

Bootloader

  • GRUB_TIMEOUT=0 (or 1) on servers; keep the menu reachable by holding Shift/Esc
  • systemd-boot or a direct UKI boot avoids GRUB's filesystem probing

Kernel

  • quiet — writing to a slow serial or framebuffer console at 115200 baud can cost hundreds of milliseconds
  • Build in the storage and filesystem drivers you need; drop the ones you don't (custom kernels for appliances, VMs, and microVMs)
  • lz4 or zstd compression on fast storage
  • Use initcall_debug to find slow driver probes

initramfs

  • dracut --hostonly or mkinitcpio's autodetect hook to include only this machine's drivers
  • For VMs with built-in virtio drivers and no LUKS/LVM, skip the initramfs entirely and boot with root=/dev/vda1

Userspace

  • Read systemd-analyze critical-chain, not just blame
  • Mask what you don't need: systemctl mask plymouth-quit-wait.service ModemManager.service
  • Don't make your service depend on network-online.target unless it truly needs a routable network at start; most servers just need to bind a socket, which works before the network is up (or use socket activation)
  • Use Type=notify so dependents start as soon as you're ready, not after a fixed sleep
  • Move one-time work (package installs, key generation) into the image build

Debugging a Machine That Won't Boot

Work out which stage failed from the last thing you saw, then use the right tool for that stage.

flowchart TD
    START["Machine does not come up"] --> Q1{"Any firmware logo or POST output?"}
    Q1 -->|No| HW["Hardware or firmware: power, RAM, BMC/IPMI event log"]
    Q1 -->|Yes| Q2{"Bootloader menu or 'grub rescue' prompt?"}
    Q2 -->|"grub rescue or no bootloader"| BLF["Bootloader: reinstall GRUB from rescue media, check ESP and efibootmgr"]
    Q2 -->|"Menu shows"| Q3{"Kernel messages appear?"}
    Q3 -->|"No, or panic early"| KF["Kernel: try older kernel from menu, remove quiet, add console=ttyS0"]
    Q3 -->|Yes| Q4{"Dropped to dracut emergency shell?"}
    Q4 -->|Yes| IRF["initramfs: root device missing, regenerate with missing driver, check root= UUID"]
    Q4 -->|No| Q5{"systemd emergency or rescue mode?"}
    Q5 -->|Yes| SDF["Userspace: bad fstab entry, failed fsck, journalctl -xb"]
    Q5 -->|"Boots but service down"| SVC["systemctl status, journalctl -u unit -b"]

The recovery ladder

From least to most drastic, all via the kernel command line (press e in GRUB, append, Ctrl+X to boot):

AppendYou getUse when
systemd.unit=rescue.targetRoot shell, local filesystems mounted, no network/servicesA service hangs or breaks boot
systemd.unit=emergency.targetRoot shell, root mounted read-only, nothing elsefstab or local-fs problems; rescue itself fails
rd.breakShell inside the initramfs, real root at /sysroot (read-only)Reset a root password, fix things before switch_root
init=/bin/bashBash as PID 1, no systemd at allsystemd itself is broken; remember mount -o remount,rw /
systemd.debug_shellA root shell on tty9 while boot proceedsInspect a hang live (disable afterwards — it's unauthenticated)
# Inside emergency mode: make root writable to fix files
mount -o remount,rw /

# The classic: an fstab entry for a disk/NFS share that isn't there
# Mark non-critical mounts so they can't block boot
# /etc/fstab
# nas:/backups  /mnt/backups  nfs  defaults,nofail,_netdev,x-systemd.device-timeout=10s  0 0

# Reset a forgotten root password via rd.break (RHEL family)
mount -o remount,rw /sysroot
chroot /sysroot
passwd root
touch /.autorelabel        # SELinux systems: relabel on next boot
exit; exit

Reading the journal across boots

# Logs from the current boot, with explanations for errors
journalctl -b -x

# Logs from the PREVIOUS boot (requires persistent journal)
journalctl -b -1 -p err

# List recorded boots
journalctl --list-boots

# Only kernel messages, previous boot — find why it crashed or hung
journalctl -k -b -1 | tail -50

# One unit, this boot
journalctl -u myapp.service -b --no-pager

# Units that failed this boot
systemctl --failed

# Make the journal persistent if 'journalctl -b -1' says no such boot
sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald

No persistent journal, no post-mortem

On many minimal and cloud images the journal lives only in /run (a tmpfs) and is lost on reboot. The one boot you care about — the one that crashed — is then gone. Create /var/log/journal (or set Storage=persistent in /etc/systemd/journald.conf) on every server, and ship logs off-box for machines that may never come back.

War Story (illustrative)

A database replica rebooted for a kernel update and never came back. The console showed You are in emergency mode. Someone had added an iSCSI backup volume to /etc/fstab months earlier with plain defaults. The iSCSI target had since been decommissioned, so local-fs.target — which every fstab mount without nofail is required by — failed, and systemd dropped to emergency mode. The fix was one word (nofail) and _netdev for the network mount; the outage lasted 40 minutes because nobody had console access configured. Two takeaways: only the filesystems the OS truly needs belong in the boot path, and out-of-band console access (IPMI, iDRAC, cloud serial console) is part of your infrastructure, not an afterthought.


Decision Guide

If you need...Choose
To protect against bootkits and tampered kernels on physical hardwareUEFI Secure Boot + measured boot with TPM-bound disk encryption; keep dbx updated
The simplest modern bootloader on UEFIsystemd-boot, or UKIs booted directly
BIOS support, complex storage in /boot, or multi-bootGRUB2
A root filesystem on LUKS, LVM, RAID, or network storageAn initramfs (host-only unless hardware changes are expected)
The fastest VM boot you controlCustom kernel with built-in drivers, no initramfs, quiet, minimal services
Faster autoscaling in the cloudGolden images instead of cloud-init package installs; IMDSv2 required
Containers that shut down cleanlyExec-form CMD, SIGTERM handler, tini if you spawn children
Strong isolation for untrusted code with container-like speedFirecracker / Kata microVMs
Smallest, fastest, single-purpose applianceConsider a unikernel — and accept the debugging cost
Services that start in parallel without racesType=notify + After=/Requires= together, or socket activation

Key Takeaways

  1. Each boot stage exists because the previous one can't do the next job — firmware can't read ext4, the kernel can't run LVM tools, and nothing before PID 1 knows your service graph.
  2. Secure Boot is a chain: firmware verifies shim, shim verifies GRUB and the kernel, the kernel verifies modules. It's broken by signed-but-vulnerable code, so revocation (dbx) matters as much as signing.
  3. initramfs solves a chicken-and-egg problem — the drivers and tools needed to mount root live on root. Most "won't boot after a hardware change" failures live here.
  4. systemd separates requirement from ordering. Requires= without After= starts things in parallel. Type=notify and socket activation are how you get parallel boot without races.
  5. Use systemd-analyze critical-chain, not blame, to find what actually delays boot — and journalctl -b -1 (with a persistent journal) to find why the last boot failed.
  6. Containers don't boot; your app becomes PID 1 — it ignores unhandled signals and must reap orphans. Use exec-form commands and tini.
  7. Fast-booting VMs come from deleting stages: Firecracker skips firmware and bootloader and emulates five devices to reach ~125 ms; unikernels skip the general-purpose OS entirely.