On this page

Cryptography Primer, PKI & Web of Trust for Distributed Systems

The eight fallacies of distributed computing include "the network is secure". Every message between your services crosses links you don't control — shared switches, cloud provider fabric, other tenants' hypervisors, the public Internet. Cryptography is how a distributed system gets guarantees that the network itself won't give: that nobody read the message, nobody changed it, and it really came from who it claims.

This page is a working engineer's primer. It covers what each primitive guarantees and — just as important — what it doesn't, then shows how they combine into identity: certificates, public key infrastructure, revocation, certificate transparency, mutual TLS between services, and the alternative "web of trust" model from PGP. It's written for using cryptography correctly through well-reviewed libraries; it is not a guide to designing primitives, and nothing here should be read as "write your own".

Deeper treatments elsewhere on the site: TLS Handshake (the protocol in detail), Encryption Overview, Symmetric vs Asymmetric, Cryptography for Engineers (RSA/ECC math, post-quantum), Hashing Algorithms, Key Management, Envelope Encryption, Encryption in Transit and Encryption at Rest.


Why It Exists: The Security Goals

Distributed systems need four distinct properties. Different primitives provide different subsets, and mixing them up is the root of many real vulnerabilities.

GoalQuestionProvided byNot provided by
ConfidentialityCan an eavesdropper read it?Encryption (AES-GCM, ChaCha20-Poly1305)Hashes, MACs, signatures
IntegrityWas it modified in transit or at rest?MACs, AEAD, signaturesPlain encryption (non-AEAD), plain hashes over untrusted channels
AuthenticationWho sent it / who am I talking to?MACs (shared key), signatures and certificates (public key)Encryption alone
Non-repudiationCan the sender later deny sending it, to a third party?Digital signatures (only the private-key holder could sign)MACs (either key holder could have produced the tag)

Two additional properties appear constantly in distributed protocols:

  • Freshness / replay protection — the message isn't an old valid message replayed. Provided by nonces, timestamps, sequence numbers, or session keys — never by the primitives alone.
  • Forward secrecy — compromise of a long-term key doesn't expose past sessions. Provided by ephemeral key exchange (ECDHE).

Kerckhoffs's principle

A system must remain secure even if everything about it except the key is public. In practice: use published, standardised algorithms; keep secrets in keys, never in obscure code; and assume attackers have your source.


Hash Functions

A cryptographic hash function maps any input to a fixed-size digest (SHA-256 → 32 bytes) with three properties:

PropertyMeaning
Preimage resistanceGiven a digest, you can't find an input that produces it
Second-preimage resistanceGiven an input, you can't find a different input with the same digest
Collision resistanceYou can't find any two inputs with the same digest

By the birthday bound, an n-bit hash offers about n/2 bits of collision resistance, which is why 256-bit hashes are the modern baseline.

Where distributed systems use plain hashes:

  • Content addressing and integrity of data you fetched from a trusted reference: Git object IDs, container image digests (sha256:…), package lock files, Merkle trees in replication anti-entropy and in certificate transparency logs.
  • Deduplication and change detection.
  • Building blocks for MACs, signatures and key derivation.

What a plain hash does not give you: authentication. If an attacker can change the message, they can change the hash too. A hash proves integrity only when the digest itself arrives over a trusted path (a signed manifest, a pinned digest in your deployment config).

Status of common hashes: MD5 and SHA-1 are broken for collision resistance — a practical SHA-1 collision was published in 2017 (the "SHAttered" work by CWI Amsterdam and Google), and MD5 collisions were used in the 2012 Flame malware to forge a code-signing certificate chaining to Microsoft. Use SHA-256/SHA-512, SHA-3, or BLAKE2/BLAKE3. Password storage is a separate problem requiring deliberately slow functions (Argon2id, scrypt, bcrypt) — see Hashing Algorithms.


Message Authentication Codes (HMAC)

A MAC is a keyed checksum: tag = MAC(key, message). Anyone holding the key can compute and verify tags; anyone without it can't forge a valid tag for a new message. That gives integrity and authentication between parties sharing a secret.

HMAC (RFC 2104) builds a MAC from a hash function:

where K′ is the key padded to the hash block size and ipad/opad are fixed constants. The nested structure matters: the naive construction SHA-256(key || message) is unsafe because SHA-256's Merkle–Damgård design lets anyone who knows a valid digest compute digests of extended messages without the key (length extension). HMAC isn't affected. Never build MACs by concatenating keys and messages yourself.

Distributed-systems uses of HMAC:

UseExample
Webhook signaturesThe sender signs the body with a shared secret; the receiver verifies (Stripe, GitHub and Slack all use HMAC-SHA256 for webhooks)
Request signingAWS Signature Version 4 derives a signing key with chained HMACs and signs the canonical request
TokensJWTs with HS256 are HMAC-SHA256 over header and payload
Cookies / session IDsSigned cookies prevent client tampering
Key derivationHKDF (RFC 5869) is built from HMAC

Two implementation rules:

  1. Compare tags in constant time (hmac.Equal in Go, crypto.timingSafeEqual in Node, hmac.compare_digest in Python). A byte-by-byte early-exit comparison leaks how many leading bytes matched through timing.
  2. Include freshness. A valid signed message can be replayed. Sign a timestamp along with the body and reject anything outside a short window; for stronger guarantees, track recently seen message IDs.

Code: verifying a webhook (TypeScript)

// verify-webhook.ts — Node.js
import { createHmac, timingSafeEqual } from "node:crypto";

const TOLERANCE_SECONDS = 300;

/**
 * Header format (a common convention): "t=<unix seconds>,v1=<hex hmac>"
 * Signed payload: `${t}.${rawBody}` — the timestamp is covered by the MAC.
 */
export function verifyWebhook(rawBody: string, header: string, secret: string, now = Date.now()): boolean {
  const parts = Object.fromEntries(header.split(",").map((kv) => kv.split("=", 2) as [string, string]));
  const t = Number(parts.t);
  if (!Number.isFinite(t) || !parts.v1) return false;
  if (Math.abs(now / 1000 - t) > TOLERANCE_SECONDS) return false;      // replay window

  const expected = createHmac("sha256", secret).update(`${t}.${rawBody}`).digest();
  const given = Buffer.from(parts.v1, "hex");
  return given.length === expected.length && timingSafeEqual(given, expected); // constant time
}

// Self-check
const secret = "whsec_test";
const body = JSON.stringify({ event: "invoice.paid", id: "inv_881" });
const t = Math.floor(Date.now() / 1000);
const sig = createHmac("sha256", secret).update(`${t}.${body}`).digest("hex");
console.assert(verifyWebhook(body, `t=${t},v1=${sig}`, secret), "valid signature rejected");
console.assert(!verifyWebhook(body + " ", `t=${t},v1=${sig}`, secret), "tampered body accepted");
console.assert(!verifyWebhook(body, `t=${t - 3600},v1=${sig}`, secret), "stale timestamp accepted");

Verify against the raw request bytes, before any JSON parsing or re-serialisation — a re-encoded body won't match the sender's bytes.


Symmetric Encryption

Symmetric encryption uses one secret key to both encrypt and decrypt. It's fast — modern CPUs have AES instructions (AES-NI on x86, ARMv8 Cryptography Extensions) that encrypt at multiple GB/s per core — so it does all the bulk work in TLS, disk encryption and application-level encryption.

AES and why the mode matters

AES is a block cipher: it encrypts exactly 16 bytes under a 128-, 192- or 256-bit key. To encrypt longer data you need a mode of operation, and the mode decides almost everything about security:

ModeWhat it doesUse?
ECBEncrypts each block independentlyNever. Identical plaintext blocks → identical ciphertext blocks; patterns leak (the "ECB penguin" image)
CBCEach block XORed with previous ciphertext; needs random IVLegacy only. No integrity on its own; padding-oracle bugs plagued it in TLS
CTRTurns AES into a stream cipher with a counterBuilding block; no integrity on its own
GCMCTR encryption + GHASH authentication tagYes — default AEAD in TLS 1.2/1.3
XTSTweakable mode for disk sectorsDisk/full-volume encryption (see Encryption at Rest)

Authenticated encryption (AEAD)

Encryption without integrity is a trap: an attacker who can't read a ciphertext may still be able to flip bits in it and cause predictable changes in the plaintext, or use the receiver's error behaviour as an oracle. Authenticated Encryption with Associated Data (AEAD) fixes this by producing a ciphertext and a tag; decryption fails outright if anything was modified.

flowchart LR
    K["Key (256-bit)"] --> E["AEAD Seal"]
    N["Nonce (96-bit, unique per key)"] --> E
    P["Plaintext"] --> E
    A["Associated data — authenticated, not encrypted"] --> E
    E --> C["Ciphertext"]
    E --> T["Tag (128-bit)"]

Associated data is how you bind a ciphertext to its context: a tenant ID, a record ID, a protocol header. Decryption with a different context fails, so an attacker can't move an encrypted value from one row or tenant to another. Use it.

The two AEADs you'll meet everywhere:

  • AES-GCM — fastest where AES hardware exists.
  • ChaCha20-Poly1305 (RFC 8439) — fast in software, constant-time without special instructions; popular on mobile and in WireGuard.

Nonces: the one rule

A GCM nonce (96 bits) must never repeat under the same key. Reusing one reveals the XOR of the two plaintexts and allows recovery of the authentication subkey, after which tags can be forged. NIST SP 800-38D limits randomly generated 96-bit nonces to 232 encryptions per key to keep collision probability negligible. Practical consequences:

  • Random nonces are fine at moderate volume; at very high volume per key, rotate keys or use a counter-based nonce with a guarantee of no reuse (hard in distributed systems where many nodes encrypt with the same key).
  • XChaCha20-Poly1305 uses a 192-bit nonce, making random nonces safe at essentially any volume.
  • Nonce-misuse-resistant modes such as AES-GCM-SIV (RFC 8452) degrade gracefully if a nonce repeats.
  • Envelope encryption — a fresh data key per object — keeps per-key volumes small.

Code: HMAC, AES-GCM and Ed25519 (Go)

// primitives.go — go run primitives.go
package main

import (
	"crypto/aes"
	"crypto/cipher"
	"crypto/ed25519"
	"crypto/hmac"
	"crypto/rand"
	"crypto/sha256"
	"encoding/hex"
	"fmt"
)

// ---- Integrity + authenticity with a shared key: HMAC-SHA256 ----
func sign(key, msg []byte) []byte {
	m := hmac.New(sha256.New, key)
	m.Write(msg)
	return m.Sum(nil)
}

func verify(key, msg, tag []byte) bool {
	return hmac.Equal(sign(key, msg), tag) // constant-time comparison
}

// ---- Confidentiality + integrity: AES-256-GCM (AEAD) ----
// Returns nonce || ciphertext||tag. aad is authenticated but not encrypted.
func seal(key, plaintext, aad []byte) ([]byte, error) {
	block, err := aes.NewCipher(key) // 32-byte key -> AES-256
	if err != nil {
		return nil, err
	}
	gcm, err := cipher.NewGCM(block)
	if err != nil {
		return nil, err
	}
	nonce := make([]byte, gcm.NonceSize()) // 12 bytes; MUST be unique per key
	if _, err := rand.Read(nonce); err != nil {
		return nil, err
	}
	return gcm.Seal(nonce, nonce, plaintext, aad), nil
}

func open(key, sealed, aad []byte) ([]byte, error) {
	block, err := aes.NewCipher(key)
	if err != nil {
		return nil, err
	}
	gcm, err := cipher.NewGCM(block)
	if err != nil {
		return nil, err
	}
	n := gcm.NonceSize()
	if len(sealed) < n+gcm.Overhead() {
		return nil, fmt.Errorf("ciphertext too short")
	}
	return gcm.Open(nil, sealed[:n], sealed[n:], aad) // fails if anything was modified
}

func main() {
	// HMAC
	macKey := make([]byte, 32)
	rand.Read(macKey)
	body := []byte(`{"event":"invoice.paid","id":"inv_881"}`)
	tag := sign(macKey, body)
	fmt.Println("HMAC tag:", hex.EncodeToString(tag)[:16]+"…")
	fmt.Println("verify original:", verify(macKey, body, tag))
	fmt.Println("verify tampered:", verify(macKey, []byte(`{"event":"invoice.paid","id":"inv_882"}`), tag))

	// AES-GCM
	encKey := make([]byte, 32)
	rand.Read(encKey)
	aad := []byte("tenant=42;record=7") // binds ciphertext to its context
	ct, _ := seal(encKey, []byte("card ending 4242"), aad)
	pt, err := open(encKey, ct, aad)
	fmt.Printf("GCM round trip: %q err=%v (sealed %d bytes)\n", pt, err, len(ct))
	ct[len(ct)-1] ^= 1
	_, err = open(encKey, ct, aad)
	fmt.Println("GCM tampered:", err)
	ct[len(ct)-1] ^= 1
	_, err = open(encKey, ct, []byte("tenant=43;record=7"))
	fmt.Println("GCM wrong context:", err)

	// Ed25519 signatures
	pub, priv, _ := ed25519.GenerateKey(rand.Reader)
	release := []byte("artifact sha256=9f86d0... version=1.4.2")
	sig := ed25519.Sign(priv, release)
	fmt.Println("Ed25519 sig bytes:", len(sig), "verify:", ed25519.Verify(pub, release, sig))
	fmt.Println("Ed25519 tampered:", ed25519.Verify(pub, append(release, '!'), sig))
}

Output (the tag differs per run because keys are random):

HMAC tag: 2180631f799951e1…
verify original: true
verify tampered: false
GCM round trip: "card ending 4242" err=<nil> (sealed 44 bytes)
GCM tampered: cipher: message authentication failed
GCM wrong context: cipher: message authentication failed
Ed25519 sig bytes: 64 verify: true
Ed25519 tampered: false

The sealed size is 12 (nonce) + 16 (plaintext) + 16 (tag) = 44 bytes: AEAD costs a fixed 28 bytes per message, not a percentage.


Asymmetric Cryptography and Key Exchange

Symmetric crypto has a bootstrapping problem: both sides need the same secret key, and you can't send it over the network you're trying to protect. With n parties, pairwise keys need n(n−1)/2 secrets. Public-key (asymmetric) cryptography solves this: each party has a key pair — a private key it never shares and a public key it can publish.

RSA and elliptic curves, intuitively

  • RSA relies on the difficulty of factoring the product of two large primes. Operations are modular exponentiations; keys are large (2048-bit minimum today, 3072-bit for ~128-bit security). Use RSA-OAEP for encryption and RSA-PSS for signatures; the older PKCS#1 v1.5 padding is still widespread for signatures but has a long history of implementation pitfalls for encryption.
  • Elliptic-curve cryptography (ECC) relies on the difficulty of the elliptic-curve discrete logarithm problem. Keys are much smaller for the same security: a 256-bit curve gives roughly the security of 3072-bit RSA. Common curves: P-256 (NIST), Curve25519 — used as X25519 for key exchange (RFC 7748) and Ed25519 for signatures (RFC 8032).
RSA-2048RSA-3072P-256 / X25519 / Ed25519
Approx. security level~112 bits~128 bits~128 bits
Public key size256 bytes384 bytes32–65 bytes
Signature size256 bytes384 bytes64 bytes (Ed25519)
Signing speedSlowSlowerFast
Verification speedFastFastFast

Security-level equivalences are from NIST SP 800-57. See Cryptography for Engineers for the underlying math.

Asymmetric operations are far slower than symmetric ones, so real protocols use them only to authenticate and to agree on a symmetric key, then switch to AEAD for the data. That pattern is called hybrid encryption, and it's how TLS, SSH, Signal and envelope encryption all work.

Diffie–Hellman key exchange

Diffie–Hellman (1976) lets two parties who have never met agree on a shared secret over a public channel. The classic intuition is mixing paint:

  1. Alice and Bob agree publicly on a common colour (yellow).
  2. Each picks a secret colour, mixes it with yellow, and sends the mixture.
  3. Each adds their own secret colour to the mixture they received.
  4. Both end up with the same three-colour mix; an observer has only the two two-colour mixtures and can't "unmix" them.

In the finite-field version, with public prime p and generator g: Alice sends ga mod p, Bob sends gb mod p, both compute gab mod p. In ECDH, the "exponentiation" is scalar multiplication of a curve point.

sequenceDiagram
    participant A as Alice
    participant B as Bob
    Note over A: picks secret a, computes A_pub = a x G
    Note over B: picks secret b, computes B_pub = b x G
    A->>B: A_pub
    B->>A: B_pub
    Note over A: shared = a x B_pub
    Note over B: shared = b x A_pub
    Note over A,B: both derive the same session keys via HKDF

Two critical points:

  • Plain DH is unauthenticated. An active man-in-the-middle can run one exchange with each side. DH must be combined with authentication — in TLS, the server signs the handshake with the private key matching its certificate. Key exchange gives you a secret; signatures and certificates tell you with whom.
  • Ephemeral DH (DHE/ECDHE) gives forward secrecy. If fresh key pairs are generated for each session and discarded afterwards, a later compromise of the server's long-term signing key doesn't let an attacker decrypt recorded sessions. TLS 1.3 requires ephemeral key exchange and removed static RSA key transport for this reason. See TLS Handshake.

Post-quantum note

A large quantum computer would break RSA and ECC (Shor's algorithm) but only weaken symmetric ciphers and hashes (Grover's algorithm, countered by 256-bit keys). NIST standardised ML-KEM (FIPS 203) for key establishment and ML-DSA (FIPS 204) for signatures in August 2024. Because recorded traffic could be decrypted later ("harvest now, decrypt later"), key exchange is migrating first: hybrid X25519 + ML-KEM key exchange is already deployed in major browsers and CDNs. Cryptography for Engineers covers the migration.


Digital Signatures

A signature is the public-key analogue of a MAC: sig = Sign(private_key, message), and anyone with the public key can run Verify(public_key, message, sig). Because only the private-key holder can sign, signatures provide non-repudiation and — more importantly for distributed systems — verification by parties who don't share any secret with the signer.

MAC (HMAC)Signature (Ed25519, ECDSA, RSA-PSS)
KeysOne shared secretPrivate key signs, public key verifies
Who can verifyOnly key holdersAnyone with the public key
Who can forgeAny key holderOnly the private-key holder
Non-repudiationNoYes
SpeedVery fastSlower (verification usually fast)
DistributionSecret must reach every verifier securelyPublic key can be published
Typical useService-to-service with a shared secret, webhooksCertificates, JWTs verified by many services (RS256/ES256/EdDSA), software releases, consensus votes

Where distributed systems use signatures:

  • Certificates — a CA signs a binding between a public key and a name.
  • Tokens — an identity provider signs a JWT once; every service verifies it with the provider's published public key (JWKS) without calling back. See JWT Deep Dive.
  • Software supply chain — signed packages, container images and commits; transparency logs for signatures (Sigstore).
  • Byzantine fault-tolerant protocols — replicas sign votes so that others can verify them; see Byzantine Fault Tolerance.

Nonce hazards in ECDSA: ECDSA requires a fresh secret random value per signature; reusing or biasing it leaks the private key. This happened famously with Sony's PlayStation 3 code-signing key (disclosed in 2010), where a constant value was used. Ed25519 derives its per-signature value deterministically from the key and message, eliminating this failure class — one reason it's a good default for new systems. RFC 6979 gives a deterministic construction for ECDSA too.


Certificates and PKI

Public keys solve the shared-secret problem but create a new one: how do you know a public key really belongs to payments.internal or example.com? If an attacker can substitute their key, every guarantee collapses. A certificate answers by having a trusted third party sign the binding.

What's in an X.509 certificate

FieldPurpose
Subject / Subject Alternative Names (SAN)The identity: DNS names, IPs, URIs (e.g. SPIFFE IDs), emails
Subject public keyThe key being vouched for
IssuerWho signed this certificate
Validity (notBefore, notAfter)When it may be trusted
Serial numberUnique per issuer; used in revocation
Key usage / extended key usageWhat the key may do (digitalSignature, serverAuth, clientAuth, keyCertSign …)
Basic constraintsWhether this certificate is a CA, and path-length limits
ExtensionsRevocation pointers (CRL distribution points, OCSP URLs), SCTs, name constraints, policies
SignatureThe issuer's signature over all of the above

Chains of trust

Browsers and operating systems ship a root store — a curated list of root CA certificates (Mozilla's NSS store, Apple's, Microsoft's, the Chrome Root Store). Roots are kept offline and sign intermediate CAs; intermediates sign leaf (end-entity) certificates. A verifier builds a chain from the leaf to a trusted root and checks every link.

flowchart TB
    Root["Root CA (in trust store, kept offline)"] -->|"signs"| Int["Intermediate CA"]
    Int -->|"signs"| Leaf["Leaf: api.example.com"]
    Leaf --> V{"Verifier checks"}
    V --> C1["Each signature valid up the chain"]
    V --> C2["Validity dates, key usage, basic constraints"]
    V --> C3["Name matches SAN"]
    V --> C4["Not revoked"]
    V --> C5["SCTs present (public web)"]

Validation steps a correct implementation performs:

  1. Signature chain: each certificate is signed by the next one up, ending at a trusted root.
  2. Validity period of every certificate.
  3. Constraints: intermediates must have CA:true; path-length and name constraints are honoured; key usage allows the operation.
  4. Name matching: the requested hostname (or identity) matches a SAN. The legacy Common Name field is not used for hostname matching by modern clients.
  5. Revocation status (with the caveats below).
  6. Certificate Transparency proofs, for publicly trusted certificates.

Don't implement these yourself — use your platform's verifier, and never ship code that disables verification (InsecureSkipVerify: true, rejectUnauthorized: false, verify=False) outside a test.

Lifetimes are shrinking

Publicly trusted TLS certificates have been capped at 398 days since September 2020, and the CA/Browser Forum approved a schedule (ballot SC-081, 2025) that steps the maximum down further, reaching 47 days in 2029. Short lifetimes limit the damage of a compromised key and of revocation not working — and they make automation (ACME, as used by Let's Encrypt) mandatory rather than optional. See TLS Handshake.

Revocation: CRL, OCSP and their limits

When a private key leaks or a certificate was mis-issued, it must be revoked before it expires. The mechanisms:

MechanismHowProblems
CRL (Certificate Revocation List)CA publishes a signed list of revoked serials; clients download itLists can be large; clients cache them, so revocation is delayed
OCSPClient asks the CA's responder "is serial X revoked?"Adds latency, leaks browsing to the CA, and responders go down — so clients usually soft-fail (treat no answer as "fine"), which an attacker who can block traffic can exploit
OCSP staplingServer fetches a signed, time-limited OCSP response and includes it in the handshakeFixes privacy and latency; only effective if clients require it (Must-Staple), which few do
Browser-pushed setsBrowser vendors aggregate revocations and push compact sets (Chrome's CRLSets, Firefox's CRLite)Coverage and freshness depend on the vendor
Short-lived certificatesCertificates expire before revocation would matterRequires reliable automation

The industry trend is clear: revocation checking at handshake time has never worked well, so the ecosystem is moving to short lifetimes plus browser-distributed revocation data. Let's Encrypt, for example, announced in 2024 that it would end OCSP support in favour of CRLs, and completed the transition in 2025.

For internal PKI the same lesson applies, more strongly: issue certificates that live hours or days, rotate them automatically, and you rarely need revocation at all.

Certificate Transparency

Any of the hundreds of trusted CAs can technically issue a certificate for any domain. Mis-issuance — through compromise, error or coercion — used to be discovered only by luck. The 2011 DigiNotar breach is the canonical case: attackers obtained fraudulent certificates including one for *.google.com that was used to intercept users in Iran; it was spotted largely because Chrome pinned Google's keys. DigiNotar was distrusted by all major browsers and went bankrupt. Later, Symantec's CA business was distrusted by browsers (in stages through 2018) after repeated mis-issuance findings.

Certificate Transparency (CT, RFC 6962) makes issuance public:

  1. Before issuing, the CA submits the certificate (as a precertificate) to multiple independent, append-only CT logs, built as Merkle trees.
  2. Each log returns a Signed Certificate Timestamp (SCT) promising to include it.
  3. The CA embeds SCTs in the certificate; browsers (Chrome has required CT for publicly trusted certificates since 2018) reject certificates without valid SCTs.
  4. Monitors watch the logs; domain owners can be alerted to any certificate issued for their names (e.g. via crt.sh or commercial monitoring).
sequenceDiagram
    participant CA as CA
    participant L as CT Log
    participant B as Browser
    participant M as Monitor / Domain owner
    CA->>L: submit precertificate
    L-->>CA: SCT, signed promise to log it
    CA->>CA: issue certificate with embedded SCTs
    B->>B: TLS handshake, verify SCTs from trusted logs
    M->>L: watch new entries
    Note over M: alert on unexpected certificates for my domains

CT doesn't prevent mis-issuance; it guarantees that mis-issuance is detectable, which in turn makes CAs accountable. The Merkle-tree logs allow anyone to verify inclusion and that the log is append-only — the same data structure distributed systems use for anti-entropy.


mTLS Between Services

Standard TLS authenticates only the server. In a microservice architecture both directions matter: payments needs to know that the caller really is checkout, not a compromised pod in another namespace. Mutual TLS adds a client certificate to the handshake, so each side proves possession of a private key bound to an identity. It's a core building block of zero trust networking: identity is cryptographic, not inferred from IP addresses.

sequenceDiagram
    participant C as checkout (client)
    participant S as payments (server)
    C->>S: ClientHello, key share
    S->>C: ServerHello, key share, Certificate, CertificateRequest, CertificateVerify, Finished
    Note over C: verify server chain to internal CA, check identity is payments
    C->>S: Certificate, CertificateVerify, Finished
    Note over S: verify client chain, extract identity, authorize checkout
    C->>S: application data over the encrypted channel

Workload identity: SPIFFE

Using DNS names as service identities is awkward (many services, ephemeral pods, multiple clusters). SPIFFE (Secure Production Identity Framework For Everyone, a CNCF project) standardises workload identity as a URI such as spiffe://example.org/ns/prod/sa/payments, carried in an X.509 certificate's URI SAN — an X.509-SVID. SPIRE is its reference implementation: it attests workloads (by node and process properties) and issues short-lived SVIDs automatically. Service meshes such as Istio use SPIFFE-format identities and issue and rotate workload certificates automatically, with lifetimes on the order of a day by default.

Operational shape of internal mTLS

ConcernRecommended approach
CA hierarchyOffline root; online intermediate(s) per environment or cluster; never issue leaves from the root
IdentitySPIFFE IDs or service names in SANs; authorise on identity, not just "has a valid cert"
LifetimesHours to a few days; automated rotation well before expiry (e.g. at half-life)
RevocationRely on short lifetimes; keep a deny-list for emergencies
DistributionSidecar/mesh or an agent (SPIRE, cert-manager, Vault PKI) — not files copied by hand
Trust bundle rotationPublish the new CA alongside the old one; switch issuing; retire the old one after every leaf has rolled
AuthorisationAfter authentication, apply policy: which identities may call which methods
ObservabilityAlert on certificates nearing expiry and on handshake failure rates

Authentication is not authorisation

tls.RequireAndVerifyClientCert proves that the client holds a certificate from your CA. If every workload gets a certificate from the same CA, that includes every compromised workload. You still need to check which identity connected and whether it's allowed to do this — as the code below does.

Code: a private CA, SPIFFE identities and mTLS (Go)

This self-contained program creates a CA, issues short-lived workload certificates with SPIFFE URI SANs, and runs a TLS 1.3 server that requires client certificates and authorises a specific identity. An identity from the same CA but outside the allow-list is rejected.

// mtls.go — go run mtls.go
// Builds a private CA, issues SPIFFE-style workload certificates, and runs an
// mTLS connection where each side verifies the other's identity.
package main

import (
	"crypto/ecdsa"
	"crypto/elliptic"
	"crypto/rand"
	"crypto/tls"
	"crypto/x509"
	"crypto/x509/pkix"
	"errors"
	"fmt"
	"io"
	"math/big"
	"net"
	"net/url"
	"time"
)

type issued struct {
	cert *x509.Certificate
	key  *ecdsa.PrivateKey
	der  []byte
}

func newCA() issued {
	key, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
	tmpl := &x509.Certificate{
		SerialNumber:          big.NewInt(1),
		Subject:               pkix.Name{CommonName: "demo internal CA"},
		NotBefore:             time.Now().Add(-time.Minute),
		NotAfter:              time.Now().Add(24 * time.Hour),
		IsCA:                  true,
		BasicConstraintsValid: true,
		KeyUsage:              x509.KeyUsageCertSign | x509.KeyUsageCRLSign,
	}
	der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key)
	if err != nil {
		panic(err)
	}
	cert, _ := x509.ParseCertificate(der)
	return issued{cert, key, der}
}

// issue creates a short-lived leaf certificate whose identity is a SPIFFE ID URI SAN.
func issue(ca issued, serial int64, spiffeID string, usage x509.ExtKeyUsage, dns ...string) issued {
	key, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
	id, _ := url.Parse(spiffeID)
	tmpl := &x509.Certificate{
		SerialNumber: big.NewInt(serial),
		NotBefore:    time.Now().Add(-time.Minute),
		NotAfter:     time.Now().Add(1 * time.Hour), // short-lived: rotation, not revocation
		KeyUsage:     x509.KeyUsageDigitalSignature,
		ExtKeyUsage:  []x509.ExtKeyUsage{usage},
		URIs:         []*url.URL{id},
		DNSNames:     dns,
	}
	der, err := x509.CreateCertificate(rand.Reader, tmpl, ca.cert, &key.PublicKey, ca.key)
	if err != nil {
		panic(err)
	}
	cert, _ := x509.ParseCertificate(der)
	return issued{cert, key, der}
}

func tlsCert(i issued) tls.Certificate {
	return tls.Certificate{Certificate: [][]byte{i.der}, PrivateKey: i.key, Leaf: i.cert}
}

// requireSPIFFEID runs AFTER normal chain verification and pins the peer's identity.
func requireSPIFFEID(allowed ...string) func([][]byte, [][]*x509.Certificate) error {
	return func(_ [][]byte, chains [][]*x509.Certificate) error {
		if len(chains) == 0 || len(chains[0]) == 0 {
			return errors.New("no verified chain")
		}
		for _, u := range chains[0][0].URIs {
			for _, a := range allowed {
				if u.String() == a {
					return nil
				}
			}
		}
		return fmt.Errorf("peer identity %v not authorized", chains[0][0].URIs)
	}
}

func main() {
	ca := newCA()
	pool := x509.NewCertPool()
	pool.AddCert(ca.cert)

	server := issue(ca, 2, "spiffe://example.org/ns/prod/sa/payments", x509.ExtKeyUsageServerAuth, "payments.internal")
	client := issue(ca, 3, "spiffe://example.org/ns/prod/sa/checkout", x509.ExtKeyUsageClientAuth)
	rogue := issue(ca, 4, "spiffe://example.org/ns/dev/sa/scratch", x509.ExtKeyUsageClientAuth)

	serverCfg := &tls.Config{
		MinVersion:            tls.VersionTLS13,
		Certificates:          []tls.Certificate{tlsCert(server)},
		ClientAuth:            tls.RequireAndVerifyClientCert, // the "m" in mTLS
		ClientCAs:             pool,
		VerifyPeerCertificate: requireSPIFFEID("spiffe://example.org/ns/prod/sa/checkout"),
	}

	ln, err := tls.Listen("tcp", "127.0.0.1:0", serverCfg)
	if err != nil {
		panic(err)
	}
	defer ln.Close()
	go func() {
		for {
			c, err := ln.Accept()
			if err != nil {
				return
			}
			go func(c net.Conn) {
				defer c.Close()
				if err := c.(*tls.Conn).Handshake(); err != nil {
					return
				}
				io.WriteString(c, "hello, authenticated checkout\n")
			}(c)
		}
	}()

	dial := func(who issued) (string, error) {
		cfg := &tls.Config{
			MinVersion:            tls.VersionTLS13,
			RootCAs:               pool,
			ServerName:            "payments.internal",
			Certificates:          []tls.Certificate{tlsCert(who)},
			VerifyPeerCertificate: requireSPIFFEID("spiffe://example.org/ns/prod/sa/payments"),
		}
		conn, err := tls.Dial("tcp", ln.Addr().String(), cfg)
		if err != nil {
			return "", err
		}
		defer conn.Close()
		b, err := io.ReadAll(conn) // in TLS 1.3 a rejected client cert surfaces here
		return string(b), err
	}

	msg, err := dial(client)
	fmt.Printf("checkout -> payments: %q err=%v\n", msg, err)
	if err != nil || msg == "" {
		panic("authorized client should succeed")
	}
	msg, err = dial(rogue)
	fmt.Printf("scratch  -> payments: %q err=%v\n", msg, err)
	if err == nil && msg != "" {
		panic("unauthorized client must be rejected")
	}
	fmt.Println("self-check passed")
}

Output:

checkout -> payments: "hello, authenticated checkout\n" err=<nil>
scratch  -> payments: "" err=remote error: tls: bad certificate
self-check passed

Notice the comment in dial: in TLS 1.3 the client sends its certificate after it has already considered the handshake complete, so a server-side rejection appears on the first read rather than in Dial. Code that only checks the dial error will think an unauthorised connection succeeded.

In production you'd get certificates from SPIRE, a mesh, cert-manager or Vault rather than generating them in-process, and reload them on rotation (Go's tls.Config.GetCertificate / GetClientCertificate callbacks make hot rotation straightforward). Encryption in Transit shows a Node.js equivalent.


Hierarchical PKI vs. the PGP Web of Trust

X.509 PKI is hierarchical: trust flows down from a small set of roots chosen by software vendors. There's another model.

The web of trust

Phil Zimmermann released PGP in 1991; its standard form is OpenPGP (RFC 4880, updated by RFC 9580 in 2024), implemented by GnuPG and others. PGP has no mandatory central authority. Instead:

  • Each user generates a key pair and publishes the public key (keyservers, websites, in person).
  • Users sign each other's keys to attest "I've verified this key belongs to this person" — classically at in-person key-signing parties, checking ID and fingerprints.
  • Each user assigns owner trust to others ("how much do I trust Bob's judgement when he signs keys?").
  • A key is considered valid if it's signed by enough keys you trust — in GnuPG's classic model, one fully trusted signer or several marginally trusted ones, within a limited path length.
flowchart LR
    subgraph Hierarchical["Hierarchical PKI (X.509)"]
        R["Root CA"] --> I1["Intermediate"]
        I1 --> L1["alice.example.com"]
        I1 --> L2["bob.example.com"]
    end
    subgraph WoT["Web of trust (PGP)"]
        A["Alice"] -->|"signed"| B["Bob"]
        B -->|"signed"| C["Carol"]
        A -->|"signed"| D["Dave"]
        D -->|"signed"| C
        C -->|"signed"| A
    end

Comparing the models

Hierarchical PKIWeb of trust
Trust anchorsRoot stores curated by vendorsEach user's own key and chosen introducers
Who vouchesCommercial or organisational CAsPeers
ScalabilityExcellent — billions of TLS connections dailyPoor beyond communities; trust paths get long and sparse
Single point of failureA compromised CA can issue for any name (mitigated by CT)None central, but a careless signer weakens everyone who trusts them
RevocationCRL / OCSP / short lifetimesRevocation certificates published by the key owner; distribution is unreliable
UsabilityInvisible to end usersNotoriously hard: fingerprints, trust levels, key management
Identity typeDomain names, services, organisationsPeople (emails)
Where it lives todayTLS, code signing, mTLS, S/MIMESigning Linux distribution packages and release artifacts, some secure email

The web of trust's practical weaknesses became concrete in 2019, when the SKS keyserver network — which by design never deletes data — was hit by certificate-flooding attacks that attached tens of thousands of signatures to specific public keys, making them impractical for GnuPG to import. Newer keyservers such as keys.openpgp.org dropped unverified third-party signatures, effectively stepping back from the open web-of-trust model.

Trust models in modern distributed systems

Most real systems sit between the two extremes:

  • Private hierarchical PKI for services (a company CA, a mesh CA) — hierarchical, but you choose the root.
  • TOFU (trust on first use) — SSH's known_hosts: accept a key the first time, alert if it changes. Simple, vulnerable only on first contact.
  • Pinned keys / trust bundles — a service ships with the exact public keys it trusts (e.g. JWKS endpoints of a specific IdP, a SPIFFE trust bundle).
  • Transparency logs — CT for certificates, Sigstore's Rekor for software signatures: anyone can issue, but every issuance is publicly auditable.
  • Federation — SPIFFE trust-domain federation exchanges trust bundles between organisations: hierarchical within each domain, explicit peer agreements between them.

Key Management and Rotation Basics

The cryptography is rarely the weak point; key handling is. A short checklist — Key Management and Secrets Management go much deeper.

Where keys live

TierExampleUse for
HSMCloud HSMs, on-prem HSM appliancesRoot CA keys, KMS master keys; keys never leave hardware
KMSAWS KMS, Google Cloud KMS, Azure Key VaultKey-encryption keys; envelope encryption; signing via API
Secrets managerVault, cloud secret managersDistributing data keys, API keys, database credentials
Process memoryShort-lived data keys, TLS session keysBulk crypto operations
NeverSource code, container images, CI logs, environment dumps—

Envelope encryption ties this together: a KMS-held key-encryption key wraps per-object data keys, so bulk encryption happens locally while the master key never leaves the KMS. See Envelope Encryption.

Rotation without downtime

Rotation in a distributed system is a protocol, not an event, because you can't update every node simultaneously:

stateDiagram-v2
    [*] --> Generated
    Generated --> Published: "distribute new key / CA to all verifiers"
    Published --> Active: "start signing / encrypting with new key"
    Active --> Retiring: "old key: verify / decrypt only"
    Retiring --> Destroyed: "no data or sessions depend on it"
    Destroyed --> [*]
  1. Overlap. Verifiers must trust the new key before signers start using it, and keep trusting the old key until everything signed or encrypted with it has expired or been re-encrypted.
  2. Key IDs. Tag every ciphertext, token and signature with a key identifier (kid in JWTs, key version in KMS ciphertexts) so verifiers know which key to use during overlap.
  3. Automate. Rotation that requires a human is rotation that doesn't happen. Short-lived certificates force this discipline.
  4. Separate keys by purpose. Don't reuse an encryption key as a MAC key or one signing key across environments; derive purpose-specific keys with HKDF if you must start from one secret.
  5. Plan for compromise. Know how you'd revoke a CA, rotate a JWT signing key with immediate effect, or re-encrypt data — and rehearse it.

War Story (illustrative)

A platform team ran an internal CA whose intermediate certificate had a five-year lifetime, created by an engineer who had since left. Leaf certificates rotated automatically every 30 days, so everyone assumed rotation was "solved". When the intermediate expired, every newly issued leaf chained to an expired certificate and every mTLS handshake in the fleet began failing as leaves rotated, over about a day. The fix was quick once diagnosed; the lessons were to monitor expiry of every certificate in the chain, keep intermediate lifetimes short enough that their rotation is routine, and practise trust-bundle rotation with overlapping CAs.


Failure Modes and Common Mistakes

MistakeConsequenceCorrect approach
Disabling certificate verification "temporarily"Any on-path attacker can impersonate the serverFix the trust store; use a test CA in tests
Encryption without authentication (CBC/CTR without MAC)Ciphertext tampering, padding oraclesUse an AEAD
Reusing AES-GCM noncesPlaintext leakage and tag forgeryRandom nonces within limits, key rotation, XChaCha20 or GCM-SIV
SHA256(secret + message) as a MACLength-extension forgeryHMAC
Non-constant-time tag comparisonTiming side channelLibrary constant-time compare
Signing without context (bare payloads)Signature reused in another protocol or fieldSign structured data with a purpose/domain separator
mTLS without authorisationAny workload with a cert can call anythingCheck peer identity against policy
Long-lived certificates with no expiry monitoringSurprise fleet-wide outagesShort lifetimes, automation, alerting on every cert in the chain
Keys in source or imagesCredential leaks via repos and registriesKMS / secrets manager, scanning in CI
Rolling your own primitive or protocolSubtle, catastrophic flawsStandard constructions in well-reviewed libraries (libsodium, Go's crypto, BoringSSL, Tink)

Decision Guidance

NeedUse
Encrypt data in transit between servicesTLS 1.3 (mTLS internally), via a mesh or standard libraries
Encrypt a field or blob at restAEAD (AES-256-GCM or XChaCha20-Poly1305) with envelope encryption and associated data
Prove a message came from a party sharing a secretHMAC-SHA256 with timestamp and constant-time verify
Let many services verify tokens without sharing a secretSignatures (Ed25519 / ECDSA P-256 / RSA-PSS) with published keys and kid
Content integrity for artifactsSHA-256 digest delivered over a trusted path, plus a signature
Service identityPrivate PKI with SPIFFE IDs and short-lived certificates
Public website identityWebPKI certificates via ACME, monitored through CT
Developer or release signing among a communityOpenPGP or Sigstore
Store passwordsArgon2id (not encryption, not a fast hash)

Key Takeaways

  1. Know which goal you need — confidentiality, integrity, authentication, non-repudiation — and pick the primitive that provides it.
  2. Hashes give integrity only with a trusted digest; MACs add authentication between key holders; signatures let anyone verify and only one party sign.
  3. Use AEAD (AES-GCM, ChaCha20-Poly1305) for encryption, bind context with associated data, and never reuse a nonce under the same key.
  4. Key exchange (ECDHE) gives shared secrets and forward secrecy, but must be authenticated with signatures and certificates.
  5. PKI binds keys to names through chains of signatures; revocation is weak, so the ecosystem relies on short lifetimes and certificate transparency.
  6. mTLS gives services cryptographic identity; you still need authorisation on that identity.
  7. The web of trust decentralises vouching but scales poorly; modern systems combine private hierarchies, pinning, federation and transparency logs.
  8. Most failures are key management failures: automate rotation with overlap, use key IDs, and keep master keys in KMS/HSMs.

Interview Questions

1. What's the difference between a hash, a MAC and a signature?

A hash is keyless: anyone can compute it, so it only proves integrity if the digest arrives via a trusted path. A MAC uses a shared secret: it proves integrity and that the sender holds the key, but any key holder could have produced it (no non-repudiation). A signature uses a private key to sign and a public key to verify: anyone can verify, only the private-key holder can sign, giving non-repudiation and verification without shared secrets.

2. Why is AES-GCM preferred over AES-CBC, and what's its main operational hazard?

GCM is an AEAD: it provides confidentiality and integrity together, so tampered ciphertexts are rejected rather than decrypted into attacker-influenced plaintext (CBC without a MAC enabled padding-oracle attacks). It's also parallelisable and hardware-accelerated. Its hazard is nonce reuse: repeating a nonce under one key leaks plaintext relationships and the authentication key, enabling forgeries. Manage nonce uniqueness, limit per-key volume, or use XChaCha20-Poly1305 / AES-GCM-SIV.

3. How does Diffie–Hellman provide a shared secret over a public channel, and why does TLS still need certificates?

Each side publishes g^a and g^b; both compute g^ab, which an eavesdropper can't derive from the public values. But DH alone doesn't authenticate: an active attacker can perform separate exchanges with each side. TLS authenticates the exchange by having the server sign the handshake with the private key matching its certificate, and the certificate binds that key to the server's name.

4. What is forward secrecy and how is it achieved?

Forward secrecy means that compromising a long-term key later doesn't expose past session contents. It's achieved with ephemeral key exchange (DHE/ECDHE): each session uses fresh key pairs that are discarded afterwards; the long-term key only signs the handshake. TLS 1.3 requires ephemeral key exchange.

5. Why doesn't certificate revocation work well, and what replaced it?

CRLs are large and cached, so they're slow to propagate. OCSP adds latency, leaks browsing to CAs and is unreliable, so clients soft-fail, letting an attacker who blocks OCSP use a revoked certificate. Stapling helps only if required. The ecosystem moved to short certificate lifetimes, browser-pushed revocation sets (CRLSets, CRLite) and Certificate Transparency to detect mis-issuance.

6. What does Certificate Transparency guarantee?

That every publicly trusted certificate is recorded in public, append-only, Merkle-tree logs (browsers require SCTs proving submission). It doesn't prevent mis-issuance, but makes it detectable by domain owners and monitors, holding CAs accountable. The Merkle structure lets anyone check inclusion and that logs haven't been rewritten.

7. Design service-to-service authentication for 500 microservices.

Use mTLS with a private PKI: offline root, per-cluster intermediates, workload identities as SPIFFE IDs in URI SANs, issued automatically (SPIRE or a service mesh) with lifetimes of hours to a day and rotation at roughly half-life. Authorise on identity with policy (which service may call which method), not just "has a valid cert". Monitor expiry of every certificate in the chain, rotate trust bundles with overlapping CAs, and keep CA keys in an HSM/KMS. For end-user context, propagate signed tokens alongside mTLS.

8. Compare hierarchical PKI and the PGP web of trust.

Hierarchical PKI anchors trust in a few roots chosen by vendors; it scales to the whole web and is invisible to users, but a compromised CA can issue for any name (mitigated by CT). The web of trust lets peers vouch for each other's keys and has no central authority, but trust paths are sparse, usability is poor, revocation is unreliable, and open keyservers proved vulnerable to flooding. Modern systems use private hierarchies, pinning, federation and transparency logs instead.


Further Reading

  • Jean-Philippe Aumasson, Serious Cryptography (2nd ed., 2024)
  • Dan Boneh and Victor Shoup, A Graduate Course in Applied Cryptography (free online)
  • RFC 2104 (HMAC), RFC 5869 (HKDF), RFC 8439 (ChaCha20-Poly1305), RFC 7748 (X25519), RFC 8032 (Ed25519), RFC 8446 (TLS 1.3), RFC 5280 (X.509 PKI profile), RFC 6962 (Certificate Transparency), RFC 9580 (OpenPGP)
  • NIST SP 800-38D (GCM), SP 800-57 (key management and security strengths), FIPS 203/204 (ML-KEM, ML-DSA)
  • SPIFFE specifications and SPIRE documentation (spiffe.io)
  • Related pages: TLS Handshake, Encryption in Transit, Key Management, Envelope Encryption, Zero Trust, Service Mesh, JWT Deep Dive