On this page

Timeouts, Retries, Backoff & Hedging

Every remote call can fail in three ways: it can return an error, it can return the right answer too late, or it can never return at all. Timeouts turn the third case into the first. Retries turn some errors into successes. Backoff and jitter keep retries from turning a small failure into a large one. Hedging attacks the second case — the slow-but-successful response — which in large systems is often the dominant problem.

These four tools look simple and are routinely misconfigured. A missing timeout leaks a thread per stuck call until the process dies. A retry at every layer of a five-layer stack multiplies load by hundreds during an outage. Backoff without jitter synchronises clients into waves. Hedging without a budget doubles load exactly when the backend is weakest.

This page covers how to set each of them from data, how they interact, and how to stop them from amplifying failures. It complements Circuit Breaker (which stops calling a broken dependency), Rate Limiting (which caps how fast callers can go), Backpressure Patterns (which push load signals upstream), and Stability Patterns / Stability Antipatterns (which catalogue the broader family).


Why This Matters

The cost of waiting

A call with no timeout holds resources — a thread, a connection-pool slot, a file descriptor, request-scoped memory — until the peer responds or the TCP stack gives up, which with default kernel settings can take many minutes for an unresponsive peer. One slow dependency then drains the caller's pool, and the caller becomes slow for everything, including requests that never touch the slow dependency. That is the "blocked threads" and "slow responses" antipattern in Stability Antipatterns.

The cost of retrying

Retries are a bet that the failure was transient. When the bet is right (a dropped packet, a rolling deploy, a single overloaded replica), retries hide failures from users cheaply. When the bet is wrong (the dependency is overloaded), retries add load to the thing that is already failing from too much load. The difference between those two cases is the difference between a resilient system and a self-inflicted outage.

History in one paragraph

Exponential backoff dates back to Ethernet's collision handling (binary exponential backoff in CSMA/CD) and TCP's retransmission timer, where Van Jacobson's 1988 congestion-control work made backoff central to Internet stability. In service architectures, Marc Brooker's 2015 AWS Architecture Blog post "Exponential Backoff And Jitter" popularised jittered variants with simulation results; the Amazon Builders' Library article "Timeouts, retries, and backoff with jitter" and Google's Site Reliability Engineering book (chapters on handling overload and cascading failures) codified retry budgets and deadline propagation; and Jeff Dean and Luiz André Barroso's 2013 "The Tail at Scale" (CACM) introduced hedged and tied requests to a wide audience.


Timeouts

What a timeout actually bounds

A "timeout" is several different timers, and you usually need more than one:

TimerBoundsTypical value
Connect timeoutTCP (and TLS) handshakeLow: a few × the network RTT, e.g. 100 ms–1 s inside a datacenter
Request / response timeoutTime from sending the request to receiving the full responseDerived from downstream latency percentiles
Idle / read timeoutGap between bytes on a streaming responseSeconds; protects against stalled streams
Overall deadlineThe whole operation, including retriesSet by the caller's own budget (user-facing SLA)
Pool acquisition timeoutWaiting for a free connection from the poolShort; a long wait here is the overload signal

A common mistake is to configure a request timeout but no overall deadline, so three retries of a 2 s timeout plus backoff silently turn into an 8-second user wait.

Choosing a timeout from latency percentiles

The Amazon Builders' Library advice is a good starting rule: pick an acceptable rate of false timeouts — calls that would have succeeded if you'd waited — and set the timeout at the corresponding latency percentile of the downstream, measured from the client's side (so it includes network and queueing).

Worked example. Your client-side measurements of GET /inventory/{sku} over a week:

PercentileLatency
p508 ms
p9015 ms
p9945 ms
p99.9140 ms
p99.99900 ms
max12 s
  • Timeout at 150 ms (just above p99.9): about 0.1% of calls that would eventually have succeeded are abandoned. A retry will usually land on a faster path, so effective success is high.
  • Timeout at 50 ms (≈ p99): 1% false timeouts — every one of those becomes a retry, adding ~1% load and noise.
  • Timeout at 10 s: false timeouts are near zero, but a stuck backend holds your resources for 10 s each. With 200 req/s, a full outage puts 200 × 10 = 2,000 requests in flight at once (Little's Law: L = λW — see Queueing Theory). That will exhaust most connection pools.

So the timeout is a trade-off between wasted successful work (too short) and resource exhaustion during failure (too long). The latency distribution tells you where the knee is.

xychart-beta
    title "False-timeout rate vs timeout value (example distribution)"
    x-axis "Timeout (ms)" [10, 20, 50, 100, 150, 300, 1000]
    y-axis "Percent of calls abandoned" 0 --> 60
    line [45, 8, 0.9, 0.3, 0.1, 0.05, 0.01]

Some practical refinements:

  • Watch the tail over time, not a single snapshot. Latency distributions shift with deploys, traffic mix and data growth. Revisit timeouts when SLOs or dependencies change, and alert on timeout rate.
  • Different operations deserve different timeouts. A cached point read and a report export should not share a 30 s default.
  • Beware defaults. Many HTTP clients have no request timeout by default, or a very long one. Envoy's default route timeout is 15 s. Make every timeout an explicit decision.
  • Timeouts should shrink as you go down the stack. If the edge gives a request 2 s, a service three hops down cannot sensibly use a 5 s timeout.

Deadlines and deadline propagation

A timeout is relative ("give up after 200 ms"). A deadline is absolute ("give up at 12:00:00.200"). Deadlines compose across hops; timeouts don't.

Consider a user request with a 1 s budget passing through three services. With independent timeouts, each hop might wait its full timeout, and the bottom service might keep working long after the user has given up. With deadline propagation, every hop knows how much time is left and stops when it's gone.

sequenceDiagram
    participant U as User
    participant G as Gateway
    participant O as Orders
    participant I as Inventory
    U->>G: request, budget 1000 ms
    Note over G: deadline = now + 1000 ms
    G->>O: call, remaining 980 ms
    Note over O: 30 ms of local work
    O->>I: call, remaining 940 ms
    Note over I: checks remaining before expensive query
    I-->>O: result after 120 ms
    O-->>G: result
    G-->>U: response at about 200 ms

Rules for propagation:

  1. Carry the remaining time, not the absolute timestamp, across machines — or carry an absolute deadline only if clocks are well synchronised. gRPC sends a relative timeout in the grpc-timeout header and each server reconstructs a local deadline, which avoids clock-skew problems (see Clock Synchronization).
  2. Subtract a little for the return trip and for local work after the downstream call.
  3. Check the deadline before starting expensive work. If 3 ms remain and your query takes 50 ms, fail immediately.
  4. Propagate cancellation. When the caller gives up, downstream work should stop — Go's context.Context, gRPC cancellation, and AbortSignal in JavaScript all support this.
  5. Don't let a retry exceed the deadline. The retry loop must check remaining time before each attempt and before each backoff sleep.

RPC Fundamentals and gRPC Deep Dive cover the gRPC mechanics.

Go makes propagation natural, because the deadline travels in the context:

package orders

import (
	"context"
	"errors"
	"net/http"
	"strconv"
	"time"
)

const deadlineHeader = "X-Request-Deadline-Ms" // remaining budget, relative

// Middleware: rebuild a local deadline from the caller's remaining budget.
func WithPropagatedDeadline(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		ctx := r.Context()
		if ms, err := strconv.Atoi(r.Header.Get(deadlineHeader)); err == nil && ms > 0 {
			var cancel context.CancelFunc
			ctx, cancel = context.WithTimeout(ctx, time.Duration(ms)*time.Millisecond)
			defer cancel()
		}
		next.ServeHTTP(w, r.WithContext(ctx))
	})
}

// Outbound call: pass the remaining budget minus a safety reserve.
func CallInventory(ctx context.Context, client *http.Client, url string) (*http.Response, error) {
	const reserve = 10 * time.Millisecond
	if dl, ok := ctx.Deadline(); ok {
		remaining := time.Until(dl) - reserve
		if remaining <= 0 {
			return nil, context.DeadlineExceeded // don't start work we can't finish
		}
		req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
		if err != nil {
			return nil, err
		}
		req.Header.Set(deadlineHeader, strconv.FormatInt(remaining.Milliseconds(), 10))
		return client.Do(req)
	}
	return nil, errors.New("refusing to call downstream without a deadline")
}

Retries

What is safe to retry

A retry repeats an operation whose outcome you don't know. Whether that's safe depends on two questions: did it already happen? and does doing it twice matter?

Failure observedDid the server act?Safe to retry?
Connection refused / DNS failureNoYes
Connect timeoutNo (request never sent)Yes
TLS handshake failureNoUsually yes (but a cert problem won't fix itself)
HTTP 503 with Retry-After, gRPC UNAVAILABLEUsually noYes, with backoff, honouring Retry-After
HTTP 429 / gRPC RESOURCE_EXHAUSTEDNoYes, but slowly — you are being told to back off
Request timeout after sendingUnknownOnly if idempotent
HTTP 500UnknownOnly if idempotent, and sparingly
HTTP 400 / 401 / 403 / 404 / 409 / 422Server rejected itNo — retrying won't change the answer
Response received, client crashed before processingYesOnly if idempotent

The dangerous row is the timeout after the request was sent: the server may have completed the work and the response was lost. Retrying a non-idempotent operation (charge a card, append to a list, send an email) then performs it twice.

Two ways to make the dangerous row safe:

  1. Idempotent operations by design. PUT with a full representation, DELETE by ID, "set balance to X if version = 7". HTTP's method semantics define which methods are idempotent — see Idempotency.
  2. Idempotency keys. The client generates a key per logical operation (not per attempt) and sends it with every retry; the server stores the result keyed by it and replays the stored result on duplicates. See Idempotency and, for messaging, Exactly-Once Semantics.

Generate the idempotency key once, outside the retry loop

A key generated inside the loop is a new key for every attempt, which makes the deduplication useless. The key identifies the user's intent ("pay invoice 881"), not the HTTP request.

Retry where it helps: one layer, not every layer

Retrying is most effective close to the failure and least harmful when it happens in exactly one place in the call path. The Amazon Builders' Library article recommends retrying at a single point in the stack. Here is why.

Retry amplification across layers

Suppose each layer makes up to a attempts (1 original + a − 1 retries) and there are L layers between the user and a failing database. If the database fails every request, the worst-case number of database calls per user request is:

Attempts per layer (a)Layers (L)Calls hitting the bottom per user request
238
3327
3481
35243
451,024
flowchart LR
    U["User — 1 request"] --> GW["Gateway: 3 attempts"]
    GW -->|"3"| API["API: 3 attempts each"]
    API -->|"9"| SVC["Service: 3 attempts each"]
    SVC -->|"27"| DB[("Database (overloaded)")]
    style DB fill:#c92a2a,color:#fff

And it gets worse with timeouts: if the top layer's timeout is shorter than the time the lower layers spend retrying, the top layer gives up and retries while the lower layers are still retrying the first attempt. The work the user abandoned keeps running.

Mitigations, in order of effectiveness:

  1. Retry at one layer only — typically the layer closest to the user that can still make a meaningful decision, or the client library of the failing dependency. Other layers fail fast.
  2. Propagate deadlines so abandoned work stops.
  3. Use retry budgets so even misconfigured layers can't multiply load beyond a fixed fraction.
  4. Signal "don't retry". A lower layer that has already retried can return a status meaning "overloaded, do not retry" (the SRE book suggests exactly this), and the layers above respect it.

Retry budgets

A per-request limit ("at most 3 attempts") bounds the work per request but not the total extra load: if every request fails, a 3-attempt policy triples load. A retry budget bounds the total, usually as a ratio of retries to original requests over a sliding window.

  • Google's SRE book describes a per-client retry budget: retries are allowed only while the retry-to-request ratio is below 10%, combined with a per-request cap of three attempts.
  • Finagle (Twitter's RPC library) ships a RetryBudget that permits retries up to a percentage of recent requests plus a small minimum rate, so low-traffic clients can still retry.
  • Envoy supports a retry budget in its circuit-breaker settings, expressed as a percentage of active requests that may be retries.
  • gRPC's retry throttling keeps a token bucket per server: each failure removes a token, each success adds tokenRatio, and retries are allowed only while the bucket is above half full.

Worked example with a 10% budget:

SituationRequests/sFailure rateRetries without budget (3 attempts)Retries with 10% budget
Healthy1,0000.5%5/s (+0.5%)5/s (+0.5%)
Partial failure1,00020%~250/s (+25%)100/s (+10%)
Total outage1,000100%2,000/s (+200%, i.e. 3×)100/s (+10%)

In the healthy case the budget changes nothing. In the outage it caps your contribution to the dependency's overload at 1.1× instead of 3×. That is the entire point: retries should help with rare, independent failures and switch themselves off during correlated ones.

flowchart TD
    F["Call failed"] --> R{"Retryable error?"}
    R -->|"no"| E["Return error"]
    R -->|"yes"| I{"Idempotent or has idempotency key?"}
    I -->|"no"| E
    I -->|"yes"| B{"Budget has tokens?"}
    B -->|"no"| E
    B -->|"yes"| D{"Time left before deadline greater than backoff + expected latency?"}
    D -->|"no"| E
    D -->|"yes"| S["Sleep with jittered backoff, then retry"]

Client-side adaptive throttling

A related technique from the SRE book's Handling Overload chapter: each client tracks, over a window of a couple of minutes, how many requests it sent and how many the backend accepted. When the backend starts rejecting, the client begins rejecting locally with probability:

with K typically 2. When the backend accepts everything, requests ≈ accepts and nothing is rejected locally. When it accepts only a fraction, the client sheds the excess itself, so rejected requests don't even cost the backend the CPU of rejecting them.


Exponential Backoff and Jitter

Why back off at all

If a dependency is failing because it's overloaded, the most useful thing a client can do is go away for a while. If it's failing for some other transient reason, a short pause is enough. Exponential backoff covers both cases: start with a short delay and double it on each failure, up to a cap.

With base = 100 ms and cap = 10 s: 100, 200, 400, 800 ms, 1.6, 3.2, 6.4, 10, 10 s …

Why exponential backoff alone isn't enough

If 1,000 clients fail at the same moment (a dependency restarts, a network blip ends), pure exponential backoff makes them retry at the same moments: all at +100 ms, all at +300 ms, all at +700 ms. Each wave is a synchronised spike. The server sees 1,000 requests in a millisecond, then nothing, then 1,000 again. Backoff spread the waves out in time but not the clients within each wave.

Jitter adds randomness so that clients spread out within each window.

The three jitter strategies

These are the formulations from Brooker's AWS post (attempt counts from 0, random(a, b) is uniform):

StrategyFormulaCharacter
Full jittersleep = random(0, min(cap, base × 2^attempt))Maximum spread; some sleeps are very short
Equal jittert = min(cap, base × 2^attempt), sleep = t/2 + random(0, t/2)Guarantees at least half the backoff; less spread
Decorrelated jittersleep = min(cap, random(base, previous_sleep × 3))Each sleep depends on the last, not on the attempt count
xychart-beta
    title "Backoff window per attempt (base 100 ms, cap 10 s)"
    x-axis "Attempt" [0, 1, 2, 3, 4, 5, 6, 7]
    y-axis "Max sleep (ms)" 0 --> 10000
    bar [100, 200, 400, 800, 1600, 3200, 6400, 10000]

With full jitter the actual sleep is uniform under each bar; with equal jitter it's uniform in the top half of each bar.

Simulation intuition

You don't have to take the comparison on faith. The script below models 100 clients that all fail at the same instant against a server that can accept only 5 requests per 10 ms slot; excess requests fail and the client backs off. It counts total calls (load on the server) and the time until every client has succeeded.

# jitter_sim.py — python jitter_sim.py
import random, statistics

BASE, CAP = 10, 10_000      # ms
SLOT, CAPACITY = 10, 5      # server accepts 5 requests per 10 ms slot; extras fail

def no_jitter(attempt, prev):    return min(CAP, BASE * 2 ** attempt)
def full(attempt, prev):         return random.uniform(0, min(CAP, BASE * 2 ** attempt))
def equal(attempt, prev):
    t = min(CAP, BASE * 2 ** attempt); return t / 2 + random.uniform(0, t / 2)
def decorrelated(attempt, prev): return min(CAP, random.uniform(BASE, prev * 3))

def simulate(policy, clients=100):
    # every client fires at t=0 (e.g. after a shared dependency recovers)
    pending = [(0.0, i, 0, BASE) for i in range(clients)]  # (time, id, attempt, prev_sleep)
    calls, done_at = 0, 0.0
    while pending:
        pending.sort()
        slot = int(pending[0][0] // SLOT)
        batch = [p for p in pending if int(p[0] // SLOT) == slot]
        pending = [p for p in pending if int(p[0] // SLOT) != slot]
        calls += len(batch)
        random.shuffle(batch)
        for k, (t, cid, attempt, prev) in enumerate(batch):
            if k < CAPACITY:
                done_at = max(done_at, t)
            else:
                s = policy(attempt, prev)
                pending.append((t + s, cid, attempt + 1, s))
    return calls, done_at

if __name__ == "__main__":
    random.seed(7)
    for name, pol in [("no jitter", no_jitter), ("full jitter", full),
                      ("equal jitter", equal), ("decorrelated", decorrelated)]:
        runs = [simulate(pol) for _ in range(200)]
        print(f"{name:13} calls={statistics.mean(r[0] for r in runs):7.0f} "
              f"time_to_drain_ms={statistics.mean(r[1] for r in runs):8.0f}")

One run (seed 7, mean of 200 trials each):

StrategyTotal calls (server load)Time until all 100 succeed
No jitter~1,050~100 s
Full jitter~441~335 ms
Equal jitter~468~509 ms
Decorrelated jitter~387~337 ms

The exact numbers depend on the model, but the shape matches Brooker's published results: un-jittered backoff is dramatically worse, because synchronised clients keep colliding until the backoff hits its cap; full and decorrelated jitter are close, and both beat equal jitter. The intuition: the server's capacity is a rate, and jitter converts a spike into a rate. Full jitter is the usual default because it's simple, stateless and nearly the best.

Jitter everything periodic

The same synchronisation problem affects cron jobs, cache TTLs, heartbeat intervals, token refreshes and health checks. If many machines do something "every 60 s", add jitter so they don't all do it at :00.

Honour server hints

When a server returns Retry-After (HTTP 429/503) or a gRPC pushback, use it as a floor for the next sleep. The server knows more about its state than the client's exponential guess does.


Hedged and Tied Requests

The tail-at-scale problem

Retries handle failures. Hedging handles slowness. In a system that fans out, slowness compounds:

Fan-out NRequests slower than a single server's p99
11%
109.6%
10063.4%
1,000~100%

This is the central observation of The Tail at Scale: if each leaf server is slow 1% of the time and a request touches 100 leaves, almost two-thirds of user requests are slow. Rare per-server hiccups — GC, compaction, a noisy neighbour, a cache miss — become the common case at the top.

Hedged requests

Send the request to one replica. If it hasn't answered within some delay — typically around the p95 of expected latency — send the same request to a second replica, and use whichever answer comes first (cancelling the other).

sequenceDiagram
    participant C as Client
    participant R1 as Replica 1
    participant R2 as Replica 2
    C->>R1: GET key
    Note over C: wait hedge delay, about p95
    Note over R1: stuck behind a GC pause
    C->>R2: GET key (hedge)
    R2-->>C: value
    Note over C: use first response, cancel the other
    C-->>R1: cancel

The cost is bounded: if you hedge at the p95, at most ~5% of requests get a second copy. The benefit can be large. The paper reports a Google BigTable benchmark where reading 1,000 keys spread across 100 servers and hedging after a 10 ms delay cut the 99.9th-percentile latency from 1,800 ms to 74 ms, while sending only about 2% more requests.

When hedging works:

  • Reads, or idempotent writes. You will sometimes do the work twice.
  • Variance comes from the server side (GC, queueing, disk). If the request is slow because it's expensive (a huge scan), a second copy will be just as slow and you've doubled the cost.
  • Replicas are independent. Hedging to a replica on the same overloaded host doesn't help.

When to disable it: when the backend is overloaded. Hedges are extra load; they must be covered by the same kind of budget as retries (gRPC's hedging policy shares the retry throttling mechanism).

Tied requests

Hedging still waits a delay before sending the second copy. Tied requests send both copies immediately, to two servers' queues, each tagged with the identity of the other. When one server starts executing the request, it tells the other to cancel its copy. Most of the time the duplicate is cancelled while still queued, so the extra work is tiny, and the request runs on whichever server got to it first. The paper notes a small random delay between the two enqueues helps avoid both servers starting simultaneously when both are idle.

Tied requests need server cooperation (cross-server cancellation), so they're mostly found inside storage and search systems rather than in general-purpose RPC.

TechniqueIdea
Micro-partitioningMany more partitions than machines, so load can be rebalanced finely
Selective replicationExtra replicas for hot items
Latency-induced probationTemporarily stop sending to a replica that's slow, keep probing it
Good-enough responsesIn search-like fan-out, return results from the leaves that answered by the deadline
Canary requestsSend a risky query to one or two leaves first, before fanning out to thousands

Load Shedding and Bulkheads

Timeouts, retries and hedges are client-side tools. Their server-side counterparts decide what to do when more work arrives than can be done.

Load shedding

When a server is saturated, every queued request gets slower; past a point, requests time out at the client after the server has spent effort on them — pure waste. Load shedding rejects excess work early and cheaply, so the work that is accepted completes within its deadline. The Amazon Builders' Library article "Using load shedding to avoid overload" argues for exactly this: an overloaded server should maintain its useful throughput ("goodput") by refusing work rather than collapsing.

Effective shedding policies:

  • Concurrency limits. Cap in-flight requests; reject above the limit. Adaptive limiters (Netflix's open-source concurrency-limits library, for example) adjust the limit from observed latency, in the spirit of TCP congestion control.
  • Queue-time based shedding. If a request has already waited in the queue longer than its client's deadline (or longer than some target), drop it — nobody is waiting for the answer. Facebook's "Fail at Scale" (ACM Queue, 2015) describes using a CoDel-style controlled-delay queue and switching to adaptive LIFO under load, so fresh requests (whose clients are still waiting) are served before stale ones.
  • Priority / criticality. Shed batch and prefetch traffic before user-facing traffic. The SRE book describes tagging requests with criticality and propagating it.
  • Cheap rejection. The rejection path must cost far less than serving — return before parsing large bodies or touching the database.

See Backpressure Patterns for queue-side shedding and Rate Limiting for per-client caps.

Bulkheads

A ship's bulkheads stop one flooded compartment from sinking the vessel. In software, a bulkhead gives each dependency (or tenant, or traffic class) its own bounded pool of resources, so one slow dependency can exhaust only its own pool.

flowchart LR
    subgraph Service["Checkout service"]
        P1["Pool: payments (20)"]
        P2["Pool: recommendations (5)"]
        P3["Pool: inventory (20)"]
    end
    P1 --> Pay["Payments API"]
    P2 --> Rec["Recommendations (slow!)"]
    P3 --> Inv["Inventory"]
    style Rec fill:#c92a2a,color:#fff
    style P2 fill:#e8590c,color:#fff

When recommendations hang, only its 5 slots fill up; checkout keeps working and simply shows no recommendations. Without bulkheads, those calls would share one pool of 45 and eventually block payments too.

Sizing a bulkhead uses Little's Law: slots = target throughput × expected latency. For 100 req/s at 50 ms p99, about 5 concurrent slots carry the load; add headroom, and set the pool acquisition timeout short so that a full pool fails fast rather than queueing. Stability Patterns shows thread-pool and semaphore variants.

Circuit breakers

A circuit breaker sits one level above all of this: after enough failures it stops calling the dependency entirely for a cool-down period, then probes. It pairs naturally with retry budgets — the budget limits retry amplification, the breaker limits all traffic to a dependency that is clearly down. The full state machine and implementation are in Circuit Breaker.

How the pieces fit

ToolProtectsAgainstSide
Timeout / deadlineCaller's resourcesSlow or hung calleeClient
Retry (+ idempotency)Success rateTransient, independent failuresClient
Backoff + jitterCalleeSynchronised retry wavesClient
Retry budget / adaptive throttlingCalleeRetry amplification during correlated failureClient
Hedging / tied requestsTail latencySlow replicasClient (+ server for tied)
Circuit breakerBothCalling a dependency that's clearly downClient
Load sheddingCallee's goodputMore work than capacityServer
BulkheadCaller's other featuresOne dependency exhausting shared resourcesClient/server

Implementation: A Resilient Client (TypeScript)

The client below combines the ideas: an overall deadline, per-attempt timeouts that never exceed the remaining budget, retries only for safe cases, full-jitter backoff, Retry-After support, a token-bucket retry budget, a stable idempotency key, and optional hedging for idempotent reads. It uses only the platform fetch and AbortSignal APIs (Node 20+ or modern browsers).

// resilient-client.ts
type Method = "GET" | "HEAD" | "PUT" | "DELETE" | "POST" | "PATCH";

export interface CallOptions {
  method?: Method;
  body?: unknown;
  deadlineMs: number;          // overall budget, including retries
  attemptTimeoutMs: number;    // per-attempt cap (from downstream p99.9)
  maxAttempts?: number;        // default 3
  idempotencyKey?: string;     // required to retry POST/PATCH
  hedgeAfterMs?: number;       // enable hedging for idempotent calls (≈ p95)
}

/** Retry budget: retries may consume at most `ratio` tokens per request. */
export class RetryBudget {
  private tokens: number;
  constructor(private readonly max = 10, private readonly ratio = 0.1) {
    this.tokens = max;
  }
  onRequest() { this.tokens = Math.min(this.max, this.tokens + this.ratio); }
  tryRetry(): boolean {
    if (this.tokens < 1) return false;
    this.tokens -= 1;
    return true;
  }
}

const IDEMPOTENT: ReadonlySet<Method> = new Set(["GET", "HEAD", "PUT", "DELETE"]);
const RETRYABLE_STATUS = new Set([429, 502, 503, 504]);

export function fullJitter(attempt: number, baseMs = 50, capMs = 2_000): number {
  return Math.random() * Math.min(capMs, baseMs * 2 ** attempt);
}

function sleep(ms: number, signal: AbortSignal): Promise<void> {
  return new Promise((resolve, reject) => {
    if (signal.aborted) return reject(signal.reason);
    const t = setTimeout(resolve, ms);
    signal.addEventListener("abort", () => { clearTimeout(t); reject(signal.reason); }, { once: true });
  });
}

function retryAfterMs(res: Response): number | undefined {
  const h = res.headers.get("retry-after");
  if (!h) return undefined;
  const secs = Number(h);
  if (!Number.isNaN(secs)) return secs * 1000;
  const date = Date.parse(h);
  return Number.isNaN(date) ? undefined : Math.max(0, date - Date.now());
}

export class ResilientClient {
  constructor(private readonly budget = new RetryBudget()) {}

  async call(url: string, opts: CallOptions): Promise<Response> {
    const method = opts.method ?? "GET";
    const maxAttempts = opts.maxAttempts ?? 3;
    const canRetry = IDEMPOTENT.has(method) || opts.idempotencyKey !== undefined;
    const deadline = AbortSignal.timeout(opts.deadlineMs);
    const startedAt = Date.now();
    const remaining = () => opts.deadlineMs - (Date.now() - startedAt);

    this.budget.onRequest();
    let lastError: unknown;

    for (let attempt = 0; attempt < maxAttempts; attempt++) {
      const attemptBudget = Math.min(opts.attemptTimeoutMs, remaining());
      if (attemptBudget <= 0) break;

      try {
        const res = await this.attempt(url, method, opts, attemptBudget, deadline);
        if (!RETRYABLE_STATUS.has(res.status)) return res; // success, or non-retryable error
        lastError = new Error(`HTTP ${res.status}`);
        if (!canRetry || attempt === maxAttempts - 1 || !this.budget.tryRetry()) return res;

        const wait = Math.max(fullJitter(attempt), retryAfterMs(res) ?? 0);
        if (wait >= remaining()) return res;              // can't afford to wait
        await sleep(wait, deadline);
      } catch (err) {
        lastError = err;
        if (deadline.aborted) break;                      // overall deadline hit
        // Network error or per-attempt timeout: outcome unknown for unsafe methods.
        if (!canRetry || attempt === maxAttempts - 1 || !this.budget.tryRetry()) break;
        const wait = fullJitter(attempt);
        if (wait >= remaining()) break;
        await sleep(wait, deadline).catch(() => {});
      }
    }
    throw lastError ?? new Error("deadline exceeded");
  }

  /** One logical attempt. With hedging, may issue a second copy after hedgeAfterMs. */
  private async attempt(
    url: string, method: Method, opts: CallOptions, timeoutMs: number, deadline: AbortSignal,
  ): Promise<Response> {
    const headers: Record<string, string> = {
      "content-type": "application/json",
      "x-request-deadline-ms": String(Math.floor(timeoutMs)), // propagate remaining budget
    };
    if (opts.idempotencyKey) headers["idempotency-key"] = opts.idempotencyKey;
    const body = opts.body === undefined ? undefined : JSON.stringify(opts.body);

    const send = (signal: AbortSignal) => fetch(url, { method, headers, body, signal });

    const hedge = opts.hedgeAfterMs !== undefined && IDEMPOTENT.has(method);
    if (!hedge) {
      return send(AbortSignal.any([deadline, AbortSignal.timeout(timeoutMs)]));
    }

    // Hedged: primary now, backup after hedgeAfterMs; first success wins, loser is cancelled.
    const controllers = [new AbortController(), new AbortController()];
    const signals = controllers.map((c) =>
      AbortSignal.any([deadline, AbortSignal.timeout(timeoutMs), c.signal]));
    const primary = send(signals[0]);
    const backup = new Promise<Response>((resolve, reject) => {
      const t = setTimeout(() => {
        if (!this.budget.tryRetry()) return reject(new Error("hedge budget exhausted"));
        send(signals[1]).then(resolve, reject);
      }, opts.hedgeAfterMs);
      signals[0].addEventListener("abort", () => clearTimeout(t), { once: true });
    });
    try {
      const winner = await Promise.any([primary, backup]);
      controllers.forEach((c) => c.abort());              // cancel the loser
      return winner;
    } catch (e) {
      throw e instanceof AggregateError ? e.errors[0] : e;
    }
  }
}

// --- Usage ---
// const client = new ResilientClient();
// const res = await client.call("https://inventory.internal/sku/123", {
//   deadlineMs: 800, attemptTimeoutMs: 150, hedgeAfterMs: 40,
// });
// await client.call("https://payments.internal/charges", {
//   method: "POST", body: { invoice: 881, cents: 4200 },
//   idempotencyKey: "invoice-881-charge",   // generated ONCE per logical operation
//   deadlineMs: 3000, attemptTimeoutMs: 1000,
// });

Design notes worth keeping when you adapt it:

  • The idempotency key comes from the caller, per logical operation, so retries reuse it. POST without a key is never retried.
  • Every attempt's timeout is clamped to the remaining overall budget, and the loop refuses to sleep past the deadline.
  • Hedges draw from the same budget as retries, so a struggling backend automatically stops getting duplicates.
  • 4xx responses other than 429 are returned immediately — the server has answered.
  • In production you'd also record metrics per outcome (success, retried-success, budget-exhausted, deadline-exceeded) and integrate with a circuit breaker.

Failure Modes and War Stories

Retry storms

A retry storm is a positive feedback loop: the dependency slows → callers time out → callers retry → the dependency gets more load → it slows further. Once started, it can persist even after the original trigger is gone, because the retry traffic alone exceeds capacity. Breaking it usually requires shedding load at the dependency or turning traffic off and ramping it back up.

flowchart LR
    A["Dependency slows"] --> B["Callers time out"]
    B --> C["Callers retry"]
    C --> D["Load rises"]
    D --> A
    style A fill:#c92a2a,color:#fff

War Story — Amazon DynamoDB, US-East-1, September 2015

AWS's published post-incident summary of the September 20, 2015 DynamoDB disruption describes this pattern. DynamoDB storage servers periodically fetch their partition membership from an internal metadata service. The adoption of Global Secondary Indexes had made some membership responses much larger and slower to serve. After a brief network disruption, many storage servers requested membership at the same time; responses took longer than the storage servers' timeout, so they retried, keeping the metadata service overloaded. Storage servers that couldn't obtain membership removed themselves from service, which raised error rates for customers. Recovery required pausing requests to the metadata service so capacity could be added. The lessons — timeouts that don't fit the real latency distribution, synchronised retries, and retries against an overloaded dependency — are exactly the ones on this page. (Source: AWS, "Summary of the Amazon DynamoDB Service Disruption and Related Impacts in the US-East Region", 2015.)

Timeouts that don't nest

If an upstream timeout (1 s) is shorter than the downstream's retry schedule (3 × 500 ms), the upstream gives up and retries while the downstream is still working on the first request. Work multiplies and nobody receives the answer. Rule: an outer timeout must exceed the inner timeout × inner attempts + backoff, or the inner layer must not retry. Deadline propagation fixes this automatically.

Retrying non-idempotent operations

Retrying a POST /orders after a timeout creates duplicate orders. Retrying a Kafka produce without the idempotent producer creates duplicate messages. The fix is always the same: idempotency keys or idempotent design — never "just don't retry and hope".

Thundering herd on recovery

When a dependency comes back, every client whose circuit breaker was open, every cache that expired, and every queued retry fires at once. Jittered backoff, jittered breaker cool-downs, and gradually ramping traffic (half-open probes, slow-start in load balancers) all help.

War Story (illustrative)

A mobile app retried failed API calls immediately, three times, and also re-sent everything on reconnect. During a 4-minute backend incident, several million phones accumulated pending requests. When the backend recovered, the reconnect wave was roughly ten times normal peak and knocked it over again — twice. The fix shipped in the next app release: full-jitter exponential backoff with a 5-minute cap, a client-side retry budget, honouring Retry-After, and a server-side "slow start" that admitted a growing fraction of traffic over several minutes.


Real Numbers and Defaults

System / guidanceRelevant behaviour
AWS SDKsExponential backoff with jitter by default; "standard" and "adaptive" retry modes, the latter adding client-side rate limiting
gRPC retry policymaxAttempts (capped at 5), initialBackoff, maxBackoff, backoffMultiplier, retryableStatusCodes; backoff is randomised between 0 and the current ceiling; retry throttling via maxTokens / tokenRatio
gRPC hedging policymaxAttempts, hedgingDelay, nonFatalStatusCodes; shares retry throttling
EnvoyDefault route timeout 15 s; configurable retry policy with jittered backoff; retry budgets in circuit-breaker thresholds
Google SRE bookPer-request cap of 3 attempts; per-client retry ratio below 10%; adaptive throttling with K≈2
The Tail at ScaleHedging after a 10 ms delay: p99.9 from 1,800 ms to 74 ms with ~2% extra requests (BigTable benchmark)

An example gRPC service config combining retries and throttling:

{
  "methodConfig": [{
    "name": [{ "service": "inventory.v1.Inventory", "method": "GetItem" }],
    "timeout": "0.3s",
    "retryPolicy": {
      "maxAttempts": 3,
      "initialBackoff": "0.05s",
      "maxBackoff": "1s",
      "backoffMultiplier": 2,
      "retryableStatusCodes": ["UNAVAILABLE"]
    }
  }],
  "retryThrottling": { "maxTokens": 10, "tokenRatio": 0.1 }
}

Decision Guidance

QuestionGuidance
What timeout should I set?Client-measured p99.9 of the downstream (or the percentile matching your acceptable false-timeout rate), clamped by the remaining deadline
Should every layer retry?No. Retry in one layer; others fail fast and propagate deadlines
How many attempts?2–3 is typical; more rarely helps and increases amplification
Which backoff?Full jitter exponential, base 10–100 ms, cap 1–30 s depending on context; honour Retry-After
Do I need a retry budget?Yes, for any service-to-service traffic at scale. ~10% retry ratio is a common ceiling
Should I hedge?For latency-sensitive, idempotent reads with server-side variance and spare capacity. Hedge at ~p95, budgeted
What about writes?Retry only with idempotency keys or idempotent semantics; never hedge non-idempotent writes
Server overloaded?Shed early and cheaply, prioritise by criticality, use bounded queues and concurrency limits
One dependency slowing everything?Bulkhead it; add a circuit breaker

Key Takeaways

  1. Every remote call needs a timeout, and every request needs a deadline that is propagated and checked downstream.
  2. Pick timeouts from data — the client-observed latency distribution — balancing wasted work against resource exhaustion.
  3. Retry only what is safe: errors where the server didn't act, or idempotent operations, or operations with a stable idempotency key.
  4. Back off exponentially with jitter. Full jitter is simple and close to optimal; no jitter is dramatically worse.
  5. Retries multiply across layers (a^L). Retry in one place and cap the total with a retry budget.
  6. Hedging cuts tail latency cheaply when variance is server-side — at the cost of duplicate work that must be budgeted.
  7. Servers must protect themselves with load shedding and bulkheads; client politeness is not a guarantee.

Interview Questions

1. How would you choose the timeout for a call to a downstream service?

Measure the latency distribution from the client side (so it includes network and queueing). Decide the acceptable false-timeout rate, e.g. 0.1%, and set the timeout near the matching percentile (p99.9). Then check it against the caller's own deadline and resource limits via Little's Law: in-flight requests = rate × timeout during an outage must fit in the connection/thread pool. Revisit as the distribution changes.

2. Why is retrying at every layer dangerous? Quantify it.

Attempts multiply: with a attempts per layer and L layers, a failing bottom dependency sees up to a^L calls per user request — 3 attempts across 5 layers is 243×. It also wastes work when upper timeouts fire before lower retries finish. Retry at one layer, propagate deadlines, return "don't retry" signals from lower layers, and use retry budgets.

3. Compare full, equal and decorrelated jitter.

Full jitter: uniform in [0, min(cap, base·2^n)] — maximum spread, stateless. Equal jitter: half the backoff fixed plus a random half — guarantees some wait but spreads less. Decorrelated: random in [base, 3 × previous sleep], capped — depends on the previous sleep. In simulations (including AWS's published ones), full and decorrelated perform similarly and better than equal jitter; all are far better than no jitter, which leaves clients synchronised.

4. What is a retry budget and why isn't "max 3 attempts" enough?

A per-request cap bounds work per request but not total load: if every request fails, 3 attempts triples traffic to a dependency that's already failing. A retry budget caps retries as a fraction of total requests (e.g. 10%) over a window, so during a correlated outage extra load is bounded at 1.1×, while rare independent failures still get retried.

5. Explain hedged requests. When would they make things worse?

Send a request, and if no reply arrives within roughly the p95, send a duplicate to another replica and take the first response, cancelling the other. It bounds extra load to about 5% while cutting tail latency when slowness is caused by transient server-side issues. It makes things worse when the request is inherently expensive (both copies are slow), when the backend is overloaded (duplicates add load), or when the operation isn't idempotent.

6. A request timed out after being sent. Can you retry a POST that creates a payment?

Only if the server supports idempotency keys and the client sends the same key generated once per logical payment. The server either completed it (and replays the stored result) or didn't (and processes it now). Without that, the timeout means the outcome is unknown and a retry may charge twice; you'd need to query the payment status first.

7. How do deadlines differ from timeouts, and how should they be propagated?

A timeout is a relative duration per call; a deadline is the absolute time the whole operation must finish. Propagate the remaining budget (relative, to avoid clock skew) to each downstream call, subtract local overhead and a reserve, check it before starting expensive work, propagate cancellation when the caller gives up, and never let retries or backoff run past it.

8. What is adaptive LIFO and why would a server switch to it under load?

Under normal load, FIFO is fair. Under overload, the oldest requests in the queue are the most likely to have already timed out at the client, so serving them is wasted work. Switching to LIFO (newest first), combined with a controlled-delay policy that drops requests that have waited too long, serves requests whose clients are still waiting, preserving goodput. Facebook described this in Fail at Scale.


Further Reading

  • Marc Brooker, Exponential Backoff And Jitter, AWS Architecture Blog, 2015
  • Amazon Builders' Library: Timeouts, retries, and backoff with jitter; Using load shedding to avoid overload
  • Jeffrey Dean and Luiz André Barroso, The Tail at Scale, Communications of the ACM, 2013
  • Betsy Beyer et al. (eds.), Site Reliability Engineering, chapters 21 (Handling Overload) and 22 (Addressing Cascading Failures)
  • Ben Maurer, Fail at Scale, ACM Queue, 2015
  • gRPC proposal A6, Client Retries (retry and hedging policies)
  • Related pages: Circuit Breaker, Idempotency, Backpressure Patterns, Queueing Theory, Stability Patterns