On this page

Concurrency Patterns in LLD

Thread-Safe Class Design was about keeping one class correct under concurrent calls. This page goes one level up: the recurring shapes of concurrent code that appear inside low-level designs. When an interviewer asks you to design a logger, a task scheduler, a notification service or a pub-sub system, the "concurrency part" is almost always one of a handful of patterns, wired into the object model in a predictable place.

The theory behind most of them already lives elsewhere on Archon. Classic Synchronization Problems proves the bounded buffer and readers–writers solutions from semaphores up, and the System Design concurrency patterns page gives the short catalogue. Here the question is practical: which library class do you reach for, where does it sit in your class diagram, what can go wrong, and which LLD problems use it?

PatternCore JDK toolLLD problems that use it
Producer–consumerBlockingQueueAsync logger, notification service, pub-sub, order pipeline
Delayed workDelayQueue, ScheduledExecutorServiceTask scheduler, hold expiry, retry with back-off, reminders
Worker poolThreadPoolExecutorNotification fan-out, web crawler, image processing, any request handler
Read-write lockReentrantReadWriteLock, StampedLockConfig registry, route table, in-memory cache, feature flags
Lazy initialisationvolatile + DCL, holder idiom, enumSingletons: logger, connection pool, ID generator
Async compositionCompletableFutureCheckout (price + stock + fraud in parallel), aggregators, search fan-out
Thread-per-request / virtual threadsExecutors.newVirtualThreadPerTaskExecutor()Any blocking-I/O service: booking API, URL shortener backend
Rate-limited workertoken bucket + queueSMS/email sender, API client, crawler politeness
Graceful shutdownshutdown / awaitTermination / shutdownNowEvery component that owns threads
Pub-sub dispatchCopyOnWriteArrayList + per-subscriber executorEvent bus, chat rooms, stock ticker, notification preferences

The same caveats about runtimes apply as on the previous page. Java has parallel OS threads and, since Java 21, virtual threads. CPython 3.12 has threads that interleave under the GIL (fine for I/O-bound work, no speedup for CPU-bound Python code) and asyncio for cooperative concurrency. Node has one event-loop thread, so most of these patterns become "limit the number of in-flight promises" rather than "manage threads", and worker_threads for CPU-bound parallelism.

Producer-consumer with a BlockingQueue

The shape. Some threads produce work items; other threads consume them; a queue sits between them. The queue decouples the two sides in time (producers don't wait for consumers to finish) and in rate (a burst of producers is absorbed by the queue). It's also the most effective way to turn shared mutable state into single-owner state: if only the consumer thread touches the file, the socket or the order book, those need no locks.

flowchart LR
    P1["Request thread 1"] -->|"offer(line)"| Q[("Bounded<br/>BlockingQueue")]
    P2["Request thread 2"] -->|"offer(line)"| Q
    P3["Request thread N"] -->|"offer(line)"| Q
    Q -->|"take()"| W["Single writer thread"]
    W --> F["Log file<br/>(touched by one thread only)"]

A BlockingQueue gives you everything the hand-rolled monitor version in Classic Synchronization Problems builds by hand: put blocks when full, take blocks when empty, and the handoff creates a happens-before edge, so whatever the producer wrote before put is visible to the consumer after take. The design decisions that remain are yours:

  1. Bounded or unbounded? Always bounded in production. An unbounded queue turns a slow consumer into an OutOfMemoryError some hours later.
  2. What happens when it's full? This is the backpressure policy, and it's a product decision:
    • block (put): the producer slows down to the consumer's rate. Right for an order pipeline where losing an item is unacceptable, wrong for a logger on the request path.
    • drop (offer returns false): count the drop and move on. Right for logs and metrics, where a lost debug line beats a stalled checkout.
    • block with a timeout (offer(item, timeout)): a middle ground.
    • reject to the caller: throw, and let the caller decide (an HTTP 503).
  3. How does it stop? A poison pill, a special item that tells the consumer to finish, or an interrupt plus a closed flag. Either way, decide whether queued items are drained or discarded on shutdown.
  4. How many consumers? One consumer preserves order and confines state. Several consumers give throughput but no global ordering.

Here is an async logger, a common LLD prompt in its own right and a component of nearly every other design. It uses a drop policy with a short timeout, a poison pill and a static factory (so the writer thread isn't started from inside the constructor).

import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicLong;
import java.util.function.Consumer;

class AsyncLoggerDemo {
    public static void main(String[] args) throws Exception {
        List<String> sink = Collections.synchronizedList(new ArrayList<>()); // stands in for a file
        AsyncLogger log = AsyncLogger.start(1024, sink::add);

        ExecutorService producers = Executors.newFixedThreadPool(8);
        for (int t = 0; t < 8; t++) {
            int id = t;
            producers.submit(() -> { for (int i = 0; i < 10_000; i++) log.log("t" + id + " line " + i); });
        }
        producers.shutdown();
        if (!producers.awaitTermination(30, TimeUnit.SECONDS)) throw new AssertionError("producers hung");
        log.close();                                       // drains the queue, then stops the writer

        long written = sink.size(), dropped = log.dropped();
        if (written + dropped != 80_000) throw new AssertionError(written + " + " + dropped);
        System.out.println("written " + written + ", dropped " + dropped + ", total 80000");
    }
}

/** Thread-safe. log() may be called from any thread; only the writer thread touches the sink. */
final class AsyncLogger implements AutoCloseable {
    private static final String POISON = new String("POISON");  // compared by identity, never equal to a real line
    private final BlockingQueue<String> queue;
    private final Consumer<String> sink;
    private final AtomicLong dropped = new AtomicLong();
    private final Thread writer;
    private volatile boolean closed;

    private AsyncLogger(int capacity, Consumer<String> sink) {
        this.queue = new ArrayBlockingQueue<>(capacity);
        this.sink = sink;
        this.writer = new Thread(this::drain, "log-writer");
    }

    /** Construct fully, then start the thread: 'this' never escapes a half-built object. */
    static AsyncLogger start(int capacity, Consumer<String> sink) {
        AsyncLogger logger = new AsyncLogger(capacity, sink);
        logger.writer.start();
        return logger;
    }

    void log(String line) {
        if (closed) throw new IllegalStateException("logger closed");
        try {
            // Backpressure policy: wait briefly, then drop rather than stall the caller.
            if (!queue.offer(line, 5, TimeUnit.MILLISECONDS)) dropped.incrementAndGet();
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            dropped.incrementAndGet();
        }
    }

    long dropped() { return dropped.get(); }

    private void drain() {
        try {
            while (true) {
                String line = queue.take();
                if (line == POISON) return;                // everything queued before the pill is written
                sink.accept(line);
            }
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
    }

    @Override public void close() throws InterruptedException {
        closed = true;
        queue.put(POISON);                                 // blocks until there is room: the pill is never dropped
        writer.join();
        // Simplification: a log() racing with close() can still enqueue after the pill and be lost;
        // guard log() and close() with a ReadWriteLock if that matters.
    }
}
import queue
import threading
from typing import Callable

_POISON = object()


class AsyncLogger:
    """Thread-safe. Only the writer thread touches the sink."""

    def __init__(self, capacity: int, sink: Callable[[str], None]) -> None:
        self._queue: queue.Queue[object] = queue.Queue(maxsize=capacity)
        self._sink = sink
        self._dropped = 0
        self._dropped_lock = threading.Lock()
        self._closed = False
        self._writer = threading.Thread(target=self._drain, name="log-writer")

    @classmethod
    def start(cls, capacity: int, sink: Callable[[str], None]) -> "AsyncLogger":
        logger = cls(capacity, sink)
        logger._writer.start()
        return logger

    def log(self, line: str) -> None:
        if self._closed:
            raise RuntimeError("logger closed")
        try:
            self._queue.put(line, timeout=0.005)  # wait briefly, then drop
        except queue.Full:
            with self._dropped_lock:
                self._dropped += 1

    @property
    def dropped(self) -> int:
        return self._dropped

    def _drain(self) -> None:
        while True:
            item = self._queue.get()
            if item is _POISON:
                return
            self._sink(item)  # type: ignore[arg-type]

    def close(self) -> None:
        self._closed = True
        self._queue.put(_POISON)  # blocking put: the pill is never dropped
        self._writer.join()


if __name__ == "__main__":
    sink: list[str] = []  # list.append is atomic in CPython, and only the writer calls it anyway
    log = AsyncLogger.start(1024, sink.append)
    producers = [threading.Thread(target=lambda t=t: [log.log(f"t{t} line {i}") for i in range(10_000)])
                 for t in range(8)]
    for p in producers:
        p.start()
    for p in producers:
        p.join()
    log.close()
    assert len(sink) + log.dropped == 80_000
    print(f"written {len(sink)}, dropped {log.dropped}, total 80000")

Even with a sink this fast, eight producers occasionally outrun the single writer for a few milliseconds: five Java runs dropped between 0 and 5 lines out of 80,000, and five Python runs dropped between 30 and 42. Make the sink sleep for a millisecond and the drop counter climbs instead of the request threads stalling, which is the whole point of the policy. If losing any line is unacceptable, switch offer with a timeout to a blocking put and accept that logging can slow requests down. In Python, before writing your own, check the standard library: logging.handlers.QueueHandler and QueueListener implement exactly this pattern. In Node, logging libraries such as pino move formatting and I/O to a worker thread for the same reason.

Where it appears in LLD problems:

  • Logger: request threads produce, one writer consumes. The writer owns the file handle, so rotation needs no locks.
  • Notification service: the API enqueues Notification objects and returns immediately; channel workers consume and send.
  • Pub-sub: each subscriber gets its own queue, so a slow subscriber can't block the publisher (see below).
  • Order processing pipeline: stages connected by queues (validate → reserve → charge → ship), each stage with its own consumers.

A variant: delayed work for a task scheduler

A task scheduler ("run this job in 30 seconds", "every 5 minutes", "expire this seat hold at 10:05") is a producer–consumer system whose queue is ordered by due time instead of arrival. The consumer sleeps until the earliest item is due. In Java that queue is DelayQueue; take() blocks until the head's delay has expired. In real code you'd use ScheduledExecutorService, which is built on the same idea, but interviewers often want to see the mechanism, because "design a task scheduler" is a common prompt.

import java.util.*;
import java.util.concurrent.*;

class DelaySchedulerDemo {
    public static void main(String[] args) throws Exception {
        List<String> ran = Collections.synchronizedList(new ArrayList<>());
        TaskScheduler scheduler = TaskScheduler.start();
        scheduler.schedule("C", 300, () -> ran.add("C"));
        scheduler.schedule("A", 100, () -> ran.add("A"));
        ScheduledTask cancelled = scheduler.schedule("X", 150, () -> ran.add("X"));
        scheduler.schedule("B", 200, () -> ran.add("B"));
        cancelled.cancel();
        Thread.sleep(500);
        scheduler.stop();
        if (!ran.equals(List.of("A", "B", "C"))) throw new AssertionError(ran);
        System.out.println("ran in due-time order: " + ran);
    }
}

final class ScheduledTask implements Delayed {
    final String name;
    final long dueNanos;
    final Runnable action;
    private volatile boolean cancelled;

    ScheduledTask(String name, long delayMillis, Runnable action) {
        this.name = name;
        this.dueNanos = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(delayMillis);
        this.action = action;
    }

    void cancel() { cancelled = true; }          // lazy cancellation: skipped when it comes due
    boolean isCancelled() { return cancelled; }

    @Override public long getDelay(TimeUnit unit) {
        return unit.convert(dueNanos - System.nanoTime(), TimeUnit.NANOSECONDS);
    }

    @Override public int compareTo(Delayed other) {
        return Long.compare(dueNanos, ((ScheduledTask) other).dueNanos);
    }
}

final class TaskScheduler {
    private final DelayQueue<ScheduledTask> queue = new DelayQueue<>();
    private final ExecutorService workers = Executors.newFixedThreadPool(4); // a slow task can't delay the next one
    private final Thread dispatcher = new Thread(this::dispatch, "scheduler");

    private TaskScheduler() { }

    static TaskScheduler start() {
        TaskScheduler s = new TaskScheduler();
        s.dispatcher.start();
        return s;
    }

    ScheduledTask schedule(String name, long delayMillis, Runnable action) {
        ScheduledTask task = new ScheduledTask(name, delayMillis, action);
        queue.put(task);
        return task;
    }

    private void dispatch() {
        try {
            while (!Thread.currentThread().isInterrupted()) {
                ScheduledTask task = queue.take();             // blocks until the earliest task is due
                if (!task.isCancelled()) workers.execute(task.action);
            }
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();                // stop() interrupts us: exit quietly
        }
    }

    void stop() throws InterruptedException {
        dispatcher.interrupt();
        dispatcher.join();
        workers.shutdown();
        workers.awaitTermination(5, TimeUnit.SECONDS);
    }
}
import heapq
import itertools
import threading
import time
from typing import Callable


class TaskScheduler:
    """A heap ordered by due time, and one dispatcher thread waiting on a Condition."""

    def __init__(self) -> None:
        self._heap: list[tuple[float, int, str, Callable[[], None]]] = []
        self._seq = itertools.count()          # tie-breaker so callables are never compared
        self._cond = threading.Condition()
        self._cancelled: set[str] = set()
        self._stopped = False
        self._thread = threading.Thread(target=self._dispatch, name="scheduler")

    @classmethod
    def start(cls) -> "TaskScheduler":
        s = cls()
        s._thread.start()
        return s

    def schedule(self, name: str, delay_s: float, action: Callable[[], None]) -> None:
        with self._cond:
            heapq.heappush(self._heap, (time.monotonic() + delay_s, next(self._seq), name, action))
            self._cond.notify()                # the new task may be due sooner than the current head

    def cancel(self, name: str) -> None:
        with self._cond:
            self._cancelled.add(name)

    def _dispatch(self) -> None:
        with self._cond:
            while not self._stopped:
                if not self._heap:
                    self._cond.wait()
                    continue
                due, _, name, action = self._heap[0]
                wait = due - time.monotonic()
                if wait > 0:
                    self._cond.wait(timeout=wait)  # woken early by schedule() or stop(): re-check
                    continue
                heapq.heappop(self._heap)
                if name not in self._cancelled:
                    action()                   # Simplification: runs on the dispatcher; hand to a pool if tasks are slow

    def stop(self) -> None:
        with self._cond:
            self._stopped = True
            self._cond.notify()
        self._thread.join()


if __name__ == "__main__":
    ran: list[str] = []
    s = TaskScheduler.start()
    s.schedule("C", 0.3, lambda: ran.append("C"))
    s.schedule("A", 0.1, lambda: ran.append("A"))
    s.schedule("X", 0.15, lambda: ran.append("X"))
    s.schedule("B", 0.2, lambda: ran.append("B"))
    s.cancel("X")
    time.sleep(0.5)
    s.stop()
    assert ran == ["A", "B", "C"], ran
    print("ran in due-time order:", ran)

Two design points to say aloud. First, cancellation is lazy: a cancelled task stays in the queue and is skipped when it comes due, which keeps cancel O(1) at the cost of memory until then. ScheduledThreadPoolExecutor.setRemoveOnCancelPolicy(true) switches to eager removal. Second, the dispatcher hands work to a pool rather than running it, so one slow job doesn't delay every job behind it. The same mechanism powers seat-hold expiry in Booking & Inventory Concurrency.

Worker pools and executors

The shape. A fixed set of threads takes tasks from a queue. Creating a thread per task is expensive for platform threads (about a megabyte of reserved stack and a kernel object each) and unbounded, so a pool caps concurrency and reuses threads. In Java, ExecutorService is the abstraction, and ThreadPoolExecutor is the implementation whose knobs you should know:

flowchart LR
    S["submit(task)"] --> C{"fewer than<br/>core threads?"}
    C -->|"yes"| N["start a new<br/>core thread"]
    C -->|"no"| Q{"queue<br/>has room?"}
    Q -->|"yes"| E["enqueue"]
    Q -->|"no"| M{"fewer than<br/>max threads?"}
    M -->|"yes"| X["start an extra thread"]
    M -->|"no"| R["RejectedExecutionHandler"]
KnobWhat to choose
corePoolSize, maximumPoolSizeCPU-bound: about the number of cores. I/O-bound: more, roughly cores × (1 + wait time / compute time), then measure
Work queueBounded ArrayBlockingQueue. Note the counter-intuitive rule above: extra threads beyond core start only when the queue is full, so with an unbounded queue maximumPoolSize is never used
RejectedExecutionHandlerAbortPolicy (throw, the default), CallerRunsPolicy (the submitting thread runs the task, a natural backpressure), DiscardPolicy, DiscardOldestPolicy
ThreadFactoryName your threads (notif-sms-3) so thread dumps are readable; set the daemon flag deliberately

The Executors.newFixedThreadPool and newCachedThreadPool factories are fine for demos, but in a design discussion construct the ThreadPoolExecutor explicitly, because the fixed pool uses an unbounded queue and the cached pool has no thread limit. Both are ways to run out of memory under load.

Bulkheads: one pool per dependency

The pool decision that matters most in LLD is how many pools. If a notification service sends email, SMS and push through one shared pool, a slow SMS provider fills every thread with blocked SMS sends, and email stops too. Giving each channel its own pool is the bulkhead pattern: a failure is contained to its compartment.

import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;

class NotificationPoolsDemo {
    public static void main(String[] args) throws Exception {
        NotificationService svc = new NotificationService(Map.of(
                "sms", new Channel("sms", 50),      // a slow provider: 50 ms per send
                "email", new Channel("email", 1)));
        long start = System.nanoTime();
        List<CompletableFuture<Void>> sms = new ArrayList<>(), email = new ArrayList<>();
        for (int i = 0; i < 40; i++) sms.add(svc.send("sms", "user" + i, "OTP " + i));
        for (int i = 0; i < 40; i++) email.add(svc.send("email", "user" + i, "Receipt " + i));

        CompletableFuture.allOf(email.toArray(CompletableFuture[]::new)).join();
        long emailMs = (System.nanoTime() - start) / 1_000_000;
        CompletableFuture.allOf(sms.toArray(CompletableFuture[]::new)).join();
        long smsMs = (System.nanoTime() - start) / 1_000_000;
        svc.shutdown();
        System.out.println("40 emails done after " + emailMs + " ms, 40 SMS after " + smsMs + " ms");
        if (emailMs >= smsMs) throw new AssertionError("email should not wait behind SMS");
    }
}

record Channel(String name, long sendMillis) {
    void deliver(String to, String body) {
        try { Thread.sleep(sendMillis); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
    }
}

/** One bounded pool per channel: a slow channel can only exhaust its own threads. */
final class NotificationService {
    private final Map<String, Channel> channels;
    private final Map<String, ThreadPoolExecutor> pools = new HashMap<>();

    NotificationService(Map<String, Channel> channels) {
        this.channels = Map.copyOf(channels);
        for (String name : channels.keySet()) pools.put(name, newPool(name, 4, 100));
    }

    private static ThreadPoolExecutor newPool(String name, int threads, int queueSize) {
        AtomicInteger n = new AtomicInteger();
        return new ThreadPoolExecutor(threads, threads, 30, TimeUnit.SECONDS,
                new ArrayBlockingQueue<>(queueSize),                       // bounded
                r -> new Thread(r, "notif-" + name + "-" + n.incrementAndGet()),
                new ThreadPoolExecutor.CallerRunsPolicy());                // full: the caller slows down
    }

    CompletableFuture<Void> send(String channel, String to, String body) {
        Channel c = channels.get(channel);
        if (c == null) throw new IllegalArgumentException("unknown channel " + channel);
        return CompletableFuture.runAsync(() -> c.deliver(to, body), pools.get(channel));
    }

    void shutdown() throws InterruptedException {
        for (ThreadPoolExecutor p : pools.values()) p.shutdown();
        for (ThreadPoolExecutor p : pools.values()) p.awaitTermination(10, TimeUnit.SECONDS);
    }
}
import time
from concurrent.futures import ThreadPoolExecutor, wait


class NotificationService:
    """One pool per channel. Threads suit this I/O-bound work despite the GIL:
    a thread blocked in a network call (here, sleep) releases the GIL."""

    def __init__(self, send_seconds: dict[str, float]) -> None:
        self._send_seconds = dict(send_seconds)
        self._pools = {name: ThreadPoolExecutor(max_workers=4, thread_name_prefix=f"notif-{name}")
                       for name in send_seconds}

    def send(self, channel: str, to: str, body: str):
        delay = self._send_seconds[channel]
        return self._pools[channel].submit(time.sleep, delay)  # stands in for the provider call

    def shutdown(self) -> None:
        for pool in self._pools.values():
            pool.shutdown(wait=True)


if __name__ == "__main__":
    svc = NotificationService({"sms": 0.05, "email": 0.001})
    start = time.perf_counter()
    sms = [svc.send("sms", f"user{i}", f"OTP {i}") for i in range(40)]
    email = [svc.send("email", f"user{i}", f"Receipt {i}") for i in range(40)]
    wait(email)
    email_ms = (time.perf_counter() - start) * 1000
    wait(sms)
    sms_ms = (time.perf_counter() - start) * 1000
    svc.shutdown()
    print(f"40 emails done after {email_ms:.0f} ms, 40 SMS after {sms_ms:.0f} ms")
    assert email_ms < sms_ms
// Node has no threads to pool for I/O; the equivalent is a cap on in-flight promises per channel.
class Limiter {
  private active = 0;
  private waiting: Array<() => void> = [];
  maxSeen = 0;

  constructor(private readonly limit: number) {}

  async run<T>(task: () => Promise<T>): Promise<T> {
    while (this.active >= this.limit) {
      await new Promise<void>((resolve) => this.waiting.push(resolve));
    }
    this.active++;
    this.maxSeen = Math.max(this.maxSeen, this.active);
    try {
      return await task();
    } finally {
      this.active--;
      this.waiting.shift()?.();
    }
  }
}

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
const limiters = { sms: new Limiter(4), email: new Limiter(4) };
const latency = { sms: 50, email: 1 };
const send = (channel: "sms" | "email") => limiters[channel].run(() => sleep(latency[channel]));

const start = performance.now();
const sms = Promise.all(Array.from({ length: 40 }, () => send("sms")));
await Promise.all(Array.from({ length: 40 }, () => send("email")));
const emailMs = performance.now() - start;
await sms;
const smsMs = performance.now() - start;
console.log(`40 emails after ${emailMs.toFixed(0)} ms, 40 SMS after ${smsMs.toFixed(0)} ms`);
if (limiters.sms.maxSeen > 4 || emailMs >= smsMs) throw new Error("limiter or bulkhead broken");

On our machine the Java version printed 40 emails done after 29 ms, 40 SMS after 513 ms or close to it on every run (emails between 27 and 34 ms, SMS between 510 and 516 ms over five runs): 40 SMS sends at 50 ms each across four threads take about 500 ms, and the email pool is untouched by that. Put both channels in one four-thread pool and the emails queue behind the SMS sends.

Where it appears in LLD problems: the notification service (a pool per channel), a web crawler (a pool of fetchers, with per-host limits), an image or video processing service (a CPU-sized pool), and any request-handling service. In an interview, mentioning "bounded queue, named threads, explicit rejection policy, one pool per dependency" covers what most interviewers are listening for.

Read-write locks

The shape. State that's read constantly and written rarely, such as a configuration registry, a routing table, a feature-flag store or a product catalogue cache. A plain lock serialises the readers for no reason. A read-write lock allows any number of concurrent readers or one writer. Classic Synchronization Problems covers reader and writer preference and starvation in depth; the practical points for design are these:

  • ReentrantReadWriteLock in Java. A thread holding the read lock can't upgrade to the write lock (it deadlocks); release the read lock first and re-check after taking the write lock. Downgrading (write → read) is allowed.
  • StampedLock adds an optimistic read: read without locking, then validate(stamp) to check that no write happened meanwhile, and fall back to a real read lock if one did. It's very fast for tiny reads, but it isn't reentrant and is easy to misuse.
  • An RW lock only pays off when read sections are long enough, or contention high enough, that the extra bookkeeping is worth it. For short reads of a small map, a plain lock or ConcurrentHashMap often wins.
  • Often the best "read-write lock" is no lock at all: keep an immutable snapshot in a volatile field and replace it on every write (copy-on-write). Readers never block and always see a consistent snapshot; writers pay for a copy. For configuration that changes a few times a day, this is the right answer.

The example keeps a price and its currency, which must always be read together. Both implementations keep the pair consistent; the test runs readers against a writer and checks that no reader ever sees a mismatched pair.

import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.locks.ReentrantReadWriteLock;

class ReadWriteDemo {
    public static void main(String[] args) throws Exception {
        for (PriceBook book : List.of(new LockedPriceBook(), new SnapshotPriceBook())) {
            int torn = run(book);
            if (torn != 0) throw new AssertionError(book.getClass().getSimpleName() + " tore " + torn);
            System.out.println(book.getClass().getSimpleName() + ": 0 inconsistent reads");
        }
    }

    /** One writer flips between (100, USD) and (90, EUR); readers check they never see a mix. */
    static int run(PriceBook book) throws Exception {
        AtomicInteger torn = new AtomicInteger();
        ExecutorService pool = Executors.newFixedThreadPool(9);
        Future<?> writer = pool.submit(() -> {
            for (int i = 0; i < 20_000; i++) book.set("sku-1", i % 2 == 0 ? 100 : 90, i % 2 == 0 ? "USD" : "EUR");
        });
        List<Future<?>> readers = new ArrayList<>();
        for (int r = 0; r < 8; r++) readers.add(pool.submit(() -> {
            for (int i = 0; i < 50_000; i++) {
                Price p = book.get("sku-1");
                if (p != null && !((p.amount() == 100 && p.currency().equals("USD"))
                                || (p.amount() == 90 && p.currency().equals("EUR")))) torn.incrementAndGet();
            }
        }));
        writer.get();
        for (Future<?> f : readers) f.get();
        pool.shutdown();
        return torn.get();
    }
}

record Price(long amount, String currency) { }

interface PriceBook { Price get(String sku); void set(String sku, long amount, String currency); }

/** Readers share the read lock; a writer takes the exclusive write lock. */
final class LockedPriceBook implements PriceBook {
    private final ReentrantReadWriteLock rw = new ReentrantReadWriteLock();
    private final Map<String, Long> amounts = new HashMap<>();     // guarded by rw
    private final Map<String, String> currencies = new HashMap<>(); // guarded by rw: changes with amounts

    public Price get(String sku) {
        rw.readLock().lock();
        try {
            Long a = amounts.get(sku);
            return a == null ? null : new Price(a, currencies.get(sku));
        } finally { rw.readLock().unlock(); }
    }

    public void set(String sku, long amount, String currency) {
        rw.writeLock().lock();
        try { amounts.put(sku, amount); currencies.put(sku, currency); }
        finally { rw.writeLock().unlock(); }
    }
}

/** Copy-on-write snapshot: readers never lock, writers replace the whole immutable map. */
final class SnapshotPriceBook implements PriceBook {
    private volatile Map<String, Price> snapshot = Map.of();

    public Price get(String sku) { return snapshot.get(sku); }

    public synchronized void set(String sku, long amount, String currency) { // writers serialise
        Map<String, Price> next = new HashMap<>(snapshot);
        next.put(sku, new Price(amount, currency));
        snapshot = Map.copyOf(next);                                    // volatile write publishes safely
    }
}
import threading
from types import MappingProxyType


class SnapshotPriceBook:
    """Python's stdlib has no read-write lock. Copy-on-write gives lock-free reads:
    rebinding self._snapshot is a single atomic reference assignment."""

    def __init__(self) -> None:
        self._snapshot: MappingProxyType[str, tuple[int, str]] = MappingProxyType({})
        self._write_lock = threading.Lock()

    def get(self, sku: str) -> tuple[int, str] | None:
        return self._snapshot.get(sku)

    def set(self, sku: str, amount: int, currency: str) -> None:
        with self._write_lock:
            nxt = dict(self._snapshot)
            nxt[sku] = (amount, currency)   # the pair is one immutable tuple
            self._snapshot = MappingProxyType(nxt)


if __name__ == "__main__":
    book, torn = SnapshotPriceBook(), []

    def writer() -> None:
        for i in range(20_000):
            book.set("sku-1", *((100, "USD") if i % 2 == 0 else (90, "EUR")))

    def reader() -> None:
        for _ in range(20_000):
            p = book.get("sku-1")
            if p is not None and p not in ((100, "USD"), (90, "EUR")):
                torn.append(p)

    threads = [threading.Thread(target=writer)] + [threading.Thread(target=reader) for _ in range(4)]
    for t in threads:
        t.start()
    for t in threads:
        t.join()
    assert not torn
    print("SnapshotPriceBook: 0 inconsistent reads")

Where it appears in LLD problems: a config or feature-flag service (snapshot), an LRU cache's read path (careful: in an LRU cache, a "read" updates recency, so it's a write and an RW lock doesn't help, which is a good follow-up to raise yourself), a URL shortener's in-memory alias cache, and a DNS-style route table.

Double-checked locking and holder singletons

Singletons are discussed on their own page (Singleton), including why to prefer dependency injection over a global. The concurrency question is narrower: if an object must be created lazily, exactly once, and shared, how do you do it without locking on every access?

Double-checked locking, done correctly

The idea: check without the lock (fast path), and only if the instance is missing, take the lock and check again (because another thread may have created it while you waited). The classic broken version omits volatile. Without it, the write of the reference can become visible to another thread before the writes inside the constructor, so a second thread can skip the lock, see a non-null reference and use a half-constructed object. That isn't theory; it was the reason the Java Memory Model was revised in Java 5 (JSR 133), after which volatile makes the pattern correct.

Correct DCL has three ingredients: a volatile field, a second check inside the lock, and (for speed) a local variable so the volatile field is read once on the fast path.

The holder idiom and the enum singleton

For a static singleton, there's a simpler and faster option: the initialisation-on-demand holder. The JVM initialises a nested class the first time it's used, and class initialisation is guaranteed thread-safe and happens once. So Holder.INSTANCE is lazy, thread-safe and lock-free after initialisation, with no volatile and no synchronized in sight. The enum singleton is the other robust option: one-element enums are created once by the JVM and are safe against reflection and serialization attacks, though they're initialised when the enum class is first touched, and can't extend a class.

DCL still has a place: lazy initialisation of an instance field (one per object, not one per JVM), where there's no class to hold it.

All three versions below are run with 32 threads released at once by a start gate, and each counts how many times the expensive constructor ran.

import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.Supplier;

class LazyInitDemo {
    public static void main(String[] args) throws Exception {
        check("double-checked locking", DclRegistry::get, DclRegistry.CREATED);
        check("holder idiom", HolderRegistry::get, HolderRegistry.CREATED);
        check("enum", () -> EnumRegistry.INSTANCE, EnumRegistry.CREATED);
        Show show = new Show();
        check("per-instance DCL", show::seatMap, Show.CREATED);
    }

    static void check(String name, Supplier<Object> getter, AtomicInteger created) throws Exception {
        ExecutorService pool = Executors.newFixedThreadPool(32);
        CountDownLatch go = new CountDownLatch(1);
        List<Future<Object>> results = new ArrayList<>();
        for (int i = 0; i < 32; i++) results.add(pool.submit(() -> { go.await(); return getter.get(); }));
        go.countDown();
        Set<Object> distinct = Collections.newSetFromMap(new IdentityHashMap<>());
        for (Future<Object> f : results) distinct.add(f.get());
        pool.shutdown();
        if (created.get() != 1 || distinct.size() != 1) throw new AssertionError(name + ": created " + created);
        System.out.println(name + ": constructed once, 32 threads saw the same instance");
    }

    static void slowConstruction() {
        try { Thread.sleep(50); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
    }
}

final class DclRegistry {
    static final AtomicInteger CREATED = new AtomicInteger();
    private static volatile DclRegistry instance;            // volatile is what makes DCL correct

    private DclRegistry() { CREATED.incrementAndGet(); LazyInitDemo.slowConstruction(); }

    static DclRegistry get() {
        DclRegistry local = instance;                          // one volatile read on the fast path
        if (local == null) {
            synchronized (DclRegistry.class) {
                local = instance;                              // second check: someone may have won while we waited
                if (local == null) {
                    local = new DclRegistry();
                    instance = local;                          // volatile write: publishes a fully built object
                }
            }
        }
        return local;
    }
}

final class HolderRegistry {
    static final AtomicInteger CREATED = new AtomicInteger();
    private HolderRegistry() { CREATED.incrementAndGet(); LazyInitDemo.slowConstruction(); }

    private static final class Holder {                        // initialised on first use of Holder, exactly once
        static final HolderRegistry INSTANCE = new HolderRegistry();
    }

    static HolderRegistry get() { return Holder.INSTANCE; }
}

enum EnumRegistry {
    INSTANCE;
    // The counter lives in another class: while INSTANCE is being constructed,
    // EnumRegistry's own static fields haven't been initialised yet.
    static final AtomicInteger CREATED = EnumCounter.CREATED;
    EnumRegistry() { EnumCounter.CREATED.incrementAndGet(); LazyInitDemo.slowConstruction(); }
}

final class EnumCounter { static final AtomicInteger CREATED = new AtomicInteger(); }

/** DCL for a per-object lazy field: the holder idiom only works for statics. */
final class Show {
    static final AtomicInteger CREATED = new AtomicInteger();
    private volatile Map<String, Boolean> seatMap;

    Map<String, Boolean> seatMap() {
        Map<String, Boolean> local = seatMap;
        if (local == null) {
            synchronized (this) {
                local = seatMap;
                if (local == null) {
                    CREATED.incrementAndGet();
                    LazyInitDemo.slowConstruction();          // e.g. load the layout from storage
                    local = new ConcurrentHashMap<>(Map.of("A1", false, "A2", false));
                    seatMap = local;
                }
            }
        }
        return local;
    }
}

The enum version has a wrinkle worth noticing: an enum's constants are constructed before its other static fields are initialised, so a constructor that touches the enum's own statics sees null (our first draft of this sample failed with exactly that NullPointerException, and the compiler rejects the direct reference outright). That's the JVM's class-initialisation order showing through, and the same order is what makes the holder idiom work.

The other languages have their own idioms:

import threading
import time
from concurrent.futures import ThreadPoolExecutor


class Registry:
    created = 0

    def __init__(self) -> None:
        Registry.created += 1
        time.sleep(0.05)  # slow construction widens the race window


_instance: Registry | None = None
_lock = threading.Lock()


def get_registry() -> Registry:
    """Double-checked locking. In CPython, reading and binding a module global is atomic,
    so the unlocked first check is safe. The lock makes construction happen once."""
    global _instance
    if _instance is None:
        with _lock:
            if _instance is None:
                _instance = Registry()
    return _instance


if __name__ == "__main__":
    gate = threading.Barrier(32)

    def worker(_: int) -> Registry:
        gate.wait()
        return get_registry()

    with ThreadPoolExecutor(max_workers=32) as pool:
        instances = set(map(id, pool.map(worker, range(32))))
    assert Registry.created == 1 and len(instances) == 1
    print("constructed once, 32 threads saw the same instance")
    # The simplest Python singleton is a module-level object: modules are imported once,
    # under the import lock. functools.cache is NOT a once-only guarantee under threads.
// Node: synchronous lazy init is trivially safe (one thread). The trap is ASYNC init:
// cache the promise, not the result, or concurrent callers each start their own init.
let created = 0;

class Registry {
  private constructor() { created++; }
  static async load(): Promise<Registry> {
    await new Promise((r) => setTimeout(r, 50)); // e.g. read config from disk
    return new Registry();
  }
}

let pending: Promise<Registry> | undefined;

function getRegistry(): Promise<Registry> {
  // Assigned synchronously, before any await, so every caller shares one in-flight load.
  pending ??= Registry.load().catch((err) => {
    pending = undefined; // let a later call retry after a failure
    throw err;
  });
  return pending;
}

const all = await Promise.all(Array.from({ length: 32 }, () => getRegistry()));
if (created !== 1 || new Set(all).size !== 1) throw new Error(`created ${created}`);
console.log("constructed once, 32 callers got the same instance");

The TypeScript version shows the asynchronous form of double-checked locking: storing the in-flight Promise is the "volatile field", and assigning it before the first await is the lock. Caching the result instead (if (!instance) instance = await load()) runs load() once per concurrent caller.

Where it appears in LLD problems: the logger, an ID generator, a connection pool and a configuration registry are the usual singletons. Lazy per-key initialisation (a seat map loaded the first time a show is opened, a per-user rate limiter created on first request) is the same problem keyed by ID, and ConcurrentHashMap.computeIfAbsent solves it in one line.

Futures and async composition

The shape. A request needs several independent pieces of information, such as a price, a stock level and a fraud score, each from a slow dependency. Fetching them one after another costs the sum of the latencies; fetching them concurrently costs the maximum. A Future is a handle to a result that isn't ready yet, and CompletableFuture adds composition: chain a step after a result (thenApply, thenCompose), combine two results (thenCombine), wait for many (allOf), recover from failure (exceptionally, handle), and bound the wait (orTimeout, completeOnTimeout).

sequenceDiagram
    participant C as CheckoutService
    participant P as Pricing
    participant I as Inventory
    participant F as FraudCheck
    C->>P: quote(sku) async
    C->>I: available(sku) async
    C->>F: score(user) async, 200 ms budget
    P-->>C: price after 120 ms
    I-->>C: stock after 80 ms
    F-->>C: no reply in budget, default to "review"
    C->>C: thenCombine all three into a CheckoutQuote

Design rules that matter in an interview:

  • Pass your own executor. The *Async methods without an executor run on the common ForkJoinPool, which is sized for CPU work. Blocking I/O there starves everything else in the JVM that uses it, parallel streams included.
  • Every remote call gets a timeout, and you decide what a timeout means: fail the request (orTimeout) or use a safe default (completeOnTimeout). For fraud scoring, "send to manual review" is a reasonable default; for price, there's no safe default, so fail.
  • Handle failure at the edge of each branch, so one failed branch produces a defined result instead of an exception surfacing from somewhere unexpected in join().
  • thenCompose for dependent calls, thenCombine for independent ones. Using thenApply with a function that returns a future gives you a CompletableFuture<CompletableFuture<T>>, a common slip.
import java.util.concurrent.*;

class CheckoutDemo {
    public static void main(String[] args) throws Exception {
        ExecutorService io = Executors.newFixedThreadPool(8);
        CheckoutService checkout = new CheckoutService(io);

        long start = System.nanoTime();
        CheckoutQuote q = checkout.quote("sku-42", "user-7").join();
        long ms = (System.nanoTime() - start) / 1_000_000;
        System.out.println(q + " in " + ms + " ms (sequential would be about 400 ms)");
        if (ms > 350) throw new AssertionError("calls did not run concurrently");
        if (!q.fraudDecision().equals("REVIEW")) throw new AssertionError("timeout default not applied");

        CheckoutQuote failed = checkout.quote("missing", "user-7").join();
        if (failed.priceCents() != -1) throw new AssertionError("failure not handled");
        System.out.println("missing sku -> " + failed);
        io.shutdown();
    }
}

record CheckoutQuote(long priceCents, int stock, String fraudDecision) { }

final class CheckoutService {
    private final Executor io;
    CheckoutService(Executor io) { this.io = io; }

    CompletableFuture<CheckoutQuote> quote(String sku, String user) {
        CompletableFuture<Long> price = CompletableFuture
                .supplyAsync(() -> slowPrice(sku), io)
                .orTimeout(1, TimeUnit.SECONDS);                       // no safe default: fail
        CompletableFuture<Integer> stock = CompletableFuture
                .supplyAsync(() -> slow(80, 5), io);
        CompletableFuture<String> fraud = CompletableFuture
                .supplyAsync(() -> slow(200, "APPROVE"), io)
                .completeOnTimeout("REVIEW", 150, TimeUnit.MILLISECONDS); // safe default on timeout

        return price.thenCombine(stock, (p, s) -> new long[] {p, s})
                .thenCombine(fraud, (ps, f) -> new CheckoutQuote(ps[0], (int) ps[1], f))
                .exceptionally(e -> new CheckoutQuote(-1, 0, "ERROR: " + rootMessage(e)));
    }

    private static long slowPrice(String sku) {
        if (sku.equals("missing")) throw new IllegalArgumentException("no such sku");
        return slow(120, 1_999L);
    }

    private static <T> T slow(long millis, T value) {
        try { Thread.sleep(millis); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
        return value;
    }

    private static String rootMessage(Throwable e) {
        while (e.getCause() != null) e = e.getCause();
        return e.getMessage();
    }
}
import asyncio
import time


async def slow(seconds: float, value):
    await asyncio.sleep(seconds)
    return value


async def price(sku: str) -> int:
    if sku == "missing":
        raise ValueError("no such sku")
    return await slow(0.12, 1999)


async def fraud_or_review() -> str:
    try:
        return await asyncio.wait_for(slow(0.2, "APPROVE"), timeout=0.15)
    except TimeoutError:
        return "REVIEW"  # safe default when the fraud service is slow


async def quote(sku: str) -> tuple[int, int, str]:
    try:
        p, s, f = await asyncio.gather(
            asyncio.wait_for(price(sku), timeout=1.0),
            slow(0.08, 5),
            fraud_or_review(),
        )
        return p, s, f
    except (ValueError, TimeoutError) as e:
        return -1, 0, f"ERROR: {e}"


async def main() -> None:
    start = time.perf_counter()
    q = await quote("sku-42")
    ms = (time.perf_counter() - start) * 1000
    print(q, f"in {ms:.0f} ms")
    assert ms < 350 and q[2] == "REVIEW"
    failed = await quote("missing")
    assert failed[0] == -1
    print("missing sku ->", failed)


asyncio.run(main())
const sleep = <T>(ms: number, value: T) => new Promise<T>((r) => setTimeout(() => r(value), ms));

function withTimeout<T>(p: Promise<T>, ms: number, fallback?: T): Promise<T> {
  let timer: NodeJS.Timeout | undefined;
  const timeout = new Promise<T>((resolve, reject) => {
    timer = setTimeout(() => (fallback !== undefined ? resolve(fallback) : reject(new Error("timeout"))), ms);
  });
  return Promise.race([p, timeout]).finally(() => clearTimeout(timer));
}

async function price(sku: string): Promise<number> {
  if (sku === "missing") throw new Error("no such sku");
  return sleep(120, 1999);
}

async function quote(sku: string): Promise<[number, number, string]> {
  try {
    return await Promise.all([
      withTimeout(price(sku), 1000),
      sleep(80, 5),
      withTimeout(sleep(200, "APPROVE"), 150, "REVIEW"),
    ]);
  } catch (e) {
    return [-1, 0, `ERROR: ${(e as Error).message}`];
  }
}

const start = performance.now();
const q = await quote("sku-42");
const ms = performance.now() - start;
console.log(q, `in ${ms.toFixed(0)} ms`);
if (ms > 350 || q[2] !== "REVIEW") throw new Error("not concurrent or no default");
const failed = await quote("missing");
if (failed[0] !== -1) throw new Error("failure not handled");
console.log("missing sku ->", failed);

Each version finished the happy path in roughly 150 ms on our machine, the length of the fraud budget, instead of the 400 ms that sequential calls would take. The Promise.race timeout in TypeScript doesn't cancel the underlying work; it only stops waiting for it. Real cancellation needs an AbortSignal passed into the call (fetch accepts one, and AbortSignal.timeout(ms) creates one). Java's orTimeout likewise completes the future without interrupting the task running on the executor.

Where it appears in LLD problems: checkout and quote aggregation, a search service fanning out to several indexes, a travel aggregator querying many providers with a deadline, and a notification service returning a future per send so callers can wait or not.

Thread-per-request vs virtual threads

The simplest server design gives each request its own thread and lets that thread block on the database, the payment provider and the file system. The code reads top to bottom, stack traces make sense, and ThreadLocal-style request context just works. Its limit is the cost of platform threads: each one maps to an OS thread with a reserved stack, so a server tops out at a few thousand concurrent requests, and most of those threads are idle, waiting on I/O.

The industry's first answer was asynchronous code: callbacks, CompletableFuture chains, reactive streams, Node's event loop, Python's asyncio. It scales to huge numbers of concurrent connections, at the cost of "coloured" functions, fragmented stack traces and harder debugging.

Virtual threads (JEP 444, final in Java 21) are Java's second answer: threads that the JVM schedules onto a small pool of carrier OS threads. When a virtual thread blocks on I/O, a lock or sleep, the JVM unmounts it and runs another on the same carrier. Blocking becomes cheap, so you can go back to thread-per-request with millions of threads. The test below runs 10,000 tasks that each block for 50 ms:

import java.util.*;
import java.util.concurrent.*;
import java.util.stream.IntStream;

class VirtualThreadsDemo {
    public static void main(String[] args) throws Exception {
        long platform = time(Executors.newFixedThreadPool(200));
        long virtual = time(Executors.newVirtualThreadPerTaskExecutor());
        System.out.println("10000 blocking tasks: 200 platform threads " + platform
                + " ms, virtual threads " + virtual + " ms");
        if (virtual >= platform) throw new AssertionError("expected virtual threads to finish first");
    }

    static long time(ExecutorService executor) throws Exception {
        long start = System.nanoTime();
        try (executor) {                                     // ExecutorService is AutoCloseable since Java 19
            List<Future<Integer>> fs = new ArrayList<>();
            IntStream.range(0, 10_000).forEach(i -> fs.add(executor.submit(() -> {
                Thread.sleep(50);                            // stands in for a blocking database call
                return i;
            })));
            for (Future<Integer> f : fs) f.get();
        }
        return (System.nanoTime() - start) / 1_000_000;
    }
}
import asyncio
import time


async def handle(i: int) -> int:
    await asyncio.sleep(0.05)  # an awaitable I/O call; the event loop runs others meanwhile
    return i


async def main() -> None:
    start = time.perf_counter()
    results = await asyncio.gather(*(handle(i) for i in range(10_000)))
    ms = (time.perf_counter() - start) * 1000
    assert results == list(range(10_000))
    print(f"10000 concurrent asyncio tasks in {ms:.0f} ms")


asyncio.run(main())
// Node is event-loop concurrency by default: 10,000 pending timers cost almost nothing.
const handle = (i: number) => new Promise<number>((r) => setTimeout(() => r(i), 50));

const start = performance.now();
const results = await Promise.all(Array.from({ length: 10_000 }, (_, i) => handle(i)));
const ms = performance.now() - start;
if (results.length !== 10_000 || results[9_999] !== 9_999) throw new Error("lost results");
console.log(`10000 concurrent promises in ${ms.toFixed(0)} ms`);

On our machine the 200-thread pool took 2,552–2,555 ms over five runs (10,000 tasks ÷ 200 threads × 50 ms is 2,500 ms, plus overhead) and virtual threads 93–250 ms. The asyncio version took 110–133 ms and the Node version 61–73 ms, because they're the same idea with explicit awaits.

What changes in your design when you adopt virtual threads:

  • Don't pool them. A virtual thread is cheap to create; use one per task. If you need to limit concurrency against a dependency (only 50 database connections), use a Semaphore or the connection pool itself, not a thread pool.
  • Pinning. Until Java 23, a virtual thread blocking inside a synchronized block pinned its carrier thread. JEP 491 fixed this in Java 24. Native calls still pin.
  • ThreadLocal caches stop working as caches. A per-thread cache with a million threads is a million caches. Scoped values (JEP 506, final in Java 25) are the intended replacement for passing request context.
  • CPU-bound work gains nothing. Virtual threads make waiting cheap, not computing. A CPU-bound pipeline still wants a pool sized to the cores.

In an LLD interview, the practical summary is: write the service as plain blocking, thread-per-request code, say it runs on virtual threads (or a bounded pool on older JVMs), and put the concurrency limits where the scarce resource is.

A rate-limited worker

The shape. Work arrives faster than a downstream provider allows: an SMS gateway that accepts 20 messages a second, an API with a quota, a crawler that must be polite to each host. A rate-limited worker is a producer–consumer queue whose consumers must take a token from a token bucket before each send. The bucket refills at the allowed rate and holds up to a burst capacity, so short bursts go through immediately and sustained load is smoothed to the rate. The rate limiter pattern page covers the algorithms; this is how it sits inside an object design.

One design detail matters: don't sleep while holding the bucket's lock. The bucket below hands out reservations: under the lock it computes when the caller's token will be available (possibly in the future, by letting the token count go negative as a debt), and the caller sleeps after releasing the lock. Waiting callers queue up fairly by reservation time, and the lock is held for nanoseconds.

import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;

class RateLimitedWorkerDemo {
    public static void main(String[] args) throws Exception {
        AtomicInteger sent = new AtomicInteger();
        RateLimitedWorker worker = RateLimitedWorker.start(new TokenBucket(20, 5), 4);
        long start = System.nanoTime();
        for (int i = 0; i < 45; i++) worker.submit(sent::incrementAndGet);
        worker.shutdownAndDrain();
        double seconds = (System.nanoTime() - start) / 1e9;
        // 5 go at once (the burst), the other 40 at 20 per second: about 2.0 s.
        System.out.printf("sent %d messages in %.2f s%n", sent.get(), seconds);
        if (sent.get() != 45 || seconds < 1.8 || seconds > 3.0) throw new AssertionError(seconds);
    }
}

/** Thread-safe token bucket. Callers reserve under the lock and sleep outside it. */
final class TokenBucket {
    private final double ratePerSecond, burst;
    private double tokens;                     // guarded by this, may go negative (a queue of reservations)
    private long lastNanos;                    // guarded by this

    TokenBucket(double ratePerSecond, double burst) {
        this.ratePerSecond = ratePerSecond;
        this.burst = burst;
        this.tokens = burst;
        this.lastNanos = System.nanoTime();
    }

    private synchronized long reserveNanos() {
        long now = System.nanoTime();
        tokens = Math.min(burst, tokens + (now - lastNanos) / 1e9 * ratePerSecond);
        lastNanos = now;
        tokens -= 1;
        return tokens >= 0 ? 0 : (long) (-tokens / ratePerSecond * 1e9);
    }

    void acquire() throws InterruptedException {
        long wait = reserveNanos();
        if (wait > 0) TimeUnit.NANOSECONDS.sleep(wait);   // outside the lock
    }
}

final class RateLimitedWorker {
    private final BlockingQueue<Runnable> queue = new LinkedBlockingQueue<>(10_000);
    private final TokenBucket bucket;
    private final ExecutorService threads;
    private volatile boolean accepting = true;

    private RateLimitedWorker(TokenBucket bucket, int n) {
        this.bucket = bucket;
        this.threads = Executors.newFixedThreadPool(n);
    }

    static RateLimitedWorker start(TokenBucket bucket, int n) {
        RateLimitedWorker w = new RateLimitedWorker(bucket, n);
        for (int i = 0; i < n; i++) w.threads.execute(w::loop);
        return w;
    }

    void submit(Runnable task) {
        if (!accepting || !queue.offer(task)) throw new RejectedExecutionException("worker full or stopping");
    }

    private void loop() {
        try {
            while (accepting || !queue.isEmpty()) {
                Runnable task = queue.poll(50, TimeUnit.MILLISECONDS);   // poll so we notice shutdown
                if (task == null) continue;
                bucket.acquire();
                task.run();
            }
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
    }

    void shutdownAndDrain() throws InterruptedException {
        accepting = false;                      // stop intake, keep draining what's queued
        threads.shutdown();
        if (!threads.awaitTermination(30, TimeUnit.SECONDS)) threads.shutdownNow();
    }
}
import asyncio
import time


class TokenBucket:
    """asyncio version: single-threaded, so no lock is needed. The reservation is
    computed synchronously (no await inside), which makes it atomic on the event loop."""

    def __init__(self, rate: float, burst: float) -> None:
        self.rate, self.burst = rate, burst
        self.tokens, self.last = burst, time.monotonic()

    async def acquire(self) -> None:
        now = time.monotonic()
        self.tokens = min(self.burst, self.tokens + (now - self.last) * self.rate)
        self.last = now
        self.tokens -= 1
        if self.tokens < 0:
            await asyncio.sleep(-self.tokens / self.rate)


async def main() -> None:
    bucket, sent = TokenBucket(20, 5), []
    queue: asyncio.Queue[int] = asyncio.Queue()
    for i in range(45):
        queue.put_nowait(i)

    async def worker() -> None:
        while True:
            try:
                item = queue.get_nowait()
            except asyncio.QueueEmpty:
                return
            await bucket.acquire()
            sent.append(item)

    start = time.perf_counter()
    await asyncio.gather(*(worker() for _ in range(4)))
    seconds = time.perf_counter() - start
    print(f"sent {len(sent)} messages in {seconds:.2f} s")
    assert len(sent) == 45 and 1.8 <= seconds <= 3.0


asyncio.run(main())
class TokenBucket {
  private tokens: number;
  private last = performance.now();

  constructor(private readonly ratePerSec: number, private readonly burst: number) {
    this.tokens = burst;
  }

  // The reservation runs synchronously, so it's atomic on Node's single thread.
  async acquire(): Promise<void> {
    const now = performance.now();
    this.tokens = Math.min(this.burst, this.tokens + ((now - this.last) / 1000) * this.ratePerSec);
    this.last = now;
    this.tokens -= 1;
    if (this.tokens < 0) await new Promise((r) => setTimeout(r, (-this.tokens / this.ratePerSec) * 1000));
  }
}

const bucket = new TokenBucket(20, 5);
const queue = Array.from({ length: 45 }, (_, i) => i);
let sent = 0;

async function worker(): Promise<void> {
  for (let item = queue.shift(); item !== undefined; item = queue.shift()) {
    await bucket.acquire();
    sent++;
  }
}

const start = performance.now();
await Promise.all([worker(), worker(), worker(), worker()]);
const seconds = (performance.now() - start) / 1000;
console.log(`sent ${sent} messages in ${seconds.toFixed(2)} s`);
if (sent !== 45 || seconds < 1.8 || seconds > 3.0) throw new Error(`unexpected ${seconds}`);

All three printed times between 2.00 and 2.06 seconds on our machine. The bucket is deliberately per-provider: if the SMS gateway allows 20 per second across your whole fleet, a per-process bucket is wrong once you run three instances, and the limit has to move to a shared store (a Redis counter, or the provider's own 429 responses with back-off). Saying that unprompted shows you know where the in-process design stops.

Where it appears in LLD problems: the notification service (per-channel buckets in front of each provider), an API client SDK, a web crawler (one bucket per host), and the "design a rate limiter" prompt itself, where this class is the answer and the follow-up is per-user buckets in a ConcurrentHashMap created with computeIfAbsent.

Graceful shutdown

Every component that owns threads must also own their shutdown. A process that's killed mid-task loses whatever was in memory: queued notifications, half-written log lines, seat holds that never get released. Graceful shutdown is a small protocol:

  1. Stop accepting new work (reject, or return 503).
  2. Drain: let in-flight and queued work finish, up to a deadline.
  3. Cancel what's left after the deadline: interrupt running tasks, and collect tasks that never started so they can be persisted or re-queued.
  4. Release resources: flush and close files, connections and sockets.
  5. Report: log what completed, what was cancelled and what was handed back.

The lifecycle is a state machine, which is a useful diagram to draw when the interviewer asks:

stateDiagram-v2
    [*] --> RUNNING: start()
    RUNNING --> DRAINING: shutdown() — stop intake
    DRAINING --> TERMINATED: all tasks done before deadline
    DRAINING --> CANCELLING: deadline passed
    CANCELLING --> TERMINATED: interrupt running, return unstarted
    TERMINATED --> [*]

ExecutorService already implements most of this: shutdown() stops intake and lets queued tasks run, awaitTermination(timeout) waits, and shutdownNow() interrupts running tasks and returns the ones that never started. Your tasks must cooperate by responding to interruption: blocking calls such as sleep, take and Future.get throw InterruptedException, and CPU loops should check Thread.currentThread().isInterrupted().

import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.*;

class GracefulShutdownDemo {
    public static void main(String[] args) throws Exception {
        OrderProcessor p = new OrderProcessor(4);
        for (int i = 0; i < 20; i++) p.submit("order-" + i);
        Thread.sleep(50);
        ShutdownReport r = p.shutdown(250);                      // 250 ms grace period
        System.out.println(r);
        try { p.submit("late"); throw new AssertionError("accepted after shutdown"); }
        catch (RejectedExecutionException expected) { }
        if (r.completed() + r.interrupted() + r.neverStarted().size() != 20) throw new AssertionError(r);
    }
}

record ShutdownReport(int completed, int interrupted, List<String> neverStarted) { }

final class OrderProcessor {
    private final ThreadPoolExecutor pool;
    private final AtomicInteger completed = new AtomicInteger(), interrupted = new AtomicInteger();
    private final Map<Runnable, String> idsByTask = new ConcurrentHashMap<>();

    OrderProcessor(int threads) {
        pool = new ThreadPoolExecutor(threads, threads, 0, TimeUnit.SECONDS, new ArrayBlockingQueue<>(100));
        Runtime.getRuntime().addShutdownHook(new Thread(() -> {    // also runs on SIGTERM / Ctrl+C
            if (!pool.isShutdown()) shutdown(1_000);
        }));
    }

    void submit(String orderId) {
        Runnable task = () -> {
            try {
                Thread.sleep(100);                                 // process the order (interruptible)
                completed.incrementAndGet();
            } catch (InterruptedException e) {
                interrupted.incrementAndGet();                     // a real system would mark it for retry
                Thread.currentThread().interrupt();
            }
        };
        idsByTask.put(task, orderId);
        pool.execute(task);                                        // throws RejectedExecutionException once shut down
    }

    ShutdownReport shutdown(long graceMillis) {
        pool.shutdown();                                           // 1. stop intake, 2. let queued work run
        List<String> unstarted = new ArrayList<>();
        try {
            if (!pool.awaitTermination(graceMillis, TimeUnit.MILLISECONDS)) {
                for (Runnable r : pool.shutdownNow()) unstarted.add(idsByTask.get(r)); // 3. cancel the rest
                pool.awaitTermination(1, TimeUnit.SECONDS);       // let interrupted tasks record themselves
            }
        } catch (InterruptedException e) {
            pool.shutdownNow();
            Thread.currentThread().interrupt();
        }
        return new ShutdownReport(completed.get(), interrupted.get(), unstarted);  // 5. report
    }
}
import threading
import time
from concurrent.futures import Future, ThreadPoolExecutor, wait


class OrderProcessor:
    def __init__(self, threads: int) -> None:
        self._pool = ThreadPoolExecutor(max_workers=threads)
        self._stop = threading.Event()  # Python threads can't be interrupted: tasks poll this instead
        self._futures: list[Future[None]] = []
        self.completed = 0
        self.abandoned = 0
        self._lock = threading.Lock()

    def submit(self, order_id: str) -> Future[None]:
        future = self._pool.submit(self._process, order_id)  # raises RuntimeError after shutdown
        self._futures.append(future)
        return future

    def _process(self, order_id: str) -> None:
        for _ in range(10):  # 100 ms of work in 10 ms steps, checking for cancellation
            if self._stop.is_set():
                with self._lock:
                    self.abandoned += 1
                return
            time.sleep(0.01)
        with self._lock:
            self.completed += 1

    def shutdown(self, grace_s: float) -> None:
        self._pool.shutdown(wait=False)              # 1. stop intake, queued work keeps running
        _, not_done = wait(self._futures, timeout=grace_s)  # 2. drain until the deadline
        if not_done:
            self._stop.set()                         # 3. ask running tasks to stop...
            self._pool.shutdown(wait=True, cancel_futures=True)  # ...and cancel unstarted ones


if __name__ == "__main__":
    p = OrderProcessor(4)
    futures = [p.submit(f"order-{i}") for i in range(20)]
    time.sleep(0.05)
    p.shutdown(grace_s=0.25)
    cancelled = sum(f.cancelled() for f in futures)
    print(f"completed {p.completed}, abandoned mid-task {p.abandoned}, never started {cancelled}")
    assert p.completed + p.abandoned + cancelled == 20
    try:
        p.submit("late")
        raise AssertionError("accepted after shutdown")
    except RuntimeError:
        pass
// Node: stop intake, wait for in-flight work up to a deadline, then abort the rest.
class OrderProcessor {
  private inFlight = new Set<Promise<void>>();
  private accepting = true;
  private readonly abort = new AbortController();
  completed = 0;
  aborted = 0;

  submit(orderId: string): void {
    if (!this.accepting) throw new Error(`rejected ${orderId}: shutting down`);
    const job = this.process(this.abort.signal).finally(() => this.inFlight.delete(job));
    this.inFlight.add(job);
  }

  private async process(signal: AbortSignal): Promise<void> {
    try {
      await new Promise<void>((resolve, reject) => {
        const t = setTimeout(resolve, 100 + Math.random() * 300);
        signal.addEventListener("abort", () => { clearTimeout(t); reject(new Error("aborted")); }, { once: true });
      });
      this.completed++;
    } catch {
      this.aborted++;
    }
  }

  async shutdown(graceMs: number): Promise<void> {
    this.accepting = false;
    const timer = setTimeout(() => this.abort.abort(), graceMs);
    await Promise.allSettled([...this.inFlight]);
    clearTimeout(timer);
  }
}

const p = new OrderProcessor();
// In production: process.once("SIGTERM", () => p.shutdown(10_000).then(() => process.exit(0)));
for (let i = 0; i < 20; i++) p.submit(`order-${i}`);
await p.shutdown(250);
console.log(`completed ${p.completed}, aborted ${p.aborted}`);
if (p.completed + p.aborted !== 20) throw new Error("lost a job");
let rejected = false;
try { p.submit("late"); } catch { rejected = true; }
if (!rejected) throw new Error("accepted after shutdown");

A typical Java run printed ShutdownReport[completed=12, interrupted=4, neverStarted=[order-16, order-17, order-18, order-19]]: three waves of four orders finished inside the grace period, four were interrupted mid-task, and four never started and were handed back. Over five runs the split varied (one run finished only two waves and handed back eight orders), but the three numbers always added up to 20, which is what the test asserts. The Python and TypeScript versions vary the same way. Note the shutdown hook: the JVM runs it on normal exit, System.exit, Ctrl+C and SIGTERM, but not on SIGKILL or a crash, so anything that truly must not be lost belongs in durable storage before it's acknowledged, not in a queue waiting for a graceful exit.

Python threads can't be interrupted from outside, so tasks must poll a threading.Event, as the sample does. In asyncio, task.cancel() raises CancelledError at the task's next await, which is the closest analogue to Java interruption. In Node, the AbortController/AbortSignal pair is the standard cancellation channel, and most modern APIs (fetch, timers in node:timers/promises, streams) accept a signal.

Where it appears in LLD problems: all of them. The logger must flush. The task scheduler must decide whether pending jobs run, persist or get dropped. The notification service must not lose queued sends. The booking service must release holds it can't confirm. A sentence on shutdown at the end of a design is cheap and distinctive.

Pub-sub: per-subscriber queues

The Observer pattern notifies listeners synchronously, in the publisher's thread. That's fine in a single-threaded UI, but in a server it couples the publisher to the slowest subscriber: if one listener makes a network call, every publish waits for it, and if one throws, the remaining listeners may never run. A concurrent pub-sub broker fixes this by giving each subscriber its own queue and its own consumer, which keeps per-subscriber ordering while isolating subscribers from each other.

import java.util.*;
import java.util.concurrent.*;
import java.util.function.Consumer;

class PubSubDemo {
    public static void main(String[] args) throws Exception {
        Broker broker = new Broker();
        List<String> fast = Collections.synchronizedList(new ArrayList<>());
        List<String> slow = Collections.synchronizedList(new ArrayList<>());
        broker.subscribe("orders", fast::add);
        broker.subscribe("orders", m -> { sleep(20); slow.add(m); });
        broker.subscribe("orders", m -> { throw new IllegalStateException("buggy subscriber"); });

        long start = System.nanoTime();
        for (int i = 0; i < 20; i++) broker.publish("orders", "order-" + i);
        long publishMs = (System.nanoTime() - start) / 1_000_000;
        broker.close();

        List<String> expected = new ArrayList<>();
        for (int i = 0; i < 20; i++) expected.add("order-" + i);
        if (!fast.equals(expected) || !slow.equals(expected)) throw new AssertionError("order or delivery broken");
        if (publishMs > 100) throw new AssertionError("publisher waited for subscribers: " + publishMs + " ms");
        System.out.println("published 20 in " + publishMs + " ms; both subscribers got all 20 in order");
    }

    static void sleep(long ms) {
        try { Thread.sleep(ms); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
    }
}

final class Broker implements AutoCloseable {
    private record Subscription(Consumer<String> handler, ExecutorService lane) { }
    private final Map<String, List<Subscription>> topics = new ConcurrentHashMap<>();

    void subscribe(String topic, Consumer<String> handler) {
        // One single-thread executor per subscriber: its own queue, its own order, its own failures.
        ExecutorService lane = Executors.newSingleThreadExecutor();
        topics.computeIfAbsent(topic, t -> new CopyOnWriteArrayList<>()).add(new Subscription(handler, lane));
    }

    void publish(String topic, String message) {
        for (Subscription s : topics.getOrDefault(topic, List.of())) {
            s.lane().execute(() -> {
                try { s.handler().accept(message); }
                catch (RuntimeException e) { /* log and continue: one bad subscriber can't stop the others */ }
            });
        }
    }

    @Override public void close() throws InterruptedException {
        for (List<Subscription> subs : topics.values()) for (Subscription s : subs) s.lane().shutdown();
        for (List<Subscription> subs : topics.values()) for (Subscription s : subs) s.lane().awaitTermination(10, TimeUnit.SECONDS);
    }
}

CopyOnWriteArrayList holds each topic's subscribers because subscriptions change rarely and publishes iterate constantly, and its iterator never throws ConcurrentModificationException when a subscription is added mid-publish. A thread per subscriber is fine for dozens of subscribers. For thousands, each lane becomes a queue drained by a shared pool with a "one task per lane at a time" rule, or each lane gets a virtual thread. The single-thread executors here use unbounded queues, so a permanently stuck subscriber grows its queue forever; a production broker bounds each lane and applies the logger's drop or block policy per subscriber. In Node, EventEmitter is synchronous Observer, so a slow listener blocks emit; wrapping each listener in its own promise chain gives the same per-subscriber ordering.

Putting it together: a notification service

Interviewers rarely ask for one pattern in isolation. "Design a notification service" exercises nearly all of them at once, and seeing where each one sits in the class diagram is the best way to remember them.

classDiagram
    class NotificationService {
        +send(Notification) Receipt
        +shutdown(Duration)
    }
    class Dispatcher {
        -BlockingQueue~Notification~ inbox
        -run()
    }
    class ChannelWorker {
        -ThreadPoolExecutor pool
        -TokenBucket bucket
        +submit(Notification)
    }
    class TokenBucket {
        +acquire()
    }
    class RetryScheduler {
        -DelayQueue~RetryTask~ due
        +retryLater(Notification, Duration)
    }
    class PreferenceStore {
        -Map snapshot
        +channelsFor(userId) List
    }
    class ChannelSender {
        <<interface>>
        +deliver(Notification)
    }
    NotificationService --> Dispatcher : enqueues
    Dispatcher --> PreferenceStore : reads
    Dispatcher --> ChannelWorker : routes to
    ChannelWorker --> TokenBucket : acquires
    ChannelWorker --> ChannelSender : calls
    ChannelWorker --> RetryScheduler : on failure
    RetryScheduler --> Dispatcher : re-enqueues

Reading the diagram as a set of decisions:

  • NotificationService.send returns immediately with a receipt ID after putting the notification on the dispatcher's bounded inbox. That's producer–consumer, and its backpressure policy is the first thing to settle with the interviewer. For password-reset codes, blocking briefly (or returning 503) is better than dropping; for marketing pushes, dropping under overload is fine. Two priorities can mean two queues.
  • PreferenceStore is read on every notification and written when a user changes settings. That's the read-mostly shape, and a copy-on-write snapshot in a volatile field (or a ConcurrentHashMap keyed by user, since each user's preferences are independent) beats a read-write lock.
  • Each ChannelWorker owns a bounded pool, so SMS, email and push are bulkheaded from each other, and a token bucket per provider keeps within its rate limit.
  • Failures go to a RetryScheduler, a delay queue that re-enqueues each failed notification after an exponential back-off. That's the scheduler pattern, and it must cap attempts, or a provider outage becomes an ever-growing retry storm.
  • shutdown cascades in dependency order: stop accepting in the service, drain the dispatcher's inbox, shut down each channel pool with a deadline, then stop the retry scheduler and persist whatever it still holds so that retries survive a deploy.
  • Idempotency. A retry after a timeout can send the same SMS twice if the first attempt actually succeeded. Each notification carries an ID that the sender passes to the provider as an idempotency key, or that a dedupe table checks. Booking & Inventory Concurrency covers the same idea for payments.

None of these classes contains a hand-written synchronized block. Every piece of shared state is either confined (each worker's in-flight notification), immutable (the notification itself, and the preference snapshot), or delegated to a JDK concurrent class (queues, pools, atomics). That's the target for any LLD answer: the concurrency lives in a few well-understood library components at the boundaries, and the domain classes stay simple.

A few more LLD prompts, mapped the same way:

PromptPatterns, and where they sit
LoggerProducer–consumer into one writer thread; drop policy; poison-pill shutdown; holder or DI-provided singleton
Task scheduler / cronDelay queue plus dispatcher thread; worker pool for execution; lazy cancellation; graceful shutdown that persists pending jobs
Pub-sub / event busPer-subscriber queues; CopyOnWriteArrayList subscriber lists; bounded lanes; at-least-once delivery means subscribers must be idempotent
Rate limiterToken bucket per key in ConcurrentHashMap.computeIfAbsent; reservation under lock, sleep outside; eviction of idle buckets
Web crawlerFrontier queue (producer–consumer, deduplicated with a concurrent set); fetcher pool; per-host token buckets; CompletableFuture for parse-then-store chains
Movie or hotel bookingThread-per-request (virtual threads); per-show locks or CAS; delay queue for hold expiry. See Booking & Inventory Concurrency
Cache with refreshRead-mostly snapshot or ConcurrentHashMap; computeIfAbsent for once-per-key loading; scheduled refresh

War Story (illustrative)

A team moved its product-page aggregator to CompletableFutures and saw page latency improve, until a traffic spike. Then every request in the service slowed to a crawl, including endpoints that never touched the aggregator. The cause: the aggregator called supplyAsync(() -> httpClient.get(url)) without passing an executor, so a few hundred blocking HTTP calls sat on the JVM-wide common ForkJoinPool, which has only about as many threads as there are cores. Parallel streams elsewhere in the service used the same pool and queued behind the blocked HTTP calls. The fix was one argument, a dedicated bounded executor for I/O, plus a lint rule that bans the executor-less *Async methods on I/O paths. The lesson for an interview is the same one as the bulkhead: every blocking dependency gets its own bounded concurrency, and you name it.

Choosing a pattern

Symptom in the requirementsPattern
"Don't make the caller wait for X"Producer–consumer queue, with a backpressure policy
"Run X at time T", "every N minutes", "expire after N minutes"Delay queue or scheduled executor
"Handle many requests", "process many items"Worker pool (or virtual threads for blocking I/O)
"One slow dependency must not take down the others"One bounded pool or limiter per dependency (bulkhead)
"Read constantly, change rarely"Copy-on-write snapshot; RW lock if writes are frequent or data is large
"Created once, shared, created lazily"Holder idiom or enum; DCL with volatile for instance fields; promise memoisation in Node
"Needs data from several services"CompletableFuture / Promise.all / asyncio.gather, with per-call timeouts
"Provider allows N per second"Rate-limited worker with a token bucket
"Many independent listeners"Per-subscriber queues
Every component that starts threadsGraceful shutdown

Common mistakes

MistakeConsequenceFix
Unbounded queues and newFixedThreadPool everywhereMemory grows without limit under overloadBounded queue plus an explicit rejection or drop policy
Blocking I/O in CompletableFuture.supplyAsync without an executorStarves the common pool that the whole JVM sharesPass a dedicated I/O executor, or use virtual threads
DCL without volatileOther threads may see a half-constructed objectvolatile field, or the holder idiom
Upgrading a read lock to a write lockDeadlockRelease the read lock, take the write lock, re-check
Swallowing InterruptedExceptionShutdown can't stop the taskRestore the flag with Thread.currentThread().interrupt() and exit
Sleeping while holding a rate limiter's lockEvery caller serialises behind the sleeperReserve under the lock, sleep outside
Pooling virtual threadsThrows away their main benefitOne virtual thread per task; limit with a Semaphore
Synchronous Observer in a serverSlowest subscriber sets publish latency; one exception stops the restPer-subscriber queues, exceptions caught per subscriber
No shutdown pathQueued work lost on deployStop intake, drain with a deadline, cancel, report
Caching the result of async init instead of the promiseConcurrent callers each run the initCache the in-flight promise

Interview questions

  1. "Design an async logger. What happens when the disk is slow?" Request threads put lines on a bounded BlockingQueue; one writer thread owns the file. When the queue fills, apply a backpressure policy: for logs, drop and count (or keep only warnings and errors), because blocking request threads to wait on log I/O turns a disk hiccup into an outage. On shutdown, enqueue a poison pill with a blocking put and join the writer so everything already queued is flushed.

  2. "Why does ThreadPoolExecutor never grow past its core size?" Because with an unbounded queue the queue never fills, and extra threads beyond core are only created when the queue is full. Use a bounded queue if you want the pool to grow, and choose a rejection policy for when both are full.

  3. "What's wrong with this double-checked locking?" (A version without volatile.) The reference write can be reordered with the constructor's field writes, so another thread can see a non-null reference to a partly constructed object. Make the field volatile, or use the holder idiom for statics.

  4. "When would a read-write lock be slower than a plain lock?" When read sections are very short, because the RW lock's bookkeeping costs more than the time readers would have waited, or when writes are frequent. For read-mostly configuration, a copy-on-write snapshot in a volatile field beats both.

  5. "Your checkout calls pricing, inventory and fraud. How do you keep latency down?" Call them concurrently with CompletableFutures on a dedicated executor, give each a timeout, decide per call whether a timeout fails the request or falls back to a default (fraud → manual review, price → fail), and combine with thenCombine or allOf. Latency becomes the maximum, not the sum.

  6. "What do virtual threads change about your design?" Blocking becomes cheap, so you can write simple thread-per-request code without a large platform-thread pool. Don't pool virtual threads; limit concurrency at scarce resources with semaphores or connection pools. They don't help CPU-bound work, and per-thread caches in ThreadLocals stop making sense.

  7. "How do you rate-limit SMS sends to 20 per second across four worker threads?" A shared token bucket in front of the send. Workers take a task from the queue, acquire a token (reserve under the lock, sleep outside it) and send. If the service runs on several instances, the limit is global, so the bucket must move to shared storage or the design must react to the provider's 429 responses.

  8. "How does your service shut down without losing work?" Stop intake, let queued and in-flight work finish within a deadline, then interrupt what's running and collect unstarted tasks to persist or re-queue, then release resources. Tasks must respond to interruption. Anything that must survive a crash should be durable before it's acknowledged, since shutdown hooks don't run on SIGKILL.

  9. "Observer or pub-sub with queues for a notification fan-out?" Queues. Synchronous Observer makes publish latency equal to the slowest subscriber's and lets one exception stop delivery to the rest. Per-subscriber queues keep per-subscriber ordering, isolate failures and let each subscriber go at its own pace, at the cost of memory and the need for bounded queues per subscriber.

  10. "How would you design the task scheduler's cancellation?" Lazy cancellation by default: mark the task cancelled (O(1)) and skip it when it comes due, accepting that it occupies memory until then. Switch to eager removal if cancellations are frequent and tasks are far in the future. Either way, a task that has already started must check for cancellation (interruption or a flag) itself.

  11. "In Node, why does if (!instance) instance = await load() load twice?" Every caller that arrives before the first load() resolves sees instance unset and starts its own load. Store the promise synchronously (pending ??= load()) so later callers share the in-flight one, and clear it on failure so a retry is possible.