On this page

Distributed Locking

Distributed locking is one of the most deceptively dangerous problems in distributed systems. The idea seems simple: multiple processes need mutual exclusion over a shared resource. In a single-process world, you use a mutex. In a distributed world, you need a distributed lock. But the properties that make a local mutex safe — atomicity, no partial failure, no clock skew — simply do not exist in a distributed system.

Getting distributed locks wrong leads to double-charging customers, corrupting data, overselling inventory, or processing the same job multiple times. Getting them right requires understanding exactly what guarantees your lock provides and, critically, what guarantees it does not provide.

What Locks Are For: Efficiency vs Correctness

Before choosing a lock algorithm, answer one question: what happens if two holders ever run at the same time? Martin Kleppmann's framing of this question is the most useful idea in the whole topic, and it splits locks into two very different kinds.

Efficiency lockCorrectness lock
PurposeAvoid doing the same work twicePrevent two writers from corrupting shared state
If mutual exclusion fails onceSome wasted work, a duplicate email, a cache filled twiceLost update, double charge, oversold inventory, corrupt file
Acceptable failure rate"Rarely" is fineMust be "never", or the damage must be blocked elsewhere
Typical examplesOne cron instance generates the nightly report. One worker rebuilds a cache entry. Deduplicate expensive API calls.One process appends to a log segment. One writer moves money between accounts. One scheduler assigns a job to a machine.
What you needA cheap, available lock (one Redis node is fine)A lock plus a check at the resource: fencing tokens, compare-and-set, or a transaction

The key point is that no lock, on its own, can make a correctness lock safe once process pauses and network delays are possible. The reason is simple. A lock service can only decide who holds the lock at the lock service. It cannot reach into a client and stop it from acting on an old belief. The protected resource is the only place that sees every write in order, so the final check must happen there.

That leads to a design rule that the rest of this page keeps returning to:

Use the lock to make conflicts rare. Use the resource to make conflicts harmless.

For efficiency locks you can stop after the first sentence. For correctness locks you need both.

Why Distributed Locks Are Hard

A correct distributed lock must satisfy three properties:

  1. Mutual exclusion (safety): At most one client holds the lock at any time
  2. Deadlock freedom (liveness): Even if a client that holds the lock crashes, eventually the lock becomes available
  3. Fault tolerance: The lock service survives node failures

The fundamental problem is that in a distributed system, you cannot have all three of the following simultaneously:

  • No clock assumptions — The algorithm works regardless of clock drift or process pauses
  • No network assumptions — The algorithm works regardless of message delays
  • Mutual exclusion — At most one client holds the lock

This is a consequence of the CAP Theorem and the FLP impossibility result. Every practical distributed lock makes compromises.

Clarification

CAP and FLP are related, but the more direct reason is this. In an asynchronous system (see System Models) nobody can tell a crashed lock holder from a slow one. Deadlock freedom therefore needs a timeout that releases the lock from a holder that has gone quiet. A timeout is a guess, so sometimes it fires on a holder that is only paused. That holder wakes up still believing it owns the lock. So "the lock is released if the holder dies" and "at most one client ever acts as holder" cannot both be guaranteed by the lock service alone. Fault-tolerant lock services such as ZooKeeper and etcd solve consensus internally (which is how they get around FLP, using timeouts for liveness), but that only makes the service's own view consistent. It does not stop a paused client from acting on a stale view.

graph TD
    subgraph "What Can Go Wrong"
        A["Client A acquires lock"] --> B["Client A pauses (GC, page fault, swap)"]
        B --> C["Lock expires (TTL)"]
        C --> D["Client B acquires lock"]
        D --> E["Client A resumes, thinks it still has the lock"]
        E --> F["BOTH clients write to shared resource"]
        F --> G["DATA CORRUPTION"]
    end

    style G fill:#f44336,color:white

Single-Node Redis Lock

The simplest approach. Works well when you only have one Redis instance and can tolerate the lock being unavailable if that instance dies.

import redis
import uuid
import time

class RedisLock:
    def __init__(self, redis_client: redis.Redis, resource: str,
                 ttl_ms: int = 10000):
        self.redis = redis_client
        self.resource = f"lock:{resource}"
        self.ttl_ms = ttl_ms
        self.token = str(uuid.uuid4())

    def acquire(self, timeout_ms: int = 5000) -> bool:
        """Try to acquire the lock within timeout."""
        deadline = time.time() + timeout_ms / 1000
        while time.time() < deadline:
            # SET NX with TTL — atomic acquire
            if self.redis.set(self.resource, self.token,
                              nx=True, px=self.ttl_ms):
                return True
            time.sleep(0.01)  # Retry interval
        return False

    def release(self) -> bool:
        """Release the lock only if we still hold it."""
        # Must be atomic: check-and-delete
        script = """
        if redis.call("GET", KEYS[1]) == ARGV[1] then
            return redis.call("DEL", KEYS[1])
        else
            return 0
        end
        """
        result = self.redis.eval(script, 1, self.resource, self.token)
        return result == 1

    def extend(self, additional_ms: int) -> bool:
        """Extend the lock TTL if we still hold it."""
        script = """
        if redis.call("GET", KEYS[1]) == ARGV[1] then
            return redis.call("PEXPIRE", KEYS[1], ARGV[2])
        else
            return 0
        end
        """
        result = self.redis.eval(script, 1, self.resource,
                                  self.token, str(additional_ms))
        return result == 1

The Release Script Is Critical

You must use a Lua script for release. A naive GET + DEL has a race condition: between the GET and the DEL, the lock could expire and another client could acquire it. Your DEL would then delete the other client's lock.

What SET NX PX Actually Guarantees

SET key value NX PX ttl does three things in one atomic command: it creates the key only if it does not exist (NX), stores the caller's random token as the value, and sets an expiry in milliseconds (PX). Before Redis 2.6.12 added these options to SET, people used SETNX followed by EXPIRE, which was two commands, so a crash between them left a lock that never expired. Use the single SET.

On one Redis node, this gives you:

PropertyHolds?Notes
Only one client's SET NX succeeds while the key existsYesRedis runs commands one at a time on its main thread
Lock is freed if the holder crashesYes, after the TTLHolder cannot release, so others wait up to the full TTL
Only the holder can releaseYes, with the token-checking Lua scriptWithout it, a late client deletes someone else's lock
Lock survives a Redis restartOnly with persistenceWith no AOF, or AOF with everysec fsync, a crash can lose the key
Lock survives failover to a replicaNoReplication is asynchronous. A lock written to the primary and not yet copied is gone after promotion.
Holder knows when its lock expiredNoThe holder only knows when it acquired the lock, from its own clock

The failover row is the one that surprises people. Redis Sentinel and Redis Cluster both promote replicas automatically, and neither waits for a lock key to reach the replica. The timeline is short and ugly:

sequenceDiagram
    participant A as "Client A"
    participant P as "Primary"
    participant R as "Replica"
    participant B as "Client B"
    A->>P: SET lock tokenA NX PX 30000
    P-->>A: OK
    Note over P,R: async replication has not sent the key yet
    Note over P: primary crashes
    Note over R: Sentinel promotes replica to primary
    B->>R: SET lock tokenB NX PX 30000
    R-->>B: OK, the key does not exist here
    Note over A,B: both A and B believe they hold the lock

The WAIT command, which blocks until a write reaches N replicas, narrows the window but does not close it. It does not make replication synchronous, and a replica that acknowledged can still lose the write. This failure is the main motivation for Redlock's design of independent masters with no replication between them.

Renewal and Watchdogs

A TTL has to be long enough to cover the critical section and short enough that a crashed holder does not block everyone for long. Those goals conflict when the work takes an unpredictable amount of time. The usual answer is to renew.

  • Take the lock with a moderate TTL, for example 30 seconds.
  • A background task extends it (with the token-checking PEXPIRE script, as extend() above does) every third of the TTL while the work runs.
  • If an extension fails because the token no longer matches, the holder must stop work at once, because someone else may now hold the lock.

Redisson, the popular Java Redis client, builds this in as a "lock watchdog". By default, a lock taken without an explicit lease time gets a 30-second expiry that the watchdog renews every 10 seconds for as long as the owning client is alive.

Renewal reduces how often a lock expires under a live holder. It does not remove the pause problem. If the process freezes for 40 seconds, the watchdog freezes with it, the lock expires, and the process resumes believing it still holds the lock. Renewal turns "the work took longer than expected" from a common failure into a rare one. Only fencing makes the rare case safe.

Redis Redlock Algorithm

Redlock, proposed by Salvatore Sanfilippo (antirez), extends the single-node lock to multiple independent Redis instances. The idea is to acquire the lock on a majority of N independent Redis masters (not replicas).

The Algorithm

Given N Redis instances (recommended N=5):

sequenceDiagram
    participant Client
    participant R1 as Redis 1
    participant R2 as Redis 2
    participant R3 as Redis 3
    participant R4 as Redis 4
    participant R5 as Redis 5

    Client->>Client: Generate unique token, record start time
    Client->>R1: SET lock:resource token NX PX 10000
    Client->>R2: SET lock:resource token NX PX 10000
    Client->>R3: SET lock:resource token NX PX 10000
    Client->>R4: SET lock:resource token NX PX 10000
    Client->>R5: SET lock:resource token NX PX 10000

    R1-->>Client: OK
    R2-->>Client: OK
    R3-->>Client: FAIL (another client holds it)
    R4-->>Client: OK
    R5-->>Client: OK

    Client->>Client: 4/5 acquired ≥ majority (3) ✓
    Client->>Client: elapsed time < TTL ✓
    Client->>Client: Lock validity = TTL - elapsed

Step by step:

  1. Get the current time in milliseconds ()
  2. Try to acquire the lock on all N instances sequentially, using the same key and random token, with a small per-instance timeout (e.g., 5-50ms)
  3. Get the current time again (). The lock is acquired if and only if:
    • The lock was acquired on at least instances
    • The total elapsed time () is less than the lock TTL
  4. The effective lock validity time is:
  5. If the lock was not acquired, release it on all instances
import time
import uuid
import redis

class Redlock:
    CLOCK_DRIFT_FACTOR = 0.01  # 1% clock drift
    RETRY_DELAY_MS = 200
    RETRY_COUNT = 3

    def __init__(self, redis_instances: list[redis.Redis], ttl_ms: int = 10000):
        self.instances = redis_instances
        self.quorum = len(redis_instances) // 2 + 1
        self.ttl_ms = ttl_ms

    def acquire(self, resource: str) -> dict | None:
        token = str(uuid.uuid4())
        key = f"lock:{resource}"

        for attempt in range(self.RETRY_COUNT):
            acquired_count = 0
            start_time = time.monotonic()

            for instance in self.instances:
                try:
                    if instance.set(key, token, nx=True, px=self.ttl_ms):
                        acquired_count += 1
                except redis.RedisError:
                    pass

            elapsed_ms = (time.monotonic() - start_time) * 1000
            drift = self.ttl_ms * self.CLOCK_DRIFT_FACTOR + 2
            validity = self.ttl_ms - elapsed_ms - drift

            if acquired_count >= self.quorum and validity > 0:
                return {
                    "resource": resource,
                    "token": token,
                    "validity_ms": validity,
                }

            # Failed — release all
            self._release_all(key, token)
            time.sleep(self.RETRY_DELAY_MS / 1000 * (0.5 + 0.5 *
                       __import__('random').random()))

        return None

    def release(self, lock_info: dict) -> None:
        key = f"lock:{lock_info['resource']}"
        self._release_all(key, lock_info['token'])

    def _release_all(self, key: str, token: str) -> None:
        script = """
        if redis.call("GET", KEYS[1]) == ARGV[1] then
            return redis.call("DEL", KEYS[1])
        else
            return 0
        end
        """
        for instance in self.instances:
            try:
                instance.eval(script, 1, key, token)
            except redis.RedisError:
                pass

Martin Kleppmann's Critique of Redlock

In his influential 2016 blog post "How to do distributed locking," Martin Kleppmann identified fundamental problems with Redlock. His critique is essential reading for anyone building distributed locks.

The GC Pause Problem

Time →
Client A:  [acquire lock]............[GC PAUSE 15 sec]............[write data]
                                                                      ↑
Lock:      [████████ VALID ████████][EXPIRED]                    A thinks it
                                                                 still has lock
Client B:                           [acquire lock][write data]
                                                       ↑
                                                  B also writes

Result: BOTH clients wrote. Mutual exclusion violated.

The problem is not specific to GC — any process pause (page faults, thrashing, CPU scheduling) can cause a client to hold a lock past its expiry without realizing it. And there is no way for the client to detect this after the pause ends, because it cannot know how long it was paused.

The Clock Jump Problem

Redlock depends on clock accuracy. If a Redis node's clock jumps forward (due to NTP correction, VM migration, or operator error), its locks expire prematurely, breaking the quorum guarantee.

Kleppmann's Solution: Fencing Tokens

The fundamental fix is to not rely on the lock for safety at all. Instead, use the lock for efficiency (preventing duplicate work) and use fencing tokens for safety:

sequenceDiagram
    participant C1 as Client A
    participant Lock as Lock Service
    participant Store as Storage

    C1->>Lock: Acquire lock
    Lock-->>C1: Lock granted, fence_token = 34

    Note over C1: GC PAUSE
    Note over Lock: Lock expires

    participant C2 as Client B
    C2->>Lock: Acquire lock
    Lock-->>C2: Lock granted, fence_token = 35
    C2->>Store: Write (fence_token = 35)
    Store-->>C2: OK (stored max_token = 35)

    Note over C1: Resumes from GC
    C1->>Store: Write (fence_token = 34)
    Store-->>C1: REJECTED (34 < 35)
The lock can't stop a paused client from waking up and writing. The storage can: it remembers the highest token it has seen and rejects any write with a lower one, so A's stale write with 34 bounces off B's 35.
class FencedLock:
    """Lock with monotonically increasing fencing tokens."""

    def __init__(self, lock_service, resource: str):
        self.lock_service = lock_service
        self.resource = resource
        self.fence_token: int | None = None

    def acquire(self) -> int:
        """Returns a fencing token (monotonically increasing integer)."""
        lock_info = self.lock_service.acquire(self.resource)
        if lock_info is None:
            raise LockAcquisitionError("Could not acquire lock")
        self.fence_token = lock_info['fence_token']
        return self.fence_token

    def write_with_fence(self, storage, key: str, value: any) -> bool:
        """Write to storage, rejected if a higher fence token has been seen."""
        return storage.conditional_write(key, value, self.fence_token)


class FencedStorage:
    """Storage that rejects writes with stale fencing tokens."""

    def __init__(self):
        self.data = {}
        self.max_tokens = {}  # Per-resource max fence token seen

    def conditional_write(self, key: str, value: any,
                          fence_token: int) -> bool:
        max_seen = self.max_tokens.get(key, 0)
        if fence_token < max_seen:
            return False  # Stale lock holder
        self.max_tokens[key] = fence_token
        self.data[key] = value
        return True

The Real Lesson

If you need the lock for correctness (not just efficiency), you must use fencing tokens. If your storage system does not support fencing, your distributed lock cannot guarantee mutual exclusion in the presence of process pauses, regardless of which lock algorithm you use.

The Redlock Debate, Presented Fairly

Kleppmann's post drew a direct response from Salvatore Sanfilippo, "Is Redlock safe?", published shortly afterwards, and then a further round of replies from both sides. The exchange is often summed up as "Redlock is broken", but the two authors agreed on more than that suggests, and the real disagreement is narrower and more interesting.

Kleppmann's argument

  1. Timing assumptions. Redlock is safe only if network delays, process pauses and clock errors are all small compared to the TTL. That is a synchronous model with bounded delays. Real systems violate those bounds now and then: GC pauses of seconds, packets delayed in queues, VM migrations, clocks stepped by NTP. A safety property that holds "unless timing is bad" is not one you can rely on for correctness.
  2. Clock jumps on the lock servers. Expiry is driven by each Redis node's clock. If one node's clock jumps forward, its copy of the lock expires early. Example: client A holds the lock on nodes 1, 2 and 3. Node 3's clock jumps and its key expires. Client B gets nodes 3, 4 and 5. Two clients now each have a majority.
  3. No fencing. Redlock's token is a random value, not a number that increases with every grant. So the storage cannot order two holders and reject the older one, which means the standard fix for pauses is not available.
  4. Wrong tool either way. For efficiency, one Redis node is simpler and good enough, so five nodes are wasted cost. For correctness, Redlock is not safe enough. So it is in an awkward middle ground.

Sanfilippo's response

  1. Pauses hit every lock with expiry. A process that pauses after acquiring a lock and then acts after the lease has ended is a problem for any auto-releasing lock, including ZooKeeper's session-based one. That is not specific to Redlock. Redlock does check the time elapsed during acquisition and discards the lock if it took too long, which handles delays before the lock is granted.
  2. Bounded drift, not synchronised clocks. Redlock assumes that clocks on different nodes advance at about the same rate, not that they show the same time. He argued that this is a reasonable assumption for real hardware, and that the clock-jump scenario comes from operational mistakes (letting NTP step clocks, or an admin changing the time), which should be prevented by configuration.
  3. Fencing with unique tokens. If the resource can check tokens, a random unique token can be used with compare-and-set: the resource stores the token of the current writer and refuses writes from a token that does not match. He also argued that a system able to do that kind of check can often avoid needing a distributed lock at all.
  4. Restarts. To cover a node that crashes and loses its keys (no fsync), Redlock recommends delayed restarts: keep a crashed node out for at least one maximum TTL, so any lock it held has expired everywhere before it rejoins.

Where they agree and where they do not

PointStatus
A paused client can outlive its lockAgreed. It applies to any lock with expiry.
Correctness needs a check at the resourceBroadly agreed. They differ on whether ordered tokens are required.
Single Redis is fine for efficiency locksAgreed
Redlock's safety depends on timing assumptionsAgreed that it does. They disagree about whether those assumptions are acceptable.
Random tokens are as good as increasing tokensDisputed. Kleppmann: an ordered token lets any storage reject the older holder even after the newer one finishes. A random token needs a compare-and-set against the current holder's token, which is a stronger requirement on the storage.
Clock jumps are a real riskDisputed. Operational discipline vs "it will happen in production eventually".

What practitioners took from it

  • For efficiency, use one Redis node with SET NX PX and a token-checked release. Redlock's extra nodes buy availability of the lock service, not safety of the protected resource.
  • For correctness, do not rely on any lock alone. Use a consensus-backed lock service (etcd, ZooKeeper, Consul) for good behaviour under failover, and add fencing or compare-and-set at the resource for safety.
  • The debate made the efficiency-versus-correctness split standard vocabulary, and made "where is the fencing check?" the first question in any design review that involves a lock.

The general principle that came out of it matches what consensus protocols already do: timing assumptions are acceptable for liveness (progress), but safety should not depend on them.

Further reading on the debate

Read both posts in order: Kleppmann, "How to do distributed locking" (February 2016), and Sanfilippo, "Is Redlock safe?" (February 2016). The Redis documentation page on distributed locks links to both.

ZooKeeper Distributed Locks

ZooKeeper provides stronger guarantees than Redis for locking because it uses consensus (ZAB protocol) for replication and supports ephemeral sequential nodes.

ZooKeeper Lock Recipe

sequenceDiagram
    participant C1 as Client A
    participant ZK as ZooKeeper
    participant C2 as Client B

    C1->>ZK: Create /locks/resource/lock-0001 (ephemeral, sequential)
    C2->>ZK: Create /locks/resource/lock-0002 (ephemeral, sequential)

    C1->>ZK: Get children of /locks/resource
    ZK-->>C1: [lock-0001, lock-0002]
    C1->>C1: My node (0001) is lowest → I have the lock

    C2->>ZK: Get children of /locks/resource
    ZK-->>C2: [lock-0001, lock-0002]
    C2->>C2: My node (0002) is NOT lowest
    C2->>ZK: Watch lock-0001 (the node just before mine)

    Note over C1: Client A finishes work
    C1->>ZK: Delete /locks/resource/lock-0001
    ZK-->>C2: Watch triggered: lock-0001 deleted
    C2->>ZK: Get children of /locks/resource
    ZK-->>C2: [lock-0002]
    C2->>C2: My node is now lowest → I have the lock
import org.apache.zookeeper.*;
import java.util.*;

public class ZooKeeperLock {
    private final ZooKeeper zk;
    private final String lockPath;
    private String myNode;

    public ZooKeeperLock(ZooKeeper zk, String resource) {
        this.zk = zk;
        this.lockPath = "/locks/" + resource;
    }

    public void acquire() throws Exception {
        // Ensure parent path exists
        ensurePath(lockPath);

        // Create ephemeral sequential node
        myNode = zk.create(
            lockPath + "/lock-",
            new byte[0],
            ZooDefs.Ids.OPEN_ACL_UNSAFE,
            CreateMode.EPHEMERAL_SEQUENTIAL
        );

        while (true) {
            List<String> children = zk.getChildren(lockPath, false);
            Collections.sort(children);

            String myNodeName = myNode.substring(
                myNode.lastIndexOf('/') + 1);

            if (children.get(0).equals(myNodeName)) {
                return; // I have the lock
            }

            // Watch the node just before mine
            int myIndex = children.indexOf(myNodeName);
            String watchNode = lockPath + "/" + children.get(myIndex - 1);

            final Object lock = new Object();
            Stat stat = zk.exists(watchNode, event -> {
                synchronized (lock) {
                    lock.notifyAll();
                }
            });

            if (stat != null) {
                synchronized (lock) {
                    lock.wait(); // Wait for predecessor to be deleted
                }
            }
            // Loop back and check again
        }
    }

    public void release() throws Exception {
        if (myNode != null) {
            zk.delete(myNode, -1);
            myNode = null;
        }
    }
}

Key advantages over Redis:

PropertyRedis (Redlock)ZooKeeper
ConsensusNo (independent nodes)Yes (ZAB protocol)
Ephemeral locksTTL-based (clock-dependent)Session-based (heartbeat)
OrderingNo guaranteed orderingSequential nodes → FIFO ordering
Watch mechanismPub/Sub (not atomic)Native watches (atomic with read)
Fencing tokensNot built inSequential node number = fence token

ZooKeeper Is Not Immune Either

ZooKeeper makes the lock service itself consistent, but a client can still act on a lock it has lost. The lock is an ephemeral znode tied to the client's session. The session stays alive while the client heartbeats. If ZooKeeper hears nothing for the session timeout (negotiated between 2 and 20 times the server's tickTime), it expires the session and deletes the ephemeral nodes. The problem is the client's view during a partition:

sequenceDiagram
    participant A as "Client A"
    participant ZK as "ZooKeeper ensemble"
    participant B as "Client B"
    A->>ZK: holds /locks/job/lock-0007
    Note over A,ZK: network partition starts, heartbeats stop
    Note over A: connection lost, state SUSPENDED, A does not know if the session survives
    Note over ZK: session timeout passes, session expired, lock-0007 deleted
    B->>ZK: lock-0008 is now lowest
    ZK-->>B: B holds the lock
    Note over A: A is still running its critical section
    Note over A,ZK: partition heals
    ZK-->>A: session expired
    Note over A: only now does A learn it lost the lock

Apache Curator, the standard Java client library, models this with connection states: SUSPENDED when the connection drops, and LOST when the session is known to be gone (or, in newer versions, when the session timeout has passed since suspension). The correct behaviour is to treat SUSPENDED as "you may have lost the lock" and pause work at once, rather than waiting for LOST. That narrows the window, but it does not close it: a long GC pause in client A prevents it from noticing anything. So the conclusion is the same as for Redis. ZooKeeper gives you a far better lock service, and you still need fencing for correctness.

The token to fence with: each znode carries a czxid, the transaction id of the write that created it. zxids increase across the whole ensemble, so the creating zxid of the lock node is a valid fencing token. The sequential suffix also increases, but only within one parent node.

etcd Distributed Locks

etcd uses Raft consensus and provides native lease-based locking:

package main

import (
	"context"
	"log"
	"time"

	clientv3 "go.etcd.io/etcd/client/v3"
	"go.etcd.io/etcd/client/v3/concurrency"
)

func main() {
	cli, err := clientv3.New(clientv3.Config{
		Endpoints:   []string{"localhost:2379"},
		DialTimeout: 5 * time.Second,
	})
	if err != nil {
		log.Fatal(err)
	}
	defer cli.Close()

	// Create a session with a 10-second TTL
	session, err := concurrency.NewSession(cli,
		concurrency.WithTTL(10))
	if err != nil {
		log.Fatal(err)
	}
	defer session.Close()

	// Create a distributed lock
	mutex := concurrency.NewMutex(session, "/locks/my-resource")

	// Acquire the lock
	ctx, cancel := context.WithTimeout(context.Background(),
		5*time.Second)
	defer cancel()

	if err := mutex.Lock(ctx); err != nil {
		log.Fatal("failed to acquire lock:", err)
	}
	log.Println("Lock acquired!")

	// The lock's revision acts as a fencing token
	log.Printf("Fencing token (revision): %d", mutex.Header().Revision)

	// Do work under the lock...
	time.Sleep(2 * time.Second)

	// Release the lock
	if err := mutex.Unlock(context.TODO()); err != nil {
		log.Fatal("failed to release lock:", err)
	}
	log.Println("Lock released")
}

etcd's lease mechanism is superior to TTL-based expiry because the session automatically renews the lease through keepalive RPCs. If the client process dies, the session ends, and the lock is released within one TTL period.

How the etcd Mutex Works

etcd's concurrency.Mutex is a small recipe on top of three etcd primitives: leases, revisions, and transactions.

  1. The session creates a lease with a TTL and keeps it alive in the background.
  2. Lock() writes a key /locks/my-resource/<lease-id> attached to that lease, in a transaction that only creates it if it does not already exist.
  3. Every key in etcd has a create revision, the global revision number of the write that created it. Revisions increase monotonically across the entire cluster.
  4. The waiter with the lowest create revision under the prefix holds the lock. Every other waiter watches the key with the next-lower create revision and waits for it to be deleted, the same "watch your predecessor" trick as ZooKeeper, which avoids waking every waiter at once (the herd effect).
  5. If the holder's lease expires (missed keepalives), etcd deletes its key, and the next waiter's watch fires.

Because revisions are global and increasing, the lock key's create revision is a natural fencing token. etcd also lets you turn the token check into a guard on etcd writes themselves. A transaction can say "only apply this write if my lock key still exists with my create revision", which makes writes to etcd safe even if the client was paused:

// Write only while we still own the lock. If our lease expired and the
// key was deleted, the comparison fails and nothing is written.
// IsOwner() is the comparison CreateRevision(myKey) == myRev.
txn := cli.Txn(ctx).
	If(mutex.IsOwner()).
	Then(clientv3.OpPut("/config/leader-settings", newValue))
resp, err := txn.Commit()
if err != nil {
	return err
}
if !resp.Succeeded {
	return errors.New("lost the lock before the write, not applied")
}

That only protects data stored in etcd. For writes to anything else, such as a database, a file system or an external API, the token has to travel with the write and be checked there.

Fencing Tokens in Practice

The fencing idea is easy to state: every lock grant comes with a number that is larger than every earlier grant's number, and the resource rejects any write carrying a number smaller than the largest it has seen. Making it work end to end requires answering three questions.

Where the token comes from

SourceMonotonic?Notes
etcd create revision of the lock keyYes, cluster-wideFree with the etcd mutex
ZooKeeper czxid of the lock znodeYes, ensemble-wideOr the sequence number, which is per parent
A database sequence, incremented in the same transaction that records the lockYesSimple if the lock already lives in Postgres or MySQL
Redis INCR alongside SET NXOnly until failoverAsync replication can lose increments and hand out a number twice
Random UUID (Redlock's token)NoWorks only with compare-and-set against the current holder, not with "reject older"
Wall-clock timestampNoClocks jump, and two clients can read the same millisecond

Where the check happens

The check must be made by the resource, atomically with the write. In a SQL database that means one conditional statement:

-- One row per protected resource remembers the highest token seen.
CREATE TABLE fence (
    resource   text PRIMARY KEY,
    max_token  bigint NOT NULL
);

-- The write and the fence check happen in one transaction.
BEGIN;
UPDATE fence
   SET max_token = $1
 WHERE resource = 'invoice-batch-2026-11'
   AND max_token <= $1;          -- reject writers older than the newest seen
-- If 0 rows were updated, a newer holder exists: ROLLBACK and stop.
UPDATE invoices SET status = 'sent' WHERE batch = '2026-11';
COMMIT;

The UPDATE ... WHERE max_token <= $1 takes a row lock on the fence row, so two concurrent writers are serialised by the database, and the older one either waits and then fails the condition, or commits first and is then overtaken. Either way no write from an older token lands after a newer token has written.

The same pattern works elsewhere:

  • Object storage: a conditional write (for example an If-Match on the ETag of a manifest object) gives compare-and-set, which is the unique-token variant of fencing.
  • Kafka: idempotent and transactional producers carry a producer epoch. When a new instance with the same transactional.id starts, the broker bumps the epoch and rejects the old instance as a "zombie". That is fencing built into the protocol.
  • Raft and Paxos: every message carries a term or ballot number, and replicas reject messages from older terms. That is fencing at the heart of consensus. See Leases and Fencing.

What fencing cannot protect

Fencing needs a resource that can say no. Some cannot:

  • Sending an email, charging a card through a third-party API, or firing a webhook. The provider does not know your token. Here, use the provider's idempotency key feature, and derive the key from the job rather than the lock holder, so that two holders send the same key and the provider deduplicates.
  • Writing to a plain file on a shared disk with no conditional write support.
  • Anything where the side effect happens before the write that could be fenced.

If the resource cannot check, the lock is an efficiency lock whatever you call it, and the design must tolerate an occasional duplicate.

Lock-Service Failure Timelines

Most lock bugs only show up under specific timing. Walking through the timelines is the fastest way to see which guarantees survive which failures. In every row below, assume a 30-second TTL or session timeout.

FailureSingle RedisRedlock (5 nodes)etcd / ZooKeeperWith fencing at the resource
Holder crashesFreed after up to 30 sFreed after up to 30 sFreed after lease or session timeoutSame, and no stale writes possible
Holder paused 45 s (GC, VM stall)Two holdersTwo holdersTwo holdersStale write rejected
Lock server crashes, no persistenceLock lost, second holder possibleSafe if a majority survives and restarts are delayedSafe, data is replicated by consensusSafe
Primary fails over to replicaLock can be lostNot applicable (no replicas)Safe, a new leader has every committed writeSafe
Clock jumps forward on a lock serverEarly expiry, two holdersEarly expiry on one node can break the majorityLease timing is driven by the leader, and a jump can shorten a lease, but the key's revision still orders holdersSafe
Network partition isolates holderHolder keeps working until it noticesSameSame, until SUSPENDED or expiry is seenStale write rejected
Lock server overloaded, slow repliesAcquire latency risesAcquisition may exceed validity and be discardedKeepalives may be missed, leases expireSafe, but throughput drops

Read down the columns and the pattern is clear. Better lock services fix the rows about the lock service's own failures (crashes, failover). Only fencing fixes the rows about the client's failures (pauses, partitions). Both kinds happen in production.

A concrete timeline for the most common real incident, a long pause during a batch job:

gantt
    title Lease expiry under a stop-the-world pause, 30 s TTL
    dateFormat ss
    axisFormat %S s
    section Client A
    Holds lock, working         :a1, 00, 10s
    Paused by GC               :crit, a2, 10, 35s
    Resumes and writes          :a3, 45, 5s
    section Lock service
    Lease valid for A           :l1, 00, 30s
    Lease valid for B           :l2, 31, 19s
    section Client B
    Waiting                     :b1, 00, 31s
    Holds lock, writes          :b2, 31, 19s

From second 45 to second 50 both clients write. Without fencing, whichever write lands last wins, and the data is a mix of both. With fencing, A's writes carry the older token and are rejected from second 45 on.

Other Lock Backends

Redis, ZooKeeper and etcd are not the only options, and the database you already run is often the best lock service you have.

BackendMechanismStrengthsWatch out for
PostgreSQL advisory lockspg_advisory_lock(key) (session) or pg_advisory_xact_lock(key) (released at transaction end)No new infrastructure. Transaction-scoped locks are released automatically on commit or rollback.Session locks leak through connection poolers like PgBouncer in transaction mode. The lock is only as available as the primary.
SELECT ... FOR UPDATE on a rowRow lock held for the transactionThe lock and the write are in the same transaction, so the database does the fencing for youLong transactions hold locks and block vacuum
DynamoDB conditional writesPut an item with attribute_not_exists, store a lease duration and a version, refresh by conditional updateServerless, highly availableExpiry is decided by clients reading the lease, so client clocks and pauses matter
Consul sessions and KVacquire on a key bound to a session with health checksIntegrates with service healthSame pause problem as any session-based lock
Kubernetes Lease objectsOptimistic updates on a coordination.k8s.io Lease resourceAlready there for controllers, used by the built-in leader electionBuilt for leader election with second-scale timing, not high-rate locking

The row-lock option is worth stressing. If the protected resource is a database row, a transaction with SELECT ... FOR UPDATE (or an optimistic version check) gives you mutual exclusion and the guarantee that stale writers cannot commit, with no separate lock service at all.

When to Avoid Locks

A distributed lock is a serialisation point, a failure domain, and a source of subtle bugs. Many problems that look like they need one can be solved more simply.

ProblemInstead of a lockHow it works
Prevent double processing of a message or requestIdempotency key plus a unique constraintInsert (idempotency_key) into a table with a unique index. The second attempt fails the insert and returns the saved result.
Two writers updating the same recordOptimistic concurrencyStore a version column and UPDATE ... WHERE id = ? AND version = ?. The loser retries with fresh data.
Only one consumer per entityPartition by keyRoute all events for an entity to one Kafka partition. One consumer owns each partition at a time, and the group protocol's generation numbers fence old owners.
Only one instance runs a cron jobA scheduler with single-run semantics, or leader electionKubernetes CronJob with concurrencyPolicy: Forbid, or elect a leader once instead of locking per job
Counter or set updated by many writersAtomic operations or CRDTsINCR, UPDATE x = x + 1, or a CRDT that merges concurrent updates
Stock reservationConditional decrementUPDATE stock SET qty = qty - 1 WHERE sku = ? AND qty > 0
Exclusive ownership for a long time (minutes to hours)Leader election with leases and fencingSee Leader Election and Leases and Fencing

A good habit in design reviews: when someone proposes a distributed lock, ask "what is the resource, and can it check a condition on write?" If it can, a conditional write is usually simpler and safer than the lock. If it cannot, the lock can only ever be an efficiency lock, and the design has to tolerate duplicates.

Signs that a lock is the wrong tool:

  • High contention. If many clients queue for the same lock all the time, the lock has become a global serialisation point. Shard the resource or redesign so that writers do not conflict.
  • Long hold times. A lock held for minutes is really a lease on ownership, and should be designed as leader election, with fencing.
  • Locks across several resources. Taking locks on A and then B in different orders across clients can deadlock, and a distributed deadlock with TTLs turns into a livelock of timeouts. Order lock acquisition consistently, or use a transaction.

When to Use Which Lock

ScenarioRecommended ApproachWhy
Efficiency (prevent duplicate work)Single Redis node + TTLSimple, fast, failures cause at most double work
Correctness (prevent data corruption)ZooKeeper/etcd + fencing tokensConsensus-backed, sequential ordering
Already running RedisSingle Redis lock (not Redlock)Simpler, same practical guarantees as Redlock
Already running ZooKeeperZooKeeper lock recipeBattle-tested, well-understood semantics
Already running etcdetcd concurrency APINative support, Raft-backed
Need orderingZooKeeper or etcdSequential nodes provide fair ordering

Kleppmann's Advice (paraphrased)

If you need locks only for efficiency, use a single Redis instance with automatic lock expiration. If you need locks for correctness, do not use Redlock — use a proper consensus system like ZooKeeper, and use fencing tokens.

Correction: an earlier version of this box put the advice in quotation marks. It is a faithful summary of the conclusion of Kleppmann's 2016 post, not a verbatim quote. In the original, he recommends the single-node Redis algorithm for efficiency locks and, for correctness, a consensus system such as ZooKeeper, probably through one of the Curator lock recipes, with fencing.

The "Already running Redis" row reflects Kleppmann's view. Sanfilippo would add that Redlock gives a lock service that survives the loss of a minority of Redis nodes, which a single instance does not. Both are true. The question is whether that availability is worth five independent Redis deployments when it does not change the safety story for the protected resource.

Lock Safety Checklist

Before deploying distributed locks in production, verify:

  1. Do you actually need a distributed lock? Could you use an idempotency key or database constraint instead?
  2. Is the lock for efficiency or correctness? This determines your entire approach
  3. Do you have fencing tokens? If correctness matters, your storage must reject stale writes
  4. What happens when the lock holder crashes? TTL/lease expiry must release the lock
  5. What happens during network partitions? Can two clients both believe they hold the lock?
  6. Is your clock reliable? If using TTL-based locks, NTP drift can violate assumptions
  7. What is your lock contention pattern? High contention may indicate a design problem
  8. Does the holder stop when renewal fails? A failed extension, a SUSPENDED ZooKeeper connection, or a lost etcd lease must stop the critical section, not just log a warning
  9. Is the TTL tied to real work time? Measure the p99.9 of the critical section and set the TTL with headroom, or use renewal
  10. Is the release token-checked? Never DEL a lock key without comparing the token

War Story (illustrative)

A billing team used a Redis lock to ensure only one worker generated each customer's monthly invoice. The lock had a 60-second TTL. The invoice generator usually finished in 5 seconds and then wrote the invoice and emailed it. One month, a large customer's invoice took 70 seconds, because its usage data had grown tenfold and nobody had looked at the critical-section time since launch. The lock expired at 60 seconds, a second worker picked up the same job, and both workers wrote invoices and sent emails. The customer got two invoices with different numbers, and their accounts-payable system paid both. The fix had three parts. Each invoice row got a unique constraint on (customer_id, billing_period), so the second insert failed. The email provider was called with an idempotency key derived from the invoice id. The lock stayed, but only as an efficiency measure, with a watchdog renewal. After that, the lock's only job was to stop two workers from wasting time on the same customer. Correctness came from the database and the email provider.

Interview Questions

1. What is the difference between an efficiency lock and a correctness lock?

An efficiency lock prevents duplicate work, and an occasional double run costs only time or money. A correctness lock prevents corruption, and a single failure of mutual exclusion is a bug. Efficiency locks can use a single Redis node. Correctness locks need a check at the resource (fencing or compare-and-set), because no lock service can stop a paused client from acting.

2. Why is SET key value NX PX ttl with a random value better than SETNX then EXPIRE?

It is atomic, so a crash cannot leave a key with no expiry, and the random value lets the release script confirm that the caller still holds the lock before deleting it.

3. Explain how a GC pause breaks a lease-based lock.

The client acquires a lock with a 30-second lease, then pauses for 40 seconds. The lease expires, another client acquires the lock and writes, then the first client resumes and writes too, still believing it holds the lock. It cannot detect the pause after the fact. Fencing tokens make the resource reject its write.

4. Summarise the Redlock debate.

Kleppmann argued that Redlock depends on bounded delays, pauses and clock drift, has no monotonic fencing token, and is therefore unsafe for correctness and unnecessary for efficiency. Sanfilippo replied that pauses after acquisition affect every expiring lock, that Redlock only assumes bounded clock drift rate, that unique tokens with compare-and-set can fence, and that delayed restarts cover crash recovery. They agree that correctness needs a check at the resource, and differ over whether Redlock's timing assumptions are acceptable.

5. How does a ZooKeeper lock avoid the herd effect?

Each waiter creates an ephemeral sequential node and watches only the node immediately before it. When the holder releases, only the next waiter is notified, instead of every waiter re-reading the children.

6. Does ZooKeeper make fencing unnecessary?

No. A client partitioned from ZooKeeper, or paused, can keep working after its session has expired and its lock node is gone. ZooKeeper fixes failures of the lock service, not of the client. Use the lock node's czxid as a fencing token.

7. Where would you get a fencing token, and how would a Postgres-backed resource check it?

From the etcd lock key's create revision, the ZooKeeper node's czxid, or a database sequence. In Postgres, keep a fence row with max_token and run UPDATE fence SET max_token = $t WHERE resource = $r AND max_token <= $t in the same transaction as the write, rolling back if no row was updated.

8. A Redis primary with a lock key fails over to a replica. What can go wrong?

Replication is asynchronous, so the key may not have reached the replica. After promotion the lock does not exist, and a second client can acquire it while the first still believes it holds it. WAIT narrows but does not close the window.

9. Give three alternatives to a distributed lock for preventing double processing.

A unique constraint on an idempotency key, optimistic concurrency with a version column, and partitioning so that one consumer owns each key (for example a Kafka partition). Conditional decrements and CRDTs cover counters and stock.

10. Your lock protects calls to a third-party payment API that does not know about your tokens. How do you make it safe?

You cannot fence the API, so the lock can only reduce duplicates. Make the call idempotent with the provider's idempotency-key feature, using a key derived from the business operation (for example the order id), so two holders send the same key and the provider charges once.

Further Reading

  • CAP Theorem — Why you cannot have both safety and liveness
  • Consistency Models — Linearizability and its relationship to locking
  • Leader Election — Related coordination problem
  • Distributed Transactions — Alternative approaches to distributed coordination
  • Raft Full Walkthrough — The consensus protocol behind etcd locks
  • Martin Kleppmann, "How to do distributed locking" (2016)
  • Salvatore Sanfilippo, "Is Redlock safe?" (2016)
  • Apache Curator documentation — Production ZooKeeper lock recipes
  • Martin Kleppmann, Designing Data-Intensive Applications, Chapter 8