ZooKeeper: A Coordination Service
Almost every distributed system needs a small amount of data that everyone must agree on: who the current leader is, which servers are alive, what the current configuration is, who holds a lock. That data is tiny — kilobytes — but getting it wrong causes split brain, double processing and lost writes. In the mid-2000s, each team at Yahoo! was building its own fragile version of this on top of databases, heartbeats and timeouts.
ZooKeeper was Yahoo!'s answer: a single, replicated, highly available service whose only job is coordination. Patrick Hunt, Mahadev Konar, Flavio Junqueira and Benjamin Reed described it in "ZooKeeper: Wait-free coordination for Internet-scale systems" at USENIX ATC 2010. It became an Apache top-level project and the coordination backbone of Hadoop, HBase, Kafka (until KRaft), Solr, Storm, Druid, Pulsar and many more.
This page covers the design from the paper and the Apache ZooKeeper documentation (Programmer's Guide, Administrator's Guide, Recipes and Solutions). The replication protocol itself, ZAB, has its own page: ZAB: ZooKeeper Atomic Broadcast. Here we only summarise it and focus on what ZooKeeper offers to the people building on it.
1. The problem and its constraints
Why a coordination service?
You can build leader election or locking into each application with Paxos or Raft. In 2007 that was a research project, not a library call. The ZooKeeper authors observed that the same few primitives kept showing up — configuration, group membership, leader election, locks, rendezvous, barriers — and that each team implementing them got the corner cases wrong in different ways.
Google had answered the same problem with Chubby (Burrows, OSDI 2006): a lock service with a file-system-like namespace. ZooKeeper's authors made a different choice, and it's the core idea of the paper:
Rather than implement specific primitives such as locks on the server, expose an API that lets clients build those primitives, using wait-free data objects.
"Wait-free" means no ZooKeeper operation ever blocks waiting for another client. There is no lock() call that hangs until someone else releases. Every call returns promptly; clients combine ordered writes, sequence numbers, ephemeral nodes and watches (notifications) to implement blocking behaviour themselves. That keeps the server simple and fast, and means one slow or faulty client can't make the service stall for others.
Constraints the design works within
| Constraint | Design consequence |
|---|---|
| Coordination data is small (configuration, membership, lock state) | The whole data tree lives in memory on every server; znode data is limited to about 1 MB by default |
| Reads vastly outnumber writes in coordination workloads (clients poll config, check membership) | Every server answers reads locally without talking to the others; writes go through a leader and a quorum |
| Clients must not see the system go backwards | Every client tracks the last transaction it has seen and won't connect to a server that is behind |
| Clients fail and partition all the time | Sessions with timeouts; ephemeral znodes that disappear when a session dies |
| Polling is wasteful and slow | Watches: one-time notifications pushed to clients |
| Must stay available through failures | Replicated ensemble (typically 3 or 5 servers), majority quorum; survives ⌊(n−1)/2⌋ failures |
One thing it is not built for: bulk data. ZooKeeper is a terrible place to store user records, large blobs or queues with millions of items. That rule, broken often, causes a large share of production ZooKeeper incidents.
2. Architecture
A ZooKeeper deployment is an ensemble of servers. One is the leader, the others are followers; optionally there are observers, which receive updates but don't vote. Every server holds a full copy of the data tree in memory, backed by a transaction log and periodic snapshots on disk.
flowchart TB
subgraph Clients
C1["Client A<br/>(session 0x1a)"]
C2["Client B<br/>(session 0x2b)"]
C3["Client C<br/>(session 0x3c)"]
end
subgraph Ensemble["ZooKeeper ensemble (5 voters)"]
L["Leader<br/>orders all writes, assigns zxid"]
F1["Follower 1"]
F2["Follower 2"]
F3["Follower 3"]
F4["Follower 4"]
O1["Observer<br/>(non-voting, read scaling)"]
end
C1 -->|"TCP session: reads served here"| F1
C2 -->|"TCP session"| L
C3 -->|"TCP session"| O1
F1 -->|"forward write"| L
O1 -->|"forward write"| L
L -->|"PROPOSAL / COMMIT"| F1
L -->|"PROPOSAL / COMMIT"| F2
L -->|"PROPOSAL / COMMIT"| F3
L -->|"PROPOSAL / COMMIT"| F4
L -->|"INFORM (committed txns)"| O1
Each server has the same components, described in Section 4 of the paper:
- Request processor. On the leader, it turns each client write into an idempotent transaction. The client says "
setData(/x, data, version=5)"; the leader checks the current version, computes the resulting state, and produces a transaction like "setDataTXN(/x, data, newVersion=6, mtime=…)" — or anerrorTXNif the version check fails. Because transactions describe the result rather than the request, they can be applied more than once without changing the outcome. That matters during recovery, when ZAB may redeliver transactions. - Atomic broadcast (ZAB). Transactions go through ZAB to every server in the same order.
- Replicated database. An in-memory tree of znodes. Each server logs transactions to disk before applying them, and periodically writes a fuzzy snapshot.
- Client connection handling. Each client connects to one server over TCP. Reads are answered from that server's local replica. Writes are forwarded to the leader. Watches are registered and fired on the server the client is connected to.
Fuzzy snapshots
ZooKeeper snapshots without stopping the world. A snapshot thread walks the tree depth-first, reading each znode's data and metadata atomically, while writes keep arriving. The resulting snapshot may not match any single point in time — some znodes reflect later transactions than others. That's fine because transactions are idempotent: on recovery, a server loads the snapshot and replays every transaction from the log that started after the snapshot began. Replaying a transaction that was already reflected in the snapshot produces the same state, so the final state is correct. The paper gives an example of exactly this: two setData transactions on two znodes interleaving with the snapshot scan, with the replay producing the correct final values.
Why reads are local
Serving reads from the local replica is what lets ZooKeeper scale reads nearly linearly with server count (see the numbers in Section 7). The price is that a read can return stale data — a follower may be a few transactions behind the leader. Section 5 explains exactly what ZooKeeper does and doesn't promise about that, and how sync narrows the gap.
3. Data model
Znodes
ZooKeeper exposes a hierarchical namespace, like a file system: paths such as /app/config or /election/n_0000000007. Each node in the tree is a znode. Unlike a file system, every znode can hold data and children at the same time. Data is an opaque byte array, read and written whole — no partial reads or appends. Data size is limited by jute.maxbuffer (just under 1 MB by default) and should in practice be kept to kilobytes.
Each znode carries a Stat structure:
| Field | Meaning |
|---|---|
czxid / mzxid | zxid of the transaction that created / last modified the znode |
pzxid | zxid of the last change to its children |
ctime / mtime | Wall-clock creation / modification time (informational only) |
version | Number of changes to the data — used for conditional updates |
cversion | Number of changes to the children |
aversion | Number of changes to the ACL |
ephemeralOwner | Session ID of the owner if ephemeral, else 0 |
dataLength, numChildren | Size of data, number of children |
A zxid (ZooKeeper transaction ID) is a 64-bit number: the high 32 bits are the leader's epoch, the low 32 bits a counter within that epoch. Every write gets a unique, increasing zxid, so zxids give a total order of all state changes.
Znode types
The paper defines two kinds of znodes plus a flag:
- Regular (persistent) znodes are created and deleted explicitly by clients.
- Ephemeral znodes are deleted automatically when the session that created them ends (or explicitly). Ephemeral znodes cannot have children.
- The sequential flag, available on both kinds, makes ZooKeeper append a monotonically increasing counter to the name:
create("/q/item-", SEQUENTIAL)produces/q/item-0000000042. The paper's guarantee: the sequence number of a new znode is never smaller than that of any other child already created under the same parent. The counter is the parent'scversion, formatted as ten zero-padded digits, so lexicographic sort order matches creation order.
Later versions added two more types (both in 3.5.x/3.6): container znodes, deleted automatically once their last child is removed (handy as parents for lock and election children), and TTL znodes, persistent nodes that are deleted if they aren't modified within a time-to-live and have no children (disabled by default; enabled with zookeeper.extendedTypesEnabled).
The API
The paper's API (Section 2.2) is small, and it hasn't changed much:
| Call | What it does |
|---|---|
create(path, data, flags) | Create a znode; flags: ephemeral, sequential. Returns the actual name |
delete(path, version) | Delete if the version matches (−1 = any version) |
exists(path, watch) | Return Stat or null; optionally set a watch |
getData(path, watch) | Return data and Stat; optionally set a watch |
setData(path, data, version) | Write data if the version matches (−1 = any) |
getChildren(path, watch) | Return child names; optionally set a watch |
sync(path) | Wait until all updates pending at the start of the call have propagated to the server the client is connected to |
Since 3.4 there is also multi(), which applies a list of operations (create, delete, setData, and check — a version assertion) atomically: all succeed or none do. It's the closest ZooKeeper has to a transaction.
Notice what's missing: no handles and no open. Every call takes a full path. Every update can take an expected version, which turns it into a compare-and-swap: "set this data only if nobody changed it since I read version 5". This is how clients implement safe read-modify-write without locks.
All calls have synchronous and asynchronous variants. The asynchronous form lets a client have many operations in flight at once, and ZooKeeper guarantees their callbacks run in issue order. That's important for performance: the paper's example is a new leader that needs to write hundreds of configuration znodes — sending them asynchronously takes about one round trip instead of hundreds.
Watches
A watch is a one-time trigger. getData("/cfg", watch=true) returns the current data and registers interest; the next change to /cfg sends a notification to the client and removes the watch. To keep watching, the client must read again with a new watch.
Key properties (from the paper and the Programmer's Guide):
- A watch tells you that something changed, not what changed. The notification carries the event type (
NodeDataChanged,NodeCreated,NodeDeleted,NodeChildrenChanged) and the path, not the new data. The client re-reads to find the new state. This keeps notifications tiny and avoids sending stale intermediate values. - You can miss intermediate changes. Between the notification and your re-read, the znode may change several times. You'll see the latest state, not every state. Watches are for "wake up and look", not for change data capture.
- Ordering. A client sees a watch event for a znode before it sees the new data for that znode. Watch events, responses and other notifications are delivered in order. This is what makes "read with watch, react, re-read with watch" correct.
- Watches are tied to the session and the server. They live on the server the client is connected to. If the client reconnects to another server, the client library re-registers its watches there, and the server fires any watch whose znode changed in the meantime (comparing zxids). The documented gap: an
existswatch for a znode that was created and deleted while the client was disconnected can be missed. - Session events. Clients also receive connection-state notifications:
Disconnected,SyncConnected,Expired, and others. These are critical — see sessions below.
ZooKeeper 3.6 added persistent and persistent recursive watches (addWatch), which don't need re-registering and can cover a whole subtree. They reduce round trips for clients that track many nodes, but the one-shot model is still what most recipes and the paper assume.
Sessions
A client connects by creating a session, with a timeout the client proposes and the server bounds (by default between 2 × and 20 × tickTime; with the conventional 2-second tick that's 4–40 seconds). The session, not the TCP connection, is what owns ephemeral znodes and watches.
The paper (Section 4.3) describes the liveness mechanism: if the client has sent nothing for one third of the session timeout, it sends a heartbeat; if it has heard nothing from its server for two thirds of the timeout, it switches to another server. The leader decides when a session has expired, based on the last time any server heard from it.
That gives three distinct client states, and confusing them is the single most common bug in ZooKeeper client code:
| State | What it means | What the client must do |
|---|---|---|
| Connected | Session alive, connected to a server | Normal operation |
| Disconnected (connection loss) | TCP connection to its server is gone; the session may still be alive | Don't assume you lost locks or leadership yet — but stop doing work that requires them, because you can't know. The library is reconnecting. Any in-flight operation has an unknown outcome |
| Expired | The ensemble has expired the session; ephemeral znodes are gone; watches are gone | Everything held by the session is lost. Create a new ZooKeeper handle and rebuild all state (re-create ephemeral nodes, re-register watches) |
The asymmetry is deliberate: the client cannot learn that its session expired until it reconnects, because expiry is decided by the ensemble. A client that is partitioned from the ensemble will see Disconnected and must assume the worst after roughly the session timeout passes. That's why a leader elected through ZooKeeper must step down on disconnect after a timeout, not only on Expired — otherwise a partitioned old leader and a new leader overlap. Even that isn't enough on its own: a long GC pause can stop a process before it notices anything. For anything that writes to shared storage, combine the ZooKeeper lock with a fencing token — the znode's czxid or sequence number makes a good one, since it only grows.
4. Read path and write path
Write path
sequenceDiagram
autonumber
participant C as Client
participant F as Follower (client's server)
participant L as Leader
participant F2 as Other followers
C->>F: setData /cfg, data, expected version 5
F->>L: forward request
L->>L: check version, build idempotent txn, assign zxid
L->>L: append txn to own log
L->>F: PROPOSAL zxid
L->>F2: PROPOSAL zxid
F->>F: append to log and fsync
F2->>F2: append to log and fsync
F-->>L: ACK
F2-->>L: ACK
Note over L: quorum of ACKs, counting the leader itself
L->>F: COMMIT zxid
L->>F2: COMMIT zxid
F->>F: apply to in-memory tree, fire watches
F-->>C: OK, new Stat with version 6
Every write goes to the leader, which assigns a zxid and broadcasts it as a proposal. Followers write the proposal to their transaction log and fsync before acknowledging (this is why the transaction log disk latency dominates ZooKeeper write latency). Once a quorum has acknowledged, the leader commits; each server applies committed transactions in zxid order, firing any watches the change triggers. The server the client is connected to answers the client once it has applied the transaction locally.
Read path
sequenceDiagram
autonumber
participant C as Client
participant F as Follower (client's server)
participant L as Leader
C->>F: getData /cfg, watch = true
F->>F: read from local in-memory tree
F->>F: register watch for this session
F-->>C: data + Stat, tagged with last applied zxid
Note over C,F: The answer may be stale relative to the leader
C->>F: sync /cfg
F->>L: sync request
L-->>F: placed after all writes pending at the leader
F->>F: apply those pending writes
F-->>C: sync done
C->>F: getData /cfg
F-->>C: data at least as new as when sync was issued
Reads are local: no network hop, no quorum. Every response carries the zxid the server had applied when it answered. The client library remembers the highest zxid it has seen. If the client later reconnects to a different server, that server will not accept the session until it has caught up to the client's zxid. That's how ZooKeeper guarantees a client never sees state go backwards, even across failovers.
For readers who need fresher data, there's sync. The paper describes its implementation (Section 4.2): the follower sends the sync to the leader, which places it at the end of the queue of pending writes between it and that follower. When the follower sees the sync come back, it has applied every write that the leader had at the time. A read issued after the sync completes (in FIFO order, so you can issue sync then getData asynchronously back-to-back) observes at least that state.
5. Consistency guarantees
The paper states two basic ordering guarantees (Section 2.3):
- Linearizable writes. All requests that update ZooKeeper state are serializable and respect precedence. More precisely, the paper defines A-linearizability (asynchronous linearizability): standard linearizability assumes one outstanding operation per client, but a ZooKeeper client can have many in flight. A-linearizability means the writes are linearizable, and a client's own operations are additionally ordered FIFO.
- FIFO client order. All requests from a given client are executed in the order the client sent them.
Reads are not linearizable. A read is processed locally at whichever server the client is connected to, so it can miss writes that have committed at the leader but not yet been applied there. The usual way to describe the read guarantee: each client sees a sequentially consistent history — its own writes and reads appear in order, the state it observes only moves forward, and every write it observes is in the global write order — but two clients connected to different servers can see different, slightly stale views at the same instant.
The Programmer's Guide lists the guarantees in operational terms:
| Guarantee | Meaning |
|---|---|
| Sequential consistency | Updates from a client are applied in the order they were sent |
| Atomicity | Updates either succeed or fail — no partial results |
| Single system image | A client sees the same view of the service regardless of which server it connects to; a client never sees an older view, even if it fails over to a different server with the same session |
| Reliability | Once an update has been applied, it persists until overwritten |
| Timeliness | A client's view is guaranteed up to date within a certain time bound (on the order of tens of seconds); either changes are seen within that bound, or the client sees a disconnect |
And the guide explicitly warns: ZooKeeper does not guarantee that two different clients have identical views at every instant. If client A updates /x and then tells client B over some other channel (a direct TCP message, a database row), B can read /x from a lagging server and see the old value. That's the "hidden channel" problem. The fix is for B to call sync before reading.
The fine print on sync
Is sync + read a linearizable read? Nearly, but not strictly — and the paper says so. For sync to be correct, the follower must be sure the server it forwarded to is still the leader. The paper's design: if there are pending transactions committing, the follower can tell the leader is alive; if there are none, the leader would need to issue a null transaction to confirm its leadership. But the implementation as described doesn't: it relies on timeouts being set so that a leader realises it's no longer leader before followers abandon it. So sync depends on timing assumptions, not on a quorum round. In an unlucky partition, a deposed leader could answer a sync and the following read could be stale. For nearly all coordination use cases this is acceptable, but if a design needs a strictly linearizable read, do a write instead (for example, a conditional setData or a multi with a check), which goes through the quorum.
The guarantee that makes recipes work
The combination of FIFO client order, ordered watch notifications and linearizable writes gives a powerful pattern. The paper's example (Section 2.3): a new leader must change many configuration parameters, and no process should read a half-updated configuration.
- The leader deletes a
readyznode. - It updates the configuration znodes (asynchronously, so it's fast).
- It creates
readyagain.
A process reading the configuration first checks exists(ready, watch=true). Because of FIFO order, if it sees ready, every configuration change made before ready was created is visible to it. And if the leader starts a new round of changes while the reader is reading, the reader's watch on ready fires before it can see any of the new partial changes, so the reader knows to start over. No locks, no blocking — just ordering.
Durability and liveness
The paper's other two guarantees: if a majority of servers are active and communicating, the service is available; and if ZooKeeper responds successfully to a change request, the change persists across any number of failures as long as a quorum eventually recovers.
6. ZAB at a glance
ZooKeeper's replication protocol is ZAB (ZooKeeper Atomic Broadcast). The full walk-through, including its phases and a simulation, is on the ZAB page; here is what you need to reason about ZooKeeper's behaviour:
- Primary-backup with a total order. One leader proposes all state changes, each with a zxid. Followers log and ack; the leader commits after a quorum of acks. All servers apply committed transactions in zxid order.
- Epochs fence old leaders. A new leader starts a new epoch (the high 32 bits of the zxid). Followers that have joined the new epoch reject proposals from the old one.
- Recovery preserves everything committed and in order. During recovery, ZAB ensures that every transaction a previous leader committed is delivered by the new leader before it broadcasts anything new, and that transactions which were only proposed by the old leader and skipped by the new one are discarded. Practically: the new leader is chosen so that it has the most up-to-date history among a quorum (highest epoch, then highest zxid).
- FIFO channel assumption. ZAB relies on TCP's ordered delivery between leader and each follower. That's part of why ZooKeeper guarantees primary order — a leader's proposals are delivered in the order it sent them — which a plain Multi-Paxos log doesn't promise per proposer.
- ZAB vs Paxos/Raft. ZAB was designed for a primary-backup system where the primary executes requests and broadcasts state changes, not a state machine replicating requests. In practice it is close to Raft: a single leader, a log, majority quorums, and term/epoch-based leadership. The Paxos page and the ZAB page compare them in detail.
Fast leader election finishes in well under a second in a healthy ensemble; the paper measured under 200 ms in its failure-injection test (Section 5.1). The full unavailability during a leader failure is longer, because followers first have to notice the leader is gone (bounded by syncLimit × tickTime) and then sync with the new one.
7. Recipes
The Apache documentation's Recipes and Solutions page and Section 2.4 of the paper describe how to build primitives on the API. The Apache Curator library implements production-hardened versions, and you should normally use it (Java) or kazoo's recipes (Python) instead of rolling your own. But the recipes are worth knowing exactly, because they show how ZooKeeper's guarantees compose — and because interviewers ask about them.
The examples below use the plain Java client (org.apache.zookeeper) and Python's kazoo client, without their built-in recipe helpers, so the mechanics are visible.
Configuration management
Store the configuration in one znode. Each process reads it with a watch, and re-reads when the watch fires.
from kazoo.client import KazooClient
zk = KazooClient(hosts="zk1:2181,zk2:2181,zk3:2181")
zk.start()
CONFIG = "/app/config"
current = {}
def load(event=None):
# Re-read with a fresh one-shot watch. The watch fires at most once, so the
# callback must re-register it, and it may coalesce several changes.
data, stat = zk.get(CONFIG, watch=load)
current["value"], current["version"] = data, stat.version
print("config version", stat.version)
zk.ensure_path(CONFIG)
load()
def update(new_bytes: bytes, expected_version: int) -> bool:
# Compare-and-swap: fails with BadVersionError if someone else wrote first.
from kazoo.exceptions import BadVersionError
try:
zk.set(CONFIG, new_bytes, version=expected_version)
return True
except BadVersionError:
return False
For multi-znode configuration, use the ready-znode pattern from Section 5 or write everything in one multi() transaction.
Group membership
Each member creates an ephemeral child under a group znode. The children list is the live membership. If a member crashes or partitions long enough for its session to expire, its znode disappears and everyone watching the children is notified.
import socket
GROUP = "/app/workers"
zk.ensure_path(GROUP)
# Ephemeral + sequential gives a unique name even if two members share a hostname.
me = zk.create(f"{GROUP}/{socket.gethostname()}-", value=b"10.0.0.7:8080",
ephemeral=True, sequence=True)
def on_members(event=None):
members = zk.get_children(GROUP, watch=on_members)
print("live members:", sorted(members))
on_members()
The paper notes this works even though ZooKeeper has no notion of "liveness" beyond sessions: membership is exactly "sessions currently alive that created a child". It's the basis of service discovery in many ZooKeeper-backed systems (Kafka's broker registry under /brokers/ids worked this way before KRaft).
Simple lock — and the herd effect
The obvious lock: create an ephemeral znode /lock; if creation fails because it exists, watch it and retry when it's deleted. This is correct but suffers from the herd effect: when the lock is released, every waiting client wakes up and races to create /lock, though only one can win. With a thousand waiters, every release causes a thousand notifications and a thousand failed creates. It also gives no fairness — no first-come-first-served order.
Lock without the herd effect
The paper's improved lock uses ephemeral sequential children and has each waiter watch only its immediate predecessor:
- Create
lockdir/lock-withEPHEMERAL | SEQUENTIAL→ you getlock-0000000042. - List children of
lockdir(no watch) and sort them. - If yours has the lowest sequence number, you hold the lock.
- Otherwise, call
existswith a watch on the child immediately before yours. - If
existsreturns null (it's already gone), go to step 2. Otherwise wait for the watch, then go to step 2.
Releasing is just deleting your znode. When a holder releases or crashes, exactly one client — the next in line — is notified. Clients are served in the order they arrived. Note that step 5 goes back to step 2 rather than assuming you now hold the lock: the predecessor might have given up (its session died while waiting) rather than released, in which case there's still someone ahead of you.
flowchart LR
subgraph lockdir["/locks/orders"]
A["lock-0000000040<br/>(holder)"]
B["lock-0000000041"]
C["lock-0000000042"]
D["lock-0000000043"]
end
B -.->|"watches"| A
C -.->|"watches"| B
D -.->|"watches"| C
import org.apache.zookeeper.*;
import org.apache.zookeeper.data.Stat;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.CountDownLatch;
/** Fair, herd-free lock from the ZooKeeper paper (Section 2.4). */
public final class ZkLock {
private final ZooKeeper zk;
private final String dir; // e.g. "/locks/orders", must exist
private String myPath; // e.g. "/locks/orders/lock-0000000042"
public ZkLock(ZooKeeper zk, String dir) {
this.zk = zk;
this.dir = dir;
}
public void lock() throws KeeperException, InterruptedException {
myPath = zk.create(dir + "/lock-", new byte[0],
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL_SEQUENTIAL);
String myName = myPath.substring(dir.length() + 1);
while (true) {
List<String> children = zk.getChildren(dir, false);
Collections.sort(children); // zero-padded => lexical == numeric
int idx = children.indexOf(myName);
if (idx < 0) {
throw new IllegalStateException("lock node vanished: session expired?");
}
if (idx == 0) {
return; // lowest sequence number: we hold it
}
String predecessor = dir + "/" + children.get(idx - 1);
CountDownLatch changed = new CountDownLatch(1);
// Watch only the node just before ours: no herd on release.
Stat stat = zk.exists(predecessor, event -> changed.countDown());
if (stat != null) {
changed.await(); // deleted, or a session event
}
// Loop: the predecessor may have given up rather than released.
}
}
public void unlock() throws KeeperException, InterruptedException {
if (myPath != null) {
zk.delete(myPath, -1);
myPath = null;
}
}
/** Monotonic fencing token: pass it to the storage system you protect. */
public long fencingToken() {
return Long.parseLong(myPath.substring(myPath.lastIndexOf('-') + 1));
}
}
Two production details this simple version omits, and which Curator's InterProcessMutex handles:
- The create-then-connection-loss problem. If the connection drops after the server created your sequential znode but before you received the reply, you don't know its name. Retrying the create makes a second znode, and the first one blocks everyone until your session ends. Curator's "protected" mode embeds a GUID in the name (
_c_<guid>-lock-) so that on retry it can list the children and find the node it already created. - Connection state. On
Disconnected, a lock holder should stop work that needs the lock until it reconnects or the session expires, and onExpiredit has definitely lost the lock. And as always with distributed locks, the protected resource should check a fencing token — the sequence number above works because it's assigned by the totally ordered write log.
Read/write locks extend the same idea (paper, Section 2.4): readers create read- nodes and wait only for the nearest earlier write- node; writers create write- nodes and wait for the nearest earlier node of either kind. Readers that arrive together share the lock; the herd effect remains only for readers waiting on one writer, which is desired.
Leader election
Leader election is the herd-free lock where "holding the lock" means "being leader" and nobody releases voluntarily. The Leader Election page sketches it; here's a working version:
import threading
ELECTION = "/app/election"
zk.ensure_path(ELECTION)
class Candidate:
def __init__(self, zk, on_elected):
self.zk, self.on_elected = zk, on_elected
# Data = our address, so followers can find the leader by reading the
# lowest child instead of keeping a separate /leader znode in sync.
self.path = zk.create(f"{ELECTION}/n_", value=b"10.0.0.7:9000",
ephemeral=True, sequence=True)
self.name = self.path.rsplit("/", 1)[1]
self.check()
def check(self, event=None):
children = sorted(self.zk.get_children(ELECTION))
if self.name not in children:
raise RuntimeError("our candidate znode is gone: session expired")
i = children.index(self.name)
if i == 0:
self.on_elected()
return
prev = f"{ELECTION}/{children[i - 1]}"
# exists() with a watch: if prev is already gone, re-check right away.
if self.zk.exists(prev, watch=self.check) is None:
self.check()
def become_leader():
print("I am the leader; start leader-only work with fencing token", me.name)
me = Candidate(zk, become_leader)
threading.Event().wait() # keep the process alive
Two rules are easy to get wrong:
- Step down on session trouble. Register a connection listener (
zk.add_listenerin kazoo). OnSUSPENDED(disconnected), stop leader-only work; onLOST(expired), drop leadership and re-enter the election with a new session. - Leadership in ZooKeeper isn't leadership in the world. A paused old leader can wake up and act before it notices its session expired. Anything the leader writes must be fenced — by the sequence number, the epoch of the election znode, or a conditional write.
Double barrier
A double barrier lets a group of processes start a computation together and leave together (paper, Section 2.4). With a barrier znode b and a threshold N:
- Enter: create an ephemeral child of
b. If the number of children is at least N, createb/readyand proceed. Otherwise wait forb/readyto exist (watch withexists). - Leave: delete your child. Wait until
bhas no process children left; to avoid a herd, watch one specific remaining child rather than the whole list.
import threading
from kazoo.exceptions import NodeExistsError
class DoubleBarrier:
def __init__(self, zk, path, n, name):
self.zk, self.path, self.n = zk, path, n
self.me = f"{path}/{name}"
zk.ensure_path(path)
def _wait_for(self, predicate):
ev = threading.Event()
while not predicate(lambda e: ev.set()):
ev.wait()
ev.clear()
def enter(self):
self.zk.create(self.me, ephemeral=True)
ready = f"{self.path}/ready"
def is_ready(watch):
if self.zk.exists(ready, watch=watch):
return True
members = [c for c in self.zk.get_children(self.path) if c != "ready"]
if len(members) >= self.n:
try:
self.zk.create(ready)
except NodeExistsError:
pass
return True
return False
self._wait_for(is_ready)
def leave(self):
self.zk.delete(self.me)
def all_left(watch):
while True:
members = [c for c in self.zk.get_children(self.path) if c != "ready"]
if not members:
return True
# Watch a single remaining member, not the whole list: no herd.
if self.zk.exists(f"{self.path}/{sorted(members)[0]}", watch=watch):
return False
# It left between the two calls: look again instead of waiting.
self._wait_for(all_left)
The ready node stays behind after the round, so use a fresh barrier path per round. Note the inner loop in leave: if the member we chose leaves between get_children and exists, exists returns None and sets a watch for a creation that will never happen — waiting on it would hang, so we re-list instead. Curator's DistributedDoubleBarrier handles the same race.
Rendezvous and queues
Two more from the paper. Rendezvous: a master creates a znode r and passes its path to workers it starts; workers exists(r, watch) and wait for the master to write its address into r. Queues: producers create sequential children; consumers take the lowest. ZooKeeper queues work for small, low-rate coordination queues only — every consumer listing a large children list is expensive, and Kafka-era experience is that a real message queue is the right tool.
8. Fault tolerance and recovery
What the ensemble survives
An ensemble of n voting servers keeps working as long as a majority (⌊n/2⌋ + 1) are up and connected. So:
| Voting servers | Quorum | Failures tolerated |
|---|---|---|
| 3 | 2 | 1 |
| 4 | 3 | 1 |
| 5 | 3 | 2 |
| 7 | 4 | 3 |
Even numbers add a voter without adding fault tolerance, and every extra voter makes writes slower (the leader waits for more acks, and more servers must fsync). Most production deployments run 3 or 5. To scale reads or serve remote data centres, add observers: they receive committed transactions and serve reads and sessions, but don't vote, so they add read capacity without slowing writes or affecting quorum. Observers arrived in ZooKeeper 3.3.
Server failure
- Follower dies: the leader keeps going if a quorum remains. Clients connected to the dead follower see
Disconnected, reconnect to another server within the session timeout, and keep their sessions, ephemeral nodes and watches. - Leader dies: followers notice when they stop hearing from it (
syncLimitticks), run fast leader election, the winner synchronises the others to its history (sending a diff, a truncation, or a full snapshot depending on how far behind each is), and service resumes. The ensemble serves no requests during this window — servers that are electing a leader drop their client connections, and clients seeDisconnecteduntil a server rejoins a working quorum. (Servers started withreadonlymode.enabledcan keep serving reads to clients that opted in to read-only mode while partitioned from the quorum.) - Disk loss: a server that loses its data directory rejoins as an empty server and syncs from the leader. Losing the data on a majority at once can lose committed writes — the log is the source of durability.
Client failure and session expiry
When a client dies or is partitioned, its session expires after the timeout: ephemeral znodes are deleted, watches waiting on them fire, and other clients react (new election, membership change, lock handoff). The session timeout is the key tuning knob: too short and GC pauses or network blips cause spurious expiries and leadership flapping; too long and real failures take that long to detect.
Recovery on restart
A restarting server loads its latest fuzzy snapshot, replays the transaction log from that point, and then syncs with the current leader. Because the data set is in-memory and small, this typically takes seconds — unless someone stored gigabytes in ZooKeeper, in which case restarts, leader syncs and snapshot transfers all get slow, and an ensemble can struggle to re-form at all.
9. Performance numbers from the paper
The paper's evaluation (Section 5) ran on a cluster of 50 servers, each with a dual-core Xeon 2.1 GHz, 4 GB of RAM, gigabit Ethernet and two SATA disks; ZooKeeper's transaction log was on a dedicated disk. Load came from 250 simulated clients on 35 machines, each request reading or writing 1 KB.
Throughput at the extremes (paper's Table 1, approximate):
| Servers | 100% reads (ops/s) | 0% reads, all writes (ops/s) |
|---|---|---|
| 3 | ~87,000 | ~21,000 |
| 5 | ~165,000 | ~18,000 |
| 7 | ~257,000 | ~14,000 |
| 9 | ~296,000 | ~12,000 |
| 13 | ~460,000 | ~8,000 |
This table is the clearest picture of ZooKeeper's trade-offs:
- Reads scale with servers, because every server answers reads locally.
- Writes get slower with more servers, because every write is broadcast, logged and fsynced by more machines and the leader must process more acks.
- So ZooKeeper is built for read-dominated workloads, and the paper notes that coordination workloads typically have read-to-write ratios of 10:1 or higher. Its Yahoo! applications included the Fetching Service (the crawler's master election, configuration and failure recovery), Katta (a distributed indexer) and the Yahoo! Message Broker (pub-sub, using ZooKeeper for topic ownership, failure detection and configuration).
Failure behaviour (paper's Figure 8): with failures injected — followers killed and restarted, then the leader killed — throughput dips briefly and recovers. The leader-election part took less than 200 ms, and throughput recovered quickly once the new leader was in place.
Modern hardware and ZooKeeper versions are much faster in absolute terms, but the shape — reads scale, writes don't, fsync latency on the log disk dominates write latency — is unchanged. A write takes at least one network round trip from leader to quorum plus one fsync; on SSDs in one data centre that's typically single-digit milliseconds.
10. Operating ZooKeeper
ZooKeeper failures in production are usually operational, not algorithmic. From the Administrator's Guide and common practice:
- Give the transaction log its own fast disk (
dataLogDirseparate fromdataDir). Every write waits on an fsync to that log. A shared disk with other I/O (or snapshots) causes write latency spikes, which cause follower timeouts, which cause leader elections. An SSD is strongly preferred. - Purge old snapshots and logs. ZooKeeper does not delete them by default. Set
autopurge.snapRetainCount(default 3) andautopurge.purgeInterval(in hours; default 0, which means disabled), or runzkCleanup.shfrom cron. Full disks are a classic outage. - Watch GC pauses. A server JVM pausing longer than the follower sync timeout drops out of the quorum; a client JVM pausing longer than its session timeout loses its session. Size heaps sensibly, use a low-pause collector, and log GC.
- Tune timeouts together.
tickTime(2000 ms conventionally),initLimit(ticks a follower has to connect and sync to the leader at startup — raise it for large data sets),syncLimit(ticks a follower can lag before being dropped), and client session timeouts (bounded by 2× and 20×tickTimeunless overridden byminSessionTimeout/maxSessionTimeout). - Keep data small. Respect
jute.maxbuffer: a znode's data, and agetChildrenresponse, must fit. A parent with hundreds of thousands of children can produce a response too large to return, which breaks every client that lists it. Raisingjute.maxbufferon servers and clients is a workaround, not a fix. - Limit watches and connections. Millions of watches or tens of thousands of connections per server cost memory and make every change expensive to fan out. Per-IP limits are set with
maxClientCnxns. - Monitor. The
mntrfour-letter command (whitelisted with4lw.commands.whitelistsince 3.5.3) or the AdminServer HTTP endpoint (port 8080 by default, since 3.5) reports average/max latency, outstanding requests, znode and watch counts, followers synced, and pending syncs. Alert on rising outstanding requests and on fsync latency warnings in the log. - Change membership safely. Since 3.5, dynamic reconfiguration (
reconfig) adds and removes ensemble members without a rolling restart of everyone. Before that, changing membership required careful rolling restarts that could lose quorum if done in the wrong order. - Secure it. Enable SASL (Kerberos or DIGEST-MD5) and ACLs, and TLS (supported for client and quorum connections since 3.5). An unauthenticated ZooKeeper on a network often means anyone can delete your cluster's metadata.
- Don't use it as a database or a queue. The ZooKeeper community's most common advice. Keep bulk data in a real store and put only a pointer (a version, a path, a location) in ZooKeeper.
War Story (illustrative)
A team stored per-job status for a batch scheduler in ZooKeeper, one znode per job, under one parent. After a year, the parent had 400,000 children. One day a new dashboard called getChildren on it every few seconds. Each response was larger than jute.maxbuffer, the client failed and retried, and the server spent its time serialising huge responses. Write latency rose, followers fell behind, the ensemble went through repeated elections, and every service using that ensemble for leader election — including unrelated ones — lost their leaders at once. The fix was to move job status into a database and keep only scheduler leadership in ZooKeeper, plus a separate ensemble for the most critical consumers.
11. What it got right, what it got wrong, and successors
What it got right
- Primitives over policies. Exposing ordered, versioned, watched znodes instead of fixed lock/election calls let people build recipes the authors never anticipated. Curator's recipe list (locks, semaphores, leader latches, barriers, caches, queues, shared counters) is the result.
- Local reads. Most coordination traffic is reads; serving them from any replica gives near-linear read scaling.
- Sessions and ephemeral nodes. Tying liveness to a session, with automatic cleanup, turned "detect a dead process" into a one-line create call. It's been copied by every successor (etcd leases, Consul sessions).
- Idempotent transactions and fuzzy snapshots. Small decisions that made recovery simple.
- Wait-free server. No client can block the service for others.
What it got wrong or made hard
- Reads aren't linearizable by default, and
syncrelies on timing. Developers routinely assumed ZooKeeper was "strongly consistent" in every respect and built hidden-channel bugs. - One-shot watches are easy to misuse. Missing intermediate changes, forgetting to re-register, and missed
existswatches across reconnects were common bugs. 3.6's persistent watches came much later. - Client complexity. Connection loss vs expiry, retries with unknown outcomes, and the sequential-node-orphan problem meant raw client code was hard to get right. Curator became effectively mandatory.
- Operational fragility with large data, many watches, or GC pauses — and a Java stack that many teams didn't want to run just for coordination.
- Static membership until 3.5's dynamic reconfiguration.
- An external dependency for every system that needs it. Kafka's move to KRaft is the clearest signal: KIP-500 replaced ZooKeeper with a Raft-based metadata quorum inside Kafka, ZooKeeper mode was deprecated in the 3.x line, and Kafka 4.0 (2025) removed it. The motivations were scalability of metadata (partition count), faster controller failover, and running one system instead of two.
etcd and Consul
The two systems most often compared with ZooKeeper both use Raft:
| ZooKeeper | etcd | Consul | |
|---|---|---|---|
| Consensus | ZAB | Raft | Raft (servers) + gossip (Serf) for membership |
| Data model | Hierarchical znodes with data and children | Flat key space (ranges by prefix), MVCC with a global revision for every change | Flat KV with prefix queries, plus a service catalog |
| Liveness | Sessions + ephemeral znodes | Leases (TTL, kept alive by the client), keys attached to a lease | Sessions tied to health checks, plus agent-level health checking |
| Notifications | One-shot watches (persistent since 3.6) | Streaming watches from a revision: you can resume from revision r and get every change since — no missed intermediate events as long as history hasn't been compacted | Blocking queries (long-poll with an index) |
| Read consistency | Local reads, sequentially consistent; sync for fresher | Linearizable by default (ReadIndex through the leader); serializable (local) on request | Modes: default (leader with lease), consistent (leader verifies quorum), stale (any server) |
| Conditional writes | Version checks, multi | Transactions: If(compare...).Then(ops).Else(ops) | Check-and-set by index, transactions |
| Typical users | Hadoop, HBase, Kafka (pre-KRaft), Solr, Druid | Kubernetes, many Go systems | HashiCorp stack, service discovery and service mesh |
| Language / ops | Java, JVM tuning | Go single binary | Go single binary, agents on every node |
The design lessons the successors took from ZooKeeper: keep the sessions-plus-ephemeral idea (as leases), make watches resumable from a revision so clients don't miss events, make the default read linearizable, and expose a compare-and-swap transaction. etcd's MVCC revision is essentially a generalisation of ZooKeeper's zxid: a global, monotonically increasing version stamp that can be used both for ordering and as a fencing token.
12. Lessons for system design interviews
- Know what coordination data is. Small, critical, read-mostly: leader identity, membership, configuration, locks, shard assignments. Put that in ZooKeeper/etcd. Put everything else somewhere else.
- Never put your own consensus in an interview design unless asked. Say "leader election via ZooKeeper/etcd" and explain the ephemeral-sequential pattern, the herd effect, and fencing.
- Leases and sessions are the liveness primitive. Everything that "disappears when a process dies" is a session or lease with a timeout. Be ready to say what happens during the timeout window and how fencing tokens make that window safe.
- Be precise about read consistency. "ZooKeeper is CP" is true for writes, but its default reads can be stale. If the design needs linearizable reads, mention
sync(with its caveat) or etcd's linearizable reads. - Watches are notifications, not a change log. If a design needs every change, use a log (Kafka) or etcd watches from a revision, not one-shot watches.
- Read scaling vs write scaling. More ZooKeeper servers mean more read capacity and less write capacity. Use observers for read scale and keep voters at 3 or 5.
- Shared coordination infrastructure is a blast-radius risk. One overloaded ensemble can take down every system that depends on it. Mention separate ensembles for critical systems.
13. Interview questions
1. Why doesn't ZooKeeper provide a lock() call that blocks until the lock is free?
The design is wait-free: no operation waits on other clients. A blocking lock on the server would need the server to track waiters, queue them and handle their failures, and a slow or failed client holding a server-side lock could stall others. Instead, ZooKeeper gives ordered sequential znodes, ephemeral nodes and watches, and clients build blocking locks themselves. The server stays simple and fast, and different lock semantics (fair, read/write, semaphores) are all just client recipes.
2. Explain the herd effect and how the sequential-node lock avoids it.
In the naive lock, every waiter watches the single lock znode. On release, all waiters are notified and all try to create the znode; only one wins, so N−1 clients did useless work and the server got a burst of N requests. In the improved recipe each waiter creates an ephemeral sequential node and watches only the node immediately before its own. A release (or crash) deletes one node and notifies exactly one waiter — the next in line — giving O(1) work per handoff and FIFO fairness.
3. A client gets a ConnectionLoss error after calling create("/locks/x/lock-", EPHEMERAL_SEQUENTIAL). What could have happened and what should it do?
The outcome is unknown: the create may or may not have been applied before the connection dropped. If the client blindly retries, it may create a second node, and the first (orphaned) node will block the lock until the session ends. It should reconnect (the session is likely still alive), then list the children and look for a node it created — which requires embedding a unique ID (a GUID or session ID) in the name, as Curator's protected mode does. Only if none exists should it create again.
4. Are ZooKeeper reads linearizable? How would you get a fresh read?
No. Reads are served from the local replica of the server the client is connected to, which can lag the leader. The client never sees state go backwards (single system image) and its own writes are ordered, but another client's committed write may not be visible yet. Calling sync(path) before reading makes the server catch up with the leader's pending writes first. The paper notes sync relies on timing assumptions about leader timeouts rather than a quorum round, so it isn't strictly linearizable in every failure case. For a strictly linearizable check, perform a conditional write.
5. Why do more ZooKeeper servers make writes slower? How do you scale reads without that cost?
Every write must be proposed to all voters, logged and fsynced by a quorum, and acknowledged to the leader; more voters means a larger quorum and more work at the leader. The paper's measurements show write throughput falling from about 21k ops/s with 3 servers to about 8k with 13, while read throughput rose from about 87k to 460k. Observers receive committed transactions and serve reads and sessions without voting, so they add read capacity without enlarging the quorum.
6. Your service's leader was elected via ZooKeeper. A 40-second GC pause hits it. What can go wrong and how do you make it safe?
During the pause its session expires (if the timeout is shorter than 40 s), its ephemeral znode is deleted, and another candidate becomes leader. When the old process wakes up, it may not yet have processed the Expired event and may keep acting as leader — writing to a database or sending commands — so two leaders act at once. Mitigations: step down on Disconnected after a timeout; treat Expired as immediate loss; and fence every side effect with a monotonic token (the znode sequence number or czxid) that the downstream system checks, rejecting writes from older tokens.
7. How does the ready-znode pattern let a leader update many configuration znodes without readers seeing a partial update?
The leader deletes ready, writes the configuration znodes, then re-creates ready. Readers check exists(ready, watch). By FIFO order and ordered notifications, a reader that sees ready sees all configuration writes made before ready was created; if the leader starts another update while a reader is mid-read, the reader's watch on ready fires before it can observe any of the new partial writes, so it restarts its read.
8. What does etcd do differently from ZooKeeper, and when would you choose each?
etcd uses Raft, a flat MVCC key space with a global revision, leases instead of sessions, linearizable reads by default, transactions with compare clauses, and watches that stream every change from a given revision. ZooKeeper has a hierarchical namespace, sessions with ephemeral nodes, local (sequentially consistent) reads and one-shot watches. Choose etcd for new systems in a Go/Kubernetes ecosystem or when you need resumable watches and linearizable reads. ZooKeeper remains the right choice when the systems you run already depend on it (HBase, older Kafka, Solr, Hadoop HA) and you have JVM operations experience.
9. Why did Kafka remove ZooKeeper?
KIP-500 moved Kafka's metadata into an internal Raft quorum (KRaft). With ZooKeeper, the controller had to load all metadata from ZooKeeper on failover, which was slow for clusters with hundreds of thousands of partitions; metadata changes went through ZooKeeper and then had to be propagated to brokers; and operators had to run and secure two distributed systems. With KRaft, metadata is a replicated log inside Kafka, standby controllers have it in memory, and failover is fast. ZooKeeper mode was removed in Kafka 4.0.
Further reading
- Patrick Hunt, Mahadev Konar, Flavio P. Junqueira, Benjamin Reed. ZooKeeper: Wait-free coordination for Internet-scale systems. USENIX ATC 2010.
- Flavio P. Junqueira, Benjamin C. Reed, Marco Serafini. Zab: High-performance broadcast for primary-backup systems. DSN 2011.
- Apache ZooKeeper documentation: Programmer's Guide, Administrator's Guide, ZooKeeper Recipes and Solutions, Dynamic Reconfiguration.
- Mike Burrows. The Chubby lock service for loosely-coupled distributed systems. OSDI 2006.
- Apache Curator documentation (recipes and "tech notes" on connection handling).
- KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum.
- Related on Archon: ZAB Protocol, Leader Election, Distributed Locking, Consistency Models, Failure Detectors, GFS & HDFS, Bigtable & HBase.