Raft in Plain English
The consensus overview explained why distributed systems need agreement. This lesson explains how the most widely deployed consensus algorithm of the last decade actually does it — without proofs, without RPC field lists, just the moving parts and the reasoning behind them.
Raft was published in 2014 by Diego Ongaro and John Ousterhout with an unusual design goal: understandability. Paxos was proven correct but famously hard to learn and harder to implement. Raft solves the same problem by splitting it into pieces a person can hold in their head: elect a leader, let the leader replicate a log, and add a couple of rules that keep everything safe.
The Problem: One Log, Many Copies
Imagine a key-value store that must survive machine failures. You run it on 5 servers. Every write (SET balance 100) must end up on all of them, in the same order, so that every server's copy of the data is identical.
The trick Raft uses — called state machine replication — is to stop thinking about the data and think about the log of commands:
- If every server starts from the same empty state,
- and applies the same commands,
- in the same order,
- then every server ends in the same state.
So the entire problem reduces to: get every server to agree on the contents and order of a log, even when servers crash and messages get lost.
Why a Majority?
Raft never waits for all servers. It waits for a majority (a quorum): 3 of 5, 2 of 3, 4 of 7.
The magic property: any two majorities overlap in at least one server. If 3 of 5 servers stored a command, then any future group of 3 must include at least one server that knows about it. That single overlapping server is how knowledge survives a leader crash.
| Cluster size | Majority | Failures tolerated |
|---|---|---|
| 1 | 1 | 0 |
| 3 | 2 | 1 |
| 5 | 3 | 2 |
| 7 | 4 | 3 |
| 4 | 3 | 1 (no better than 3 servers) |
Why odd numbers
A 4-node cluster needs 3 votes and tolerates 1 failure — the same as 3 nodes, but with one more machine to pay for and one more server in every write's critical path. That is why Raft clusters are almost always 3 or 5 nodes.
Terms: Raft's Logical Clock
Raft divides time into terms, numbered 1, 2, 3, … Each term begins with an election. If the election succeeds, one leader rules for the rest of that term. If it fails (nobody wins), the term ends with no leader and a new term starts.
Terms work like a version number for leadership:
- Every message carries the sender's current term.
- If a server sees a higher term than its own, it immediately updates its term and becomes a follower — whatever it was doing.
- If a server receives a message with a lower term, it rejects it as stale.
That single rule — higher term wins — is what dethrones old leaders after a partition, as you will see below.
The Three Roles
At any moment each server is in exactly one of three roles.
| Role | What it does |
|---|---|
| Follower | Passive. Responds to the leader and to candidates. Never initiates anything. All servers start here. |
| Candidate | A follower that stopped hearing from the leader and is now asking for votes. |
| Leader | Handles all client writes, appends them to its log, replicates them to followers, and sends heartbeats so followers know it is alive. At most one per term. |
stateDiagram-v2
[*] --> Follower
Follower --> Candidate: Election timeout, no heartbeat heard
Candidate --> Candidate: Split vote, timeout again, new term
Candidate --> Leader: Wins votes from a majority
Candidate --> Follower: Discovers current leader or higher term
Leader --> Follower: Discovers higher term
Note there is no arrow from Follower directly to Leader, and no arrow from Leader to Candidate. Leadership is always won in an election and only ever lost to a higher term.
Leader Election
Heartbeats and Timeouts
The leader sends periodic heartbeats (empty AppendEntries messages) to every follower — typically every 50–100 ms. Each follower runs an election timer. Every heartbeat resets the timer. If the timer fires, the follower concludes the leader is dead.
The crucial detail: each follower picks its timeout randomly, say between 150 and 300 ms. That randomness is the whole trick to avoiding endless tied elections.
Running for Office
When a follower's timer fires, it:
- Increments its term (say from 3 to 4).
- Becomes a candidate and votes for itself.
- Sends
RequestVoteto every other server.
Each server grants at most one vote per term, first come first served — with one restriction we will get to in the safety section. The candidate:
- wins if it collects votes from a majority, and immediately starts sending heartbeats to assert leadership;
- steps down if it hears from a legitimate leader with a term at least as high as its own;
- times out and starts a new election with a higher term if nobody wins.
Split Votes
Suppose a 5-node cluster has one server down, and two of the remaining four followers time out at nearly the same instant. Each votes for itself and wins one more vote from the other two servers. Both candidates now hold 2 votes — but a majority of 5 is 3, so neither wins. Term 4 ends leaderless.
Because the timeouts are randomized, in term 5 one of them almost certainly times out first, collects votes before the other wakes up, and wins. In practice, elections usually settle in a single round, and a split vote costs only a few hundred milliseconds.
Log Replication
Once there is a leader, normal operation is simple.
- A client sends a command (
SET x 5) to the leader. Followers that receive client requests redirect them to the leader. - The leader appends the command to its own log as a new entry, tagged with the current term and an index.
- The leader sends the entry to all followers in
AppendEntriesmessages, in parallel. - Each follower appends the entry and acknowledges.
- As soon as a majority (counting the leader itself) has stored the entry, it is committed. The leader applies it to its state machine and replies to the client.
- The leader tells followers the new commit index in its next message; they apply the entry too.
sequenceDiagram
participant C as "Client"
participant L as "Leader S1"
participant F2 as "Follower S2"
participant F3 as "Follower S3"
participant F4 as "Follower S4"
participant F5 as "Follower S5 (slow)"
C->>L: SET x 5
L->>L: Append entry #7 (term 2)
par Replicate
L->>F2: AppendEntries #7
L->>F3: AppendEntries #7
L->>F4: AppendEntries #7
L->>F5: AppendEntries #7
end
F2-->>L: OK
F3-->>L: OK
Note over L: S1 + S2 + S3 = 3 of 5, entry #7 committed
L->>L: Apply SET x 5
L-->>C: OK
F4-->>L: OK (late, still fine)
L->>F5: Next AppendEntries carries commitIndex = 7
The leader does not wait for the slowest server. A lagging follower simply catches up on later messages.
Keeping Logs Consistent
Every AppendEntries message includes the index and term of the entry just before the new ones. A follower accepts the message only if its own log has a matching entry at that position. If not, it rejects, and the leader steps back one entry and tries again until it finds the point where the two logs agree — then overwrites everything after it with its own entries.
The result is the Log Matching property: if two logs have an entry with the same index and term, the logs are identical up to that point. The leader's log is always the source of truth, and followers are repaired to match it.
When Things Go Wrong
The Leader Crashes
Say S1 is leader of a 5-node cluster in term 2 and crashes.
- Followers stop receiving heartbeats.
- The follower with the shortest random timeout — say S3 — fires first, moves to term 3, and requests votes.
- S2, S4 and S5 grant their votes (assuming S3's log is up to date — see below). S3 has 4 of 5 votes and becomes leader.
- S3 starts heartbeating and repairing any followers whose logs differ from its own.
Total unavailability: roughly one election timeout plus one round trip — typically well under a second.
What about writes S1 had accepted but not committed when it crashed? If they reached a majority, they are committed and the new leader is guaranteed to have them. If they did not, they may be lost — but the client never received a success reply for them, so no promise was broken. The client will time out and retry.
A Network Partition
Now the interesting case. A 5-node cluster with leader S1 is split by a network failure:
flowchart LR
subgraph Minority["Minority side (2 nodes)"]
S1["S1<br/>old leader, term 2"]
S2["S2<br/>follower"]
end
subgraph Majority["Majority side (3 nodes)"]
S3["S3<br/>new leader, term 3"]
S4["S4<br/>follower"]
S5["S5<br/>follower"]
end
S1 -. "network partition" .- S3
On the minority side, S1 still believes it is leader. A client sends it a write. S1 appends it and replicates to S2 — but it can never reach 3 acknowledgements. The write is never committed, and S1 never tells the client it succeeded. The minority side cannot make progress, which is exactly what we want.
On the majority side, S3, S4 and S5 stop hearing heartbeats. One of them times out, starts term 3, wins 3 of 3 votes, and becomes leader. Clients that can reach this side keep working normally.
When the partition heals, S1 receives a message carrying term 3. Higher term wins: S1 immediately steps down to follower. Its uncommitted entries conflict with S3's log, so S3 overwrites them during the normal consistency check. No committed data is lost, and there was never a moment where two leaders could both commit.
Stale reads from the old leader
A deposed leader that does not yet know it is deposed can still answer reads from its local state — which may be stale. Production systems avoid this by serving linearizable reads only after confirming leadership with a majority (etcd's ReadIndex) or by using time-bound leader leases. Raft's core paper handles writes; safe reads are an extra step you must not forget.
War Story (illustrative)
A team ran a 3-node cluster with two nodes in one availability zone and one in another. When the first zone lost power, the surviving node held 1 of 3 votes and could not elect itself — the cluster went read-only even though "one node was fine". Raft was doing its job. The fix was placement, not code: spreading 3 nodes across 3 zones means losing any one zone still leaves a majority.
Why It Is Safe: The Election Restriction
There is one hole in everything above. What if a server that missed some committed entries becomes leader? Since the leader's log always wins, it would overwrite committed data on the other servers.
Raft closes this with the election restriction: a server refuses to vote for a candidate whose log is less up-to-date than its own. "Up to date" is compared simply:
- The log whose last entry has the higher term is more up to date.
- If the last terms are equal, the longer log is more up to date.
Now combine it with majority overlap:
- A committed entry is stored on a majority of servers.
- A winning candidate needs votes from a majority.
- Those two majorities share at least one server — and that server has the committed entry.
- That server will refuse to vote for any candidate missing it.
So every new leader already holds every committed entry. This is the Leader Completeness property, and it is why Raft can let the leader's log overwrite followers without ever destroying committed data.
The full walkthrough covers one more subtle rule (a leader only counts replicas to commit entries from its own term, carrying older entries along implicitly), along with the proof. For intuition, the election restriction plus overlapping majorities is the heart of it.
Where Raft Runs
| System | What Raft does there |
|---|---|
| etcd | The replicated key-value store at its core; Kubernetes stores all cluster state in etcd, so every kubectl apply goes through Raft |
| HashiCorp Consul | Replicates the service catalog and KV store among server nodes |
| CockroachDB | Runs one Raft group per range (a slice of the keyspace) — thousands of small Raft groups per cluster |
| TiKV / TiDB | Multi-Raft: each region of data is its own Raft group, built on the etcd Raft library |
| Apache Kafka (KRaft) | Replaced ZooKeeper with a Raft-based controller quorum that manages cluster metadata |
| HashiCorp Vault, Nomad | Integrated storage and scheduler state replicated with Raft |
The pattern in large databases is worth noticing: they do not run one giant Raft cluster for all data. They shard data and run many small Raft groups in parallel, which spreads both load and leadership across the machines.
Raft in One Paragraph
Servers agree on a log, not on data. Time is divided into terms; a higher term always wins. A follower that stops hearing heartbeats waits a random timeout, becomes a candidate and asks for votes; whoever gets a majority leads that term. The leader appends client commands to its log and replicates them; once a majority stores an entry, it is committed and applied. Votes go only to candidates whose logs are at least as up to date as the voter's, so any new leader already has every committed entry. A leader cut off in a minority cannot commit anything, and steps down the moment it sees a higher term.
Key Takeaways
- Consensus is reduced to agreeing on an ordered log; identical logs produce identical state machines.
- Majorities overlap, which is what lets knowledge survive failures. Clusters of 2f + 1 nodes tolerate f failures.
- Terms act as a logical clock for leadership; any higher term forces a step-down.
- Randomized election timeouts make split votes rare and short-lived.
- An entry is committed when a majority stores it; the leader never waits for the slowest follower.
- A minority-side leader cannot commit, so partitions cost availability on the minority side, never correctness.
- The election restriction guarantees every new leader holds all committed entries.
Further Reading
- Raft: The Definitive Walkthrough — every RPC field, the commitment rule for older terms, membership changes, log compaction, and a full TypeScript implementation
- Paxos Made Simple — the algorithm Raft was designed to replace, and how its ideas map onto Raft
- Leader Election — leader election patterns beyond Raft
- Consensus in Distributed Systems — the overview this lesson builds on
- Ongaro & Ousterhout — In Search of an Understandable Consensus Algorithm (USENIX ATC 2014)