Landmark Systems
A handful of systems, most of them described in papers between 2003 and 2012, set the patterns that almost every modern database, data platform and cloud service still uses. When an interviewer says "partition it with consistent hashing and replicate with quorums", they are describing Dynamo. "A sorted map on an LSM tree" is Bigtable. "Externally consistent transactions across regions" is Spanner.
Each page here reads one system as a case study:
- the problem it was built to solve;
- its architecture;
- its read and write paths;
- the exact guarantees it gives, and the caveats its authors admitted;
- how it handles faults;
- what its successors changed.
How They Fit Together
graph LR
GFS["GFS / HDFS"] --> MR["MapReduce / Spark"]
GFS --> BT["Bigtable / HBase"]
CH["Chubby / ZooKeeper"] --> BT
CH --> ZK["ZooKeeper recipes"]
BT --> SP["Spanner"]
DY["Dynamo"] --> CS["Cassandra, Riak, DynamoDB"]
BT --> CS
SP --> CR["CockroachDB, YugabyteDB"]
CA["Calvin"] --> FA["FaunaDB"]
DL["Distributed Ledgers"]
| System | Era | Key idea | Read it to learn |
|---|---|---|---|
| GFS & HDFS | 2003 | Single master, large chunks, relaxed consistency | Metadata/data separation, leases, re-replication |
| MapReduce & Spark | 2004 / 2012 | Re-execution as fault tolerance, then lineage | Batch processing, shuffles, stragglers |
| Bigtable & HBase | 2006 | Sorted sparse map on SSTables | LSM storage, tablets, row-level atomicity |
| Amazon Dynamo | 2007 | Always writable, leaderless | Consistent hashing, sloppy quorums, vector clocks |
| ZooKeeper | 2010 | Coordination kernel, not a lock service | Znodes, watches, sessions, lock and election recipes |
| Spanner & TrueTime | 2012 | Bounded clock uncertainty + commit wait | External consistency, 2PC over Paxos groups |
| Calvin & FaunaDB | 2012 | Order first, then execute deterministically | Avoiding 2PC, deterministic locking |
| Distributed Ledgers | 2008 → | Shared state among mutually distrusting parties | Nakamoto vs BFT consensus, permissioned ledgers |
Before You Read
These pages assume the theory from the Distributed Systems Course: replication and quorums, clocks and causality, consensus, and distributed transactions. They are the course's final module.