System Design & Architecture
How to Articulate System Design Trade-Offs: Beyond the CAP Theorem to PACELC & Consensus Protocols
In Senior and Staff system design interviews at FAANG and Tier-1 tech companies, reciting the basic CAP theorem ("Consistency, Availability, Partition Tolerance") is an entry-level signal. Principal-level candidates differentiate themselves by framing trade-offs through the PACELC Theorem (Abadi, 2012) and articulating fine-grained consistency levels: Linearizability, Sequential Consistency, Causal Consistency, and Read-Your-Writes.
This guide provides the formal trade-off taxonomy and interview frameworks required to demonstrate distributed systems mastery.
+───────────────────────────────────────────────────────────────────────────+
| THE PACELC THEOREM TRADE-OFF MATRIX |
+───────────────────────────────────────────────────────────────────────────+
| |
| If there is a PARTITION (P): |
| ├── Availability (A) vs. Consistency (C) |
| |
| ELSE (E) — normal operating conditions: |
| └── Latency (L) vs. Consistency (C) |
| |
| ========================= CANONICAL CLASSIFICATIONS =================== |
| - PC/EC: Spanner, CockroachDB (Strict Serializability / Raft) |
| - PA/EL: Cassandra, DynamoDB (Tunable Consistency / Vector Clocks) |
| - PC/EL: MongoDB (Single-Leader Primary / Secondary Eventual Read) |
| |
+───────────────────────────────────────────────────────────────────────────+
1. The PACELC Framework Explained
While CAP only addresses behavior during catastrophic network partitions, PACELC accounts for the fundamental trade-off that occurs 99.99% of the time during normal operation:
Partition Availability / Consistency Else Latency / Consistency
When there is no network partition, a distributed system must choose between:
- Low Latency ($L$): Return write confirmations immediately from local memory/replica without waiting for synchronous cross-region replication (e.g. AWS DynamoDB with eventual consistency).
- Strong Consistency ($C$): Incur cross-region network round-trip times (RTT) to achieve quorum consensus (e.g. Google Cloud Spanner with TrueTime atomic clocks).
2. Consistency Hierarchy: From Strongest to Weakest
| Consistency Model | Formal Invariant | Performance Overhead | Example Technology |
|---|---|---|---|
| Strict Serializability | Global real-time ordering of all transactions | High (Multi-Paxos / Raft 2PC) | Google Spanner, CockroachDB |
| Linearizability | Individual read/write operations appear instantaneous in real-time | Moderate (Single-leader Raft quorum) | etcd, Apache ZooKeeper |
| Sequential Consistency | Operations ordered consistently across all processes (program order) | Moderate (Memory barrier synchronization) | JVM Memory Model, ZooKeeper async |
| Causal Consistency | Causally related operations seen in same order; concurrent operations arbitrary | Low (Vector Clocks / Lamport Timestamps) | Riak KV, Cosmos DB |
| Eventual Consistency | Replicas converge to same state if no new updates occur | Lowest (Asynchronous Gossip Protocol) | Apache Cassandra, DynamoDB |
3. High-Scoring Interview Response Patterns
Scenario: High-Velocity Financial Ledger (Stripe / Coinbase)
- Trade-off Choice: PC/EC (Prioritize Consistency in Partition; Prioritize Consistency in Normal Operation).
- Candidate Explanation:
“For account balances and double-entry ledgers, double-spending cannot be resolved with eventual convergence. We choose a PC/EC model using Raft leader leases and PostgreSQL multi-region synchronous replication with strict serializability, accepting a 45ms cross-region write latency overhead to guarantee financial invariants.”
Scenario: Social Media Activity Feed (Instagram / TikTok)
- Trade-off Choice: PA/EL (Prioritize Availability in Partition; Prioritize Low Latency in Normal Operation).
- Candidate Explanation:
“For video view counts and timeline generation, displaying a post 2 seconds delayed is completely acceptable compared to failing the user's feed request. We choose PA/EL using Cassandra with Local Quorum reads and asynchronous fan-out to achieve sub-10ms P99 feed delivery.”
4. Real-Time System Design Support with Kairo
During live technical interviews, remembering intricate database isolation levels under cognitive stress can be difficult. Kairo's Live Invisible Copilot HUD:
- Detects architectural prompts and surfaces PACELC matrix cheat sheets in <180ms.
- Formats suggested trade-off justifications tailored to the specific problem statement.
- Operates completely cloaked via native OS display exclusion flags.