The Core Building Blocks
System design interviews aren't about memorizing architectures. They're about showing you can reason through tradeoffs under pressure. I've sat on both sides of these interviews, and the candidates who stand out are the ones who ask clarifying questions before drawing boxes.
Let's break down the three pillars that come up in almost every system design question: load balancing, caching, and database sharding.
Load Balancers: More Than Round-Robin
A load balancer sits between clients and your server fleet, distributing incoming requests. Simple enough in theory. The interesting part is how it distributes them.
Round-robin is the default everyone reaches for. It cycles through servers sequentially. Works fine when all your servers are identical and all requests cost roughly the same to process. In practice, neither assumption holds.
Algorithm Choices That Actually Matter
Least-connections routing sends traffic to the server handling the fewest active requests. This handles uneven request costs better than round-robin. If one server gets stuck processing a slow database query, it won't keep receiving new work at the same rate.
Weighted round-robin lets you assign capacity scores. Got a mix of 4-core and 16-core machines? Give the bigger ones 4x the weight. Not perfect, but it's simple and it works.
Consistent hashing is what you want when session affinity matters. It maps requests to servers using a hash ring, so the same user tends to hit the same server. When a server goes down, only its portion of traffic gets redistributed — not everything.
# Simplified consistent hashing
import hashlib
class ConsistentHash:
def __init__(self, nodes, virtual_nodes=150):
self.ring = {}
self.sorted_keys = []
for node in nodes:
for i in range(virtual_nodes):
key = self._hash(f"{node}:{i}")
self.ring[key] = node
self.sorted_keys.append(key)
self.sorted_keys.sort()
def _hash(self, key):
return int(hashlib.md5(key.encode()).hexdigest(), 16)
def get_node(self, item):
h = self._hash(item)
for key in self.sorted_keys:
if h <= key:
return self.ring[key]
return self.ring[self.sorted_keys[0]]L4 vs L7 Load Balancing
Layer 4 (transport) load balancers work with TCP/UDP packets. They're fast because they don't inspect the payload. AWS Network Load Balancer operates at this level — it can handle millions of requests per second with single-digit millisecond latencies.
Layer 7 (application) load balancers understand HTTP. They can route based on URL paths, headers, cookies. Need /api/* going to your backend fleet and /static/* to a CDN origin? That's L7. AWS ALB, Nginx, and HAProxy all operate here.
In interviews, I'd recommend starting with L7 unless you have a specific reason for L4. Most systems benefit from content-aware routing.
Caching: The Fastest Code Is Code That Doesn't Run
Caching is deceptively simple to explain and surprisingly hard to get right. The core idea: store frequently accessed data closer to where it's needed.
Cache Placement
Client-side caching (browser cache, mobile app cache) eliminates network round-trips entirely. Set your Cache-Control headers correctly and you've solved the problem for repeat visits.
CDN caching (Cloudflare, CloudFront) puts content at edge locations geographically close to users. Static assets are obvious candidates. But you can also cache API responses for data that doesn't change every second — like product catalog pages or user profiles that update every few minutes.
Application-level caching with Redis or Memcached is where most interview discussions focus. A Redis instance with 64GB of RAM can serve 100,000+ reads per second at sub-millisecond latency. Compare that to a PostgreSQL query that might take 5-50ms.
Cache Invalidation Strategies
Phil Karlton famously said there are only two hard things in computer science: cache invalidation and naming things. He wasn't wrong about the first part.
TTL-based expiration is the simplest approach. Set a time-to-live of 300 seconds and accept that data might be stale for up to 5 minutes. For many use cases, this is perfectly fine. Your user's profile photo doesn't need to update in real-time across every page.
Write-through updates the cache and database simultaneously on every write. Guarantees consistency but adds latency to write operations. Works well when reads heavily outnumber writes.
Write-behind (write-back) updates the cache immediately and asynchronously writes to the database. Faster writes, but you risk data loss if the cache node crashes before the write propagates. I'd only recommend this for data you can afford to lose — like view counts or analytics events.
Cache-aside (lazy loading) is the most common pattern. On a cache miss, the application reads from the database, stores it in cache, then returns the result. On writes, you invalidate the cache entry and let the next read repopulate it.
# Cache-aside pattern
def get_user(user_id):
# Try cache first
cached = redis.get(f"user:{user_id}")
if cached:
return json.loads(cached)
# Cache miss — hit the database
user = db.query("SELECT * FROM users WHERE id = %s", user_id)
if user:
redis.setex(f"user:{user_id}", 300, json.dumps(user))
return user
def update_user(user_id, data):
db.execute("UPDATE users SET ... WHERE id = %s", user_id)
redis.delete(f"user:{user_id}") # InvalidateDatabase Sharding: When One Machine Isn't Enough
Sharding splits your data across multiple database servers. Each shard holds a subset of the total data. It's the nuclear option for database scaling — powerful but complex.
When You Actually Need Sharding
Don't shard prematurely. Seriously. A single PostgreSQL instance can handle tens of millions of rows without breaking a sweat if you've got proper indexes. Read replicas can absorb read traffic. Vertical scaling (bigger hardware) is simpler than horizontal scaling.
You need sharding when: your write volume exceeds what a single primary can handle, your dataset doesn't fit in one machine's storage, or you need geographic data locality.
Sharding Strategies
Range-based sharding splits data by value ranges. Users with IDs 1-1M on shard 1, 1M-2M on shard 2. Simple to implement but creates hotspots — the shard holding the newest users gets hammered while older shards sit idle.
Hash-based sharding applies a hash function to the shard key. shard = hash(user_id) % num_shards. Distributes data evenly but makes range queries across shards expensive. Adding or removing shards requires rehashing — unless you use consistent hashing.
Directory-based sharding maintains a lookup table mapping keys to shards. Most flexible but introduces a single point of failure (the directory service). You'll need to replicate and cache the directory itself.
The Cross-Shard Query Problem
Here's the catch nobody mentions until you're deep into implementation. Any query that spans multiple shards is painful. A simple SELECT COUNT(*) FROM orders that used to be one query now requires querying every shard and summing the results.
Joins across shards? Basically impossible at the database level. You'll end up doing them in application code, which is slow and error-prone.
The mitigation is choosing your shard key wisely. Shard by the dimension you query most. For a multi-tenant SaaS app, sharding by tenant_id means most queries stay within a single shard. For a social network, sharding by user_id keeps a user's data co-located.
Putting These Together in an Interview
When you get a system design question — "Design Twitter" or "Design a URL shortener" — don't jump straight into drawing boxes. Start with requirements gathering.
How many users? What's the read-to-write ratio? What's the latency requirement? Is strong consistency required, or is eventual consistency acceptable?
Then build up from simple to complex. Start with a single server, identify the bottleneck, and introduce the appropriate technique. Overloaded database reads? Add a cache layer. Single point of failure on your web server? Add a load balancer and multiple servers. Dataset outgrowing one machine? Time to discuss sharding.
The best candidates I've interviewed don't just know what these components do — they know when to introduce them and what tradeoffs they bring. That's the difference between reciting architecture diagrams and actually designing systems.