System Design for L5 Full-Stack Interviews

A cram guide. Mechanics first, folklore never.

Key numbers card

Thing	Number
Redis, single node	~1M QPS (simple gets)
SQL database, single node	~10K QPS
Page load budget	100-200ms
One machine holds comfortably	~1-2TB
Network hop vs RAM access	~1000x slower
B-tree lookup, 1B rows	3-4 disk reads
Seconds per day (round it)	~100K (86,400)
100M DAU, 10 reads/user/day	~12K QPS avg, ~50K peak
Char / int / UUID	1B / 4B / 16B
Tweet-sized record	~1KB with metadata
Latency ladder	RAM ~100ns · SSD ~100µs · same-region network ~1ms · disk seek ~10ms · cross-region ~50-150ms
Image / 1 min of video	~500KB / ~50MB → object storage + CDN, never the DB
Cache hit vs DB read	~100x cheaper
WebSocket connections per gateway box	~100K-1M

Chapter 1: The default skeleton

The diagram you start from

Nearly every system in this guide is a bend of one skeleton. Learn it cold so the first boxes on the whiteboard cost you zero thought and you can spend the interview on the parts that are actually specific to the problem.

            ┌── CDN ─────────────── static assets, media
client → DNS
            └── load balancer → app servers (stateless)
                                      │
                    ┌─────────────────┼─────────────────┐
                  cache            database           queue
                 (Redis)        (+ replicas)            │
                                      │              workers
                               object storage (S3)

Every arrow earns its place. The CDN answers most requests before they ever reach you. The load balancer spreads the rest across identical app servers. The cache absorbs the read load the database can't take. The database is the source of truth. The queue takes everything that doesn't have to happen before the response goes out. Object storage holds anything measured in megabytes. When an interviewer asks "why is that box there," the answer is always one of those sentences.

Request lifecycle, end to end

Being able to narrate what happens between typing a URL and seeing a page is a cheap credibility signal, and it's where latency intuition comes from.

1. DNS. The browser resolves the domain to an IP. Answers are cached at the browser, the OS, and the resolver, each honoring a TTL. Big services use geo-DNS or anycast so users resolve to the nearest region.
2. Connect. TCP handshake, then TLS handshake. That is 1-3 round trips before a single byte of your data moves, which is why connection reuse (keep-alive) matters and why CDNs terminating TLS at a nearby edge makes everything feel faster.
3. Load balancer. Terminates TLS, picks a healthy app server, forwards the request.
4. App server. Auth, validation, cache check, DB query on a miss, compose the response.
5. Render. The browser parses HTML and pulls scripts, styles, and images from the CDN.

Budget check. 100-200ms feels instant. A single cross-region round trip is 50-150ms, most of the budget gone before you did any work. Two conclusions, serve users from a nearby region or edge, and never put chatty multi-round-trip work inside the request path.

Vertical vs horizontal scaling

Vertical is buying a bigger machine. No code changes, no coordination, and it is the right answer more often than conference talks admit. It has a ceiling, and it is still one failure domain. Horizontal is adding machines, which requires that any machine can serve any request, meaning the servers hold no state.

Stateless is the trick that makes the app tier free. Sessions go to Redis or into a signed token, uploads go to object storage, nothing lives on local disk. Now servers are interchangeable, autoscaling is safe, a deploy or a crash just means kill the box and start another. This is why the interesting scaling conversation is never about app servers. It is about the stateful pieces, the database, the cache, the queue, which is what the rest of this guide is.

Load balancing

Layer	Sees	Can do
L4 (transport)	IPs and ports, opaque bytes	Very fast dumb forwarding. Use for raw TCP throughput or non-HTTP protocols.
L7 (application)	Full HTTP request	Route by path or header, terminate TLS, retry failed requests, sticky sessions. The default (nginx, ALB, Envoy).

Algorithms. Round robin is the default. Least-connections handles uneven request costs better. Consistent hashing routes the same user to the same server, which buys cache locality. Health checks, the LB probes each server (an HTTP /health endpoint), ejects failures from rotation, and drains connections gracefully during deploys.

Who balances the balancer. DNS returning multiple IPs, anycast, or simply the fact that managed cloud LBs are already replicated fleets. One sentence in an interview and move on.

WebSockets caveat. Long-lived connections pin to a specific server, so the LB must support them (L7 with upgrade handling) and the design needs a registry of which gateway holds which user. Chapter 15's chat design covers this.

Chapter 2: SQL vs NoSQL, the mechanics

Core framing

Plain words first, because these two terms carry the whole chapter. A JOIN stitches rows from two tables together at query time, "give me each tweet, and also the author's name and avatar from the users table", one query, the database does the assembly. A transaction makes several changes succeed or fail as a single unit, "subtract $50 from this account and add $50 to that one, and never let anyone observe the halfway state". ACID is the checklist for that promise, Atomic (all or nothing), Consistent (rules like uniqueness always hold), Isolated (concurrent transactions can't see each other's half-done work), Durable (once confirmed, it survives a crash).

SQL's superpowers are JOINs and ACID transactions. Both quietly assume all the data lives together on one machine. A JOIN is cheap when both tables are on the same disk. A transaction is cheap when one lock manager sees everything. NoSQL is not magic scaling technology. It is a family of databases that refuse to do those expensive things, and that refusal is exactly what lets them spread data across hundreds of machines.

Indexes, what reads gain and writes pay

Without an index a query scans the whole table, O(n). A B-tree index is a sorted tree over one or more columns, lookups are O(log n) with a huge branching factor, which is the "billion rows in 3-4 disk reads" number. Every WHERE, JOIN, and ORDER BY on your hot path should be backed by one.

The cost lands on writes. Every insert and update must maintain every index on the table. Five indexes means roughly six write operations per row change. This is why you index the queries you actually run, not every column that might someday matter, and why write-heavy tables stay lean on indexes.

• Composite indexes obey leftmost prefix. An index on (user_id, created_at) serves "by user" and "by user ordered by time", but not "by created_at alone". Put equality columns first, range columns last.
• Covering index. If the index contains every column the query needs, the row fetch is skipped entirely. Worth mentioning for a hyper-hot query.
• What indexes can't fix. Leading-wildcard searches (LIKE '%term%', there's no prefix to seek, this is why full-text search exists, Chapter 14), low-selectivity columns like booleans, and functions applied to the column unless you index the expression.

Practical line for interviews, "I'd index the WHERE and ORDER BY of the top queries and verify with EXPLAIN." That one sentence covers the topic at L5 depth.

When SQL is actually slow

Not "big data" in general, and not the RAM boundary. A B-tree index lookup on a billion rows takes 3-4 disk reads. Disk-resident indexed lookups are fast. The real cliffs are two specific events.

• Data exceeds one machine. Now a JOIN becomes a cross-shard network operation. You are shipping rows over the wire and merging them, which is orders of magnitude slower than a local join.
• ACID across shards. A transaction touching two shards needs two-phase commit. 2PC means multiple network round trips, locks held across them, and a coordinator that can stall everyone if it dies.

Below those cliffs, a well-indexed Postgres with read replicas handles more than most people think.

NoSQL mechanisms, what it does instead

Mechanism	What it replaces	How it works
Hash partitioning on a partition key	Query planner + shared storage	hash(key) → node, O(1) routing. Every query goes to exactly one node, no scatter-gather.
Denormalization	JOINs	Pre-assemble the document at write time. A read is one fetch of one blob, no assembly.
LSM trees	B-trees	Writes are sequential appends to a memtable then flushed to SSTables. Writes are very fast. Reads may check multiple SSTables, so reads pay instead.
Tunable / eventual consistency	Full ACID	A write returns after 1-2 replicas ack instead of all. Faster and more available, readers may see stale data briefly.

The denormalization tradeoff, concretely

Take an avatar URL. In SQL it lives in one row in users and every tweet display JOINs to it. In a document store you copy it inside every tweet, comment, and notification document so each read is a single fetch.

Now the user changes their photo. You either rewrite thousands or millions of documents (a background fan-out job) or you accept staleness until documents get touched again. That is the trade in one sentence. SQL pays at read time, NoSQL pays at update time. Read-heavy data with rarely changing attributes is the sweet spot for denormalization.

Access pattern lock-in

The partition key fixes your primary query. Partition tweets by user_id and "all tweets by user" is one-node fast, but "all tweets containing a hashtag" hits every node. Escape hatches exist and each one costs a full copy of the data.

• DynamoDB GSIs. An extra copy of the table sorted by a different key, maintained async, eventually consistent.
• Cassandra dual-write tables. Application writes the same data to two tables with different partition keys.
• Stream to Elasticsearch. CDC or a queue feeds a search cluster for flexible ad hoc queries.

Rule of thumb, each additional access pattern is another full data copy plus the machinery to keep it in sync.

Chapter 3: Replication and sharding

Two different tools that beginners blur together. Replication copies all the data onto more machines, which buys durability and read scale. Sharding splits the data across machines, which buys write scale and storage headroom. Replicate first, shard as late as possible.

Replication

The default shape is leader-follower. All writes go to one leader, the leader ships its change log to followers, reads can go anywhere. This alone scales reads (add followers) and gives durability (a dead leader means promoting a follower, not losing data).

• Sync vs async. Synchronous, the leader waits for a follower to ack before confirming the write. Nothing is ever lost, but every write pays a network round trip and one dead follower can stall all writes. Asynchronous, the leader confirms immediately and ships changes in the background. Fast, but a crash can lose the last few seconds of writes. The common compromise is semi-sync, exactly one follower is synchronous.
• Failover. Leader's heartbeat lapses, the most up-to-date follower is promoted, clients redirect. The danger is split brain, the old leader comes back and both accept writes. Real systems fence the old leader before promoting. Know the word, skip the details.
• Multi-leader and leaderless, one paragraph. Multi-leader accepts writes in several regions, at the price of conflict resolution when two regions edit the same row. Leaderless (Cassandra, Dynamo) writes to any replica and uses quorums (next chapter). Name them, don't dive unless asked.

Read replicas scale reads, not writes. Every write is still applied on every replica. If writes are the bottleneck, replication does nothing, that's sharding's job.

Replication lag, the interview classic

Async followers run seconds behind. A user posts a comment (write hits the leader), refreshes (read hits a lagging follower), and their comment is gone. That's the read-your-own-writes problem, and interviewers love it because the fix requires actually understanding the topology.

• Pin the author to the leader for their own recent data, or for ~10 seconds after any write. Simple and usually sufficient.
• Version tokens. The write returns a log position, subsequent reads carry it, and a replica that hasn't caught up to it either waits or forwards to the leader.
• Just accept it for data where staleness is invisible, like counts.

Sharding

When write throughput or data size exceeds one machine, split the data. Each shard is an independent database holding a slice, usually with its own replicas.

Strategy	How	Tradeoff
Range	Shard 1 gets users A-F, shard 2 gets G-M	Range scans stay cheap, but sequential keys (timestamps, auto-increment IDs) hammer the newest shard. Hot spot machine.
Hash	hash(key) → shard	Even spread, but range queries now hit every shard. The usual pick.
Directory	A lookup service maps key → shard	Flexible rebalancing, at the cost of an extra hop and one more critical service.

Choosing the shard key is the same discussion as the partition key in Chapter 2, high cardinality, even spread, and it must match the dominant access pattern, because queries that don't include the key become scatter-gather across all shards (ask every shard, merge the answers, pay the slowest one's latency). Cross-shard transactions mean two-phase commit, avoid them by keeping each transactional entity (a user's data, an order and its lines) on one shard.

Resharding is the pain. Naive hash(key) % N remaps almost every key when N changes, which is a full data migration. That is what consistent hashing fixes. Practical posture, don't shard until forced, and reach for tooling (Vitess, Citus) before hand-rolling.

Consistent hashing

Nodes and keys hash onto a ring, a key belongs to the next node clockwise. Adding or removing a node remaps only the neighboring slice of keys, about 1/N of the data, instead of nearly everything the way hash(key) % N does. Virtual nodes, each physical machine appears at many points on the ring, smooth out load imbalance and let heterogeneous machines take proportional shares. This is how Cassandra, DynamoDB, and distributed caches place data, and it's the standard answer to "what happens when you add a cache node."

Chapter 4: Consistency, plainly

CAP without the theory. When the network partitions, and it will, a distributed system chooses between answering with possibly stale data (available) or refusing to answer until it's sure (consistent). That's the entire theorem as interviews need it. And nobody chooses once for a whole system, you choose per feature.

Data	Choice	Why
Bank balance, seat booking, inventory	Strong	A stale read causes real harm, double-spend, double-book. Refuse or wait rather than lie.
Like count, view count, follower count	Eventual	Nobody can tell 4,982 from 4,987. Availability and latency win.
Timeline, feed, notifications	Eventual	A tweet arriving 5 seconds late is invisible. This tolerance is what makes feed architecture possible.

Vocabulary, one line each. Strong consistency, every read sees the latest write, as if there were one copy. Read-your-writes, you see your own updates, others can lag. Monotonic reads, you never see data go backwards in time. Eventual, replicas converge if writes stop, no promise when. In interviews the useful move is naming which one each feature needs, not reciting definitions.

Quorums, the one formula. With N replicas, a write acked by W of them, and a read consulting R, then W + R > N guarantees the read overlaps the latest write. Cassandra's QUORUM on N=3 is W=2, R=2. Drop to W=1 for faster writes and you've traded away the guarantee, which is exactly what "tunable consistency" means, and you tune it per query.

PACELC, one sentence of extra credit. Even with no partition, you still trade latency against consistency, synchronous replication costs a round trip on every write. Saying "strong consistency isn't free even on a healthy network" signals you actually get it.

Chapter 5: Real-world choices

System	Storage	Why
Twitter timeline	Redis + Manhattan	Precomputed feeds, fan-out on write. Timeline reads must be one cheap fetch, so feeds are materialized into Redis at tweet time.
Facebook Messages	HBase, later MyRocks	Write firehose. LSM storage absorbs the constant stream of message writes with sequential appends.
Netflix viewing history	Cassandra	Partitioned by user. Every query is "this user's history", a perfect single-partition access pattern, and eventual consistency is fine.
Payments / billing (everywhere)	SQL. Spanner, Vitess, sharded MySQL	Money needs atomicity. Nobody accepts "eventually your balance will be right."
Airbnb / Uber bookings	Sharded MySQL / Postgres	Double-booking prevention needs strong consistency and transactions on the contended row.

Two corrections to folklore. Instagram ran sharded Postgres for years at hundreds of millions of users. SQL scales much further than the conference-talk narrative suggests. And NewSQL (Spanner, CockroachDB) blurs the line, giving distributed horizontal scale with real transactions, at the cost of latency and money.

Chapter 6: How interviewers probe database choices

"Why Cassandra?" Bad answer, "it scales." Good answer names the workload shape, the access pattern, and the consistency tolerance. "Write-heavy time-series data, always queried by user_id, staleness of a few seconds is fine, so an LSM-based store partitioned by user fits."

"What's your partition key?" They are testing hot partitions. Partition by celebrity_id and Justin Bieber's partition melts while thousands of nodes idle. Know your answer, compound keys, salting the key, or handling hot entities on a separate path.

"What happens when the user updates their profile photo?" They are testing whether you understand the denormalization you just chose. If you copied the avatar into every document, say out loud that updates fan out or go stale, and pick one deliberately.

CAP, one level deep. Enough for L5. Cassandra is AP-leaning with tunable consistency per query (ONE, QUORUM, ALL). DynamoDB is tunable per read. A single-cluster SQL database is CP-ish, it stays consistent and a partition can make it unavailable. Say "tunable" and you signal you know it is not a binary.

"Could you do this with Postgres?" Often yes, and saying so scores points. "Sharded Postgres works until cross-shard queries dominate, but this access pattern is single-user lookups at high write volume, which favors Cassandra" beats reflexive NoSQL every time.

Chapter 7: Back-of-envelope math

Do this out loud in the first ten minutes. It sets the scale for every later decision and it is the cheapest seniority signal available. (QPS just means queries per second, how many requests hit the system, the unit everything here is measured in.)

100M DAU, 10 reads/user/day
= 1B reads/day
÷ ~100K seconds/day  ≈ 12K QPS average
× 3-4 peak factor    ≈ 50K QPS peak

One SQL node ≈ 10K QPS  →  can't serve this raw.
So: cache (Redis ~1M QPS/node) or read replicas, decided in minute five.

Storage side, same speed. 100M users × 1KB profile = 100GB, fits one machine. 1B tweets/year × 1KB = 1TB/year, one machine for now, plan shards for year three. Photos and video blow past this instantly, which is why they go to object storage and CDN, never the database.

Keep the divisions crude. 86,400 seconds is 100K. Nobody wants three significant figures, they want to see you reason about orders of magnitude.

The latency ladder, and the two conclusions that matter. RAM ~100ns, SSD ~100µs, same-region network ~1ms, disk seek ~10ms, cross-region ~50-150ms. Conclusion one, a cache hit is roughly 100x cheaper than a database read, which is why caching shows up in every design. Conclusion two, cross-region is the killer, one round trip eats most of a 200ms budget, so data lives near its users and chatty protocols die.

Size anchors. UUID 16B, timestamp 8B, a tweet ~280B raw and ~1KB with metadata, an image ~500KB, a minute of video ~50MB. The moment media enters the design, the numbers jump three orders of magnitude, which is the signal to route bytes to object storage and a CDN (Chapter 10) and keep only metadata in the database.

Chapter 8: Read/write ratio drives design

State the ratio before choosing storage, every time. It is one sentence and it drives everything downstream.

Workload	Ratio	Design consequence
Timeline / feed	~1000:1 reads	Precompute at write time, cache aggressively, denormalize. Pay the write cost to make reads one fetch.
Metrics / logging	Write heavy	LSM stores, batch and buffer writes, reads are rare scans and can be slow.
Chat	Roughly balanced	Both paths matter. Fast append for sends, indexed fetch per conversation.
Bookings / payments	Low volume, high value	Consistency dominates throughput. SQL, transactions, no shortcuts.

Saying "this is roughly 1000 to 1 read heavy, so I'll optimize the read path and accept expensive writes" before touching the whiteboard is the habit that signals seniority.

Chapter 9: Caching

Patterns

Pattern	How it works	When it fits
Cache-aside	App checks cache, on miss reads DB and populates cache. Cache is passive.	The default. Read-heavy, tolerates a miss penalty, cache failure just means slower reads.
Write-through	Writes go to cache and DB synchronously.	Reads must never see stale data and you can pay write latency. Cache is always warm for recently written keys.
Write-behind	Writes hit cache, flushed to DB async in batches.	Extreme write volume (counters, likes). Danger, cache dies before flush and you lose writes. Needs a durable buffer or acceptance of loss.

Invalidation

TTL is the blunt instrument. Set 60s and worst-case staleness is 60s, no coordination needed. Explicit invalidation (delete the key on write) gives freshness but now every write path must know every cache key that depends on it, and a missed one serves stale data forever. "There are only two hard problems in computer science, cache invalidation and naming things" is a real warning, the dependency graph between data and cached views grows until nobody fully knows it. Practical answer, TTL as a backstop plus explicit invalidation on the hot paths you control.

Eviction and sizing

A cache is full by design, something must go when new data arrives. LRU (least recently used) is the default and usually right, recency predicts re-access. LFU (least frequently used) resists one-off scans polluting the cache but adapts slowly. Redis is configured with a maxmemory policy, allkeys-lru evicts anything, volatile-lru only evicts keys that have TTLs, know those two names.

Sizing is a hit-rate conversation, not a data-size conversation. Access is power-law distributed, so caching the hot ~20% of the working set typically serves 80-95%+ of reads. The metric to watch is hit rate, if it sags, either the cache is too small for the working set or the access pattern has no locality and caching was the wrong tool. Quick math, 100M items × 1KB with 20% hot is 20GB, a couple of Redis nodes, that sentence in an interview closes the topic.

Failure modes with fixes

Failure	What happens	Fix
Thundering herd	A hot key expires, 10K concurrent requests all miss and hit the DB at once.	Request coalescing (one request refills, others wait), jittered TTLs so keys don't expire together, serve-stale-while-refreshing.
Hot keys	One key (celebrity profile) gets so much traffic a single cache node saturates.	Local in-process cache in front of Redis, or replicate the key across N nodes and read randomly.
Cache penetration	Requests for keys that don't exist skip the cache and hammer the DB every time.	Negative caching, cache the "not found" with a short TTL. Bloom filter on IDs for higher volume.
Cold start	Deploy or restart wipes the cache, DB takes full load and may fall over.	Cache warming before taking traffic, gradual traffic ramp, persistent cache tier that survives app deploys.

Where caches live

Client (browser memory, HTTP cache headers), CDN (static assets and cacheable API responses at the edge), edge/reverse proxy (Varnish, nginx), application tier (Redis or Memcached, the layer interviews mean by default), and the database's own buffer pool (a well-provisioned Postgres serves hot pages from RAM, which is why "the DB is slow" sometimes just means the working set outgrew memory). Say the layers exist, then go deep on the application tier.

Redis specifics

Redis is a data-structure server, not a string cache. Sorted sets power leaderboards and timelines (score = timestamp, ZRANGE gives you a page of the feed in one call). Lists make simple queues, hashes store objects field by field, sets do membership. Persistence is optional, RDB snapshots (fast, can lose recent writes) or AOF logs (slower, more durable), and many shops run it as pure cache with none. It is single-threaded per core for command execution, so one slow command like KEYS * blocks everything, and throughput scales by adding shards, not threads.

Redis vs Memcached in one line. Memcached is a plain multi-threaded string cache and perfectly fine at that job. Redis adds data structures, persistence, and pub/sub, which is why it wins by default, say Redis unless asked.

Chapter 10: CDNs, object storage, and media

CDN mechanics

A CDN is thousands of cache servers (points of presence, PoPs) placed near users, run by someone else (Cloudflare, CloudFront, Akamai). It buys exactly two things. Latency, the round trip to a nearby edge is ~10ms instead of 100ms+ to your origin, and TLS terminates at the edge so the expensive handshake happens over the short hop. Origin offload, most requests are served from edge cache and never reach your servers, which is why a power-law traffic pattern (a few hot items, a long tail) is survivable at all.

• Routing. Either DNS-based (the CDN's resolver hands back the IP of a nearby edge) or anycast (every edge advertises the same IP and internet routing delivers you to the closest one). One sentence each is plenty.
• Pull vs push. Pull is the default, the edge fetches from your origin on the first miss, caches per your headers, and serves everyone after. Push means pre-positioning content on edges before anyone asks, worth it for predictable heavy content, a big launch, video libraries (Netflix Open Connect is the extreme, boxes inside ISPs).
• Origin shield. A mid-tier cache between edges and origin, so a thousand edge misses collapse into one origin fetch instead of a thousand. Say the word when someone asks "what if every edge misses at once."
• What else lives at the edge. TLS termination, HTTP/2 and 3, WAF rules, DDoS absorption (the attack hits an edge fleet built for it, not your origin), and increasingly edge compute for tiny logic like auth checks on signed URLs.

What to serve through it. Static assets always, JS, CSS, images, fonts. Video segments especially, that's most of internet traffic. Cacheable API responses sometimes, a public GET /trending with a 30s TTL is a fine CDN citizen. Personalized or authenticated responses rarely, they're marked private and pass through.

HTTP caching headers, the control surface

The CDN and the browser both obey the same headers, so this is one skill that covers two cache layers.

Header	Meaning
Cache-Control: max-age=N	Cacheable for N seconds, by browsers and CDN both.
s-maxage=N	Overrides max-age for shared caches (the CDN) only. Long at the edge, short in browsers.
public / private	Private means browser may cache, CDN must not. For per-user responses.
no-cache	Cache it, but revalidate with the origin before serving. Not "don't cache", naming trap.
no-store	Actually don't cache. Sensitive data.
ETag + If-None-Match	Conditional request, origin answers 304 Not Modified with no body if unchanged. Saves bandwidth, not the round trip.
stale-while-revalidate	Serve the stale copy instantly, refresh in the background. Same idea as SWR on the client.

Recipes worth memorizing. Fingerprinted asset, public, max-age=31536000, immutable. HTML page, no-cache or a short max-age so deploys show up. Public API GET, s-maxage=30 at the CDN, small max-age or private in browsers. Anything sensitive, private, no-store.

Invalidation, and why fingerprinting wins

CDNs offer purge APIs, but purges take seconds to minutes to propagate globally, and a broad purge points a thundering herd of edge misses at your origin. So production systems avoid needing purges at all.

The real answer is content fingerprinting. Build tools hash each file into its name, app.3f9c21.js, and the file is immutable forever, cached for a year. A deploy produces new filenames and new HTML that references them. The HTML itself carries a short TTL, so within seconds users get new HTML pointing at new assets, and no purge ever happens. Old assets age out on their own. If you say only one thing about CDN invalidation in an interview, say this.

Purging is then reserved for the rare emergency, a leaked file, a bad image, a legal takedown.

Object storage (S3)

The other half of every media story. The mental model, S3 is a giant flat key-value store for blobs behind an HTTP API. A bucket holds objects, and each object is a key (a string like uploads/user123/video.mp4) mapping to bytes plus metadata. The "folders" you see in consoles are fake, the namespace is flat, and listing a "directory" is just a prefix query over keys. There is no filesystem underneath, no appending, no editing byte 500 of a file. You PUT, GET, and DELETE whole objects, and "modifying" an object means uploading a replacement.

• Durability, engineered. 11 nines (99.999999999%), achieved by splitting each object into chunks with parity (erasure coding) spread across multiple data centers, so several simultaneous disk or even facility failures still lose nothing. The nuance worth saying out loud, durability is not availability, S3 can have a bad hour where you can't reach your data without losing a byte of it.
• Strongly consistent since 2020. After a successful PUT, any GET or LIST sees the new object immediately. Older study material warns about eventual consistency on overwrites, that caveat is dead.
• Why blobs never go in the database. The DB is your most expensive storage per GB, blobs bloat every backup and every replica, and the DB adds nothing to a 5MB image, no indexing, no transactions on pixels. S3 costs cents per GB-month and thousands of clients can pull objects in parallel without bottlenecking one machine's disk. The pattern is always, bytes in S3, key and metadata in the database.
• Presigned URLs, the actual mechanic. Your server holds credentials for the bucket, clients never do. A presigned URL is your server using those credentials to sign a URL that encodes exactly one permitted operation, this HTTP method, this key, this expiry. S3 verifies the signature cryptographically, so your server is never consulted again. Upload flow, the client calls your API, the server creates a DB row (status: uploading), signs a PUT URL for a key it chose, and the client sends the bytes straight to S3, your app servers only ever handled a tiny JSON exchange. The same trick with GET serves private files, anyone holding the link has minutes before it dies.
• Multipart upload. Initiate, upload the file in parts (5MB+ each) in parallel, each part retried independently, then a complete call has S3 assemble the object. This is the answer to "the connection dropped at 90%", you resume from the missing parts, not from zero. Say it whenever uploads exceed ~100MB.
• Event notifications are the glue. S3 emits an event to a queue or a function when an object lands. That is what makes the pipeline below run without anything polling for new files.
• Lifecycle tiers. Hot storage, then infrequent-access, then archival (Glacier) as objects age. One sentence when someone asks about storage cost.

The standard media pipeline

1. client → POST /uploads            → server returns presigned URL + upload_id
2. client → PUT bytes directly to S3   (multipart if large)
3. S3 event → queue → workers          (thumbnails, transcodes, virus scan)
4. workers write processed outputs to S3, mark DB row ready
5. serving: client → CDN → S3          (signed URLs or signed cookies if private)

Walk this and you've answered the upload question for Instagram, YouTube, Slack attachments, and every other media product. The DB row carries a status field, uploading → processing → ready, and the UI polls or gets pushed that status. Private content is served with the same CDN, using signed URLs or cookies the CDN validates at the edge.

Chapter 11: Queues and async

The rule. Anything not needed in the request path leaves the request path. The user needs the tweet accepted, not the fan-out done. Ack fast, queue the rest.

And the reliability principle underneath this whole chapter. A job is not "handed to a worker". It is durably persisted, and it only disappears when a worker proves it finished. The queue is a database of pending work, not a pipe. Kafka writes every message to disk and replicates it across brokers before acking the producer, SQS replicates across data centers. A job sitting in a queue survives crashes of everything around it.

Kafka fundamentals

A topic is a named stream. It is split into partitions, each an append-only ordered log. Producers write to a partition (usually by hashing a key, so one user's events stay ordered). A consumer group divides partitions among its members, each partition is owned by exactly one consumer in the group, and each consumer tracks its offset, the position in the log it has processed. Committing offsets is how progress survives restarts. When a consumer dies, the group rebalances, its partitions are reassigned to the survivors, and work continues from the last committed offsets, no human involved.

Key consequence, partition count caps parallelism. 8 partitions means at most 8 consumers in a group doing work, a 9th sits idle. Choose partition count for target throughput, and know that resizing later reshuffles key-to-partition mapping.

Delivery semantics, and how a job survives a worker crash

The mechanics differ by queue style, but both are designed the same way, failure is handled by default, success requires an explicit act.

• SQS-style, the visibility timeout. When a worker receives a message it is not deleted, it becomes invisible to other workers for a window, say 60 seconds. The worker does the work, then explicitly deletes the message. If the worker crashes, hangs, or is just slow, the timeout lapses and the message reappears for another worker to pick up. Losing a job would require deleting it without doing the work.
• Kafka-style, the offset commit. Each worker owns partitions and tracks an offset. The discipline is, process the message fully, then commit the offset. Crash before committing and the group rebalances, another worker takes over the partition and re-reads from the last committed offset, redoing the unfinished work.

Notice both mechanisms have the same consequence, a crashed worker means the job runs again. That is what at-least-once delivery means, and it is the default reality of every real queue. Which forces consumers to be idempotent, running a job twice must have the same effect as running it once (setting x to 5 is idempotent, adding 1 to x is not). The standard move is dedupe on message ID, keep a table or Redis set of processed IDs and skip repeats. Some work is naturally idempotent already, inserting the same member into a Redis sorted set is a no-op the second time, and noticing that in an interview saves you the dedupe machinery.

Exactly-once is mostly a myth or expensive. Kafka offers it within its own ecosystem via transactions, but the moment you touch an external system (send an email, charge a card) you are back to at-least-once plus idempotency. Say that sentence in an interview and you are done with the topic.

Use cases

• Fan-out. One tweet write becomes millions of timeline inserts, all off the request path. Big jobs get chunked. "Fan out to 2M followers" as one job is fragile, a crash at 90% redoes everything and one worker grinds alone. Instead the first worker splits it, fetch the follower list, emit sub-jobs of ~1,000 followers each back into the queue. Retries become cheap, the work spreads across the whole fleet, and a failure re-runs one small idempotent chunk.
• Write spike absorption. The queue buffers a flash-sale burst, consumers drain at the DB's sustainable rate.
• Decoupling. Order service emits an event, email, analytics, and inventory each consume independently. New consumers need no producer changes.
• Retry queues. Failed work goes to a retry topic with backoff instead of blocking the main flow.
• Dead letter queues. After N failed retries a message parks in a DLQ for human inspection instead of poisoning the stream forever.

CDC and the outbox pattern

The dual-write problem. The app writes to the database, then publishes an event to Kafka. Crash between the two and the DB and the stream disagree forever, and no transaction spans both systems. Any design that says "save it and also emit an event" has this bug until it names a fix.

The outbox pattern is the fix. Write the business row and an event row into an outbox table in the same database transaction, so they commit or fail together. A relay process tails the outbox and publishes each event to the queue, marking it sent. Delivery is at-least-once, consumers dedupe as usual. Cheap, boring, correct.

CDC (change data capture) is the generalization, tail the database's own replication log (Debezium is the name to drop) and turn every committed change into an event stream. This is the standard way to keep Elasticsearch, caches, and warehouses in sync with the source of truth without touching application code, and it can't miss a change because it reads the same log replicas do.

Webhooks, being the producer

Webhooks flip the usual setup, you deliver events by calling someone else's server, which will be slow, down, or buggy. Interview-complete treatment:

• Sign the payload. HMAC with a shared secret in a header, so receivers can verify it's really you. Include an event ID and timestamp so replayed requests can be rejected.
• Retry with exponential backoff over hours or days, not seconds. Endpoints come back. After the retry budget, park the event in a DLQ and surface a dashboard, plus an API the consumer can poll to replay missed events.
• Expect duplicate delivery. You retry on timeout, but the request may have landed. Receivers dedupe on event ID, same at-least-once story as always.
• Receivers respond 200 immediately and process async. A receiver that does real work inline will time out and trigger your retries, the polite pattern is ack, queue, process.
• Don't promise ordering. Send full current state or a version number in each event, not diffs, so a stale event applied late does no harm.

Backpressure

When consumers fall behind, lag (newest offset minus committed offset) grows. The queue absorbs it for a while, that is its job, but unbounded lag means stale downstream data and eventually retention limits eating unprocessed messages. Monitor lag per consumer group, alert on growth trend not absolute number, and have an answer ready, scale consumers up to the partition count, then add partitions, then shed or sample load.

Queue vs stream

SQS-style queues delete a message once one consumer processes it, work distribution, one job one worker. Kafka-style streams keep an ordered log that many consumer groups read independently at their own pace, and messages persist for the retention window so you can replay history. Use a queue for "do this task once", a stream for "these events happened, several systems care."

Applied to timeline fan-out, SQS-style semantics are arguably the natural fit, it is "do each job once" work and there is no partition cap on how many workers you add. Kafka wins the moment other consumers want the same events, search indexing and analytics replaying the tweet stream from the log. Making that comparison out loud is a strong nuance, most candidates just say Kafka reflexively.

Chapter 12: Failure handling

Everything fails. The L5 signal is volunteering failure modes unprompted, "and when this cache dies, here's what happens" before the interviewer asks.

• Replication for durability. Every datastore runs with replicas. A dead primary means promotion, not data loss. Know your replication lag story, an async replica promoted after a crash may miss the last writes.
• Health checks + load balancer failover. The LB probes each instance and stops routing to failures. Users see nothing, capacity dips until autoscaling replaces the node.
• Retries with exponential backoff and jitter. Naive instant retries turn a blip into a self-inflicted DDoS. Back off exponentially so load drops, add jitter so a thousand clients don't retry in lockstep. Retry only idempotent operations, or make them idempotent first.
• Circuit breakers. When a downstream keeps failing, stop calling it. Fail fast, serve a fallback, probe occasionally, close the circuit when it recovers. This stops one slow dependency from consuming every thread in the fleet.
• Graceful degradation. The Netflix example. If personalized recommendations are down, show a generic popular list, not an error page. The user never knows. Rank features by criticality and have a cheap fallback for each non-critical one.
• Timeouts on every remote call. A call with no timeout is a thread parked forever, and a slow dependency quietly consumes the whole fleet, slow is more dangerous than down because nothing trips. Set timeouts everywhere, and shrink them down the chain, if the user gives you 500ms, the DB call gets 200, not 500, or there's no time left to degrade gracefully.
• Load shedding. When overloaded, reject cheap and early (429, or drop low-priority work) instead of degrading everyone equally into timeout soup. Serving 80% of users well beats serving 100% badly until you crash.
• Distributed locks, lightly. Sometimes exactly one worker should do a job. Redis SET key value NX EX 30 is the standard, the TTL frees the lock if the holder crashes, and the value identifies the holder so you don't release someone else's lock. Mention that a lock can expire while its holder still runs (fencing tokens are the fix's name) and that atomic conditional writes on the database often remove the need for a lock at all, which is the more senior answer.

Chapter 13: Interview process and API design

The first 5-10 minutes

Do not draw a box until you have, in order, functional requirements (the 3-4 things it must do, cut everything else out loud), non-functional requirements (scale, latency targets, consistency needs, availability), QPS and storage estimates (Chapter 4 math, spoken aloud), an API sketch (the 3-5 endpoints), and core entities (the nouns and their relationships). This ordering is the difference between designing the right system and decorating the wrong one.

API design deep dive (full-stack focus)

REST resource modeling. Nouns not verbs, POST /tweets not POST /createTweet. Nest one level max, GET /users/123/tweets is fine, deeper gets brittle.

POST /tweets
{ "text": "hello", "media_ids": [...] }
→ 201 Created
{ "id": "abc123", "text": "hello", "created_at": "..." }

GET /users/123/tweets?cursor=eyJpZCI6...&limit=20
→ 200 { "tweets": [...], "next_cursor": "eyJpZCI6..." }

Status codes that matter. 200 ok, 201 created, 400 your request is malformed, 401 who are you, 403 you can't do that, 404 not found, 409 conflict (double booking), 429 rate limited, 500 our fault, 503 try later.

Pagination, cursors beat OFFSET. OFFSET 100000 forces the DB to scan and discard 100K rows, cost grows linearly with page depth, and rows shifting under you cause skips and duplicates. A cursor encodes the last-seen sort key (WHERE created_at < :cursor ORDER BY created_at DESC LIMIT 20), constant cost at any depth, stable under inserts. Say "OFFSET degrades linearly and breaks under concurrent writes" and move on.

Idempotency keys. For unsafe operations like payments, the client generates a key, sends it in a header, and the server stores key → result. A retried request returns the stored result instead of charging twice. This is the answer to "what if the response is lost and the client retries."

Versioning. /v1/ in the path is ugly and works. Additive changes (new optional fields) don't need a version bump, breaking changes do. Never break existing clients silently.

Rate limiting headers. Return 429 with X-RateLimit-Limit, X-RateLimit-Remaining, and Retry-After so well-behaved clients can back off instead of guessing.

Real-time delivery options.

Option	Mechanism	Use when
Short polling	Client asks every N seconds	Cheap to build, fine for slow-changing data, wasteful at scale.
Long polling	Server holds the request until data or timeout	Near-real-time without WebSocket infra, good fallback.
SSE	One-way server → client stream over HTTP	Notifications, live feeds, anything where the client only listens. Auto-reconnect built in.
WebSockets	Full-duplex persistent connection	Chat, collaborative editing, games. Anything truly bidirectional. Costs connection state on servers.

REST vs GraphQL vs gRPC

REST is the default. Resources over HTTP, cacheable by URL, every tool and engineer understands it. Its weakness is fixed response shapes, clients over-fetch fields they don't need or make multiple round trips for nested data.

GraphQL lets the client declare exactly the fields it wants and walks nested relationships in one round trip. It earns its keep when many differently-shaped clients (web, iOS, watch, partners) read the same data graph. The costs, everything is a POST so HTTP and CDN caching stop working for free, the server needs query-cost limits so a malicious deep query can't melt the backend, and naive resolvers produce N+1 database queries (batching via dataloader is the fix's name).

gRPC is binary protobuf over HTTP/2, typed contracts, fast serialization, streaming built in. It's the standard for internal service-to-service calls, and awkward for browsers (needs a proxy layer).

The interview line, "REST for the public API, gRPC between internal services, GraphQL if many client shapes read the same graph and I'm ready to pay the caching and complexity tax."

Auth and OAuth

Sessions vs JWTs. Sessions, server stores state, a cookie carries the session ID, revocation is instant (delete the row), but every request costs a lookup and horizontal scale needs shared session storage. JWTs, the token carries signed claims, stateless verification, scales trivially, but revocation before expiry is hard (you end up with a token blocklist, which is state again). Common compromise, short-lived JWT access token plus a long-lived refresh token that is checked against the server.

OAuth in five sentences. OAuth is delegated authorization, the "sign in with Google" shape. Your app redirects the user to the provider with a client ID, a redirect URI, and requested scopes. The user consents there, and the provider redirects back with a one-time code. Your server exchanges the code plus its client secret for tokens, that's the authorization code flow, with PKCE added when there's no server to hold a secret (SPAs, mobile). The point, the user's password never touches your system, you hold a revocable, scoped token instead.

Two practical notes worth saying, cookies for browser clients should be HttpOnly and SameSite so scripts can't steal them, and authorization (what you may do) is checked server-side on every request, hiding buttons is not access control.

Frontend-adjacent concerns for full stack

• Optimistic updates. Apply the like/send locally at once, fire the request, roll back with a toast on failure. Requires client-generated IDs so the server response can reconcile.
• Client caching, stale-while-revalidate. Show cached data instantly, refetch in the background, swap when fresh. This is the SWR/React Query model, name it and say what it buys, perceived latency near zero.
• Debouncing search-as-you-type. Wait ~200-300ms of typing silence before firing, cancel in-flight requests on new input, ignore out-of-order responses (or use AbortController).
• Offline and retry on the client. Queue mutations locally when offline, replay with idempotency keys when connectivity returns, mark UI state as pending so the user isn't lied to.

Tradeoffs stated as sentences

Every choice gets its cost attached, in one sentence. Three examples of the shape.

• "I'm choosing fan-out on write, which makes reads a single fetch, at the cost of expensive writes for celebrity accounts, which I'll handle with a hybrid read-time merge."
• "JWTs let us verify auth without a session store lookup, at the cost of hard revocation, so I'll keep access tokens under 15 minutes."
• "Denormalizing the username into each message makes conversation reads one query, at the cost of a background rewrite job when someone renames, which is rare."

Chapter 14: Search and Elasticsearch

Why the database can't do it. LIKE '%design%' can't use a B-tree, there's no prefix to seek to, so it scans every row. The moment requirements say "search posts by keyword," you need a different index shape.

The inverted index. Instead of row → words, store word → list of documents containing it (the posting list). Text is tokenized, lowercased, and stemmed ("running" → "run") at index time, a query does the same to its terms, intersects the posting lists, and ranks the matches. Ranking in one sentence, BM25 (the modern TF-IDF), a document scores higher when the term appears often in it but rarely across the corpus. That's Elasticsearch's entire core.

How it fits an architecture. Elasticsearch is a secondary index over your source of truth, never the source of truth itself. The database stays authoritative, and changes flow to ES asynchronously via CDC or a queue (Chapter 11's outbox pattern is exactly the machinery). That means search is eventually consistent, a new post is findable a few seconds after it's created, and you say that staleness window out loud. If ES loses data you reindex from the database, which is also your answer to "what if the search cluster dies."

Scope check for full stack. Know the inverted index, the CDC feed, and the staleness. Skip shard internals, analyzers in depth, and cluster tuning. And know that typeahead is a different problem, it's prefix matching against a small hot set with precomputed answers, not full-text search, the canonical design in the next chapter covers it.

Chapter 15: The ten canonical designs

1. Twitter / newsfeed

Requirements. Post tweets, follow users, view a reverse-chronological home timeline. Non-functional, read-dominated (~1000:1), timeline load under 200ms, eventual consistency fine (a tweet appearing after 5s is acceptable).

Capacity. 200M DAU, 2 timeline loads/user/day = ~5K QPS timeline reads, peak ~20K. Writes, 100M tweets/day ≈ 1K QPS. Tweets at 1KB = 100GB/day raw.

POST /tweets                     { text }
GET  /timeline?cursor=&limit=    → { tweets[], next_cursor }
POST /users/:id/follow

Data model. tweets(id, user_id, text, created_at), follows(follower_id, followee_id), timelines materialized in Redis sorted sets, key timeline:{user_id}, member tweet_id, score timestamp, capped at ~800 entries.

Architecture. Tweet write → DB → queue → fan-out workers insert the tweet ID into every follower's Redis sorted set. Timeline read is one ZRANGE plus a hydration multi-get of tweet bodies (fetch the actual tweet text for each ID in one batched call).

Why the fan-out can't silently fail, three details worth volunteering. First, the tweet row and the fan-out job commit together via the outbox pattern (Chapter 11), so a crash between "saved" and "queued" can't lose the fan-out. Second, jobs only leave the queue when a worker finishes them, a worker that crashes mid-job means the job reappears and another worker redoes it. Third, that redo is harmless because the work is naturally idempotent, ZADD of the same tweet ID into the same sorted set is a no-op the second time. Large jobs are chunked into sub-jobs of ~1,000 followers so a retry repeats one cheap chunk, not 2M inserts.

Deep dives interviewers steer toward.

• Fan-out on write vs read. Write fan-out makes reads one fetch but a 50M-follower account triggers 50M inserts per tweet. Read fan-out (merge followees' recent tweets at request time) makes writes cheap and reads expensive.
• Hybrid celebrity approach. Threshold on follower count. Normal users fan out on write. Celebrity tweets are not fanned out, readers merge them in at read time from a small per-celebrity list. Best of both, this is the expected answer.
• Hot partitions. Redis timeline keys are per-user so they distribute, but the celebrity tweet list itself is hot, replicate it across cache nodes.

Classic follow-ups. "User follows someone new, timeline?" Backfill their recent tweets into the timeline async, or accept they appear from now on. "Deleted tweet?" Tombstone check at hydration, cheaper than scanning every timeline. "Redis node dies?" Timelines are rebuildable from the social graph and tweet store, rebuild lazily on first read, serve read-fan-out meanwhile. The deeper point to volunteer, timelines are derived data, the tweets table and follow graph are the truth, so even a bug that eats fan-out jobs for an hour means degraded latency while rebuilding, never data loss.

2. URL shortener / pastebin

Requirements. Create short link, redirect fast, optional expiry, click analytics. Read-dominated, ~100:1. Redirect latency is the product.

Capacity. 100M new URLs/month ≈ 40 writes/s. 10B redirects/month ≈ 4K QPS reads. Each mapping ~500B, 100M/month = 50GB/month, ~600GB/year, single-machine territory for years.

POST /urls        { long_url, custom_alias? } → { short_code }
GET  /:code       → 301/302 redirect

Data model. One table, urls(code PK, long_url, owner, created_at, expires_at). Cache layer in front, code → long_url.

Deep dives.

• Key generation. Counter + base62 gives short sequential codes but needs a coordinated counter (single point, or ranges leased per server) and codes are guessable. Hash the URL and take a prefix, no coordination but collisions need probing, and same URL maps to same code (feature or bug). Pre-generated pool of random codes, a key service hands them out, no collision check at write time, this is the clean answer.
• 301 vs 302. 301 permanent means browsers cache the redirect and never hit you again, cheap but you lose analytics and can't change the target. 302 keeps every click flowing through you. Choose based on whether analytics matters, and say so.
• Read path. Cache-aside on codes, hot links stay in Redis, the DB sees only the long tail. Negative-cache unknown codes to block penetration.

Classic follow-ups. "Analytics without slowing redirects?" Fire the click event to a queue, aggregate async, the redirect never waits. "Expired links?" Lazy check at read plus a background sweeper. "Custom aliases?" Same table, uniqueness conflict returns 409.

3. Chat (WhatsApp)

Requirements. 1:1 and group messages, delivery receipts, online presence, ordering within a conversation. Latency near-instant, no lost messages. Roughly balanced read/write.

Capacity. 500M DAU × 40 messages/day = 20B/day ≈ 200K messages/s average. At ~1KB each, ~20TB/day, sharded LSM storage from day one. Millions of concurrent WebSocket connections, at ~100K-1M connections per gateway box you need a fleet.

WebSocket  wss://chat  (send, receive, ack, presence frames)
GET  /conversations/:id/messages?cursor=   (history, HTTP)
POST /conversations                        (create group)

Data model. messages partitioned by conversation_id, clustered by (conversation_id, message_id) where message_id is time-ordered (Snowflake-style). One partition per conversation makes "load this chat" a single-partition range scan.

Architecture. Client holds a WebSocket to a gateway. A session/registry service maps user → gateway. Send path, message hits gateway → persisted → routed to recipient's gateway if online, else queued for push notification and offline sync.

Deep dives.

• Ordering. Global ordering is impossible and unnecessary. Order per conversation via a sequence number or time-ordered ID assigned by one writer per conversation. Clients render by that order and handle late arrivals.
• Delivery receipts. Sent (server persisted), delivered (recipient device acked), read (recipient viewed). Each is just a small message flowing back through the same pipe.
• Presence. Heartbeats over the socket update a TTL'd Redis entry, expiry means offline. Don't broadcast every flap, subscribers poll or get batched updates.
• Group fan-out. A message to a 200-member group is one persist plus 200 routing lookups, done through a queue. Cap group size, this is why WhatsApp caps groups.

Classic follow-ups. "Recipient offline?" Persist, push notification, deliver on reconnect from their inbox queue. "Multiple devices?" Per-device delivery cursors into the conversation log. "Gateway dies?" Clients reconnect to another via LB, registry updates, undelivered messages replay from storage.

4. Rate limiter + notification system

Rate limiter requirements. N requests per user per window, enforced across a fleet, decision in under a millisecond, fail open or closed as a stated choice.

Deep dives.

• Token bucket vs sliding window. Token bucket, a counter refills at rate R up to burst B, allows short bursts, two numbers per key, the usual pick. Fixed windows allow 2x bursts at boundaries. Sliding window log is exact but stores a timestamp per request. Sliding window counter (weighted blend of two fixed windows) is the practical compromise.
• Distributed enforcement. Counters live in Redis, atomicity via Lua script or INCR+EXPIRE, so every app server shares state. Latency-sensitive paths use a local allowance synced to Redis periodically, slightly loose, much faster.
• Failure stance. Redis down, fail open (availability, risk abuse) or closed (protection, reject legit users). Say the choice out loud, that is the point of the question.

Notification system. Event producers → queue → notification service, which checks user preferences (channel opt-ins, quiet hours), dedupes (idempotency key per event so retries don't double-send, plus collapse rules like "3 likes → one notification"), rate-limits per user (nobody wants 200 pushes), then fans out to channel workers, push (APNs/FCM), email, SMS, in-app. Each channel worker retries with backoff into a DLQ.

Classic follow-ups. "Exactly one email despite retries?" Idempotency key on the send, checked before dispatch, at-least-once queue plus dedupe. "Priority?" Separate queues per priority so OTPs never wait behind marketing. "429 response?" Include Retry-After so clients back off properly.

5. Netflix / YouTube

Requirements. Upload video, transcode, stream globally with adaptive quality, browse metadata, track watch history. Reads (views) massively dominate writes (uploads).

Capacity. YouTube-scale, ~500 hours uploaded per minute, but views per video follow a power law, a tiny fraction of content serves most traffic, which is exactly the CDN's job. A 1-hour 1080p video at ~5Mbps is ~2GB, times renditions ≈ 5-6GB stored per hour of content.

POST /videos              → presigned upload URL (multipart, resumable)
GET  /videos/:id          → metadata + manifest URL
GET  /manifest/:id.m3u8   → rendition playlist (served from CDN)

Data model. Metadata (title, uploader, duration, status) in SQL, it is small, relational, and queried flexibly. View counts and watch history in NoSQL (Cassandra partitioned by user), a raw write firehose. Video bytes in object storage (S3), never in a database.

Architecture. Upload → object storage → queue → transcoding workers produce renditions (240p→4K) chunked into segments → segments pushed to CDN origin. Playback, client fetches manifest, then pulls segments from the nearest CDN edge.

Deep dives.

• Transcoding as async jobs. A DAG per video, split into segments, transcode segments in parallel across workers, stitch. Video is "processing" until done, status flows back via the metadata DB. Retry per-segment, not per-video.
• CDN and edge. Popular content sits on edge servers near users (Netflix Open Connect boxes inside ISPs). Origin only serves cache misses. This is why power-law traffic is survivable.
• Adaptive bitrate. Each rendition is segmented (2-10s chunks). The client measures its own bandwidth and picks the rendition per segment, switching seamlessly. The server is dumb, the client adapts, that inversion is the insight.

Classic follow-ups. "Resumable uploads?" Multipart with per-chunk offsets, client retries only failed chunks. "View count at scale?" Don't INCR a row 100K times/s, buffer counts in memory or a stream, flush aggregated deltas. "Instant start?" Preposition the first segments of popular titles on the edge and start on the lowest rendition while measuring bandwidth.

6. Ticketmaster / booking

Requirements. Browse events, view a seat map, reserve specific seats, pay. The defining property is contention, two users want the same seat, and correctness beats latency. Traffic is violently spiky, an on-sale moment brings 10M people for 50K seats.

Capacity. The data is tiny, an arena is 50K rows. The problem is peak concurrency and correctness, say that out loud, it reframes the whole design away from storage scale.

Frame it as two different problems that collide at checkout, and name them separately, that alone is a senior move. Consistency, two different users race for the same seat, solved with concurrency control. Idempotency, the same user's request arrives twice, a timeout then a retry, a double click, solved with deduplication. Everything below is one of the two.

GET  /events/:id/seats                    → seat map with statuses
POST /reservations  { seat_ids }          → 201 hold (expires_at) | 409 taken
POST /reservations/:id/confirm  (payment) → 200 booked

Data model. SQL, non-negotiable. seats(event_id, seat_id, status, hold_user, hold_expires_at), status is free → held → booked.

Deep dives.

• Two-phase booking with TTL holds. Reserve places a 5-10 minute hold, payment confirms it, expiry frees the seat (checked lazily at read plus a background sweeper). Never lock a seat for the whole checkout without a TTL, abandoned carts would strand inventory.
• Winning the race, two options. Pessimistic, SELECT ... FOR UPDATE locks the row while you decide, correct but the lock is held while your application code thinks. Optimistic conditional write, UPDATE seats SET status='held', hold_user=:u WHERE seat_id=:s AND status='free', then check rows-affected, 1 means you won, 0 means someone beat you, return 409. The WHERE status='free' is the check and the UPDATE is the set, fused into one atomic operation that the database serializes, so there is no window where both users see "free". A third variant, insert into a bookings table with a unique constraint on (event_id, seat_id), the second insert violates the constraint and the DB itself rejects it. All three work, the conditional write is the cleanest to narrate.
• Idempotency for retries. The client generates an idempotency key when the checkout button first renders and reuses it verbatim on every retry of that action. The server keeps idempotency_keys(key PK, status, response) and tries to insert the key first. Winning the insert means do the work and store the result, finding the key with a stored result means return that result without redoing anything, finding it still in-flight (a concurrent duplicate) means wait or return 409. The subtlety that separates good from great, the key row and the seat update must commit in the same database transaction. Write the booking, crash before recording the key, and the retry books again, you have rebuilt the dual-write bug inside one service.
• The payment hop. The card processor is external, no transaction spans you and them. Persist your intent first, then call the processor with the processor's own idempotency key (Stripe supports these for exactly this reason). Now the crash matrix is safe, crash before the call means the retry calls fresh, crash after the call but before recording the outcome means the retry replays the same key and gets the same result back instead of charging twice. Reconciliation (design 9) is the backstop for whatever still slips.
• Surviving the on-sale spike. A virtual waiting room, users queue and are admitted at a rate the booking path can sustain, everyone else sees a position number instead of an error. The seat map itself is served from cache with a short TTL and can be slightly stale, because the reservation write is the authoritative check anyway.

Classic follow-ups. "Hold expires while the user is paying?" Payment confirms only if the hold is still valid and still theirs, otherwise 409 and an apologetic UI. "Live seat map?" Push seat-status deltas over SSE or just poll every few seconds, staleness is safe since reserve is the gate. "Bots?" Rate limit per account and IP, and the waiting room with randomized admission removes the prize for being fastest.

7. Typeahead / autocomplete

Requirements. Suggestions under ~100ms while the user types, top-k results ranked by popularity, freshness can lag by hours. The insight to state up front, this is not search, it's a lookup of precomputed answers, latency is the entire product.

Capacity. Every keystroke is a query. 10M DAU × 25 keystrokes/day ≈ 250M queries/day ≈ 3K QPS average, 10K+ peak, with tiny payloads. Read-to-write ratio is effectively infinite, the "writes" are an offline pipeline.

GET /suggest?q=tay&limit=10 → { suggestions: ["taylor swift", ...] }

Design. An offline pipeline aggregates query logs daily, computes the top 10 completions per prefix, and bulk-loads a store of prefix → [suggestions]. Serving is one Redis GET per keystroke. That's the whole system, and saying it that plainly is the senior move.

Deep dives.

• Trie vs flat prefix table. A trie is the textbook answer, walk the prefix, collect top-k below. A flat hash table of every prefix (up to some length) mapping to its precomputed top-10 is simpler, faster, and only modestly redundant in storage. Prefer flat, mention the trie exists.
• The client half. Debounce 150-300ms, cancel in-flight requests on new input (AbortController), cache prefix results locally so backspacing is free. As a full-stack candidate, own this part loudly.
• Freshness. Daily batch handles the long tail. For trending ("breaking news"), a second fast path counts recent queries over a sliding window and merges its results in at read time.

Classic follow-ups. "Trending topic in minutes?" The fast-path stream above. "Typos?" Fuzzy matching is expensive, punt it to full search, typeahead stays exact-prefix. "Personalization?" Re-rank the top-k client-side or blend a small per-user history list, don't precompute per user.

8. Live metrics dashboard (fleet monitoring)

Requirements. Thousands of sources (vehicles, servers, devices) emit metrics continuously, dashboards show charts fresh within seconds, historical queries go back months. A write firehose meets aggregate-only reads, nobody ever reads one raw point.

Capacity. 100K sources × 100 metrics × every 10s = 1M points/s. Immediately say the agents batch, one compressed POST per source per 10s is 10K requests/s, entirely manageable, batching at the edge is the first real design decision.

POST /ingest            { source_id, points: [...] }   (batched, gzipped)
GET  /query?metric=&window=1m&from=&to=                (dashboard reads rollups)
SSE  /live?metrics=                                    (push fresh points)

Architecture. Agents → ingest gateway → Kafka → stream aggregator computing per-metric per-window rollups (1m, 1h averages, counts, percentiles) → time-series store partitioned by metric and time window. Dashboards read rollups only. Live view gets SSE pushes or honest 5-second polling.

Deep dives.

• Downsampling is the storage story. Raw 10s data kept for days, 1-minute rollups for months, 1-hour rollups forever. Queries pick resolution by time range, a 6-month chart reads hourly rollups, ~4K points, instead of 1.5M raw ones. Do the math out loud, it justifies the whole pipeline.
• Percentiles don't average. You cannot combine per-minute p95s into an hourly p95. Store histograms or sketches per window and merge those. Knowing this one fact separates people who've run dashboards from people who've read about them.
• Push vs pull to the client. SSE is the fit, one-way, auto-reconnecting. WebSockets are overkill without client → server traffic. Polling every 5s is defensible, say why, simplicity, and the data is windowed anyway.
• Freshness vs completeness. Late data arrives after its window closed. Dashboards prefer fast-and-approximate (emit the window on time, patch late points quietly). Anything billable prefers complete, which means waiting. Name the tradeoff, pick per consumer.

Classic follow-ups. "Alerting?" Evaluate rules inside the stream aggregator, not by polling the dashboard path, alerts can't depend on the pretty path being up. "One source floods?" Per-source rate limits and sampling at the gateway. "A dashboard for 500 vehicles at once?" Pre-aggregate fleet-level rollups too, don't merge 500 series at read time.

9. Payments + webhooks (Plaid / Stripe flavor)

Requirements. Accept payment requests, guarantee exactly-one charge per user intent despite retries and crashes, notify merchant systems via webhooks, reconcile against the external processor. Low QPS, maximal correctness, this inverts every previous design's priorities and you should say so.

POST /payments   Idempotency-Key: abc123   { amount, currency, source }
→ 201 { payment_id, status: "processing" }
GET  /payments/:id                          (poll status)
webhook out:  POST merchant_url  X-Signature: hmac(...)  { event_id, type, data }

Data model. SQL. A payments table as a strict state machine, created → processing → succeeded | failed, transitions are transactional and logged. An append-only double-entry ledger, every movement is a debit row and a credit row, balances are sums over entries, never UPDATEd in place. Immutability is what makes audit and dispute resolution possible, corrections are new reversing entries.

Deep dives.

• Idempotency end to end. Client sends an idempotency key, server stores key → result and replays it on retry. Calling the external processor, persist the intent first, then call with the processor's own idempotency key, so a crash-and-retry can't double-charge. Every hop repeats the same trick, that repetition is the design.
• Webhooks out. The Chapter 11 machinery in full, outbox table in the payment transaction, dispatcher signs with HMAC, retries with backoff for days, per-endpoint circuit breaker, DLQ plus a queryable events API so merchants can backfill anything missed.
• Reconciliation. A nightly job diffs the processor's settlement report against internal state, mismatches go to a human queue. This is the honest answer to distributed transactions across companies, you cannot 2PC with a bank, you detect and repair instead.

Classic follow-ups. "Crash after charging the card but before recording it?" The intent row was persisted before the call, the retry replays the processor idempotency key, and reconciliation is the backstop. "Merchant endpoint down three days?" Retry schedule covers days, then DLQ plus the replay API. "Why double-entry?" Sums always balance to zero, so bugs surface as detectable imbalances instead of silently wrong balances.

10. Collaborative docs / multi-tenant app (Retool flavor)

Requirements. Organizations (tenants) with users and roles, shared editing of documents or internal apps, permissions enforced everywhere, audit trail. The product being sold is correct isolation between tenants, state that as the top requirement.

Tenant isolation, the core table.

Model	How	Tradeoff
Shared tables + tenant_id	Every row carries tenant_id, every query filters on it	Cheapest, scales to millions of tenants. One missed WHERE clause is a data leak, so enforce centrally (query layer or Postgres row-level security as belt and braces).
Schema per tenant	Same DB, separate namespace each	Stronger isolation, painful migrations at thousands of tenants.
Database per tenant	Full separation	Best isolation and noisy-neighbor story, highest cost. Sell it to enterprise customers as a tier.

Default answer, shared tables with centrally-enforced tenant_id filtering, big customers can graduate to dedicated.

Deep dives.

• RBAC. Users get roles, roles carry permissions, scoped to a resource level (org → workspace → doc). Checks run server-side on every request, cached briefly with invalidation on permission change. Hidden buttons are UX, not security, say it explicitly.
• Concurrent editing, three tiers of answer. For form-like data, per-field last-write-wins with a version check, UPDATE ... WHERE version = :seen, a 409 triggers refetch-and-merge in the UI. For rich text, name OT (a central server transforms concurrent ops, Google Docs) and CRDTs (structures that merge without coordination, more metadata, offline-friendly) and firmly don't implement either in the interview. Most products need the version-check tier, and knowing where the line sits is the senior signal.
• Optimistic UI. Apply the edit locally, send with the version, reconcile on response. Presence and cursors ride a WebSocket, ephemeral, in Redis, never persisted.
• Noisy neighbors. Per-tenant rate limits and query quotas, separate connection pools per tier, so one tenant's runaway dashboard can't starve the fleet.

Classic follow-ups. "Audit log?" Append-only events table written via the same outbox pattern, it doubles as the activity feed. "A tenant demands their data stays in the EU?" Region-pinned tenants, the directory maps tenant → home region, requests route there. "Permission check on every request too slow?" Cache decisions with a short TTL and bust on change, and note the tradeoff window where a revoked user has seconds of residual access.

Chapter 16: Rapid-fire probe topics

Idempotency keys. Client generates a unique key per logical operation and sends it with the request. Server stores key → result and returns the stored result on any retry. Turns at-least-once delivery into effectively-once processing. The answer to every "what if the request is retried" question, especially payments.

Snowflake-style IDs. Auto-increment IDs need one coordinator, dead at scale, and UUIDs sort randomly, which fragments B-tree indexes and can't order a feed. Snowflake IDs pack timestamp + machine ID + sequence into 64 bits, generated locally with no coordination, unique, and roughly time-sorted, so they work as both primary key and sort key. The answer to "how do you generate IDs across many servers."

Tail latency, p99 thinking. Averages lie, latency is a distribution, and the p99 matters because at scale your best customers hit it constantly, one page fanning out to 10 backend calls makes ~1 in 10 page loads eat a p99. Fixes, cap fan-out, budget timeouts, and hedged requests, send a duplicate request to a second replica if the first is slow and take whichever answers first. Saying "the average hides the tail" at the right moment is a strong signal.

Connection pooling. Database connections are expensive, and Postgres handles only a few hundred well. A pool (or PgBouncer in front) keeps warm connections and multiplexes thousands of app requests over tens of connections. It's the mundane answer to "we added app servers and the database fell over", every new server brought its own connection swarm.

Geo indexing. "Find drivers near me" can't use a B-tree, two dimensions don't sort into one order. Geohashing divides the map into cells whose names share prefixes with their neighbors, so a proximity query becomes a prefix match on the cell ID plus a check of the 8 surrounding cells. Quadtrees are the same idea as a tree. Name either, describe the cell trick, done, that's Uber and Yelp's core lookup.

Cursor pagination. The cursor is an opaque token encoding the last-seen sort key. Next page is WHERE sort_key < cursor LIMIT n, an index seek, constant cost at any depth, stable when rows are inserted or deleted mid-scroll. OFFSET scans and discards everything before the offset and skips or duplicates rows under concurrent writes.

Bloom filters. A bit array plus k hash functions answering "possibly present" or "definitely absent", with no false negatives. A few bits per element. LSM stores keep one per SSTable so reads skip files that can't contain the key, and caches use them to block penetration by nonexistent-key lookups.

Leader election, hand-wavy on purpose. Distributed systems often need exactly one node doing a job (primary DB, lock holder). Nodes agree on a leader via a consensus service (ZooKeeper, etcd), and when the leader's heartbeat lapses a new one is elected. Know the shape, name the tools, and skip Paxos/Raft internals, they are deprioritized in product interviews unless the role is infra.

Chapter 17: Self-quiz

Phrased the way interviewers phrase them. Answer out loud before revealing.

1. Why would you choose Cassandra over Postgres for viewing history?

Workload shape, write-heavy append stream. Access pattern, always fetched by user_id, a perfect partition key. Consistency tolerance, staleness of seconds is fine. LSM writes plus single-partition reads fit exactly, and no query needs a JOIN or a transaction.

2. What breaks when your partition key is celebrity_id?

Hot partition. One node absorbs all traffic for that entity while the rest idle. Fix with a compound key, key salting to spread across N partitions, or a separate read-time path for entities above a threshold.

3. When is SQL actually too slow?

Not at some row count. When data exceeds one machine, so JOINs become cross-shard network operations, and when transactions span shards, forcing two-phase commit. Indexed lookups stay fast, 3-4 reads for a billion rows.

4. You denormalized the avatar URL into every comment. The user changes their photo. Now what?

Either a background job rewrites every document containing it, or you accept staleness until documents are touched. State the choice, for avatars staleness is usually acceptable, for display names maybe not. NoSQL pays at update time.

5. Why do cursors beat OFFSET for pagination?

OFFSET scans and discards all preceding rows, cost grows linearly with depth, and concurrent inserts cause skipped or duplicated rows. A cursor is an index seek on the last-seen sort key, constant cost, stable under writes.

6. What breaks when a hot cache key expires?

Thundering herd. Thousands of concurrent misses hit the database simultaneously. Fix with request coalescing so one request refills while others wait, jittered TTLs, or serve-stale-while-revalidate.

7. Why must queue consumers be idempotent?

Delivery is at-least-once. A consumer can process a message, crash before committing its offset, and process it again on restart. Dedupe on message ID or use idempotency keys so reprocessing is harmless.

8. What caps consumer parallelism in Kafka?

Partition count. Each partition is owned by exactly one consumer in a group, so 8 partitions means at most 8 active consumers. Extra consumers idle. Size partitions for target throughput upfront.

9. Why fan-out on write for timelines, and when does it fail?

Reads outnumber writes ~1000:1, so precomputing feeds at write time makes the common op one cache fetch. It fails for celebrities, one tweet becomes 50M inserts. Hybrid, fan out normal users on write, merge celebrity tweets at read time past a follower threshold.

10. 301 or 302 for a URL shortener?

302 if analytics matters, every click flows through you. 301 is cached by browsers, cheaper but you lose click data and can't retarget the link. The answer is the tradeoff, not the number.

11. What breaks when a chat gateway server dies?

Its WebSocket connections drop. Clients auto-reconnect through the LB to another gateway, the session registry updates the user → gateway mapping, and messages that arrived meanwhile replay from persistent storage. Nothing is lost because persistence happens before routing.

12. How do you prevent double-booking in a reservation system?

Strong consistency on the contended resource. A transaction with a unique constraint or row lock on (room, date), second writer gets a 409. This is why bookings stay on SQL. Optionally a short hold with TTL for the checkout flow.

13. Why is exactly-once delivery mostly a myth?

The moment a consumer touches an external system, the send and the offset commit can't be atomic, a crash between them causes a redo. Kafka transactions give exactly-once within Kafka only. Real systems use at-least-once plus idempotent consumers.

14. What does a circuit breaker buy you that retries don't?

Retries handle transient blips but amplify sustained failures, every caller retrying a dead service is a self-inflicted DDoS and threads pile up waiting. A breaker fails fast after a failure threshold, serves fallbacks, and probes for recovery, protecting both caller and callee.

15. Sessions or JWTs?

Sessions, instant revocation and simple, but every request costs a store lookup and scaling needs shared session state. JWTs, stateless verification, scales trivially, but revocation before expiry requires a blocklist, which is state again. Common answer, short-lived JWT plus refresh token.

16. What happens when consumers fall behind the queue?

Lag grows, downstream data goes stale, and eventually retention limits delete unprocessed messages. Monitor lag trend per consumer group. Remedies in order, scale consumers to partition count, add partitions, then shed or sample load.

17. Why does adaptive bitrate streaming put the logic on the client?

Only the client knows its instantaneous bandwidth and buffer state. Video is pre-transcoded into renditions and segmented, the client picks a rendition per segment. The server stays dumb and cacheable, which is what makes CDNs work for video.

18. Your Redis timeline cluster loses a node. What breaks and how do you recover?

Timelines on that node are gone but they're derived data. Rebuild lazily, on a cache miss fall back to read-time fan-out from the follow graph and tweet store, repopulate the sorted set. Degraded latency for affected users, no data loss.

19. Why negative-cache "not found" results?

Cache penetration. Requests for nonexistent keys miss the cache every time and hammer the DB, whether malicious or a buggy client. Cache the miss with a short TTL, or keep a bloom filter of valid IDs in front.

20. Interviewer asks "could you just use Postgres for all of this?" What's the strong answer?

Often yes, and say so. "Sharded Postgres carries us to roughly X scale, Instagram proved that. The piece that outgrows it first is the write firehose of Y, whose single-partition access pattern fits Cassandra. I'd keep payments and bookings on SQL regardless, money needs transactions." Nuance beats reflexive NoSQL.

21. You have an index on (user_id, created_at). Does a query filtering only on created_at use it?

No, composite indexes obey leftmost prefix. It serves "by user_id" and "by user_id ordered by created_at", but created_at alone needs its own index. And remember the cost side, every index is maintained on every write, so index the top queries, not every column.

22. A user posts a comment, refreshes, and it's gone. What happened and what's the fix?

Replication lag, the write hit the leader, the read hit a follower that hasn't caught up. Fix with read-your-own-writes, pin the author's reads to the leader briefly after a write, or carry a version token that replicas must catch up to. Everyone else's reads can stay on replicas.

23. Range or hash sharding for time-series data, and why?

Naive range sharding on time sends all current writes to the newest shard, one hot machine and the rest idle. Hash the source or entity ID for even write spread, or use a compound key like (source, time window). Range's advantage, cheap time scans, is recovered by partitioning within each shard.

24. N=3 replicas, writes ack at W=1 for speed. What did you give up, and what does W=2, R=2 buy?

At W=1 a read at any R can miss the latest write, and a crashed node can lose acked writes. W+R>N is the guarantee, with W=2, R=2 of 3, any read quorum overlaps the latest write quorum, so reads see the newest value. Tunable per query, that's what "tunable consistency" means.

25. How do you invalidate CDN-cached JavaScript after a deploy?

You don't, you avoid needing to. Content-hash the filename (app.3f9c21.js), cache it for a year as immutable, and deploy new HTML referencing the new name. The HTML has a short TTL. Purge APIs are slow to propagate and a global purge sends a thundering herd at the origin, reserve purging for emergencies.

26. Users upload 2GB videos. How do the bytes get to storage without melting your app servers?

Presigned URLs. The server signs a short-lived URL authorizing a PUT to a specific S3 key, the client uploads directly, multipart so parts upload in parallel and retry independently. An S3 event then kicks off processing through a queue. App servers only ever handle metadata.

27. Your service writes to Postgres and publishes an event to Kafka. What's the bug and the fix?

Dual writes, a crash between the two leaves the DB and stream disagreeing, and no transaction spans both. Fix with the outbox pattern, write the business row and event row in one DB transaction, a relay tails the outbox and publishes. Or CDC, tail the DB's replication log. Either way delivery is at-least-once, consumers dedupe.

28. Design the retry story for webhooks you send to merchants.

HMAC-sign payloads with an event ID and timestamp. Exponential backoff over hours to days, per-endpoint circuit breaker, then a DLQ plus a queryable events API so merchants can replay what they missed. Receivers must dedupe on event ID and ack fast, processing async. Don't promise ordering, send full state or versions.

29. Two users click the last seat at the same instant. Walk the mechanism that picks one winner.

An atomic conditional write, UPDATE seats SET status='held', hold_user=:u WHERE seat_id=:s AND status='free'. The database serializes the two updates, one affects 1 row and wins, the other affects 0 and gets a 409. TTL on the hold so an abandoned checkout frees the seat. No application-level lock needed.

30. Typeahead must answer in under 100ms at 10K QPS. What's the architecture?

Precompute. An offline pipeline aggregates query logs and stores top-10 completions per prefix, serving is one Redis GET per keystroke. It's a lookup, not a search. The client debounces 150-300ms, cancels in-flight requests, and caches locally. Trending terms get a separate fast path merged at read time.

31. Can you average per-minute p95 latencies into an hourly p95?

No, percentiles don't compose. The hourly p95 must come from the full distribution, so store histograms or percentile sketches per window and merge those. Averaging percentiles silently understates the tail, which is exactly the part you were trying to watch.

32. Why is Elasticsearch never your source of truth, and how does data get into it?

It's a secondary index optimized for search, not durability or transactions, and if it's ever wrong or lost you want something to rebuild from. The database stays authoritative and changes stream in via CDC or an outbox-fed queue, so search is eventually consistent by a few seconds, a staleness you state out loud.

33. A fan-out worker crashes halfway through a job. Walk the recovery, in SQS terms and in Kafka terms.

SQS, the message went invisible when the worker received it, was never deleted, so the visibility timeout lapses and it reappears for another worker. Kafka, the offset was never committed, the group rebalances and another worker re-reads the partition from the last committed offset. Both redeliver, so the job runs again, which is safe because the work is idempotent, re-ZADDing the same tweet IDs is a no-op.

34. Why must the idempotency-key record commit in the same transaction as the booking it guards?

If the booking commits and the server crashes before recording the key, the retry finds no key and books again, a double booking despite "having" idempotency. Committing key and state change atomically means either both exist or neither does, so a retry always sees a consistent picture. Separate writes to the same database is just the dual-write bug wearing a disguise.

35. Why is "fan out to 2M followers" as a single queue job an anti-pattern?

A crash at 90% redelivers the whole job and redoes 2M inserts, one worker grinds alone while the fleet idles, and the job blocks its queue slot for minutes. Chunk it, the first worker fetches the follower list and emits sub-jobs of ~1,000 followers each. Retries become cheap, work spreads across all workers, and a failure repeats one small idempotent chunk.

36. How does S3 validate a presigned URL without ever calling your server?

The URL carries a cryptographic signature your server produced with its credentials, covering the method, key, and expiry. S3 recomputes the signature and checks it matches, math, not a lookup. That's why the client can upload gigabytes directly to S3 while your API only handled the tiny signing exchange.