System Design NotesRohit’s interview study guide
Chapter 02

Scaling Fundamentals

Getting from one server to many: scaling up vs out, stateless services, load balancers and their algorithms, failover, DNS and CDNs, gateways, autoscaling and running in several regions.

10 topics
Parts marked Advanced are extra depth. Skip them on a first read or a quick revision.

#Vertical vs horizontal scaling

Added

When one server can't keep up, you have two ways out. Vertical scaling (scaling up) means a bigger machine: more vCPUs, more memory, faster disks. Horizontal scaling (scaling out) means more machines of the same size, with something in front sharing the work between them. Every design interview reaches this fork within the first ten minutes, usually as "what happens when traffic grows 10×?".

The mental model is one bigger lorry vs a fleet of vans. One big lorry is simple: one driver, one route, everything in one place, and no coordination. But there is a biggest lorry anyone builds, it costs disproportionately more at the top end, and if it breaks down the whole delivery stops. A fleet of vans can grow one van at a time, keeps delivering when one breaks down, and can be parked when it's quiet. But now you need a dispatcher, each van can only carry what fits in it, and anything the vans must agree on (who has which parcel) needs radio chatter.

So the trade-off isn't "horizontal good, vertical bad". Vertical scaling is the right first move far more often than interview answers admit: it needs no code change, keeps transactions on one machine, and today's largest machines are enormous. Horizontal scaling is what you need for fault tolerance (one machine is always a single point of failure) and for growth past the biggest box. The skill is knowing which tier can scale out cheaply (stateless web and API servers) and which can't (the database, anything with shared state).

Vertical (scale up)Horizontal (scale out)
What changesMachine sizeMachine count
Code changesNoneServices must be stateless, or the state must be partitioned
CeilingLargest machine on salePractically none for stateless tiers; coordination costs for stateful ones
FailureOne machine = one point of failure; a standby doubles the costLosing one of N removes 1/N of capacity
Growth stepsBig jumps (double the machine), often with a restartSmall steps, one instance at a time, no downtime
Cost curveRises steeply at the top endRoughly linear, plus the load balancer and operational overhead
Typical useDatabases, caches, early-stage appsWeb and API tiers, workers, CDNs, anything stateless

How big is "big"? The largest instance AWS has announced is the U7inh high-memory instance: 1,920 vCPUs and 32 TiB of memory on one machine (AWS announcement, December 2024; check the current instance list before quoting it as the maximum). Few application tiers will ever outgrow that on CPU alone, but buying one, and a second one as a standby, is a very different bill from buying vans.

Before choosing, put numbers on it. Here's the API tier for a product with 20 million daily users, sized both ways:

sizing.mjs
// How many vCPUs does the API tier need at peak, and what does that look like as one box vs many?
const dau = 20_000_000;          // daily active users
const requestsPerUser = 50;      // API calls per user per day
const peakFactor = 3;            // busiest minute vs the daily average
const cpuMsPerRequest = 4;       // measured CPU time per request (an assumption here)
const targetUtilisation = 0.6;   // leave headroom for spikes and GC

const avgQps = (dau * requestsPerUser) / 86_400;
const peakQps = avgQps * peakFactor;
const busyVcpus = (peakQps * cpuMsPerRequest) / 1000; // CPU-seconds needed per second
const vcpus = busyVcpus / targetUtilisation;
console.log(`average ${Math.round(avgQps).toLocaleString("en-US")} req/s, peak ${Math.round(peakQps).toLocaleString("en-US")} req/s`);
console.log(`busy vCPUs at peak ${busyVcpus.toFixed(0)}, provisioned at 60%: ${Math.ceil(vcpus)} vCPUs`);

// Horizontal: 8-vCPU instances spread over 3 availability zones, surviving the loss of one zone.
const perInstance = 8, zones = 3;
const needed = Math.ceil(vcpus / perInstance);
const perZone = Math.ceil(needed / (zones - 1)); // any 2 zones must carry the whole peak
console.log(`horizontal: ${needed} × ${perInstance}-vCPU instances, ${perZone} per zone × ${zones} zones = ${perZone * zones} instances`);

// Vertical: one big machine, plus a standby of the same size for failover.
console.log(`vertical: one ${Math.ceil(vcpus)}-vCPU machine + an identical standby = ${2 * Math.ceil(vcpus)} vCPUs bought`);

// Ten times the users later:
const later = Math.ceil((vcpus * 10) / perInstance);
console.log(`at 10× users: ${later} instances (${Math.ceil(later / (zones - 1)) * zones} with zone headroom), or a ${Math.ceil(vcpus * 10)}-vCPU box`);
Result
average 11,574 req/s, peak 34,722 req/s
busy vCPUs at peak 139, provisioned at 60%: 232 vCPUs
horizontal: 29 × 8-vCPU instances, 15 per zone × 3 zones = 45 instances
vertical: one 232-vCPU machine + an identical standby = 464 vCPUs bought
at 10× users: 290 instances (435 with zone headroom), or a 2315-vCPU box
What’s happening
  1. The assumptions are at the top so an interviewer can challenge each one: 20 M daily users × 50 calls each, spread over 86,400 seconds, gives an average of 11,574 req/s; a peak factor of 3 for the busiest minute gives 34,722 req/s. (Chapter 1's estimation topic explains where peak factors come from.)
  2. Each request costs 4 ms of CPU, so at peak the tier is doing 34,722 × 0.004 = 139 vCPU-seconds of work every second: 139 vCPUs fully busy. Running at 100% leaves no room for spikes, garbage-collection pauses or a slow dependency, so the model provisions for 60% utilisation: 139 / 0.6 → 232 vCPUs.
  3. Horizontally that's ceil(232 / 8) = 29 eight-vCPU instances. But capacity must survive losing a whole availability zone (a data centre). With three zones, any two must carry the full peak, so each zone needs ceil(29 / 2) = 15 instances: 45 in total, 360 vCPUs. That extra 55% (50% in principle for three zones, plus rounding up to whole instances) is the price of zone-level fault tolerance.
  4. Vertically, a single 232-vCPU machine does the job, but on its own it's a single point of failure, so you buy an identical standby: 464 vCPUs paid for, half of them idle. Horizontal redundancy is cheaper because the spare capacity is spread out (N+1 of small units) instead of a whole second machine (1+1).
  5. At 10× the users the horizontal answer is just "more of the same" (290 instances, 435 with zone headroom). The vertical answer is a 2,315-vCPU machine, bigger than the largest instance AWS has announced (1,920 vCPUs). That's the ceiling you hit, and you hit it earlier for real workloads because memory bandwidth, network and locks run out before vCPUs do.
  6. The lesson for the interview: compute the size, then say which tier you scale which way. Stateless API servers scale out easily; the database behind them is where vertical scaling (and later replicas or sharding) comes in.
AdvancedWhy horizontal scaling isn't linear

Adding the tenth server rarely adds as much as the second did. Two things eat the gains: contention (servers queue for something shared, like a database row lock or a single leader) and coherence (servers spend time keeping each other in sync: cache invalidations, cluster membership, distributed locks). Neil Gunther's Universal Scalability Law puts both into one formula, and it's a useful way to argue why you're removing shared state:

usl.mjs
// Universal Scalability Law: throughput of N nodes = X1 · N / (1 + α(N−1) + βN(N−1))
// α = contention (time spent waiting on something shared), β = coherence (nodes keeping each other in sync).
const X1 = 2_000; // req/s one node handles alone
const designs = {
  "shared-nothing (α .01, β 0)": [0.01, 0],
  "shared lock (α .08, β 0)": [0.08, 0],
  "chatty cluster (α .03, β .002)": [0.03, 0.002],
};
const usl = (n, a, b) => (X1 * n) / (1 + a * (n - 1) + b * n * (n - 1));

console.log("nodes".padEnd(31) + [1, 4, 16, 32, 64].map((n) => String(n).padStart(8)).join(""));
for (const [name, [a, b]] of Object.entries(designs)) {
  const row = [1, 4, 16, 32, 64].map((n) => (usl(n, a, b) / 1000).toFixed(1).padStart(7) + "k").join("");
  const peak = b > 0 ? `  peak at N=${Math.floor(Math.sqrt((1 - a) / b))}` : "";
  console.log(name.padEnd(31) + row + peak);
}
Result
nodes                                 1       4      16      32      64
shared-nothing (α .01, β 0)        2.0k    7.8k   27.8k   48.9k   78.5k
shared lock (α .08, β 0)           2.0k    6.5k   14.5k   18.4k   21.2k
chatty cluster (α .03, β .002)     2.0k    7.2k   16.6k   16.4k   11.7k  peak at N=22
What’s happening
  1. usl(n, a, b) is the USL: ideal throughput X1 · n, divided by a penalty that grows with α (contention, linear in n) and β (coherence, quadratic in n because every node talks to every other node).
  2. The shared-nothing design (each node serves its requests without asking the others, α = 0.01, β = 0) keeps growing: 64 nodes give 78.5k req/s, about 61% of the ideal 128k. The small α is the load balancer and the shared database connection setup.
  3. The shared lock design (α = 0.08, say every request briefly takes a lock on a hot row) flattens out: going from 32 to 64 nodes adds only 2.8k req/s. Throughput approaches X1 / α = 25k req/s and never passes it, however many servers you add.
  4. The chatty cluster (α = 0.03, β = 0.002, say every node must invalidate every other node's local cache on writes) is worse: it peaks at sqrt((1 − α) / β) ≈ 22 nodes and then goes down, to 11.7k at 64 nodes. Adding servers makes it slower, because coordination grows with n².
  5. The values of α and β here are illustrative, not measured. In practice you measure throughput at a few cluster sizes and fit them. The design lesson holds regardless: horizontal scaling works when the tier is stateless and shared-nothing, which is exactly the next topic.
AdvancedWhere each kind of scaling fits
  • Web and API servers: horizontal, from the start. Keep them stateless and put them behind a load balancer; run at least two (in different zones) even when one would do, for availability.
  • Background workers: horizontal. They pull from a queue, so adding consumers adds throughput without a load balancer at all.
  • Relational databases: vertical first (a bigger instance is the cheapest scaling you will ever do), then read replicas for read traffic, then partitioning when writes or data size outgrow one primary. See Replication and Partitioning and sharding strategies.
  • Caches: vertical for a single node, then horizontal by partitioning the keyspace with consistent hashing.
  • Anything with a single writer (a leader, a sequencer, a lock service): scale it up, and design so that it does as little as possible.
Advanced"Just add more servers" doesn't scale the database
Two boxes or three?

Your service needs 64 vCPUs at peak. Option A: two 64-vCPU machines (active + standby). Option B: 16-vCPU instances spread evenly over three zones, surviving the loss of any one zone. Which buys fewer vCPUs, and by how much?

Show answer

Option B buys 96 vCPUs, option A buys 128: B is 32 vCPUs (25%) cheaper.

What’s happening
  1. Option A: the standby must be as big as the active machine, so 2 × 64 = 128 vCPUs, half of them idle all the time.
  2. Option B: 64 vCPUs of 16-vCPU instances is 4 instances. Any two of the three zones must hold all 4, so each zone needs ceil(4 / 2) = 2 instances.
  3. Three zones × 2 instances × 16 vCPUs = 96 vCPUs. The spare capacity is one zone's worth (a third), not a whole second machine (a half).
  4. The more units you split capacity into, the smaller the spare fraction: with N zones you over-provision by N / (N − 1). That's the arithmetic behind "scale out for availability".

#Stateless services and where the state goes

Added

A service is stateless when any instance can handle any request, because nothing an instance needs is kept only inside that instance between requests. The state still exists (logins, carts, uploaded files, rate-limit counters); it just lives somewhere shared: a session store, the database, object storage, the client's own token. That one property is what makes horizontal scaling, rolling deploys, autoscaling and failover cheap.

The mental model is a call centre. In a good call centre, any agent can take your call, because your case history is in the shared CRM, not on a sticky note on one agent's desk. If an agent goes home sick, the next agent pulls up your case and carries on. In a bad one, "only Priya knows about your refund", so you have to wait for Priya, and if Priya leaves, your refund is gone. Sticky sessions are the "ask for Priya" workaround: they route you back to the same agent every time, which works until Priya is busy, sick or replaced.

What usually hides in an instance, and where it should go instead:

StateKept on the instance (fragile)Where it goes in a stateless design
Login sessionsIn-process session map (express-session's default memory store)Shared store (Redis, the database) or a signed token the client sends back
Uploaded filesThe server's local diskObject storage (S3-style), often uploaded directly with a presigned URL
Shopping cart, draftsIn memoryDatabase, or the session store for anonymous users
Rate-limit countersA per-process counterA shared counter (Redis) keyed by client
CachesLocal memoryFine to keep locally if it's only an optimisation; a shared cache if every instance must agree
WebSocket connectionsNecessarily on one instanceStay put, but the messages go through pub/sub so any instance can deliver to any user
Scheduled jobsA timer in every instance (runs N times)One scheduler with a lease or a queue
Clients browser · app Load balancer App 1 no local state App 2 no local state App 3 no local state Session store Redis · sessions, counters Database users, carts, orders Object storage uploads, images direct upload with a presigned URL
Any instance can serve any request: everything that must outlive a request lives in a shared store.

Here's the difference it makes, measured. 30,000 users are logged in to three servers; the load balancer routes each user to "their" server by hashing the user id. Sessions live either in each server's memory or in one shared store. Then a server crashes, or we scale out to four:

sessions.mjs
// 30,000 logged-in users, sticky routing by user id. Sessions live either in each server's
// memory or in one shared store. Who has to log in again after a crash or a scale-out?
const hash = (s) => {
  let h = 0x811c9dc5; // FNV-1a, then a murmur3-style finaliser to spread similar strings
  for (const c of s) h = Math.imul(h ^ c.charCodeAt(0), 0x01000193);
  h = Math.imul(h ^ (h >>> 16), 0x85ebca6b);
  h = Math.imul(h ^ (h >>> 13), 0xc2b2ae35);
  return (h ^ (h >>> 16)) >>> 0;
};
const users = Array.from({ length: 30_000 }, (_, i) => `user-${i}`);

const modRoute = (servers) => (u) => servers[hash(u) % servers.length];
const ringRoute = (servers) => {
  const ring = servers.flatMap((s) => Array.from({ length: 100 }, (_, v) => [hash(`${s}#${v}`), s])).sort((a, b) => a[0] - b[0]);
  return (u) => (ring.find(([p]) => p >= hash(u)) ?? ring[0])[1];
};

// Log everyone in through `before`, then send every user's next request through `after`.
function loggedOut(before, after, { shared = false, dead = null } = {}) {
  const memory = new Map(); // server -> Set of user ids it holds sessions for
  const store = new Set();  // the shared session store
  for (const u of users) {
    if (shared) store.add(u);
    else (memory.get(before(u)) ?? memory.set(before(u), new Set()).get(before(u))).add(u);
  }
  if (dead) memory.delete(dead); // a crash loses that server's memory
  const lost = users.filter((u) => !(shared ? store.has(u) : memory.get(after(u))?.has(u))).length;
  return `${((lost / users.length) * 100).toFixed(1)}%`.padStart(6);
}

const abc = ["A", "B", "C"], abcd = ["A", "B", "C", "D"];
const crashMod = (u) => modRoute(["A", "C"])(u); // after B dies, the balancer only knows A and C
console.log("event                        in-memory  shared store");
console.log(`B crashes (mod N)            ${loggedOut(modRoute(abc), crashMod, { dead: "B" })}      ${loggedOut(modRoute(abc), crashMod, { shared: true, dead: "B" })}`);
console.log(`B crashes (ring)             ${loggedOut(ringRoute(abc), ringRoute(["A", "C"]), { dead: "B" })}      ${loggedOut(ringRoute(abc), ringRoute(["A", "C"]), { shared: true, dead: "B" })}`);
console.log(`scale 3 → 4 (mod N)          ${loggedOut(modRoute(abc), modRoute(abcd))}      ${loggedOut(modRoute(abc), modRoute(abcd), { shared: true })}`);
console.log(`scale 3 → 4 (ring)           ${loggedOut(ringRoute(abc), ringRoute(abcd))}      ${loggedOut(ringRoute(abc), ringRoute(abcd), { shared: true })}`);
Result
event                        in-memory  shared store
B crashes (mod N)             66.8%        0.0%
B crashes (ring)              32.6%        0.0%
scale 3 → 4 (mod N)           74.7%        0.0%
scale 3 → 4 (ring)            25.3%        0.0%
What’s happening
  1. hash turns a user id into a 32-bit number (FNV-1a plus a mixing step so ids like user-1 and user-2 land far apart). modRoute is the simplest sticky rule: server number = hash mod N. ringRoute is a consistent-hash ring with 100 points per server, the same structure as in Consistent hashing.
  2. loggedOut logs every user in through the before routing (their session is stored on that server, or in the shared store), optionally deletes a crashed server's memory, then sends each user's next request through the after routing and counts who can't find their session.
  3. B crashes, mod N: 66.8% logged out, not the 33% you might expect. B's users lose their sessions because B's memory is gone. But the balancer now computes hash mod 2, so most of A's and C's users are also routed to the other surviving server, which has never seen them.
  4. B crashes, ring: 32.6%. On a ring, only the keys that belonged to B move; A's and C's users stay where they were. That's the whole point of consistent hashing. Still, a third of your users are logged out by one crash.
  5. Scaling out 3 → 4 is a planned, routine event, and it logs out 74.7% of users with mod N and 25.3% with a ring (everyone who now maps to D). Autoscaling would log people out every time it adds a server.
  6. With a shared store, every row is 0.0%: the session doesn't live on the server, so routing changes and crashes don't matter. That's why the stateless design wins: failures and scaling stop being user-visible events.
AdvancedSticky sessions: when they're fine

Sticky sessions (session affinity) make the load balancer send a client back to the same instance, usually with a cookie it sets itself (AWS ALB's duration-based cookie is AWSALB, with a fixed 7-day expiry, per the ELB docs in October 2026) or by hashing the client IP. They're not wrong; they're a performance optimisation that must not be required for correctness:

  • Fine: routing a user back to the instance whose local cache is warm for them, or keeping a multi-step upload on one instance, as long as a different instance can still serve them (slower) if theirs is gone.
  • Needed: long-lived connections (WebSockets, server-sent events) are inherently pinned to one instance for their lifetime; that's stickiness you can't avoid, so design for reconnects (see Scaling WebSockets).
  • Harmful: when the session data exists only on that instance (the table above), or when one "whale" client pins a large share of traffic to one box. Stickiness also fights load balancing: the balancer can't move a stuck user off a hot server, and new instances added by autoscaling get only new users.
AdvancedSessions: shared store vs signed tokens

There are two stateless ways to know who a request is from:

Session id + shared storeSigned token (e.g. a JWT)
What the client holdsAn opaque random id in a cookieThe claims themselves, signed by the server
Per-request costOne store lookup (rule of thumb: under a millisecond to Redis in the same data centre)Signature check, no network call
Logout / revokeDelete the session: instantHard: the token is valid until it expires, unless you add a deny-list (a lookup again)
Size on the wire~32 bytesHundreds of bytes to a few KB, on every request
Failure modeStore down = nobody is logged in (so replicate it)Signing key leaked = anyone can mint tokens

A common combination: short-lived access tokens (minutes) so services can verify without a lookup, plus a refresh token or session in a store so you can revoke. The Node.js site covers the implementation (cookies, express-session, JWT signing); at the design level, say which one you chose and how logout works.

AdvancedUploads and other files

A server that writes uploads to its own disk breaks three ways at once: the next request (on another instance) can't find the file, autoscaling deletes the disk when it removes the instance, and the disk fills up. Put files in object storage and keep only the metadata (owner, key, size, content type) in the database. Better still, let the browser upload directly to object storage with a short-lived presigned URL that your API hands out, so large bodies never pass through your app servers at all (the dashed path in the diagram). Object storage for files and media covers the details.

AdvancedLocal state you forgot about
Three servers become two

Sessions are in memory, routed by hash mod N across 3 servers. One server crashes and the balancer starts routing by hash mod 2. Roughly what fraction of users lose their session, and why is it so much more than a third?

Show answer

About two-thirds (exactly 4 in 6; the model above measured 66.8%).

What’s happening
  1. The old rule is h % 3 (0 → A, 1 → B, 2 → C) and the new rule is h % 2 over the survivors (0 → A, 1 → C). Both rules repeat every 6 values of h, so it's enough to check h % 6 = 0 to 5.
  2. h % 6 = 0: old A, new A, session found. = 5: old C, new C, session found. These are the only users who keep their session.
  3. h % 6 = 1 and = 4: old B. B's memory is gone, so these users are logged out wherever they land.
  4. h % 6 = 2: old C, new A. = 3: old A, new C. Their server is still alive and still has their session, but the balancer now sends them to the other server, which doesn't. Logged out too.
  5. That's 4 of 6 values, so two-thirds of users lose their session: the crashed third plus half of everyone else. A consistent-hash ring would lose only B's third, and a shared session store would lose nobody.

#Load balancers: L4 vs L7

Added

A load balancer is the component that receives traffic on one address and spreads it across a pool of backends, skipping the ones that are unhealthy. The labels L4 and L7 come from the OSI model and say how much of the traffic it reads before deciding. An L4 (transport-layer) balancer sees TCP or UDP: source and destination IPs and ports. It picks a backend when a connection opens and then shovels bytes both ways without understanding them. An L7 (application-layer) balancer speaks HTTP (and gRPC, WebSockets): it terminates the connection, reads each request (method, host, path, headers, cookies), and makes a fresh decision per request, possibly opening its own connections to the backends.

The mental model is a building's front gate vs its reception desk. The gate guard (L4) looks at your car's plate and waves you to a parking zone; he never opens the boot, so he's fast and can handle any kind of vehicle, but every passenger in the car goes to the same zone. The receptionist (L7) opens every letter, reads who it's for and walks each one to the right department; she can refuse junk mail, stamp each letter with a tracking number and spread one person's ten letters across ten clerks, but reading letters costs time, and she has to be able to read them (which means she holds the keys to sealed envelopes: TLS).

L4 · decides once per connection Client 1 TCP conn L4 balancer src/dst IP + port bytes pass through backend 1 backend 2 backend 3 every request on that conn L7 · decides per request Client HTTPS L7 balancer terminates TLS method · path · headers retries · adds X-Forwarded-For API pool /api/* Static pool /static/* each request picks its own backend, even on one client connection
L4 routes connections by addresses; L7 reads each HTTP request and routes it by content.
L4 (TCP/UDP)L7 (HTTP, gRPC)
SeesIPs, ports, protocolEverything in the request: host, path, headers, cookies, body if needed
DecidesOnce per connection (per flow for UDP)Once per request
TLSUsually passed through untouched (or terminated, if configured)Terminated: it holds the certificate and decrypts
Routing optionsBy port or IPBy host, path, header, cookie, method, weight (canaries)
ExtrasVery high throughput, low latency, any protocol, can keep the client's IPRetries, redirects, header rewrites, auth, compression, caching, WAF rules, per-request metrics
CostCheap per connectionMore CPU per request (parsing, TLS)
ExamplesAWS NLB, Google's passthrough network LB, Maglev, IPVS, HAProxy in TCP modeAWS ALB, Envoy, NGINX, HAProxy in HTTP mode, Cloudflare, API gateways

The difference that matters most in practice is per connection vs per request. Modern clients open one long-lived connection (HTTP/2, gRPC, keep-alive) and send thousands of requests over it. Here's what that does to six internal clients behind each kind of balancer:

l4-vs-l7.mjs
// Six clients keep one long-lived HTTP/2 (or gRPC) connection each and send many requests over it.
// An L4 balancer picks a backend per *connection*; an L7 balancer picks one per *request*.
const clients = [
  { name: "batch-job", rps: 600 }, { name: "web-bff", rps: 120 }, { name: "mobile-bff", rps: 90 },
  { name: "admin", rps: 10 }, { name: "reports", rps: 60 }, { name: "search", rps: 120 },
];
const show = (label, load) =>
  console.log(label.padEnd(30) + load.map((r) => `${r}`.padStart(6)).join("") + `   max/avg ${(Math.max(...load) / (load.reduce((a, b) => a + b) / load.length)).toFixed(2)}`);

function l4(backends) {
  const load = new Array(backends).fill(0);
  clients.forEach((c, i) => (load[i % backends] += c.rps)); // connection i → backend i mod N, for its whole life
  return load;
}
function l7(backends) {
  const load = new Array(backends).fill(0);
  let next = 0;
  for (const c of clients) for (let r = 0; r < c.rps; r++) load[next++ % backends]++; // every request picked afresh
  return load;
}

console.log("req/s per backend".padEnd(30) + ["b1", "b2", "b3", "b4"].map((b) => b.padStart(6)).join(""));
show("L4, 3 backends", l4(3));
show("L7, 3 backends", l7(3));

// Autoscaling adds b4. L4: existing connections stay where they are, so b4 gets nothing until clients reconnect.
const scaledL4 = [...l4(3), 0];
show("L4, b4 added (no reconnects)", scaledL4);
show("L7, b4 added", l7(4));
Result
req/s per backend                 b1    b2    b3    b4
L4, 3 backends                   610   180   210   max/avg 1.83
L7, 3 backends                   334   333   333   max/avg 1.00
L4, b4 added (no reconnects)     610   180   210     0   max/avg 2.44
L7, b4 added                     250   250   250   250   max/avg 1.00
What’s happening
  1. Six clients each hold one persistent connection; their request rates are very uneven: a batch job sends 600 req/s, the admin tool 10. This is typical for service-to-service gRPC traffic.
  2. l4 assigns each connection to a backend round-robin when it opens, and every request on it follows. Backend 1 gets the batch job and the admin tool: 610 req/s, against 180 and 210 on the others. The balancer's algorithm was perfectly fair, connection-wise (two each); the load is still 1.83× the average on one box.
  3. l7 picks a backend for every request, so the same traffic comes out at 334 / 333 / 333. The L7 balancer holds its own connections to each backend and multiplexes requests across them.
  4. Then autoscaling adds backend 4. Under L4 it gets 0 req/s: existing connections never move, so the new instance sits idle until clients reconnect (a deploy, an idle timeout, a connection age limit). Max/avg gets worse, 2.44, because the average now includes an idle box. Under L7 the new backend takes a quarter of the traffic within one request each.
  5. This is why gRPC behind an L4 balancer (a plain Kubernetes Service is L4) famously ends up with one hot pod. The fixes are an L7 proxy (Envoy, a service mesh, an L7 ingress), client-side balancing across all backends, or a maximum connection age on the server so clients reconnect and spread out.
AdvancedHealth checks: how fast dead backends drop out

Health checks: how fast a dead backend stops getting traffic

A balancer only helps availability if it notices failures. Active health checks probe each backend (a TCP connect, or an HTTP GET /healthz that must return 2xx). Passive checks (outlier detection) watch real traffic: too many connection errors or 5xx responses and the backend is ejected for a while. The defaults decide how many requests are lost:

health-checks.mjs
// A target dies at a random moment. How long does the balancer keep sending it traffic?
// Active checks: one probe every `interval` s; out of service after `unhealthy` failures in a row.
function exposure({ interval, unhealthy }) {
  // The crash lands somewhere between two probes; the first failed probe comes 0..interval s later,
  // then (unhealthy − 1) more probes must fail.
  const best = (unhealthy - 1) * interval, worst = unhealthy * interval;
  return { best, worst, avg: (best + worst) / 2 };
}
const share = 1000; // req/s the balancer was sending this target (assumption)

const configs = [
  ["ALB defaults (30 s × 2)", { interval: 30, unhealthy: 2 }],
  ["tuned (10 s × 2)", { interval: 10, unhealthy: 2 }],
  ["aggressive (5 s × 2)", { interval: 5, unhealthy: 2 }],
];
console.log("active checks".padEnd(26) + "still routed to dead target".padStart(28) + "failed requests (avg)".padStart(23));
for (const [name, cfg] of configs) {
  const { best, worst, avg } = exposure(cfg);
  console.log(name.padEnd(26) + `${best}–${worst} s`.padStart(28) + Math.round(avg * share).toLocaleString("en-US").padStart(23));
}

// Passive checks (outlier detection): the balancer watches real responses. If connections to the
// target fail fast, it trips after `maxFails` failures, which at 1,000 req/s takes about a millisecond.
const maxFails = 1; // NGINX default max_fails=1 (within fail_timeout=10s)
console.log(`passive, max_fails=${maxFails}`.padEnd(26) + `≈${((maxFails / share) * 1000).toFixed(0)} ms`.padStart(28) + String(maxFails).padStart(23));

// Coming back: healthy threshold 5 at 30 s means a fixed target waits 2.5 min before it gets traffic again.
console.log(`\nrecovery with ALB defaults: 5 passes × 30 s = ${(5 * 30) / 60} min before traffic returns`);
Result
active checks              still routed to dead target  failed requests (avg)
ALB defaults (30 s × 2)                        30–60 s                 45,000
tuned (10 s × 2)                               10–20 s                 15,000
aggressive (5 s × 2)                            5–10 s                  7,500
passive, max_fails=1                             ≈1 ms                      1

recovery with ALB defaults: 5 passes × 30 s = 2.5 min before traffic returns
What’s happening
  1. With an active check every interval seconds and unhealthy consecutive failures required, a crash that happens just before a probe is noticed after (unhealthy − 1) × interval; one just after a probe waits the full unhealthy × interval. The model takes the average of the two.
  2. AWS ALB's defaults (checked in the ELB docs, October 2026) are a 30-second interval, a 5-second timeout, 2 failures to mark a target unhealthy and 5 successes to mark it healthy again. So a dead target keeps getting its share of traffic for 30 to 60 seconds: about 45,000 failed requests at 1,000 req/s. NLB uses the same interval and thresholds.
  3. A 10-second or 5-second interval shrinks that to 10–20 s or 5–10 s, at the price of more probe traffic (every balancer node probes every target) and more risk of ejecting a backend that was only briefly slow.
  4. Passive detection reacts to real failures within the first failed request or two: NGINX's default is max_fails=1 within fail_timeout=10s (nginx docs). That's why you want both: passive for speed, active so a backend that's out gets probed back in without risking real requests.
  5. The bottom line shows the other direction: with a healthy threshold of 5 at 30 s, a fixed target waits 2.5 minutes before it gets traffic again. After a deploy, that's 2.5 minutes of reduced capacity per batch of instances.
  6. The model assumes probes fail immediately. A backend that hangs fails each probe only after the timeout (5 s on ALB), and passive checks only see it after requests time out, which is why deep health checks and request deadlines matter.
AdvancedWhat a health check should check
  • Liveness ("is the process running?") is what the balancer and orchestrator need to restart or eject an instance. Keep it cheap and local.
  • Readiness ("can it serve right now?") should fail while the instance is starting up, warming caches or draining for shutdown, so it gets no traffic in those windows.
  • Don't make the balancer's check depend on shared dependencies (the database, a downstream service). If the database blips, every instance fails its check at once, and the balancer has nowhere to send traffic. AWS ALB and NLB fail open in that case: when every target is unhealthy, they route to all of them anyway (ELB docs, 2026). Report dependency health in metrics and alerts instead.
AdvancedTLS termination and the client's IP

An L7 balancer has to decrypt to read requests, so it terminates TLS: it holds the certificate, and the hop to the backends is either plain HTTP on a private network or re-encrypted with internal certificates (needed for zero-trust networks and compliance; see the Node.js site's mTLS topic). An L4 balancer can pass TLS through so only the backends hold keys, at the price of no L7 features.

Terminating anywhere means the backend sees the balancer's IP as the source. L7 balancers add X-Forwarded-For (and X-Forwarded-Proto), which the backend must trust only from the balancer; L4 balancers either preserve the client IP at the packet level (AWS NLB does for instance targets) or prepend the PROXY protocol header. Rate limiting, geo-blocking and audit logs all depend on getting this right.

AdvancedDraining: removing a backend without errors

When you deploy or scale in, the balancer should stop sending new requests to the instance, let in-flight ones finish, and only then let it shut down. AWS calls this the deregistration delay, 300 seconds by default (ELB docs, 2026). Your app's side of the deal is graceful shutdown: on SIGTERM, fail readiness, finish what's in flight, close keep-alive connections, exit.

AdvancedThe balancer itself needs redundancy
Why is one pod hot?

A gRPC service runs 8 pods behind a Kubernetes Service (an L4 balancer). Dashboards show one pod at 90% CPU and the others near 10%. Autoscaling adds 4 more pods; nothing improves. What's going on?

Show answer

The callers hold long-lived HTTP/2 connections, and an L4 balancer assigns a backend per connection, not per request.

What’s happening
  1. Each caller opened its gRPC channel once and multiplexes every call over it. The Service chose a pod when that connection was created and never chooses again.
  2. If one caller is much busier than the rest (or a few callers landed on the same pod), that pod gets most of the requests, exactly like b1 in the model above.
  3. New pods only receive new connections. The existing callers don't reconnect, so the 4 new pods idle, which is the "b4 added" row.
  4. Fixes: balance per request (an L7 proxy such as Envoy or a mesh sidecar, or gRPC's client-side round_robin policy over all pod addresses from a headless Service), or set a maximum connection age on the server so clients periodically reconnect and spread out.

#Load-balancing algorithms

Added

Once a balancer knows which backends are healthy, it needs a rule for which one gets the next request. The rules fall into three families. Static rules ignore the backends' current state: round robin, weighted round robin, random. Dynamic rules look at load: least connections, least response time, power of two choices. Hashing rules pick by a key so the same client or object lands on the same backend: IP hash, consistent hashing. Each family answers a different question, and the right answer depends mostly on how uneven your requests are.

The mental model is the supermarket checkouts. Round robin is a staff member pointing each shopper to the next till in turn, without looking: fine if every trolley has five items, terrible when one shopper has a full trolley and the next three are stuck behind them. Least connections is pointing each shopper to the till with the fewest people queuing. Power of two choices is the shopper glancing at two random tills and joining the shorter: it sounds lazy, but it's almost as good as checking every till, and much better when your information about the queues is out of date.

AlgorithmPicksNeedsGood forWeak spot
Round robinNext backend in turnNothingUniform, short requests on identical backendsUneven request costs; mixed instance sizes
Weighted round robinIn turn, in proportion to weightsWeights you setMixed instance sizes, canaries (5% to the new version)Weights are static; a slow backend still gets its share
RandomAny backendNothingMany independent balancers (no shared state)Same as round robin, plus short-term imbalance
Least connections (least outstanding requests)Fewest in-flight requestsLive per-backend countsUneven request costs, long requests, WebSocketsHerding when counts are stale or many balancers share one view
Least response time (latency EWMA)Lowest recent latency, often × in-flightLatency measurementsBackends with different speeds, noisy neighboursReacts to the past; one slow request skews a quiet backend
Power of two choices (P2C)The less loaded of 2 random backendsCounts for 2 backendsLarge pools, many balancers, stale dataSlightly worse than full least-connections with perfect information
IP hash / consistent hashBackend owning hash(key)A key: client IP, user id, URLCache affinity, sticky routing without cookiesHot keys overload one backend; uneven when keys are skewed

Here's the key comparison as a seeded simulation: ten identical servers, each processing one request at a time from its own queue, at 80% average utilisation. Requests are uneven in the way real ones are: 90% are cheap (5 ms) and 10% are expensive (100 ms, think a report or a cache miss). Every algorithm gets exactly the same arrivals and costs:

lb-sim.mjs
// 10 identical servers, one worker each, FIFO queue. Seeded, so every run prints the same.
function mulberry32(seed) {
  return () => {
    seed = (seed + 0x6d2b79f5) | 0;
    let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}
const SERVERS = 10, REQUESTS = 200_000;
const MEAN_COST = 0.9 * 5 + 0.1 * 100; // ms: 90% cheap (5 ms), 10% expensive (100 ms)
const RATE = (0.8 * SERVERS) / MEAN_COST; // arrivals per ms for 80% average utilisation

const algorithms = {
  "round robin": (s, i) => i % SERVERS,
  random: (s, i, rnd) => Math.floor(rnd() * SERVERS),
  "least connections": (s, i) => {
    let best = i % SERVERS; // start the scan at a rotating index so ties are shared out
    for (let j = 1; j < SERVERS; j++) {
      const k = (i + j) % SERVERS;
      if (s[k].length < s[best].length) best = k;
    }
    return best;
  },
  "power of two": (s, i, rnd) => {
    const a = Math.floor(rnd() * SERVERS), b = Math.floor(rnd() * SERVERS);
    return s[a].length <= s[b].length ? a : b;
  },
};

const pct = (sorted, p) => sorted[Math.min(sorted.length - 1, Math.floor(p * sorted.length))];

for (const [name, pick] of Object.entries(algorithms)) {
  const rnd = mulberry32(42); // same arrivals and costs for every algorithm
  const queues = Array.from({ length: SERVERS }, () => []); // finish times of requests still on each server
  const busy = new Array(SERVERS).fill(0);
  const latencies = [];
  let now = 0;
  for (let i = 0; i < REQUESTS; i++) {
    now += -Math.log(1 - rnd()) / RATE; // Poisson arrivals
    const cost = rnd() < 0.9 ? 5 : 100;
    for (const q of queues) while (q.length && q[0] <= now) q.shift(); // drop finished requests
    const k = pick(queues, i, rnd);
    const start = Math.max(now, queues[k].at(-1) ?? now);
    queues[k].push(start + cost);
    busy[k] += cost;
    latencies.push(start + cost - now);
  }
  latencies.sort((a, b) => a - b);
  const util = busy.map((b) => b / now);
  console.log(
    `${name.padEnd(17)} util ${(Math.min(...util) * 100).toFixed(0)}–${(Math.max(...util) * 100).toFixed(0)}%` +
      `  p50 ${pct(latencies, 0.5).toFixed(0).padStart(3)} ms  p99 ${pct(latencies, 0.99).toFixed(0).padStart(4)} ms` +
      `  p99.9 ${pct(latencies, 0.999).toFixed(0).padStart(4)} ms`,
  );
}
Result
round robin       util 78–81%  p50  87 ms  p99  669 ms  p99.9 1181 ms
random            util 78–82%  p50 100 ms  p99  718 ms  p99.9  944 ms
least connections util 80–80%  p50   5 ms  p99  184 ms  p99.9  242 ms
power of two      util 79–80%  p50  17 ms  p99  276 ms  p99.9  384 ms
What’s happening
  1. mulberry32(42) is a tiny seeded random generator, recreated for each algorithm so all four see the same 200,000 arrivals and costs. Arrivals are Poisson (-Math.log(1 - rnd()) / RATE gives exponential gaps), at a rate chosen so the servers are 80% busy on average: mean cost is 0.9 × 5 + 0.1 × 100 = 14.5 ms, so RATE = 0.8 × 10 / 14.5 requests per ms.
  2. Each server is a FIFO queue of finish times. Before each decision the loop removes finished requests, so queues[k].length is exactly the number of requests in flight on server k: the "connections" that least connections counts. A new request starts when the last one in that queue finishes, and its latency is waiting time plus its own cost.
  3. Round robin spreads work perfectly on average (utilisation 78–81% on every server) yet has a p50 of 87 ms and a p99 of 669 ms. Averages hide the problem: it keeps sending cheap requests to a server that just started a 100 ms job, so they queue behind it while other servers sit idle. Random is no better (p99 718 ms).
  4. Least connections looks at the queues, so a request goes to an idle server whenever one exists. Median latency drops to 5 ms (the request's own cost: no waiting) and p99 to 184 ms. Same servers, same load, 3.6× better tail. The rotating start of the scan only breaks ties fairly; without it, server 0 would win every tie.
  5. Power of two samples just two servers and takes the shorter queue: p50 17 ms, p99 276 ms. Most of least-connections' gain, from looking at 2 servers instead of 10.
  6. What this demonstrates: when request costs vary, the algorithm decides your tail latency, not your average utilisation. That's why AWS ALB offers "least outstanding requests" next to its default round robin, and why Envoy's default least-request balancer is P2C.
AdvancedWhy power of two choices matters: stale information

With perfect, instant knowledge, least connections wins. Real balancers don't have that. A fleet of balancer nodes each sees only its own traffic, or shares load reports every few milliseconds. When many requests are routed on the same snapshot, least connections sends all of them to whichever server looked emptiest, and that server is suddenly the busiest (herding). This model gives the balancer a snapshot of queue lengths that's refreshed every 0, 10, 50 or 200 ms:

lb-stale.mjs
// Same 10 servers and seeded workload, but the balancer sees queue lengths from a snapshot
// refreshed every STALE ms (like many balancer nodes sharing load reports).
function mulberry32(seed) {
  return () => {
    seed = (seed + 0x6d2b79f5) | 0;
    let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}
const SERVERS = 10, REQUESTS = 200_000, RATE = (0.8 * SERVERS) / 14.5;

const pickers = {
  random: (view, rnd) => Math.floor(rnd() * SERVERS),
  "least connections": (view, rnd) => {
    const min = Math.min(...view);
    const ties = view.flatMap((n, k) => (n === min ? [k] : []));
    return ties[Math.floor(rnd() * ties.length)];
  },
  "power of two": (view, rnd) => {
    const a = Math.floor(rnd() * SERVERS), b = Math.floor(rnd() * SERVERS);
    return view[a] <= view[b] ? a : b;
  },
};

function p99(name, stale) {
  const rnd = mulberry32(42);
  const queues = Array.from({ length: SERVERS }, () => []);
  let view = new Array(SERVERS).fill(0), nextRefresh = 0, now = 0;
  const lat = [];
  for (let i = 0; i < REQUESTS; i++) {
    now += -Math.log(1 - rnd()) / RATE;
    const cost = rnd() < 0.9 ? 5 : 100;
    for (const q of queues) while (q.length && q[0] <= now) q.shift();
    if (now >= nextRefresh) (view = queues.map((q) => q.length)), (nextRefresh = now + stale);
    const k = pickers[name](view, rnd);
    const start = Math.max(now, queues[k].at(-1) ?? now);
    queues[k].push(start + cost);
    lat.push(start + cost - now);
  }
  lat.sort((a, b) => a - b);
  return lat[Math.floor(0.99 * lat.length)];
}

console.log("p99 latency (ms) by how stale the balancer's view is");
console.log("view age:          " + [0, 10, 50, 200].map((s) => `${s} ms`.padStart(8)).join(""));
for (const name of Object.keys(pickers)) {
  const row = [0, 10, 50, 200].map((s) => p99(name, s).toFixed(0).padStart(8)).join("");
  console.log(name.padEnd(19) + row);
}
Result
p99 latency (ms) by how stale the balancer's view is
view age:              0 ms   10 ms   50 ms  200 ms
random                  718     718     718     718
least connections       185     223     499    1416
power of two            276     280     306     411
What’s happening
  1. The workload and seed are the same as before. The only change is view: the queue lengths the balancer believes, copied from the real queues only when now >= nextRefresh, then reused for every request until the next refresh.
  2. At 0 ms (a fresh view every request), the results match the previous model: least connections 185 ms, P2C 276 ms, random 718 ms. Least connections breaks ties randomly here, which is why its figure differs by 1 ms.
  3. With a 50 ms old view, least connections more than doubles to 499 ms. With 200 ms it's 1,416 ms, twice as bad as random: during each 200-ms window it sends ~110 requests to the same "emptiest" server or two, then the next snapshot shows those as full and it herds onto another one.
  4. Power of two degrades gently: 276 → 411 ms. Two random samples rarely both hit the same server, so even on a stale view the requests are spread out, while still avoiding servers that looked full.
  5. Random doesn't use the view at all, so it's 718 ms in every column: the baseline P2C never falls below. This is the classic "power of two choices" result: when n requests are spread over n servers, the fullest server holds about ln n / ln ln n of them with one random choice but only about ln ln n / ln 2 with two, an exponential improvement (Azar, Broder, Karlin and Upfal's balls-into-bins proof, 1994). Michael Mitzenmacher extended it to queues (2001) and showed it holds up well with old load information (2000). It's why Envoy's least-request balancer samples 2 hosts by default (choice_count 2) and why NGINX offers random two least_conn.
AdvancedWeighted algorithms for mixed fleets

Fleets are rarely uniform for long: you add new, bigger instances next to old ones, or run a canary at 5%. Plain round robin gives every server the same number of requests, whatever its speed:

lb-weighted.mjs
// 4 servers: two new big ones (speed 2) and two old ones (speed 1). Seeded workload.
function mulberry32(seed) {
  return () => {
    seed = (seed + 0x6d2b79f5) | 0;
    let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}
const speed = [2, 2, 1, 1];
const RATE = (0.75 * 6) / 10; // total capacity is 6 "speed units"; 10 ms per request at speed 1
const wrrOrder = [0, 1, 2, 0, 1, 3]; // weights 2:2:1:1, interleaved

const pickers = {
  "round robin": (q, i) => i % 4,
  "weighted round robin": (q, i) => wrrOrder[i % wrrOrder.length],
  "least connections": (q, i) => [0, 1, 2, 3].reduce((b, k) => (q[k].length < q[b].length ? k : b), i % 4),
};

for (const [name, pick] of Object.entries(pickers)) {
  const rnd = mulberry32(7);
  const queues = [[], [], [], []], busy = [0, 0, 0, 0], lat = [];
  let now = 0;
  for (let i = 0; i < 100_000; i++) {
    now += -Math.log(1 - rnd()) / RATE;
    const work = -Math.log(1 - rnd()) * 10; // exponential, mean 10 ms at speed 1
    for (const q of queues) while (q.length && q[0] <= now) q.shift();
    const k = pick(queues, i);
    const cost = work / speed[k];
    const start = Math.max(now, queues[k].at(-1) ?? now);
    queues[k].push(start + cost);
    busy[k] += cost;
    lat.push(start + cost - now);
  }
  lat.sort((a, b) => a - b);
  const util = busy.map((b) => `${Math.round((b / now) * 100)}%`.padStart(5)).join("");
  console.log(`${name.padEnd(21)} util big,big,old,old:${util}   p99 ${lat[Math.floor(0.99 * lat.length)].toFixed(0)} ms`);
}
Result
round robin           util big,big,old,old:  56%  56% 113% 112%   p99 27120 ms
weighted round robin  util big,big,old,old:  75%  75%  76%  75%   p99 88 ms
least connections     util big,big,old,old:  73%  70%  82%  82%   p99 61 ms
What’s happening
  1. Two "big" servers work twice as fast (speed = 2) as two old ones. Total capacity is 6 units, and the arrival rate is set to 75% of it. Work per request is exponential with a mean of 10 ms at speed 1 (5 ms on a big server).
  2. Round robin sends each server a quarter of the requests. The big ones run at 56%, but the old ones need 113% and 112% of their capacity: more work arrives than they can do, so their queues grow for the whole run and p99 hits 27 seconds. Utilisation above 100% here means "the queue was still growing when the simulation ended".
  3. Weighted round robin with weights 2:2:1:1 (the order [0, 1, 2, 0, 1, 3] interleaves them so a big server never gets two in a row) gives every server ~75% and p99 88 ms. It works because you told it the truth about capacities.
  4. Least connections, with no weights at all, gets p99 61 ms: a fast server finishes sooner, so its count drops sooner, so it's picked more often. Dynamic algorithms adapt to speed differences you didn't configure, including ones you don't know about (a noisy neighbour, a slow disk, a cold cache).
  5. Weights still matter for intent: a canary at 5% or draining an old version are policy decisions, not load measurements. Envoy combines both: with unequal weights its least-request balancer divides each host's weight by (active requests + 1).
AdvancedHash-based routing

Hashing doesn't try to balance load; it tries to send the same key to the same place. ip_hash in NGINX (which hashes the first three octets of an IPv4 address) gives stickiness without cookies; hashing on a URL or object id gives cache affinity, so each cache server holds a distinct slice of the objects instead of every server caching everything. Use consistent hashing (a ring, or Maglev's lookup table, both available in Envoy) rather than hash mod N, so adding or removing a backend moves only about 1/N of the keys; Consistent hashing builds one. The cost is that load follows key popularity: one hot key (a celebrity's profile, a viral image) lands on one backend, which is where bounded-load variants and splitting hot keys come in (Hot keys and cache stampedes).

AdvancedWhat the common balancers offer (checked October 2026)
  • AWS ALB: round robin (default), least outstanding requests, weighted random (with automatic target weights that shift traffic away from targets returning errors).
  • NGINX: weighted round robin (default), least_conn, ip_hash, hash … consistent (ketama), random two least_conn (P2C), and least_time (historically an NGINX Plus feature).
  • Envoy: round robin, least request (P2C by default), ring hash, Maglev, random.
  • HAProxy: roundrobin, leastconn, source (IP hash), uri and header hashing, and random with a configurable number of draws.
AdvancedLeast connections floods a new or recovered server
How often does P2C pick the busiest server?

Ten backends. With random choice, the single busiest backend gets about 10% of new requests. With power of two choices (two independent random picks, take the less loaded), how often is the busiest backend chosen?

Show answer

About 1%: only when both random picks happen to be that backend.

What’s happening
  1. P2C picks two backends independently and uniformly (the model allows the same one twice, as the code above does).
  2. If either pick is any other backend, that other one is less loaded than the busiest, so it wins.
  3. The busiest backend is chosen only when both picks are it: (1/10) × (1/10) = 1/100 = 1%.
  4. More generally, the k-th busiest of n backends wins when both picks are among the k busiest but not both among the k − 1 busiest: (k² − (k − 1)²) / n² = (2k − 1) / n². For n = 10 that's 1% for the busiest, 3% for the second, up to 19% for the least loaded. That steady bias away from the busy end, applied to every request, keeps queues short without looking at every backend.

#Redundancy, failover and single points of failure

Added

A single point of failure (SPOF) is any component whose failure takes the whole service down: one database primary, one load balancer, one availability zone, one DNS provider, one TLS certificate that nobody renews. Redundancy means having more than one of it; failover is the process of moving work from the failed copy to a healthy one. Redundancy without working failover is just an expensive spare that nobody notices is broken.

There are two basic shapes. In active-passive, one copy (the primary) serves traffic and a standby waits, kept up to date by replication, ready to take over. In active-active, every copy serves traffic all the time, and the survivors absorb a failed copy's share. A mental model: active-passive is a car with a spare tyre. It's cheap and simple, but you have to stop, notice the flat and change the wheel (detection and promotion take time), and you only find out the spare was flat when you need it. Active-active is a plane with two engines: both are working all the time, so you know they work, and losing one is survivable only if each engine alone can fly the plane.

Active-passiveActive-active
Normal operationStandby idle (or serving only reads)All copies serve traffic
On failureDetect, promote standby, redirect traffic: seconds to minutesStop sending to the failed copy: seconds
Spare capacityA whole idle copy (1+1)Each copy runs below 100% so survivors can absorb load (N+1)
Hard partDetection, promotion, avoiding two primariesKeeping state consistent when every copy accepts writes
Typical useRelational database primaries, legacy apps, stateful singletonsStateless app tiers, load balancers, caches, multi-region reads
VariantsHot standby (running, replicated), warm (running, behind), cold (restore from backup)All-equal, or active-active with a home region per user
App tier writes db.internal virtual IP / DNS → current primary Primary epoch 7 Standby replica, read-only Witness quorum vote after failover replication heartbeat promotion needs a majority (standby + witness) and bumps the epoch to 8
Active-passive: a witness breaks ties so only one side can become primary.

Failover starts with failure detection, and detection is a trade-off you can put numbers on. A primary sends a heartbeat every second; the standby declares it dead after a timeout. Too short, and ordinary pauses (a garbage-collection stop, a network blip) trigger failovers when nothing was wrong. Too long, and a real crash means a long outage. This seeded model runs 30 days of heartbeats from a healthy primary, with rare long-tailed pauses, and counts false alarms for each timeout:

heartbeat.mjs
// A primary sends a heartbeat every second; the standby takes over if none arrives for TIMEOUT.
// Seeded model of 30 days on a healthy primary: rare pauses (GC, network blips) delay heartbeats.
function mulberry32(seed) {
  return () => {
    seed = (seed + 0x6d2b79f5) | 0;
    let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}
const rnd = mulberry32(2026);
const DAYS = 30, INTERVAL = 1000; // ms
const gaps = []; // time between consecutive heartbeats arriving at the standby
for (let i = 0; i < DAYS * 86_400; i++) {
  let gap = INTERVAL + (rnd() - 0.5) * 40; // ±20 ms of ordinary jitter
  if (rnd() < 0.0005) gap += 500 / Math.pow(1 - rnd(), 1 / 1.5); // a pause: Pareto, at least 500 ms, long tail
  gaps.push(gap);
}

const PROMOTION = 10_000; // ms to promote the standby and move the address once it decides (an assumption)
console.log("timeout   false failovers/day   real crash: detect + promote");
for (const timeout of [2_000, 3_000, 5_000, 10_000, 30_000]) {
  const falseAlarms = gaps.filter((g) => g > timeout).length / DAYS;
  // A real crash happens at a random moment after the last heartbeat, so on average
  // the standby notices TIMEOUT − INTERVAL/2 after the crash.
  const detect = timeout - INTERVAL / 2;
  console.log(
    `${(timeout / 1000 + " s").padStart(5)}   ${falseAlarms.toFixed(2).padStart(19)}   ` +
      `${(detect / 1000).toFixed(1)} s + ${PROMOTION / 1000} s = ${((detect + PROMOTION) / 1000).toFixed(1)} s down`,
  );
}
Result
timeout   false failovers/day   real crash: detect + promote
  2 s                 15.37   1.5 s + 10 s = 11.5 s down
  3 s                  5.63   2.5 s + 10 s = 12.5 s down
  5 s                  1.80   4.5 s + 10 s = 14.5 s down
 10 s                  0.57   9.5 s + 10 s = 19.5 s down
 30 s                  0.10   29.5 s + 10 s = 39.5 s down
What’s happening
  1. Each of the 2,592,000 heartbeats (30 days × 86,400 s) normally arrives 1000 ± 20 ms after the previous one. With probability 0.0005 (about 43 times a day) there's a pause: at least 500 ms, drawn from a Pareto distribution so most pauses are short but a few last many seconds, which is what GC pauses and network hiccups look like in practice. The numbers are illustrative, seeded with mulberry32(2026).
  2. A false failover is a gap longer than the timeout on a primary that was fine. At 2 s, that's 15.37 a day. Each one costs a promotion (10 s of unavailability in this model) and risks two primaries if the old one wakes up. 15 × 10 s is about 2.5 minutes of self-inflicted downtime a day, roughly 99.8% availability from detection alone.
  3. Raising the timeout cuts false alarms fast because the pauses are long-tailed: 5 s → 1.8 a day, 10 s → 0.57, 30 s → 0.1 (about three a month).
  4. The other column is the price. When the primary really crashes, the standby waits for the timeout, minus on average half an interval since the last heartbeat, then spends 10 s promoting. A 2 s timeout gives 11.5 s of downtime; 30 s gives 39.5 s.
  5. So there's no free setting: you choose between frequent short false failovers and rare long real ones. Practical systems pick a few seconds to tens of seconds, make promotion fast and safe (so a false failover is cheap), and require agreement from more than one observer before acting, which also filters out "the network between primary and standby broke" from "the primary died".
AdvancedSplit brain: when both sides think they're primary

The most dangerous failure in active-passive isn't the crash; it's the partition. The network between primary and standby breaks, both are alive, the standby stops hearing heartbeats and promotes itself, and now two primaries accept writes. Customers' orders go to different databases, and reconciling them later is manual and sometimes impossible. The defences, used together:

  • Quorum: promotion requires a majority of voters. With a primary, a standby and a witness (a tiny third node, ideally in a third zone), the side that can reach the witness has 2 of 3 votes; the isolated side has 1 and must not promote, and a primary that loses quorum should stop accepting writes. Two nodes alone can never break a tie, which is why clusters are sized 3 or 5. Leader election and consensus covers Raft, which does this properly.
  • Fencing: each promotion increments an epoch (a fencing token). Storage and downstream services reject writes carrying an older epoch, so an old primary that wakes up from a 40-second pause can't do damage. The bluntest form is STONITH ("shoot the other node in the head"): power off or detach the old primary before promoting.
  • One switch for traffic: clients find the primary through a single indirection (a virtual IP, a DNS name with a short TTL, or a proxy that knows the current leader), and only the winner of the election moves it.
AdvancedHunting single points of failure

Walk the request path in the interview and ask "what if this one disappears?" at every box. The usual suspects:

  • Load balancer, NAT gateway, VPN: managed ones are fleets; self-run ones need a pair with a floating IP.
  • Database primary: the classic SPOF. Hot standby in another zone with automatic failover, and test it.
  • A single availability zone: spread every tier across at least two (better three) zones, with N+1 capacity per zone loss, as in the sizing estimate.
  • Shared dependencies: DNS provider, identity provider, secrets manager, config service, container registry. When these are down you often can't even deploy the fix.
  • Non-machine SPOFs: a certificate or domain that expires, a cron job on one box, a single API key with a rate limit, the one engineer who knows how failover works.

Availability maths for redundant components (parallel vs serial) is in Availability, SLAs, SLOs and SLIs; blast radius and cells are in Failure modes and blast radius.

AdvancedActive-active at 70% each isn't redundant
Three zones, one fails

Your API runs in three zones with 12 instances each, all serving traffic, at a peak of 60% CPU. One zone goes down. What CPU do the remaining instances run at, and do you survive?

Show answer

90%: you survive, but with almost no headroom.

What’s happening
  1. Total work at peak is 36 instances × 60% = 21.6 instances' worth of CPU.
  2. After losing a zone, 24 instances remain to do the same work: 21.6 / 24 = 0.9, so 90% CPU each (assuming the load balancer spreads the load evenly across the survivors).
  3. That's below 100%, so requests still complete, but latency rises sharply as utilisation nears 100% (queues grow non-linearly), and any extra spike tips it over.
  4. The safe peak for "survive one of three zones" is 2/3 ≈ 67%; at 60% you're inside it with a small margin. At 70% you'd need 105% after the loss and would fail. Autoscaling helps only after its warm-up time, as the Autoscaling model shows.

#DNS, GeoDNS and anycast

Added

Before any load balancer sees a request, the client has to turn api.example.com into an IP address, and that lookup is your first chance to route traffic. Plain DNS can return several addresses (clients pick one), weight them, or drop an address whose health check fails. GeoDNS (latency- or location-based DNS) answers differently depending on where the question comes from, so users in Frankfurt get the European region's address and users in Singapore get Asia's. Anycast works one layer lower: the same IP address is announced from many locations on the internet, and ordinary internet routing (BGP) delivers each packet to the nearest one.

The mental model: DNS is a phone directory, and GeoDNS is a directory that prints a different local number depending on which town you bought it in. The catch is that people keep old directories: every resolver caches the answer for the record's TTL (time to live), so when you change a number, some callers keep dialling the old one for a while. Anycast is one national phone number that rings at the nearest branch. Nobody needs a new directory when a branch closes; the phone network just routes the call to the next-nearest one.

DNS round robin / weightedGeoDNS / latency-based DNSAnycast
HowSeveral A/AAAA records, or weights per recordAnswer depends on the resolver's location (or the client subnet it forwards)One IP announced via BGP from many sites
GranularityPer lookup, then cached for the TTLPer resolver location, cached for the TTLPer packet / per connection, by network topology
Failover speedTTL + resolver behaviour: seconds to many minutesSameSeconds to a minute: withdraw the route
Knows about load?Only via health checks and weights you setSame, plus geography or measured latencyNo: "nearest" in routing terms, not least loaded
Typical useSpreading across load balancers, simple failoverSending users to their nearest regionCDNs, DNS resolvers, DDoS absorption, global load balancers

How long does a DNS change take to work? Here's a failover: at t = 0 you change the record from the dead region's IP to a healthy one. Each resolver cached the old answer at a random moment within the last TTL, and (an assumption, for illustration) 3% of resolvers or clients ignore short TTLs and hold answers for an hour:

dns-ttl.mjs
// At t = 0 you change www's A record from the dead region's IP to a healthy one.
// Every resolver cached the old answer at some random moment within the last TTL seconds,
// and a few resolvers or clients ignore short TTLs and keep answers for an hour (an assumption).
const IGNORE_SHARE = 0.03, IGNORE_HOLD = 3600; // 3% hold any answer for 1 h

const stillOld = (ttl, t) => {
  const honest = Math.max(0, 1 - t / ttl);                       // cache age is uniform in [0, ttl)
  const sticky = Math.max(0, 1 - t / Math.max(ttl, IGNORE_HOLD)); // same, but held for an hour
  return (1 - IGNORE_SHARE) * honest + IGNORE_SHARE * sticky;
};

const times = [10, 30, 60, 300, 900, 3600];
console.log("share of users still sent to the old IP");
console.log("TTL      " + times.map((t) => (t < 60 ? `${t}s` : `${t / 60}m`).padStart(7)).join(""));
for (const ttl of [30, 60, 300, 3600]) {
  console.log(`${String(ttl).padStart(4)} s   ` + times.map((t) => `${(stillOld(ttl, t) * 100).toFixed(1)}%`.padStart(7)).join(""));
}

// The price of a short TTL: every resolver asks again once per TTL.
const RESOLVERS = 200_000; // distinct caching resolvers that look up the name (an assumption)
console.log("\nqueries per second reaching your authoritative DNS");
for (const ttl of [30, 60, 300, 3600]) console.log(`${String(ttl).padStart(4)} s   ${Math.round(RESOLVERS / ttl).toLocaleString("en-US").padStart(6)} q/s`);
Result
share of users still sent to the old IP
TTL          10s    30s     1m     5m    15m    60m
  30 s     67.7%   3.0%   2.9%   2.7%   2.3%   0.0%
  60 s     83.8%  51.5%   2.9%   2.7%   2.3%   0.0%
 300 s     96.8%  90.3%  80.5%   2.7%   2.3%   0.0%
3600 s     99.7%  99.2%  98.3%  91.7%  75.0%   0.0%

queries per second reaching your authoritative DNS
  30 s    6,667 q/s
  60 s    3,333 q/s
 300 s      667 q/s
3600 s       56 q/s
What’s happening
  1. A resolver that cached the answer a seconds ago (with a uniform between 0 and the TTL) keeps using it for another TTL − a seconds. So t seconds after the change, the share still on the old IP is 1 − t / TTL, until it reaches zero at t = TTL. That's the honest term.
  2. The sticky 3% hold the old answer for an hour regardless (some resolvers enforce a minimum TTL, and some clients and long-lived processes cache lookups themselves). They're why every row still shows 2–3% on the old IP after 1 to 15 minutes, even with a 30-second TTL. The 3% is an assumption chosen to show the shape; measure your own clients before quoting a number.
  3. With a 60 s TTL, half your users are still going to the dead region after 30 s, and the honest resolvers are all moved by 60 s. With 300 s, 80% are still on the old address after a minute. With 3,600 s, three-quarters are still lost after 15 minutes. TTL is your worst-case failover time for well-behaved clients.
  4. The second table is the price of short TTLs: each caching resolver asks your authoritative DNS once per TTL, so 200,000 resolvers generate 6,667 queries/s at a 30 s TTL and only 56 at an hour. Managed DNS providers absorb this easily (and charge per query); it matters more if you run your own name servers.
  5. The design conclusion: DNS failover is good for minutes-scale recovery. Use TTLs of 30–60 s on records you may need to move, keep a long tail of stragglers in mind, and use anycast or a global load balancer (one stable IP) when you need seconds.
AdvancedDNS failover with health checks

Managed DNS can do the switch for you: you attach a health check to each record, and the provider stops returning addresses whose checks fail. On AWS Route 53 (docs checked October 2026), health checkers in several regions probe the endpoint every 30 seconds, or 10 seconds with the "fast" option; an endpoint changes state after a failure threshold of 3 consecutive results by default; and Route 53 treats it as healthy if more than 18% of its checkers report it healthy (so one region's checkers being cut off doesn't fail you over). Detection takes roughly 3 × 30 s = 90 s with defaults, then the TTL drain from the model above starts. That's why DNS failover is minutes, not seconds.

AdvancedGeoDNS: what "the user's location" really is

The authoritative DNS server never sees the user; it sees the recursive resolver asking on their behalf. If the user is in Mumbai but uses a resolver whose nearest site is in Singapore, a naive GeoDNS sends them to Singapore. The EDNS Client Subnet extension (RFC 7871) lets resolvers forward a truncated client subnet (for example a /24) so the answer can match the user, at some cost to caching and privacy; not every resolver sends it. Latency-based routing (Route 53's "latency" policy, for instance) answers with the region that has the lowest measured latency from the resolver's network rather than the nearest on a map. Either way, GeoDNS is a coarse steering tool: good for "nearest region", unsuitable for per-request balancing.

GeoDNS also implements data-residency routing: EU users get EU addresses. It's a good first filter, but not an enforcement mechanism. Travellers, VPNs and misattributed resolvers land in the "wrong" region, so the application must still check where a user's data is allowed to live (see Multi-region architecture).

AdvancedAnycast in practice

With anycast, many sites announce the same IP prefix over BGP, and each network on the internet routes to whichever site is "closest" by its routing policy. That gives three things DNS can't:

  • Fast failover: a site that withdraws its announcement stops receiving traffic as soon as routes converge (typically seconds to a minute), with no client caches involved.
  • DDoS absorption: an attack from everywhere is split across every site instead of concentrating on one.
  • One stable address: clients, firewalls and partners can allow-list a single IP that never changes, even as regions come and go.

It's how public DNS resolvers and CDNs work: Cloudflare runs one anycast network across 348 cities and says 95% of the world's internet-connected population is within 50 ms of it (cloudflare.com/network, October 2026). Cloud providers sell it as a product too: AWS Global Accelerator gives each accelerator two static anycast IP addresses announced from AWS edge locations, which then carry traffic over AWS's backbone to the nearest healthy regional endpoint; Google Cloud's global external load balancer serves a single anycast IP worldwide.

Anycast's limits: "nearest" is decided by BGP policy, not by load or real latency, so a site can be overloaded while a slightly farther one is idle; and if routes change mid-connection, packets of a TCP connection can arrive at a site that has never heard of it. In practice routes are stable enough that TCP over anycast works well, and the providers that rely on it add tricks to handle the rare moves.

AdvancedLow TTLs don't make DNS failover instant
Picking a TTL

You may need to move api.example.com to another region during an incident, and you want 95% of well-behaved resolvers to be moved within 90 seconds of changing the record. What's the longest TTL you can use?

Show answer

About 94 seconds (in practice, 60 s with margin).

What’s happening
  1. A resolver's remaining cache time after the change is uniform between 0 and the TTL, so the fraction still on the old answer at time t is 1 − t / TTL.
  2. You need 1 − 90 / TTL ≤ 0.05, so 90 / TTL ≥ 0.95 and TTL ≤ 90 / 0.95 ≈ 94.7 s.
  3. That covers only the drain after you change the record. If a health check makes the change, add its detection time first (about 90 s with Route 53's defaults), which is why automated DNS failover is usually quoted as "a few minutes".
  4. Resolvers and clients that ignore TTLs are outside this calculation, as the 3% tail in the model shows. A 60 s TTL is the common choice: comfortably inside the budget, and still cheap in query volume.

#CDNs

Added

A content delivery network is a fleet of caching reverse proxies (points of presence, PoPs) spread around the world, sitting between users and your servers (the origin). A user's request goes to a nearby PoP; if the PoP has a fresh copy, it answers in a few milliseconds without your origin ever seeing the request. CDNs exist for two reasons that matter equally in an interview: latency (Azure measures about 200 ms for a round trip between its East US and Australia East regions, July 2026; no server upgrade beats the speed of light in fibre) and offload (every cache hit is a request your origin doesn't have to serve, which is often the cheapest scaling you can buy).

The mental model is a chain of corner shops supplied by one central warehouse. Customers shop at the corner shop on their street. When it runs out of something, it orders from a regional depot, and only when the depot runs out too does the order go to the central warehouse. Each product has a best-before date (the TTL); after it, the shop checks with the depot before selling it again. The shop's shelf labels are the cache key: if the same product is shelved under ten different labels, the shop orders it ten times.

The two ways to fill the shops:

  • Pull (the common case): the PoP fetches from the origin on the first request for an object (a miss), caches it, and serves later requests from cache until the TTL expires. Nothing to upload in advance; the first user in each PoP pays the miss.
  • Push: you upload content to the CDN's storage ahead of time (or the origin is the CDN's object storage). Good for large, predictable files (software releases, video segments) where you don't want the first viewer to pay, and for origins that can't take miss traffic.
User A first request User B same PoP, later Edge PoP near the user key: host+path Origin shield regional tier collapses misses Origin your servers or object store 1 2 miss 3 miss 4 5 cache 6 hit: served from the edge solid = request · dashed = response, cached on the way back for its TTL
Pull CDN: misses travel inward one tier at a time; hits are answered at the edge.

The scale is large: Amazon CloudFront lists 750+ PoPs in 100+ cities across 50+ countries, plus 15 regional edge caches (AWS CloudFront features page, October 2026), and Cloudflare lists 348 cities. With that many PoPs, each one sees only a slice of your traffic, so a long-tail object can miss in every PoP separately. That's what the origin shield fixes: one designated cache tier between the PoPs and the origin, so a miss in twenty PoPs becomes one request to the origin. Here's the effect, seeded: one hour, 50,000 images with Zipf-like popularity, 20 PoPs, a 10-minute TTL:

cdn-shield.mjs
// One hour of traffic for 50,000 images through 20 edge PoPs, TTL 10 minutes. Seeded.
// Popularity is Zipf-like (a few images get most views). How many requests reach the origin?
function mulberry32(seed) {
  return () => {
    seed = (seed + 0x6d2b79f5) | 0;
    let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}
const OBJECTS = 50_000, POPS = 20, REQUESTS = 2_000_000, HOUR = 3600, TTL = 600;
const cdf = [];
let sum = 0;
for (let k = 1; k <= OBJECTS; k++) cdf.push((sum += 1 / k)); // weight of rank k is 1/k
const zipf = (u) => {
  let lo = 0, hi = OBJECTS - 1; // binary search for the first rank whose CDF passes u·sum
  while (lo < hi) { const mid = (lo + hi) >> 1; if (cdf[mid] < u * sum) lo = mid + 1; else hi = mid; }
  return lo;
};

function run({ shield, noisyKeys }) {
  const rnd = mulberry32(99);
  const edges = Array.from({ length: POPS }, () => new Map()); // cache key -> expiry time
  const shieldCache = new Map();
  let edgeHits = 0, origin = 0;
  const fresh = (cache, key, t) => (cache.get(key) ?? -1) > t;
  for (let i = 0; i < REQUESTS; i++) {
    const t = (i / REQUESTS) * HOUR, pop = Math.floor(rnd() * POPS), obj = zipf(rnd());
    // Noisy keys: 30% of links carry a tracking parameter that ends up in the cache key.
    const key = noisyKeys && rnd() < 0.3 ? `img${obj}?utm=${Math.floor(rnd() * 50)}` : `img${obj}`;
    if (fresh(edges[pop], key, t)) { edgeHits++; continue; }
    edges[pop].set(key, t + TTL);
    if (shield) {
      if (fresh(shieldCache, key, t)) continue; // the shield answers; origin never sees it
      shieldCache.set(key, t + TTL);
    }
    origin++;
  }
  return `edge hit ratio ${((edgeHits / REQUESTS) * 100).toFixed(1)}%, origin requests ${origin.toLocaleString("en-US").padStart(7)} (${((origin / REQUESTS) * 100).toFixed(1)}%)`;
}

console.log(`no shield            ${run({ shield: false, noisyKeys: false })}`);
console.log(`with origin shield   ${run({ shield: true, noisyKeys: false })}`);
console.log(`shield + noisy keys  ${run({ shield: true, noisyKeys: true })}`);
Result
no shield            edge hit ratio 68.0%, origin requests 639,105 (32.0%)
with origin shield   edge hit ratio 68.0%, origin requests 181,553 (9.1%)
shield + noisy keys  edge hit ratio 52.6%, origin requests 459,582 (23.0%)
What’s happening
  1. cdf holds cumulative Zipf weights (rank k has weight 1/k), and zipf(u) binary-searches it, so a few images get most of the 2 million requests and a long tail gets one or two each. Each request goes to a random PoP at time t, spread evenly over the hour.
  2. Each PoP is a Map from cache key to expiry. A hit is a key whose expiry is still in the future. On a miss the PoP stores the object until t + TTL, and without a shield it fetches from the origin: 32.0% of all requests reach the origin (639,105), even though the edge hit ratio is 68%.
  3. With an origin shield, an edge miss first asks the shield's cache. The edge hit ratio is unchanged (68.0%: the PoPs behave the same), but the origin sees only the shield's misses: 181,553 requests, 9.1%, 3.5× fewer. The shield helps most with the long tail, where each PoP alone would rarely get a second request within the TTL, but the twenty PoPs together often do.
  4. The third run adds noisy cache keys: 30% of links carry a tracking parameter (?utm=…, 50 variants) that ends up in the key. The same image is now cached under many keys, so the edge hit ratio falls to 52.6% and origin traffic rises 2.5× to 459,582. The content didn't change; only the labels did.
  5. The fix for the third row is configuration, not hardware: build the cache key from only what changes the response (host, path, the query parameters that matter) and strip the rest. The lesson: CDN effectiveness is decided by cache keys and TTLs at least as much as by the number of PoPs.
AdvancedControlling what's cached, and for how long

The origin tells caches what to do with HTTP headers (RFC 9111 defines HTTP caching):

  • Cache-Control: public, max-age=31536000, immutable for versioned static assets (app.3f9a1c.js): the URL changes when the content does, so caches can keep it for a year and never revalidate.
  • Cache-Control: public, max-age=60, s-maxage=300 for semi-dynamic pages: browsers keep it for a minute, shared caches (CDNs) for five. CDN-Cache-Control (RFC 9213) targets CDNs only, so you can set edge TTLs without changing browser behaviour.
  • stale-while-revalidate=30 lets a cache serve a just-expired copy while it refetches in the background, so users never wait for a refresh; stale-if-error=86400 lets it serve stale content when the origin is down (both from RFC 5861). Together they turn an origin outage into stale-but-working pages.
  • Cache-Control: private or no-store for anything personal; Vary lists request headers that change the response (Accept-Encoding is fine; Vary: Cookie effectively disables caching).
AdvancedPurging

When content changes before its TTL ends, you have three options:

  1. Versioned URLs (best): never change content at a URL; publish a new URL and update the HTML that references it. No purge needed, and old and new versions can coexist during a deploy.
  2. Purge by URL or by tag: CDNs let you invalidate specific URLs, path prefixes, or every object carrying a tag (Fastly's Surrogate-Key, Cloudflare's Cache-Tag), so "product 42 changed" can purge its page, its API response and its images at once. Fastly says its purges complete globally in about 150 ms (the vendor's own figure); other CDNs range from seconds to minutes, and some charge per invalidation.
  3. Short TTLs plus stale-while-revalidate: accept up to a minute of staleness and skip purging entirely.
AdvancedWhat not to cache (or cache very carefully)
  • Personalised responses (a page with "Hello, Rohit", a cart, an account API) unless the cache key includes the identity, which usually makes caching pointless. Cache the shared shell and fetch the personal parts separately.
  • Responses that set cookies. A cached Set-Cookie hands one user's session to everyone who gets that copy.
  • Authenticated API responses keyed only by URL: user A's /api/me served to user B is a data breach. Many real incidents are this.
  • Error responses with long TTLs: a cached 500 or 404 keeps the outage going after the origin recovers. Use short negative-caching TTLs.
  • Requests with side effects: CDNs don't cache POST, but make sure state-changing operations aren't reachable by GET.
AdvancedWeb cache deception
95% vs 99% hit ratio

Your CDN serves 100,000 requests/s. How many requests per second reach the origin at a 95% hit ratio, and at 99%? What does that tell you about where to spend effort?

Show answer

5,000 req/s at 95% and 1,000 req/s at 99%: four points of hit ratio cut origin load by 5×.

What’s happening
  1. Origin load is the miss ratio times the traffic: (1 − 0.95) × 100,000 = 5,000 and (1 − 0.99) × 100,000 = 1,000.
  2. Hit ratio "sounds" nearly the same (95 vs 99), but the miss ratio, which is what the origin pays for, goes from 5% to 1%.
  3. So small improvements at the top end matter a lot: normalising cache keys, adding an origin shield (3.5× fewer origin requests in the model above), and longer TTLs with stale-while-revalidate are often worth more than adding origin servers.
  4. Track the origin request rate, not just the hit ratio. A hit ratio dominated by a few hot files can look excellent while the long tail still hammers the origin.

#Reverse proxies and API gateways

Added

A reverse proxy is a server that accepts requests on behalf of other servers and forwards them: clients think they're talking to the proxy, and the backends sit behind it unseen. (A forward proxy is the opposite: it acts on behalf of clients, like a corporate web proxy.) Load balancing is one thing a reverse proxy does, but most of its value is in the rest: TLS termination, compression, caching, buffering slow clients, request limits, routing by path. An API gateway is a reverse proxy specialised for APIs, with the cross-cutting policy every service would otherwise repeat: authentication, rate limits and quotas per client, request validation, API keys, versioned routing, request ids, and sometimes aggregation.

The mental model is a company's post room. A reverse proxy is the post room clerk: every parcel comes to one address, the clerk opens the outer box, checks it isn't leaking, and carries it to the right floor; the people upstairs never deal with couriers. An API gateway is a post room with a security desk and a ledger: it checks the sender's ID, enforces "no more than 100 parcels a day from this sender", stamps a tracking number on each one, and can bundle three replies into one envelope. What neither should do is read the letters and make business decisions. That's the departments' job.

ComponentActs forMain jobExamples
Forward proxyClientsOutbound filtering, caching, egress controlCorporate proxies, Squid
Reverse proxyServersTLS, routing, buffering, caching, compressionNGINX, HAProxy, Envoy, Caddy
Load balancerServersSpread traffic over healthy backendsALB, NLB, the reverse proxies above
API gatewayAPIsAuthN, quotas, keys, validation, versioning, analyticsAWS API Gateway, Kong, Apigee, Envoy Gateway
Service meshService-to-service callsmTLS, retries, per-request balancing, telemetry, via sidecars or node proxiesIstio, Linkerd
Backend for frontend (BFF)One client typeAggregate and reshape several services' data for one UIA Node service per app (web, iOS)

In real systems these blur: one Envoy fleet can be your load balancer, reverse proxy and gateway at once. In the interview, name the responsibilities you need at the edge and put them in one box; don't draw five boxes for one hop.

Why a gateway or BFF pays off on mobile

Aggregation is the gateway feature with the most measurable effect, because it moves round trips from the slow network (the user's phone) to the fast one (inside your data centre). A mobile home screen needs a profile, a feed and an unread count, then the authors of the feed's posts. Seeded, 20,000 page loads, the phone calling services itself vs calling one BFF:

bff.mjs
// A mobile home screen needs: profile, feed and unread count (independent), then the authors of the
// feed's posts (depends on the feed). Compare the app calling services itself with one BFF call.
function mulberry32(seed) {
  return () => {
    seed = (seed + 0x6d2b79f5) | 0;
    let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}
const rnd = mulberry32(5);
const MOBILE_RTT = 120, DC_RTT = 1; // ms, assumptions: a phone on a cellular network vs inside one data centre
// Service time: usually ~20 ms, but 2% of calls hit a slow path (cache miss, GC) at ~250 ms.
const service = () => (rnd() < 0.02 ? 250 : 20) * (0.7 + 0.6 * rnd());

const direct = [], viaBff = [];
for (let i = 0; i < 20_000; i++) {
  // Direct: wave 1 is three parallel calls from the phone, wave 2 is the authors call.
  const wave1 = Math.max(MOBILE_RTT + service(), MOBILE_RTT + service(), MOBILE_RTT + service());
  direct.push(wave1 + MOBILE_RTT + service());
  // BFF: one round trip from the phone; the BFF does both waves inside the data centre.
  const inner = Math.max(DC_RTT + service(), DC_RTT + service(), DC_RTT + service()) + DC_RTT + service();
  viaBff.push(MOBILE_RTT + 2 /* BFF's own work */ + inner);
}
const pct = (xs, p) => [...xs].sort((a, b) => a - b)[Math.floor(p * xs.length)].toFixed(0).padStart(4);
console.log("home screen        p50     p95     p99");
console.log(`app → services   ${pct(direct, 0.5)} ms ${pct(direct, 0.95)} ms ${pct(direct, 0.99)} ms`);
console.log(`app → BFF        ${pct(viaBff, 0.5)} ms ${pct(viaBff, 0.95)} ms ${pct(viaBff, 0.99)} ms`);
const slowOne = 1 - 0.98 ** 4;
console.log(`chance at least one of the 4 calls is slow: ${(slowOne * 100).toFixed(1)}%`);
Result
home screen        p50     p95     p99
app → services    284 ms  494 ms  568 ms
app → BFF         168 ms  374 ms  453 ms
chance at least one of the 4 calls is slow: 7.8%
What’s happening
  1. The assumptions: a phone on a cellular network has a 120 ms round trip to the data centre; services talk to each other with a 1 ms round trip inside it. Each service call takes about 20 ms (±30%), but 2% of calls hit a slow path at about 250 ms. All of these are illustrative numbers; the structure is the point.
  2. Direct: the app sends the three independent calls in parallel (wave 1 finishes when the slowest returns: Math.max of three MOBILE_RTT + service() values), then the authors call (wave 2), so two mobile round trips are always on the critical path: p50 284 ms.
  3. Via BFF: the app makes one call. The BFF runs both waves inside the data centre, where each hop costs 1 ms instead of 120 ms, so only one mobile round trip remains: p50 168 ms, 116 ms faster, almost exactly the round trip it removed.
  4. The tail improves by about the same amount (p99 568 → 453 ms) but stays high, because a page load makes 4 service calls and 1 − 0.98⁴ = 7.8% of page loads include at least one slow call either way. A BFF removes network round trips; it doesn't remove slow dependencies. For those you need timeouts, caching and partial responses (render the feed without the unread count).
  5. Extra benefits the numbers don't show: the BFF can trim fields the phone doesn't need (smaller payloads), the app ships fewer service contracts to keep compatible, and services can change behind the BFF without an app-store release.
AdvancedWhat belongs in the gateway, and what doesn't

Put policy that's the same for every request at the edge:

  • Authentication (validate the token or API key once), coarse authorisation ("this key may call the orders API"), rate limits and quotas per key (Rate limiting algorithms).
  • TLS termination, request size limits, schema validation of obvious junk, IP allow/deny lists, a WAF.
  • Routing by path, host, header or version; canary weights; request ids and access logs; response compression.

Keep business logic out: "orders over $500 need a fraud check" or "only the owner can cancel this order" belongs in the service that owns the data. A gateway full of business rules becomes a second monolith that every team must change and deploy, and it's on every request's path, so its bugs are everyone's outage. The Node.js site builds a small gateway in code; here the design point is the boundary.

AdvancedBuffering: the reverse proxy's quiet superpower

Many backends handle a request with a dedicated worker (a thread or process per request: PHP-FPM, Rails, Django, most Java servlet setups). A phone uploading a 5 MB photo over a weak connection can take 10 seconds to send its body, and without a proxy that worker is stuck reading slowly for 10 seconds doing nothing useful. A reverse proxy like NGINX buffers the whole request first and hands it to the backend only when it's complete, and buffers the response so the backend can move on while the proxy drips it out to the slow client. Event-loop servers such as Node suffer less from this (a slow socket doesn't hold a thread), but buffering still protects them from slow-client attacks and keeps large bodies out of the app's memory. For very large uploads, skip the path entirely with direct uploads to object storage.

AdvancedThe gateway as a bottleneck and blast radius
Sequential calls from a phone

A screen needs four API calls that must happen one after another (each needs the previous result). The phone's round trip is 120 ms, each call takes 20 ms in the service, and inside the data centre a hop costs 1 ms. Roughly how long does the screen take with direct calls, and with one BFF call that makes the four calls internally?

Show answer

About 560 ms direct and about 204 ms via the BFF.

What’s happening
  1. Direct: each call costs a mobile round trip plus service time, 120 + 20 = 140 ms, and they're sequential, so 4 × 140 = 560 ms.
  2. Via the BFF: one mobile round trip (120 ms), then inside the data centre four sequential calls at 1 + 20 = 21 ms each, 84 ms in total: 120 + 84 = 204 ms (plus a little for the BFF's own work).
  3. The saving is three mobile round trips (360 ms). Sequential chains are where aggregation pays most; independent calls can at least run in parallel from the client, as the model above shows.
  4. If the chain can be shortened, by adding an endpoint that returns the data together, or by denormalising it, that beats both: one round trip and one service call.

#Autoscaling

Added

Autoscaling adjusts the number of instances to match demand, automatically: add capacity when load rises, remove it when load falls, so you neither pay for idle machines all night nor fall over at lunchtime. It's only possible because of the earlier topics: instances must be stateless so they can come and go, and behind a load balancer that starts and stops sending them traffic. The design questions are what to measure, how fast to react, and what can't scale with it.

The mental model is a restaurant manager calling in extra staff. When the queue at the door grows, she phones people on the rota, but they take twenty minutes to arrive and put on aprons (boot and warm-up), so she has to call before the queue is out of the door, not after. When the rush ends she doesn't send everyone home the moment there's a lull, because the next wave might be ten minutes away (cooldown). And there's one thing she can't add more of: the kitchen. Twice the waiters doesn't help if the one kitchen (the database) is the bottleneck.

Kinds of scaling policy:

  • Target tracking: "keep average CPU at 60%". The autoscaler computes the instance count that would bring the metric to the target. Kubernetes' Horizontal Pod Autoscaler uses exactly desired = ceil(current × currentMetric / targetMetric) (Kubernetes docs, 2026).
  • Step scaling: "add 4 instances when CPU > 80% for 3 minutes". More knobs, more ways to get it wrong.
  • Scheduled: "40 instances from 08:30 on weekdays". For predictable peaks; also the only answer for known events (a launch, a sale, a push notification you're about to send).
  • Predictive: forecast from history and scale ahead of the curve; works for strongly periodic traffic.

What to scale on: CPU for CPU-bound services; requests or concurrency per instance for I/O-bound services (Node APIs often hit latency limits before CPU does); queue backlog per worker for consumers (messages waiting / messages per worker per minute gives the workers needed); avoid scaling on latency alone, because it rises for reasons more instances don't fix (a slow database) and you'd scale the wrong tier.

Now the timing problem, in a model with no randomness at all. Traffic sits at 3,000 req/s, then a push notification ramps it to 12,000 req/s in two minutes. Each instance handles 500 req/s at 100% CPU. A target-tracking policy checks once a minute, using a CPU metric that arrives a minute late:

autoscale.mjs
// Target-tracking autoscaler vs a traffic surge, in 10-second steps (no randomness).
// Traffic sits at 3,000 req/s, then a push notification ramps it to 12,000 req/s within 2 minutes.
const CAPACITY = 500;  // req/s one instance serves at 100% CPU
const STEP = 10;       // seconds per simulation step
const traffic = (t) => (t < 300 ? 3000 : Math.min(12_000, 3000 + ((t - 300) / 120) * 9000));

function simulate({ warmup, target, start = 10 }) {
  let running = start;         // instances serving traffic
  const pending = [];          // ready-at times of instances still booting / warming up
  const busyHistory = [];      // instances' worth of CPU in use, as the metric reports it
  let dropped = 0, overloadedSteps = 0, peakCpu = 0;
  for (let t = 0; t < 1800; t += STEP) {
    while (pending.length && pending[0] <= t) pending.shift(), running++;
    const demand = traffic(t);
    const cpu = Math.min(1, demand / (running * CAPACITY)); // CPU can't show more than 100%
    busyHistory.push(cpu * running);
    peakCpu = Math.max(peakCpu, demand / (running * CAPACITY));
    if (demand > running * CAPACITY) (dropped += (demand - running * CAPACITY) * STEP), overloadedSteps++;
    if (t % 60 === 0 && t >= 120) {
      // Once a minute, read the average of the minute before last (metrics arrive about a minute late).
      const window = busyHistory.slice(-12, -6);
      const busy = window.reduce((a, b) => a + b, 0) / window.length;
      const desired = Math.ceil(busy / target);
      const total = running + pending.length; // instances already on the way count too
      for (let k = total; k < desired; k++) pending.push(t + warmup);
    }
  }
  return `overloaded ${((overloadedSteps * STEP) / 60).toFixed(1).padStart(4)} min, ` +
    `peak demand ${(peakCpu * 100).toFixed(0).padStart(3)}% of capacity, ` +
    `${(dropped / 1e6).toFixed(2)}M requests over capacity, ends with ${running} instances`;
}

console.log(`warm-up  60 s, target 60%: ${simulate({ warmup: 60, target: 0.6 })}`);
console.log(`warm-up 300 s, target 60%: ${simulate({ warmup: 300, target: 0.6 })}`);
console.log(`warm-up 300 s, target 40%: ${simulate({ warmup: 300, target: 0.4 })}`);
console.log(`warm-up 300 s, 30 at start: ${simulate({ warmup: 300, target: 0.6, start: 30 })}`);
Result
warm-up  60 s, target 60%: overloaded  5.5 min, peak demand 240% of capacity, 1.31M requests over capacity, ends with 40 instances
warm-up 300 s, target 60%: overloaded 13.5 min, peak demand 240% of capacity, 3.83M requests over capacity, ends with 40 instances
warm-up 300 s, target 40%: overloaded  6.5 min, peak demand 225% of capacity, 1.64M requests over capacity, ends with 60 instances
warm-up 300 s, 30 at start: overloaded  0.0 min, peak demand  80% of capacity, 0.00M requests over capacity, ends with 40 instances
What’s happening
  1. Every 10-second step, the model computes demand and CPU: demand / (running × 500), capped at 100%, because a CPU metric can't report more than 100% even when demand is 240% of capacity. busyHistory records CPU × instances, which is the "instances' worth of work" the metric shows.
  2. Once a minute it reads the average of the minute before last (slice(-12, -6)), mimicking metrics that are aggregated per minute and delivered late. It computes desired = ceil(busy / target) and orders the difference; ordered instances count towards capacity immediately (as AWS does), so it doesn't order the same ones twice, but they only serve traffic after warmup seconds.
  3. With a 60-second warm-up, the tier is overloaded for 5.5 minutes and 1.31 M requests arrive above capacity (queued, slowed or dropped). Two things stretch it out: the metric lag, and the 100% cap, which means each scaling round sees at most "we need 1/0.6 = 1.67× what we have", so it takes several rounds to catch up with a 2.4× surge.
  4. With a 300-second warm-up (realistic for a VM that boots, pulls a container and warms caches), the same policy is overloaded for 13.5 minutes and 3.83 M requests are over capacity. Warm-up time, not the policy, is the dominant term.
  5. A 40% target keeps more headroom: it already asks for 15 instances while traffic is still flat (they arrive two minutes into the surge), and each later round orders more: 6.5 minutes, but the tier now runs 60 instances where 40 would do, all day. Headroom is the cost of slow scaling.
  6. Pre-scaling to 30 instances before sending the notification removes the problem: 0 minutes overloaded, peak at 80% of capacity. If you know the surge is coming (you're the one sending the push), scheduled scaling beats any reactive policy. (The model only scales out, so the 30 instances aren't scaled back in before the surge; in real life a cooldown or a scheduled minimum does that.)
AdvancedThe knobs, with real defaults (checked October 2026)
  • Kubernetes HPA: evaluates every 15 s; ignores changes within a 10% tolerance; scale-down uses a 300 s stabilisation window (it picks the highest recommendation from the last 5 minutes, so it doesn't flap); default scale-up may add 100% of current pods or 4 pods per 15 s, whichever is more; new pods' CPU samples are treated cautiously for the first 5 minutes (cpu-initialization-period), with a 30 s initial readiness delay.
  • AWS EC2 Auto Scaling: default instance warm-up is not enabled by default; AWS suggests starting at 300 s and tuning. While instances warm up they count towards capacity, but their metrics are excluded.
  • Make instances fast to become useful: small images, pre-baked dependencies, lazy initialisation off the critical path, a readiness check that passes only when the instance can really serve. Every minute saved in warm-up is worth more than any tuning of the policy.
  • Keep a minimum that survives a zone loss and a sudden spike, and a maximum that protects your wallet and your downstream dependencies.
AdvancedScaling the database is the hard part

Autoscaling the stateless tier is easy precisely because the state lives elsewhere, and "elsewhere" doesn't autoscale. The first thing that breaks is often not CPU but connections:

db-connections.mjs
// The app tier autoscales; the database doesn't. Count the connections each step of a scale-out opens.
const processesPerInstance = 4; // e.g. one Node process per vCPU on a 4-vCPU instance
const poolSize = 10;            // connections in each process's pool (a common library default)
const limits = { "PostgreSQL 18 default": 100, "MySQL 9.7 default": 151, "raised by hand": 2000 };

console.log("instances  connections  " + Object.keys(limits).map((k) => k.padStart(22)).join(""));
for (const instances of [2, 10, 40, 100]) {
  const conns = instances * processesPerInstance * poolSize;
  const fits = Object.values(limits).map((max) => (conns <= max ? "fits" : `over by ${conns - max}`).padStart(22)).join("");
  console.log(`${String(instances).padStart(9)}  ${String(conns).padStart(11)}  ${fits}`);
}

// With a pooler (PgBouncer, RDS Proxy) in front: app processes share a small set of real connections.
// Real connections needed ≈ concurrent queries at peak = query rate × query time (Little's law).
const peakQueries = 12_000; // queries per second at peak (assumption)
const queryMs = 4;          // average time a query holds its connection
console.log(`\nwith a pooler: ${peakQueries.toLocaleString("en-US")} q/s × ${queryMs} ms = ${(peakQueries * queryMs) / 1000} busy connections on average`);
Result
instances  connections   PostgreSQL 18 default     MySQL 9.7 default        raised by hand
        2           80                    fits                  fits                  fits
       10          400             over by 300           over by 249                  fits
       40         1600            over by 1500          over by 1449                  fits
      100         4000            over by 3900          over by 3849          over by 2000

with a pooler: 12,000 q/s × 4 ms = 48 busy connections on average
What’s happening
  1. Each instance runs 4 app processes, and each process keeps a pool of 10 connections (node-postgres' default max is 10, per its docs in 2026). So connections grow with instances: instances × 4 × 10.
  2. At 2 instances that's 80 connections, which fits PostgreSQL 18's default max_connections of 100 and MySQL 9.7's default of 151 (both checked in their docs, October 2026).
  3. At 10 instances it's 400: over both defaults. At the 40 instances the autoscaling model ended with, 1,600; a surge to 100 instances needs 4,000, past even a hand-raised limit of 2,000. Each PostgreSQL connection is a separate server process using memory, so raising the limit isn't free either.
  4. The fix is a connection pooler (PgBouncer, RDS Proxy, or the managed equivalent) between the app and the database. By Little's law, the connections that are actually busy are query rate × query time: 12,000 queries/s × 4 ms = 48 on average. The pooler multiplexes thousands of client connections onto about that many real ones (plus headroom for bursts).
  5. Beyond connections: reads scale with replicas and caches, writes need a bigger primary or sharding, and none of that happens in the 60 seconds an app tier takes to scale. Set the app tier's maximum so a surge can't overwhelm the database, and shed load at the edge instead (Backpressure and load shedding).
AdvancedScaling on the wrong signal
What does the HPA do?

A Deployment has 8 pods with a CPU target of 60%. Average CPU across them is 90%. How many pods will the Horizontal Pod Autoscaler ask for? What if average CPU were 63%?

Show answer

12 pods at 90%; no change at 63%.

What’s happening
  1. The HPA formula is desired = ceil(current × current / target): ceil(8 × 90 / 60) = ceil(12) = 12.
  2. The default scale-up policy allows the larger of 100% or 4 pods per 15 s, so going from 8 to 12 (+4, +50%) is allowed in one step.
  3. At 63%, the ratio is 63 / 60 = 1.05, inside the default 10% tolerance (0.9 to 1.1), so the HPA does nothing. Without the tolerance it would compute ceil(8 × 1.05) = 9 and keep adding and removing a pod as CPU wobbles around the target.
  4. The new pods don't help until they're ready, and the HPA discounts their early CPU readings, so expect the next decision 15 seconds or more later to be based on partly stale data, as in the model above.

#Multi-region architecture

Added

Running in several regions (independent data-centre groups hundreds or thousands of kilometres apart) solves three problems a single region can't: latency for users far away, survival of a whole-region outage, and data residency rules that require some users' data to stay in their country or bloc. Everything up to here (stateless tiers, load balancers, CDNs, DNS) extends to many regions easily. The hard part is the same as always, state: the speed of light means regions can't agree on anything quickly, so every multi-region design is a choice about where writes happen and how long other regions wait to see them.

The mental model is a bank with branches on three continents. Option one: every transaction is approved by head office in New York. Simple and always consistent, but a customer in Singapore waits for a call to New York every time (single region, or active-passive). Option two: each branch approves its own customers' transactions and sends a summary to the others every few seconds. Fast for everyone, but two branches can approve withdrawals from the same joint account in the same few seconds and overdraw it (active-active with asynchronous replication). Option three: every transaction needs a majority of branches to agree, which is consistent and survives losing a branch, but every transaction waits for the nearest other branch (synchronous quorum). Many banks also do option four: each account belongs to one home branch, which approves its transactions, while other branches can read balances.

US users EU users Global front door GeoDNS or anycast · health checks Region: US East Load balancer App tier (stateless) Database local reads and writes Region: West Europe Load balancer App tier (stateless) Database local reads and writes async ~85 ms
Active-active regions: users go to the nearest region; databases replicate to each other asynchronously.

What does each design cost a user, in milliseconds? This model uses Azure's published median round-trip times between East US, West Europe and Southeast Asia (Microsoft's "Azure network round-trip latency statistics", 30 days to 30 July 2026) and computes read and write latency for users in each region:

regions.mjs
// Users in three regions; where do reads and writes go, and what does each design cost them?
// Round-trip times: Azure's published P50 between East US, West Europe and Southeast Asia (30 days to 30 July 2026).
const rtt = {
  us: { us: 0, eu: 83, asia: 224 },
  eu: { us: 85, eu: 0, asia: 169 },
  asia: { us: 224, eu: 169, asia: 0 },
};
const LOCAL = 2; // ms of work inside the region. The user’s hop to their nearest region is the same in every design, so it’s left out.
const regions = ["us", "eu", "asia"];

const designs = {
  // Everything lives in us.
  "single region (us)": (u) => ({ read: rtt[u].us, write: rtt[u].us }),
  // Read replicas everywhere, one primary in us for writes.
  "active-passive writes": (u) => ({ read: 0, write: rtt[u].us }),
  // Every region takes reads and writes locally; replication to the others is asynchronous.
  "active-active (async)": () => ({ read: 0, write: 0 }),
  // Every write must be acknowledged by a majority (2 of 3 regions) before it commits.
  "majority quorum (sync)": (u) => ({ read: 0, write: Math.min(...regions.filter((r) => r !== u).map((r) => rtt[u][r])) }),
};

console.log("design".padEnd(24) + regions.map((r) => `${r} read/write`.padStart(17)).join(""));
for (const [name, cost] of Object.entries(designs)) {
  const cells = regions.map((u) => {
    const { read, write } = cost(u);
    return `${read + LOCAL}/${write + LOCAL} ms`.padStart(17);
  });
  console.log(name.padEnd(24) + cells.join(""));
}
Result
design                      us read/write    eu read/write  asia read/write
single region (us)                 2/2 ms         87/87 ms       226/226 ms
active-passive writes              2/2 ms          2/87 ms         2/226 ms
active-active (async)              2/2 ms           2/2 ms           2/2 ms
majority quorum (sync)            2/85 ms          2/87 ms         2/171 ms
What’s happening
  1. rtt holds the measured medians: 83–85 ms between the US and Europe, 169 ms Europe–Asia and 224 ms US–Asia (the table is directional; the model uses each direction as published). Every cell adds 2 ms of in-region work; the user's own hop to their nearest region is left out because it's the same in every design. Region-to-region RTT stands in for "a user in that region calling a server in another".
  2. Single region in the US: fine for US users (2 ms beyond their own hop), but every European request pays 87 ms and every Asian one 226 ms, reads and writes alike. Pages that make several sequential calls multiply that.
  3. Active-passive writes (read replicas in every region, one primary in the US): reads become local everywhere (2 ms), writes still go to the US. A good fit for read-heavy products; the cost is that a user may not see their own write in their local replica for a moment (read-your-writes needs care, see Consistency models).
  4. Active-active with asynchronous replication: 2/2 ms everywhere, the best possible, because no write waits for another region. The price isn't in this table: two regions can accept conflicting writes to the same data before they hear about each other (next model).
  5. Majority quorum across three regions (Raft or Paxos style, as Spanner and CockroachDB do): assuming the leader for the user's data sits in the user's region (per-partition leaders placed near their users), every write waits for the nearest other region to acknowledge, 85 ms in the US, 87 in Europe, 171 in Asia. That's the cost of never having conflicts and surviving a region loss with no data loss. Physics sets the floor: light in fibre covers about 200 km per millisecond, so no protocol can beat the distance.
  6. The interview move is to choose per kind of data: profile reads local, payments on a quorum or a home region, likes and view counts active-active with a merge rule.
AdvancedHow often do active-active writes conflict?

With asynchronous replication, a conflict happens when both regions write the same key within the replication lag of each other. That depends entirely on how often a single key is written from two places:

conflicts.mjs
// Active-active with asynchronous replication: two regions can both accept a write to the same key
// before either has seen the other's. How often does that happen?
const LAG = 0.15; // seconds until a write in one region is visible in the other (≈ one-way trip + queueing; assumption)

// Writes to one key arrive at random (Poisson) in each region at `perRegion` writes/s.
// A write conflicts if the other region wrote the same key within LAG either side of it.
const conflictChance = (perRegion) => 1 - Math.exp(-perRegion * 2 * LAG);

const keys = [
  ["a user's settings (two devices)", 1 / 86_400],
  ["stock count, popular product", 2],
  ["seat map, popular flight", 0.5],
  ["like counter, viral post", 200],
];

console.log("one key, written in both regions".padEnd(34) + "writes/s each".padStart(14) + "chance a write conflicts".padStart(26) + "conflicts/day".padStart(15));
for (const [name, rate] of keys) {
  const p = conflictChance(rate);
  const perDay = p * rate * 2 * 86_400; // both regions' writes to this key in a day
  const chance = p < 0.001 ? p.toExponential(1) : `${(p * 100).toFixed(1)}%`;
  console.log(name.padEnd(34) + String(+rate.toPrecision(2)).padStart(14) + chance.padStart(26) + (perDay < 1 ? `1 per ${Math.round(1 / perDay / 365)} yrs` : Math.round(perDay).toLocaleString("en-US")).padStart(15));
}
Result
one key, written in both regions   writes/s each  chance a write conflicts  conflicts/day
a user's settings (two devices)         0.000012                    3.5e-6  1 per 395 yrs
stock count, popular product                   2                     45.1%        155,931
seat map, popular flight                     0.5                     13.9%         12,035
like counter, viral post                     200                    100.0%     34,560,000
What’s happening
  1. LAG is how long a write in one region takes to become visible in the other: here 150 ms, roughly a one-way trip plus queueing and apply time (an assumption; measure replication lag in your own system).
  2. If the other region writes a key at random (Poisson) times at r writes/s, the chance it writes within ±LAG of a given write is 1 − e^(−r × 2 × LAG). Conflicts per day multiply that by both regions' writes to the key.
  3. A user's settings, edited from two devices about once a day each: a conflict roughly once per 395 years per user. Even with 50 million users that's a few hundred a day, and last-write-wins is acceptable for a settings toggle.
  4. A popular product's stock count written 2 times/s in each region: 45% of writes conflict, 155,931 a day. Last-write-wins here silently loses decrements, which means overselling. This data needs a single home (one region owns the item's inventory) or a quorum.
  5. A seat map at 0.5 writes/s: 14% conflict, and a lost write is a double-booked seat, so again a single owner per flight. A like counter at 200 writes/s conflicts on every write, but a counter can be a CRDT (each region counts its own likes; the total is the sum), so conflicts merge automatically (Conflict resolution: last-write-wins and CRDTs).
  6. The lesson: "active-active" is a per-dataset decision. Rarely-contended data replicates freely; contended data with invariants (stock, seats, balances) needs one writer; commutative data (counters, sets of likes) can merge.
AdvancedPatterns that make multi-region work
  • Home region per user (geo-partitioning): each user's data lives in one region (usually where they signed up, or where the law requires), and requests for that user are routed there, even from abroad. Writes are local for most users most of the time, with no conflicts. This is also how data residency is usually met: EU users' rows live in EU regions. Rules vary by law: the GDPR restricts transfers of personal data outside the EU rather than banning them outright, while India's central bank has required payment-system data to be stored only in India since its April 2018 circular (RBI). Treat residency as a requirement to clarify, then design storage around it.
  • Read local, write home: replicas everywhere for reads, writes forwarded to the owning region. Show users their own writes by reading from the home region (or waiting for the replica to catch up) right after they write.
  • Global services split by consistency need: an identity service with a strongly consistent store, catalogues replicated everywhere, carts per home region, analytics streamed asynchronously.
  • Routing: a global front door (GeoDNS with health checks, or anycast) sends users to the nearest healthy region; see DNS, GeoDNS and anycast.
  • Capacity for failover: in active-active, every region must be able to absorb the traffic of the region that fails, so two regions each run below 50% at peak, three below 67%, the same arithmetic as zones. Many teams accept degraded service instead (non-critical features off) to avoid paying for full spare capacity.
  • Cost: cross-region data transfer is billed per GB by every major cloud, and replicating everything everywhere multiplies storage. Replicate what you need where you need it.
Advanced"Active-active" without a conflict plan
A synchronous write across the Atlantic

Your primary database is in West Europe with a synchronous replica in East US: a write commits only after the US replica confirms it. Using the measured round trip of about 85 ms, what's the minimum commit latency, and roughly how many sequential commits per second can one database connection do?

Show answer

At least ~85 ms per commit, so at most about 11–12 sequential commits per second per connection.

What’s happening
  1. A synchronous commit sends the change to the replica and waits for its acknowledgement: one full round trip, about 85 ms from West Europe to East US, plus local work.
  2. One connection runs its transactions one after another, so it can commit at most 1000 / 85 ≈ 11.8 times a second, whatever the hardware.
  3. Throughput has to come from concurrency (many connections committing in parallel, or batching several changes per commit), not from faster commits. Latency per write stays at 85 ms+ for every user.
  4. That's why synchronous cross-region replication is reserved for data that can't lose a committed write (payments, ledgers), while most data replicates asynchronously, or synchronously only within a region, across zones whose round trip is typically a millisecond or two (a rule of thumb).

Written fresh for interviews (there was no Google Doc for this one). Runnable models were run on Node 26; figures and limits were checked against sources in October 2026.

RohitDownloads · All chapters

Sync progress across devices

Type the same private phrase on your Mac and your phone, and your Learned ticks follow you between them — on every study site.

The phrase never leaves this device: only a fingerprint of it is sent, and the server stores a fingerprint of that. Anyone who knows the phrase could see or change your ticks, so pick something you don’t use elsewhere.

Sync is on in this browser.

Sync ID

This ID must be the same on every device. If another device shows a different one, its phrase is different (capital letters count): tap “Turn off here” on it and type the phrase again exactly.