#Failure modes and blast radius
AddedAt any real scale, something is always broken. With a few thousand machines, disks die every day, a few instances are always mid-restart, one dependency is always slower than usual, and someone is always deploying. So the reliability question in a design interview is never "what if it fails?" but "when this fails, who notices, and how many of them?" That second number is the blast radius: the share of users, requests or tenants affected by a single fault.
The mental model is a ship's watertight compartments. You can't promise the hull will never be holed, so you divide it so that one hole floods one compartment, not the ship. Redundancy (two engines) protects against a component dying; compartments protect against a fault spreading. Most large outages are spreading problems: a bad config pushed everywhere at once, a "query of death" that crashes every replica it reaches, a retry storm that turns one slow service into ten.
It helps to name failures precisely, because each one has a different defence:
| Failure | What it looks like | Main defence |
|---|---|---|
| Crash or lost instance | Connection refused, health check fails | Redundancy behind a load balancer, fast replacement |
| Slow or hanging dependency | Requests pile up, threads and sockets run out | Timeouts, circuit breakers, bulkheads (next topic) |
| Overload | Latency climbs, then errors everywhere | Load shedding, backpressure, autoscaling |
| Bad deploy or config push | Errors start right after a change | Canaries, staged rollout one cell at a time, fast rollback |
| Poison request ("query of death") | One input crashes every server that handles it | Shuffle sharding, input limits, quarantining the caller |
| Third-party outage | Payments or email provider down | Fallbacks, queue the work and retry later |
| Zone or region loss | A whole location disappears | Multi-zone by default, multi-region for the critical path |
| Data corruption or operator error | Bad migration, DELETE without WHERE | Point-in-time backups, guard rails, two-person review |
| Correlated failure | Every server hits the same certificate expiry, leap second or shared-library bug at once | Staggered expiry and rollout, monitoring of expiry dates, diversity |
Two of these deserve a special warning. Gray failures — a server that is slow, or that fails one request type in ten — are worse than crashes, because health checks still pass and load balancers keep sending traffic. And correlated failures defeat redundancy completely: three replicas running the same buggy build crash together, so "we have three of them" is no defence against a bad deploy or a poison request.
Shrinking the blast radius
There is a ladder of techniques, each limiting spread a little more:
- Redundancy (N+1, multi-zone): survives a machine or zone dying. Doesn't help when every copy has the same bug or receives the same poison request.
- Sharding by customer: split customers across independent groups of servers. A fault in one shard hits only its customers —
1/shardsof them. - Cells: take sharding all the way: every cell is a complete, independent copy of the stack (load balancer, app servers, cache, database) serving a fixed slice of customers. A thin cell router in front sends each request to its customer's cell. Cells are also the unit of deployment, so a bad release goes to one cell first.
- Shuffle sharding: instead of fixed groups, give every customer its own random combination of a few servers. Two customers rarely share the same whole combination, so one customer's poison request can only take out the servers that customer touches.
The AWS whitepaper Reducing the Scope of Impact with Cell-Based Architecture (checked October 2026) describes the same shape: a routing layer that sends each request to a cell by a partition key (customer id, account id, resource id), cells that ideally share nothing, and a router kept as simple as possible because it is the one component every request still passes through.
Now shuffle sharding, with numbers. Take 8 workers. With plain sharding you'd split them into 4 fixed pairs and give each customer a pair, so a poison customer that crashes both workers in its pair takes out every other customer on that pair — a quarter of everyone. With shuffle sharding each customer gets its own random pair, out of all the pairs that exist. The model below computes the odds and then simulates them with a seeded random generator:
// Shuffle sharding: how often do two customers share ALL of their workers?
const choose = (n, k) => { let r = 1; for (let i = 1; i <= k; i++) r = (r * (n - k + i)) / i; return r; };
function mulberry32(seed) { // small seeded PRNG, so the simulation prints the same numbers every run
return () => {
seed = (seed + 0x6d2b79f5) | 0;
let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
};
}
function randomShard(workers, size, rand) { // pick `size` distinct workers
const pool = [...Array(workers).keys()];
for (let i = 0; i < size; i++) { const j = i + Math.floor(rand() * (workers - i)); [pool[i], pool[j]] = [pool[j], pool[i]]; }
return pool.slice(0, size);
}
// 1. The maths: 8 workers, each customer gets 2.
const shards = choose(8, 2);
console.log(`8 workers, shard of 2: ${shards} possible shards`);
console.log(` plain sharding (4 fixed pairs): a poison customer takes out 1/4 = 25.0% of customers`);
console.log(` shuffle sharding: P(another customer has the SAME pair) = 1/${shards} = ${(100 / shards).toFixed(1)}%`);
console.log(` P(shares exactly one worker) = ${2 * 6}/${shards} = ${((100 * 12) / shards).toFixed(1)}%, P(shares none) = ${choose(6, 2)}/${shards} = ${((100 * choose(6, 2)) / shards).toFixed(1)}%`);
// 2. Simulate it: 10,000 customers, customer 0 sends a request that crashes every worker it touches.
const rand = mulberry32(7);
const customers = Array.from({ length: 10_000 }, () => randomShard(8, 2, rand));
const poisoned = new Set(customers[0]);
let all = 0, some = 0;
for (const shard of customers.slice(1)) {
const hit = shard.filter((w) => poisoned.has(w)).length;
if (hit === shard.length) all++;
else if (hit > 0) some++;
}
console.log(`simulated (seeded), 9,999 other customers: ${all} lose every worker (${((100 * all) / 9999).toFixed(1)}%), ${some} lose one of two`);
// 3. Bigger fleets make full overlap vanishingly rare.
for (const [n, k] of [[100, 5], [2048, 4]]) {
console.log(`${n} workers, shard of ${k}: ${choose(n, k).toLocaleString("en-US")} shards, P(same shard) = ${(1 / choose(n, k)).toExponential(2)}`);
}8 workers, shard of 2: 28 possible shards
plain sharding (4 fixed pairs): a poison customer takes out 1/4 = 25.0% of customers
shuffle sharding: P(another customer has the SAME pair) = 1/28 = 3.6%
P(shares exactly one worker) = 12/28 = 42.9%, P(shares none) = 15/28 = 53.6%
simulated (seeded), 9,999 other customers: 361 lose every worker (3.6%), 4373 lose one of two
100 workers, shard of 5: 75,287,520 shards, P(same shard) = 1.33e-8
2048 workers, shard of 4: 730,862,190,080 shards, P(same shard) = 1.37e-12choose(8, 2)counts the distinct pairs you can make from 8 workers:8 × 7 / 2= 28. With shuffle sharding each customer is assigned one of those 28 pairs, instead of one of 4 fixed pairs.- Plain sharding puts a quarter of all customers on each fixed pair, so when a poison customer crashes its pair, 25% of customers lose service. With shuffle sharding, another customer loses both of its workers only if it was dealt exactly the same pair: probability
1/28= 3.6% — the "1/28th" scope of impact the AWS Builders' Library article on shuffle sharding quotes for this example (checked October 2026). - The other outcomes matter too. A customer shares exactly one worker with the poison pair with probability
12/28= 42.9% (2 poisoned workers × 6 others), and shares none with probabilityC(6,2)/28 = 15/28= 53.6%. A customer that lost one of its two workers is still served — if its clients retry on the other worker. Shuffle sharding only pays off with fault-tolerant clients. - The simulation deals random pairs to 10,000 customers using a seeded
mulberry32generator (so the output is identical every run) and poisons customer 0's pair. Of the other 9,999, 361 (3.6%) lose both workers and 4,373 (43.7%) lose one — matching the 3.6% and 42.9% from the formula, give or take sampling noise. - Bigger fleets make full overlap vanish. With 100 workers and 5 per customer there are 75,287,520 possible shards. With 2,048 workers and shards of 4 there are 730,862,190,080 — the "730 billion" AWS quotes for Route 53, which arranges its capacity into 2,048 virtual name servers and gives every customer domain four of them.
- What this demonstrates: redundancy alone (every customer on every server) gives a poison request a blast radius of 100%; fixed shards give
1/shards; shuffle sharding gives roughly1/C(n, k), at the same hardware cost. The price is complexity: the assignment must be stored, and clients must know their workers and fail over between them.
A cell-based design raises a handful of decisions, and an interviewer who hears "cells" will probe them:
- Partition key. Route by something every request carries and that keeps related data together: tenant id for B2B SaaS, user id for consumer apps, account id for payments. Requests that must span cells (an admin report across all customers) go through a separate, slower path, never through the cells' hot path.
- Cell size. Smaller cells mean a smaller blast radius and cheaper load tests (you can test the maximum size of a cell, then add cells to grow), but more cells to operate and more fixed cost per cell. A common approach is a fixed maximum cell size, and new customers go to new cells.
- The router is the shared fate. Every request still crosses it, so it should do one thing — look up the cell — from data it caches locally, so it keeps routing even if the control plane that assigns customers to cells is down. This is static stability: the data plane keeps working with the last known configuration when the control plane fails.
- Moving customers. A large tenant may outgrow its cell. Migration is copy data, dual-write or freeze writes briefly, flip the mapping, clean up — the same steps as resharding (see Resharding and hot partitions).
- Deploys go cell by cell. The first cell to get a release is the canary for the rest, so put internal or low-risk customers there.
The data plane does the real-time work (serving requests, routing, replicating); the control plane changes configuration (creating resources, assigning customers, scaling). Control planes are more complex and usually less available, so a resilient design keeps the data plane working when the control plane is down, and avoids control-plane actions in its recovery plan. AWS's disaster recovery whitepaper (checked October 2026) makes exactly this point: use data-plane operations for failover, because data planes have higher availability goals than control planes.
You have 16 workers and give each customer a shuffle shard of 4. One customer sends requests that crash every worker they reach. What fraction of other customers lose all four of their workers? What fraction would lose service with plain sharding into 4 fixed groups of 4?
Show answer
About 0.055% (1 in 1,820) with shuffle sharding, versus 25% with fixed groups.
- The number of possible shards is
C(16, 4) = 16 × 15 × 14 × 13 / 24= 1,820. - Another customer loses all four workers only if it was dealt the same four, so the probability is
1/1820≈ 0.00055 = 0.055%. - With 4 fixed groups of 4, a quarter of all customers sit on the poison customer's group and all of them go down: 25%.
- Customers that share some but not all workers with the poison shard stay up if their clients retry on their surviving workers — which is why shuffle sharding goes together with client-side failover.