#Vertical vs horizontal scaling
AddedWhen one server can't keep up, you have two ways out. Vertical scaling (scaling up) means a bigger machine: more vCPUs, more memory, faster disks. Horizontal scaling (scaling out) means more machines of the same size, with something in front sharing the work between them. Every design interview reaches this fork within the first ten minutes, usually as "what happens when traffic grows 10×?".
The mental model is one bigger lorry vs a fleet of vans. One big lorry is simple: one driver, one route, everything in one place, and no coordination. But there is a biggest lorry anyone builds, it costs disproportionately more at the top end, and if it breaks down the whole delivery stops. A fleet of vans can grow one van at a time, keeps delivering when one breaks down, and can be parked when it's quiet. But now you need a dispatcher, each van can only carry what fits in it, and anything the vans must agree on (who has which parcel) needs radio chatter.
So the trade-off isn't "horizontal good, vertical bad". Vertical scaling is the right first move far more often than interview answers admit: it needs no code change, keeps transactions on one machine, and today's largest machines are enormous. Horizontal scaling is what you need for fault tolerance (one machine is always a single point of failure) and for growth past the biggest box. The skill is knowing which tier can scale out cheaply (stateless web and API servers) and which can't (the database, anything with shared state).
| Vertical (scale up) | Horizontal (scale out) | |
|---|---|---|
| What changes | Machine size | Machine count |
| Code changes | None | Services must be stateless, or the state must be partitioned |
| Ceiling | Largest machine on sale | Practically none for stateless tiers; coordination costs for stateful ones |
| Failure | One machine = one point of failure; a standby doubles the cost | Losing one of N removes 1/N of capacity |
| Growth steps | Big jumps (double the machine), often with a restart | Small steps, one instance at a time, no downtime |
| Cost curve | Rises steeply at the top end | Roughly linear, plus the load balancer and operational overhead |
| Typical use | Databases, caches, early-stage apps | Web and API tiers, workers, CDNs, anything stateless |
How big is "big"? The largest instance AWS has announced is the U7inh high-memory instance: 1,920 vCPUs and 32 TiB of memory on one machine (AWS announcement, December 2024; check the current instance list before quoting it as the maximum). Few application tiers will ever outgrow that on CPU alone, but buying one, and a second one as a standby, is a very different bill from buying vans.
Before choosing, put numbers on it. Here's the API tier for a product with 20 million daily users, sized both ways:
// How many vCPUs does the API tier need at peak, and what does that look like as one box vs many?
const dau = 20_000_000; // daily active users
const requestsPerUser = 50; // API calls per user per day
const peakFactor = 3; // busiest minute vs the daily average
const cpuMsPerRequest = 4; // measured CPU time per request (an assumption here)
const targetUtilisation = 0.6; // leave headroom for spikes and GC
const avgQps = (dau * requestsPerUser) / 86_400;
const peakQps = avgQps * peakFactor;
const busyVcpus = (peakQps * cpuMsPerRequest) / 1000; // CPU-seconds needed per second
const vcpus = busyVcpus / targetUtilisation;
console.log(`average ${Math.round(avgQps).toLocaleString("en-US")} req/s, peak ${Math.round(peakQps).toLocaleString("en-US")} req/s`);
console.log(`busy vCPUs at peak ${busyVcpus.toFixed(0)}, provisioned at 60%: ${Math.ceil(vcpus)} vCPUs`);
// Horizontal: 8-vCPU instances spread over 3 availability zones, surviving the loss of one zone.
const perInstance = 8, zones = 3;
const needed = Math.ceil(vcpus / perInstance);
const perZone = Math.ceil(needed / (zones - 1)); // any 2 zones must carry the whole peak
console.log(`horizontal: ${needed} × ${perInstance}-vCPU instances, ${perZone} per zone × ${zones} zones = ${perZone * zones} instances`);
// Vertical: one big machine, plus a standby of the same size for failover.
console.log(`vertical: one ${Math.ceil(vcpus)}-vCPU machine + an identical standby = ${2 * Math.ceil(vcpus)} vCPUs bought`);
// Ten times the users later:
const later = Math.ceil((vcpus * 10) / perInstance);
console.log(`at 10× users: ${later} instances (${Math.ceil(later / (zones - 1)) * zones} with zone headroom), or a ${Math.ceil(vcpus * 10)}-vCPU box`);average 11,574 req/s, peak 34,722 req/s
busy vCPUs at peak 139, provisioned at 60%: 232 vCPUs
horizontal: 29 × 8-vCPU instances, 15 per zone × 3 zones = 45 instances
vertical: one 232-vCPU machine + an identical standby = 464 vCPUs bought
at 10× users: 290 instances (435 with zone headroom), or a 2315-vCPU box- The assumptions are at the top so an interviewer can challenge each one: 20 M daily users × 50 calls each, spread over 86,400 seconds, gives an average of 11,574 req/s; a peak factor of 3 for the busiest minute gives 34,722 req/s. (Chapter 1's estimation topic explains where peak factors come from.)
- Each request costs 4 ms of CPU, so at peak the tier is doing
34,722 × 0.004 = 139vCPU-seconds of work every second: 139 vCPUs fully busy. Running at 100% leaves no room for spikes, garbage-collection pauses or a slow dependency, so the model provisions for 60% utilisation:139 / 0.6→ 232 vCPUs. - Horizontally that's
ceil(232 / 8)= 29 eight-vCPU instances. But capacity must survive losing a whole availability zone (a data centre). With three zones, any two must carry the full peak, so each zone needsceil(29 / 2)= 15 instances: 45 in total, 360 vCPUs. That extra 55% (50% in principle for three zones, plus rounding up to whole instances) is the price of zone-level fault tolerance. - Vertically, a single 232-vCPU machine does the job, but on its own it's a single point of failure, so you buy an identical standby: 464 vCPUs paid for, half of them idle. Horizontal redundancy is cheaper because the spare capacity is spread out (N+1 of small units) instead of a whole second machine (1+1).
- At 10× the users the horizontal answer is just "more of the same" (290 instances, 435 with zone headroom). The vertical answer is a 2,315-vCPU machine, bigger than the largest instance AWS has announced (1,920 vCPUs). That's the ceiling you hit, and you hit it earlier for real workloads because memory bandwidth, network and locks run out before vCPUs do.
- The lesson for the interview: compute the size, then say which tier you scale which way. Stateless API servers scale out easily; the database behind them is where vertical scaling (and later replicas or sharding) comes in.
Adding the tenth server rarely adds as much as the second did. Two things eat the gains: contention (servers queue for something shared, like a database row lock or a single leader) and coherence (servers spend time keeping each other in sync: cache invalidations, cluster membership, distributed locks). Neil Gunther's Universal Scalability Law puts both into one formula, and it's a useful way to argue why you're removing shared state:
// Universal Scalability Law: throughput of N nodes = X1 · N / (1 + α(N−1) + βN(N−1))
// α = contention (time spent waiting on something shared), β = coherence (nodes keeping each other in sync).
const X1 = 2_000; // req/s one node handles alone
const designs = {
"shared-nothing (α .01, β 0)": [0.01, 0],
"shared lock (α .08, β 0)": [0.08, 0],
"chatty cluster (α .03, β .002)": [0.03, 0.002],
};
const usl = (n, a, b) => (X1 * n) / (1 + a * (n - 1) + b * n * (n - 1));
console.log("nodes".padEnd(31) + [1, 4, 16, 32, 64].map((n) => String(n).padStart(8)).join(""));
for (const [name, [a, b]] of Object.entries(designs)) {
const row = [1, 4, 16, 32, 64].map((n) => (usl(n, a, b) / 1000).toFixed(1).padStart(7) + "k").join("");
const peak = b > 0 ? ` peak at N=${Math.floor(Math.sqrt((1 - a) / b))}` : "";
console.log(name.padEnd(31) + row + peak);
}nodes 1 4 16 32 64
shared-nothing (α .01, β 0) 2.0k 7.8k 27.8k 48.9k 78.5k
shared lock (α .08, β 0) 2.0k 6.5k 14.5k 18.4k 21.2k
chatty cluster (α .03, β .002) 2.0k 7.2k 16.6k 16.4k 11.7k peak at N=22usl(n, a, b)is the USL: ideal throughputX1 · n, divided by a penalty that grows withα(contention, linear in n) andβ(coherence, quadratic in n because every node talks to every other node).- The shared-nothing design (each node serves its requests without asking the others, α = 0.01, β = 0) keeps growing: 64 nodes give 78.5k req/s, about 61% of the ideal 128k. The small α is the load balancer and the shared database connection setup.
- The shared lock design (α = 0.08, say every request briefly takes a lock on a hot row) flattens out: going from 32 to 64 nodes adds only 2.8k req/s. Throughput approaches
X1 / α= 25k req/s and never passes it, however many servers you add. - The chatty cluster (α = 0.03, β = 0.002, say every node must invalidate every other node's local cache on writes) is worse: it peaks at
sqrt((1 − α) / β)≈ 22 nodes and then goes down, to 11.7k at 64 nodes. Adding servers makes it slower, because coordination grows with n². - The values of α and β here are illustrative, not measured. In practice you measure throughput at a few cluster sizes and fit them. The design lesson holds regardless: horizontal scaling works when the tier is stateless and shared-nothing, which is exactly the next topic.
- Web and API servers: horizontal, from the start. Keep them stateless and put them behind a load balancer; run at least two (in different zones) even when one would do, for availability.
- Background workers: horizontal. They pull from a queue, so adding consumers adds throughput without a load balancer at all.
- Relational databases: vertical first (a bigger instance is the cheapest scaling you will ever do), then read replicas for read traffic, then partitioning when writes or data size outgrow one primary. See Replication and Partitioning and sharding strategies.
- Caches: vertical for a single node, then horizontal by partitioning the keyspace with consistent hashing.
- Anything with a single writer (a leader, a sequencer, a lock service): scale it up, and design so that it does as little as possible.
Your service needs 64 vCPUs at peak. Option A: two 64-vCPU machines (active + standby). Option B: 16-vCPU instances spread evenly over three zones, surviving the loss of any one zone. Which buys fewer vCPUs, and by how much?
Show answer
Option B buys 96 vCPUs, option A buys 128: B is 32 vCPUs (25%) cheaper.
- Option A: the standby must be as big as the active machine, so
2 × 64= 128 vCPUs, half of them idle all the time. - Option B: 64 vCPUs of 16-vCPU instances is 4 instances. Any two of the three zones must hold all 4, so each zone needs
ceil(4 / 2)= 2 instances. - Three zones × 2 instances × 16 vCPUs = 96 vCPUs. The spare capacity is one zone's worth (a third), not a whole second machine (a half).
- The more units you split capacity into, the smaller the spare fraction: with N zones you over-provision by
N / (N − 1). That's the arithmetic behind "scale out for availability".