#Design a news feed
AddedA news feed (a "home timeline") shows you the recent posts of everyone you follow, newest or best first, the moment you open the app. Storing posts is the easy part: a post is written once and kept forever. The hard part is assembling a different list for every reader, from hundreds of sources each, billions of times a day, in a couple of hundred milliseconds. Every design decision in this case study is about when that assembly happens: when the post is written, or when the feed is read.
A good mental model is a post office. With fan-out on write (push), when someone posts, the post office puts a slip with the post's ID into the mailbox of every follower; reading your feed is just emptying your own mailbox. With fan-out on read (pull), nothing is delivered; every time you want news, you walk to the house of each person you follow and read their noticeboard, then sort what you found. Push makes reads cheap and writes expensive; pull is the opposite. Celebrities break the post office: one letter from an account with 100 million followers means 100 million mailbox slips. So real feeds are a hybrid — deliver to mailboxes for ordinary accounts, and let readers go and look at the handful of celebrity noticeboards themselves.
The whole case study turns on one measurement: how much work each approach does on a realistic follow graph, where most accounts have a few dozen followers and a few have nearly everyone. In the interview you'd reach this in the deep dive, but it's first here because every later section leans on its numbers. The model builds a seeded follow graph of 20,000 users (a scaled-down social network: follower counts drawn from a heavy-tailed Pareto distribution, so a few accounts are vastly more popular), then counts the work for pure push, pure pull, and the hybrid at three celebrity thresholds.
// Fan-out on write vs on read vs hybrid, on a seeded follow graph with celebrities.
function mulberry32(a) {
return () => {
a = (a + 0x6d2b79f5) | 0;
let t = Math.imul(a ^ (a >>> 15), 1 | a);
t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
};
}
const rand = mulberry32(42);
const USERS = 20_000;
const pareto = (alpha) => 1 / Math.pow(1 - rand(), 1 / alpha); // heavy tail, minimum 1
// Popularity decides who gets followed: a few accounts are vastly more popular than the rest.
const cumulative = [];
let total = 0;
for (let i = 0; i < USERS; i++) cumulative.push((total += pareto(1.5)));
const pickByPopularity = () => {
const x = rand() * total;
let lo = 0, hi = USERS - 1;
while (lo < hi) { const mid = (lo + hi) >> 1; if (cumulative[mid] < x) lo = mid + 1; else hi = mid; }
return lo;
};
const following = Array.from({ length: USERS }, (_, u) => {
const want = Math.min(1000, Math.floor(20 * pareto(2))); // most follow 20-60 accounts, a few 1,000
const set = new Set();
while (set.size < want) { const v = pickByPopularity(); if (v !== u) set.add(v); }
return [...set];
});
const followers = new Array(USERS).fill(0);
for (const list of following) for (const v of list) followers[v]++;
const edges = followers.reduce((a, b) => a + b, 0);
const ranked = [...followers].sort((a, b) => b - a);
console.log(`${USERS} users, ${edges} follows; median followers ${ranked[USERS / 2]}, top 3: ${ranked.slice(0, 3).join(", ")}`);
// Every user posts once and loads the feed 10 times a day. Authors with >= T followers are "celebrities":
// their posts are not pushed; readers pull them at read time. T = Infinity is pure push, T = 0 pure pull.
function design(T) {
const celeb = followers.map((f) => f >= T);
let pushWrites = 0, worstPost = 0, pulledPerReader = 0, worstRead = 0;
followers.forEach((f, v) => { if (!celeb[v]) { pushWrites += f; worstPost = Math.max(worstPost, f); } });
for (const list of following) {
const pulled = list.filter((v) => celeb[v]).length + (T > 0 ? 1 : 0); // + own inbox
pulledPerReader += pulled;
worstRead = Math.max(worstRead, pulled);
}
const celebs = celeb.filter(Boolean).length;
return { celebs, pushWrites, worstPost, readsPerLoad: pulledPerReader / USERS, worstRead };
}
console.log("design celebs inbox writes/day worst post lists read/load worst load");
for (const [name, T] of [["push (on write)", Infinity], ["hybrid T=5,000", 5000], ["hybrid T=1,000", 1000], ["hybrid T=200", 200], ["pull (on read)", 0]]) {
const d = design(T);
console.log(`${name.padEnd(18)}${String(d.celebs).padStart(7)}${String(d.pushWrites).padStart(18)}${String(d.worstPost).padStart(12)}${d.readsPerLoad.toFixed(2).padStart(17)}${String(d.worstRead).padStart(12)}`);
}20000 users, 791735 follows; median followers 21, top 3: 17310, 12569, 8907
design celebs inbox writes/day worst post lists read/load worst load
push (on write) 0 791735 17310 1.00 1
hybrid T=5,000 6 735280 3326 3.82 7
hybrid T=1,000 40 676633 979 6.76 39
hybrid T=200 348 565647 199 12.30 186
pull (on read) 20000 0 0 39.59 1000- The graph is built first. Each user gets a popularity weight from
pareto(1.5), and each user follows 20 to 1,000 accounts picked in proportion to popularity (a binary search over the running total of weights). The result is the shape real networks have: the median account has 21 followers, but the top three have 17,310, 12,569 and 8,907 out of 20,000 users. The seeded PRNG (mulberry32(42)) makes it the same graph every run. design(T)treats every author with at leastTfollowers as a celebrity. A normal author's post is pushed: one inbox insert per follower, sopushWritesis the sum of normal authors' follower counts andworstPostis the biggest single fan-out. A reader pulls one list per celebrity they follow, plus their own inbox.- Pure push (
T = Infinity): 791,735 inbox inserts for 20,000 posts — an average write amplification of about 40, the mean follower count. Reads are perfect: 1 list per feed load. But the worst post fans out to 17,310 inboxes, and that one post is where push hurts: at real scale it's 100 million writes for one tweet. - Pure pull (
T = 0): no inbox writes at all, but every feed load merges 39.59 lists on average and 1,000 for the heaviest followers. Feeds are read about ten times as often as they're written, so this multiplies the expensive operation. - Hybrid,
T = 1,000: just 40 accounts (0.2% of users) become celebrities. Total inbox writes drop by 15% (791,735 → 676,633), but the worst single post drops 18× (17,310 → 979), because the celebrities are exactly the accounts with the giant fan-outs. Readers now merge 6.76 lists on average (their inbox plus the celebrities they follow) and 39 at worst. - Lowering the threshold to 200 keeps cutting writes, but read cost climbs (12.30 lists per load) for diminishing write savings. That curve is the trade-off you choose a point on: make the worst write small enough to meet the freshness target, and stop before reads get expensive.
- What it demonstrates: write amplification and read cost aren't fixed properties of push and pull; they depend on the follower distribution. The heavy tail is why a hybrid wins — a tiny set of accounts causes the worst write cost, and readers follow only a handful of them.
Requirements
Ask before designing. A reasonable set for "design the Twitter/Instagram home feed":
Functional
- A user can publish a post (text, plus optional images or video uploaded separately).
- A user can follow and unfollow accounts.
- The home feed shows posts from followed accounts, newest first in version 1, with ranking as an extension; it pages with infinite scroll.
- An author sees their own new post in their feed immediately.
- Out of scope unless asked: ads, search, notifications (see Design a notification system), direct messages, comments and likes (a separate case study below: Design a comments and likes system).
Non-functional
- Scale: 300 million daily active users (the assumption the estimates use).
- Feed load latency: p99 under 300 ms at the API, which is a target we set, not a standard.
- Freshness: a new post reaches followers' feeds within about 5 seconds normally. Twitter described the same goal in 2013 ("under 5 seconds, but it doesn't always work", High Scalability's write-up of Raffi Krikorian's Timelines at Scale talk).
- Availability over consistency: a feed that's a few seconds stale is fine; a feed that fails to load is not. The exception is read-your-own-writes for the author.
- Durability: posts are never lost. The feed itself is derived data — it can be rebuilt from posts and follows — so losing a cached feed is an inconvenience, not data loss.
Estimates
The assumptions: 300 M DAU, each posting 0.5 times and loading the feed 10 times a day, a mean of 200 followers per author (the mean, which celebrities pull well above the median), a 3× peak, 1 KB per post without media, and home timelines capped at 800 post IDs of 16 bytes each.
// News feed: assumptions in, QPS, storage and cache size out.
const DAU = 300e6;
const postsPerUserPerDay = 0.5;
const feedLoadsPerUserPerDay = 10;
const meanFollowers = 200; // mean, not median: celebrities pull it up
const peakFactor = 3;
const postBytes = 1_000; // text + metadata; media lives in object storage
const inboxEntries = 800; // post IDs kept per home timeline
const entryBytes = 16; // 8-byte post ID + 8-byte score/timestamp
const S = 86_400;
const posts = DAU * postsPerUserPerDay;
const loads = DAU * feedLoadsPerUserPerDay;
const fmt = (n) => (n >= 1e9 ? (n / 1e9).toFixed(1) + " B" : n >= 1e6 ? (n / 1e6).toFixed(1) + " M" : n >= 1e3 ? (n / 1e3).toFixed(1) + " k" : n.toFixed(0));
console.log(`posts: ${fmt(posts)}/day = ${fmt(posts / S)}/s avg, ${fmt((posts / S) * peakFactor)}/s peak`);
console.log(`feed loads: ${fmt(loads)}/day = ${fmt(loads / S)}/s avg, ${fmt((loads / S) * peakFactor)}/s peak`);
console.log(`push fan-out: ${fmt(posts * meanFollowers)} inbox inserts/day = ${fmt((posts * meanFollowers) / S)}/s avg`);
console.log(`pull read cost: ${fmt(loads * meanFollowers)} list reads/day if every load merges every followee`);
console.log(`post storage: ${(posts * postBytes / 1e9).toFixed(0)} GB/day, ${(posts * postBytes * 365 / 1e12).toFixed(0)} TB/year (before replication)`);
const cacheBytes = DAU * inboxEntries * entryBytes;
console.log(`timeline cache: ${(inboxEntries * entryBytes / 1e3).toFixed(1)} KB per user, ${(cacheBytes / 1e12).toFixed(2)} TB for all DAU`);
console.log(`cache nodes: ${Math.ceil((cacheBytes * 3) / 64e9)} at 64 GB usable each, with 3 replicas`);posts: 150.0 M/day = 1.7 k/s avg, 5.2 k/s peak
feed loads: 3.0 B/day = 34.7 k/s avg, 104.2 k/s peak
push fan-out: 30.0 B inbox inserts/day = 347.2 k/s avg
pull read cost: 600.0 B list reads/day if every load merges every followee
post storage: 150 GB/day, 55 TB/year (before replication)
timeline cache: 12.8 KB per user, 3.84 TB for all DAU
cache nodes: 180 at 64 GB usable each, with 3 replicas- Posts: 300 M × 0.5 = 150 M a day, which is only 1.7 k/s on average and 5.2 k/s at peak. Writing posts is not the problem.
- Feed loads: 300 M × 10 = 3 B a day, 34.7 k/s average and 104.2 k/s at peak. Reads outnumber posts 20 to 1, which is the first argument for doing work at write time.
- Push fan-out: every post becomes 200 inbox inserts, so 30 B inserts a day, 347.2 k/s on average. That's the bill for push. For scale: Twitter reported around 300 K QPS for timeline reads and roughly 5 k tweets/s on average in 2013, with fan-out to a million followers taking about 3.5 s at p50 (High Scalability, 2013) — the same order of magnitude.
- Pull read cost: if every load merged every followee's list, that's 600 B list reads a day. That's 20× the push bill, which is why pure pull doesn't survive these numbers.
- Post storage is modest: 150 GB a day, 55 TB a year before replication. Media dominates real storage and lives in object storage behind a CDN (see Object storage for files and media).
- The timeline cache: 800 × 16 bytes = 12.8 KB per user, 3.84 TB for all daily users, so about 180 cache nodes at 64 GB usable each with 3 replicas. Storing IDs, not posts, is what keeps this affordable — 800 full posts per user would be 60× bigger.
API
POST /v1/posts { text, mediaIds[] } -> 201 { postId, createdAt }
Idempotency-Key: <uuid>
GET /v1/feed?cursor=<opaque>&limit=20 -> 200 { items[], nextCursor }
items[i] = { postId, author{id,name,avatarUrl}, text, media[], likeCount, likedByMe, createdAt }
PUT /v1/users/{userId}/followers/me -> 204 (follow; repeat-safe)
DELETE /v1/users/{userId}/followers/me -> 204 (unfollow)The cursor is opaque to the client: it encodes the position in the feed (the score and post ID of the last item returned), so new posts arriving at the top don't shift the page boundaries the way an offset would (see Pagination at the API level). Follow is a PUT on a sub-resource, so retrying it is harmless; posting takes an idempotency key so a retried request doesn't publish twice (see Idempotency keys for APIs). Each feed item comes back hydrated — author, counts and "did I like it" are filled in by the feed service, so the client makes one request per page.
Data model
| Data | Key → value | Store | Why |
|---|---|---|---|
| Posts | post_id → author, text, media keys, created_at | Sharded SQL or wide-column, sharded by post_id | Written once, read by ID; Snowflake IDs sort by time |
| Author's posts | author_id → recent post_ids, newest first | Wide-column (partition author_id) | Pull for celebrities, rebuilding feeds, profile pages |
| Follows | (follower_id, followee_id) and the reverse (followee_id, follower_id) | Sharded SQL or a graph service | Fan-out needs "followers of X"; pull needs "who do I follow" — two directions, two indexes |
| Home timeline | feed:{user_id} → up to 800 (post_id, score) | Redis sorted set or list | Derived, rebuildable, in memory |
| Post cache | post_id → hydrated post | Redis / Memcached | Hydration reads; hot posts are hot keys |
post_id is a Snowflake-style ID (timestamp in the high bits, see Unique ID generation), so sorting by ID is sorting by time and the feed cache can use it directly as the score in version 1. The follow graph is stored twice, once per direction, because the two queries partition differently: "followers of a celebrity" is one huge partition read by the fan-out workers; "accounts I follow" is a small read on every pull. This is modelling data for access patterns: duplicate the data so each access is one partition.
High-level design
Walk it as two paths, out loud:
- Write path. The client posts through the gateway. The post service stores the post (the durable part, done before replying), adds it to the author's own post list, and appends a
PostCreatedevent to a log such as Kafka. It replies to the client straight away; fan-out is asynchronous, so a celebrity's post doesn't make their request take a minute. This is pub/sub and fan-out at work. - Fan-out. Workers consume the log. For each post they check the author's follower count: below the threshold
T, they page through the follower list from the social graph and insert(post_id, score)into each follower's feed cache, in pipelined batches (Twitter batched 4,000 destinations at a time in 2013). At or aboveT, they do nothing — the post already sits in the author's own list, which acts as the "celebrity posts" cache. - Read path. The feed service reads the user's feed cache (one sorted-set read), and the recent posts of each celebrity they follow (6.76 lists on average at
T = 1,000in the model), merges them by score, optionally ranks them, then hydrates the page — post bodies, authors, like counts and "liked by me" — from caches, in batched multi-gets.
The threshold isn't a magic constant; derive it from the freshness requirement. If the fan-out tier can sustain a few million inbox inserts a second in total, and one post may use at most a slice of that, then a post must finish within the 5-second target: an author with 100 M followers needs 100 M inserts, which at even 1 M inserts/s is 100 s, far over budget. An author with 10 k followers is 10 ms of work. So T sits somewhere where T / (per-post insert rate) stays well inside the freshness target — in practice somewhere in the tens to hundreds of thousands of followers at real scale, and the model shows why the exact value matters less than you'd think: the celebrity set is tiny, so readers pay for only a few extra lists either way.
Details that come up in a follow-up question:
- Inactive followers are most of a celebrity's audience at any moment. Twitter (2013) kept timelines in Redis only for users active in the last 30 days; anyone else gets their feed rebuilt on return. Skipping inactive users cuts push writes for everyone, not just celebrities.
- Hysteresis: an account that hovers around
Tshouldn't flip between push and pull on every new follower, or readers will see duplicates (pushed once, then pulled). Mark accounts as celebrities aboveTand only revert well below it, and de-duplicate bypost_idin the merge anyway. - The mirror problem: users who follow thousands of accounts are cheap under push (one inbox) but expensive under pull. The hybrid keeps them on push, which is another reason the default is push.
- A celebrity's post is a hot key: millions of readers merge the same celebrity list and hydrate the same post. Replicate it into local caches and coalesce requests (see Hot keys and cache stampedes).
Deep dive: ranking
A chronological feed sorts by post_id. A ranked feed adds a scoring step between merge and hydrate, in three stages: candidate generation (the merged inbox and celebrity posts, plus out-of-network recommendations), scoring (a model predicts how likely you are to engage with each candidate), and filtering and mixing (remove blocked authors and already-seen posts, don't show five posts from one author in a row). At Twitter in March 2023, each "For You" request pulled about 1,500 candidates, roughly half in-network and half out-of-network, and scored them with a neural network of about 48 million parameters (Twitter's engineering blog post on its recommendation algorithm). In an interview you don't build the model; you show where it sits and what it costs. A hand-written score makes the trade-off visible:
// A hand-written ranking score: engagement x affinity x time decay. Real feeds learn the weights.
const now = 100; // hours on a fake clock
const candidates = [
{ id: "p1", author: "close friend", hoursOld: 1, likes: 3, comments: 1, affinity: 0.9 },
{ id: "p2", author: "celebrity", hoursOld: 2, likes: 52_000, comments: 4_100, affinity: 0.05 },
{ id: "p3", author: "coworker", hoursOld: 0.2, likes: 0, comments: 0, affinity: 0.3 },
{ id: "p4", author: "close friend", hoursOld: 20, likes: 40, comments: 12, affinity: 0.9 },
{ id: "p5", author: "news page", hoursOld: 5, likes: 900, comments: 300, affinity: 0.2 },
{ id: "p6", author: "old classmate", hoursOld: 3, likes: 15, comments: 30, affinity: 0.1 },
];
const engagement = (p) => Math.log10(1 + p.likes + 4 * p.comments); // log: 52k likes isn't 17,000x better than 3
const decay = (p) => 1 / Math.pow(p.hoursOld + 2, 1.5); // gravity: older posts sink
const score = (p) => (0.2 + engagement(p)) * (0.1 + p.affinity) * decay(p);
const chronological = [...candidates].sort((a, b) => a.hoursOld - b.hoursOld);
const ranked = [...candidates].sort((a, b) => score(b) - score(a));
console.log("chronological:", chronological.map((p) => p.id).join(" "));
console.log("ranked: ", ranked.map((p) => p.id).join(" "));
for (const p of ranked) {
console.log(`${p.id} ${p.author.padEnd(14)} posted at t=${(now - p.hoursOld).toFixed(1).padStart(5)}h engagement ${engagement(p).toFixed(2)} decay ${decay(p).toFixed(3)} score ${score(p).toFixed(4)}`);
}chronological: p3 p1 p2 p6 p5 p4
ranked: p1 p2 p5 p6 p3 p4
p1 close friend posted at t= 99.0h engagement 0.90 decay 0.192 score 0.2123
p2 celebrity posted at t= 98.0h engagement 4.84 decay 0.125 score 0.0944
p5 news page posted at t= 95.0h engagement 3.32 decay 0.054 score 0.0571
p6 old classmate posted at t= 97.0h engagement 2.13 decay 0.089 score 0.0417
p3 coworker posted at t= 99.8h engagement 0.00 decay 0.306 score 0.0245
p4 close friend posted at t= 80.0h engagement 1.95 decay 0.010 score 0.0208- Six candidates arrive from the merge. Chronological order is just
hoursOldascending:p3 p1 p2 p6 p5 p4— the coworker's 12-minute-old post with no engagement goes first. engagementis the log oflikes + 4 × comments(comments count more, they take more effort). The log stops the celebrity's 52,000 likes from swamping everything: it scores 4.84 against the close friend's 0.90, about 5× rather than 17,000×.decayis a gravity term,1 / (age + 2)^1.5(the shape Hacker News popularised): a 1-hour-old post gets 0.192, a 20-hour-old one 0.010. The+ 2stops a brand-new post from getting an infinite boost.affinityis how much this reader interacts with this author. The close friend (0.9) wins the top slot with modest engagement; the celebrity (0.05) still comes second on sheer engagement; the coworker's fresh but empty post drops to fifth.- The close friend's 20-hour-old post (p4) has more engagement than p1 but comes last: decay of 0.010 beats everything. Every weight here is a product decision, and a real system learns them from engagement data instead of hand-tuning.
- What it demonstrates: ranking reorders the merged candidates; it doesn't change where they come from. The architecture keeps the merge cheap so that the ranking stage has time to spend.
The feed cache is derived data, so every awkward case is solved by rebuilding or filtering at read time rather than by editing millions of inboxes:
- Cache miss (a returning inactive user, or a lost cache node): fall back to pull once — read the recent posts of everyone they follow, merge, write the result into their feed cache — then let push take over again. That's slow (one load), so rate-limit rebuilds so a cache node failure doesn't trigger millions at once.
- Deleted posts: don't chase the ID through 10 million inboxes. Mark the post deleted in the posts store; hydration drops it, and the inbox entry ages out of the 800-entry cap.
- Unfollow: filter the unfollowed author at read time (the feed service already knows the follow list), and clean the inbox lazily.
- New follow: backfill the followee's latest 20 or so posts into the follower's inbox, so the follow has a visible effect immediately.
- Privacy changes (a post becomes followers-only, an account blocks you): check at hydration, every time. The inbox holds IDs, never permissions.
Trade-offs and what to say
| Push (fan-out on write) | Pull (fan-out on read) | Hybrid | |
|---|---|---|---|
| Cost per post | one insert per follower (791,735 total in the model) | one write | push for normal authors (676,633 at T = 1,000) |
| Cost per feed load | one list | one list per followee (39.59 average) | inbox + celebrities followed (6.76) |
| Worst case | celebrity post: 17,310 inserts, 100 M at real scale | heavy follower: 1,000 lists | bounded on both sides (979 / 39) |
| Freshness | delay grows with follower count | always fresh | fresh; celebrity posts are freshest |
| Memory | an inbox per active user (3.84 TB) | none beyond post lists | inbox per active user |
| Inactive users | wasted writes unless skipped | free | skip them in push |
Show answer
Pure push: 50 seconds at best, with every other author's posts queued behind it. Under the hybrid: one write, and followers see it on their next load.
- Pure push needs one insert per follower: 50 M inserts.
- At 1 M inserts/s for the whole tier, that's 50 M / 1 M = 50 s — ten times the 5-second freshness target — and only if nothing else is posting. Other authors' fan-outs queue behind it, so everyone's freshness suffers.
- Under the hybrid, the author is above the threshold, so the post is written once (to the posts store and the author's list) and nothing is fanned out.
- Each follower's next feed load merges this author's recent posts with their inbox, so the post appears as soon as it's stored — fresher than push ever managed.
- The cost moved to reads: each of those followers' loads reads one more list. With readers following only a few celebrities each, that's a few extra cache reads per load instead of a 50-second write.
For the client side of this design — infinite scroll, virtualisation, the "new posts" banner and optimistic likes — see Design a news feed UI.