Chapter 06

Messaging and Async Processing

Queues, logs and pub/sub at the design level: what each buys you, how ordering, retries and overload behave, and how events, streams and scheduled jobs fit into a system.

9 topics

Parts marked Advanced are extra depth. Skip them on a first read or a quick revision.

#Why queues

Added

A message queue sits between the part of a system that notices work and the part that does it. The producer writes a message ("order 812 was placed, send the confirmation") and carries on; a consumer picks the message up when it's ready, does the work, and acknowledges it. Until that acknowledgement, the broker keeps the message. That one small indirection changes four things at once:

Decoupling. The producer doesn't need the consumer to be up, fast, or even known. The checkout API keeps taking orders while the email service is being deployed. Teams can add a new consumer without touching the producer.
Buffering bursts. Work that arrives faster than it can be done waits in the queue instead of being refused. You size consumers for the average load plus headroom, not for the worst minute of the year.
Retries for free. A consumer that crashes halfway never acknowledges, so the broker hands the message to someone else. Failure turns into delay instead of loss.
An asynchronous user experience. The API answers 202 Accepted with a job id in milliseconds, and the slow part (transcoding a video, generating a PDF, calling a flaky partner) happens afterwards, with progress pushed or polled.

A good mental model is the ticket rail in a restaurant kitchen. The waiter clips each order to the rail and goes straight back to the tables; the cooks pull tickets at their own pace. A rush of forty orders doesn't make the waiter stand at the stove: the tickets just pile up on the rail, and if a cook burns a dish the ticket is still there to redo. But the rail doesn't make the kitchen any faster. If orders keep coming faster than the cooks can cook, the rail grows until customers give up and leave. A queue absorbs bursts; it doesn't fix sustained overload (that's Backpressure and load shedding).

The queue turns "the provider is down" from a user-facing error into a delay the user never sees.

The model below puts numbers on the buffering. A shop normally gets 300 orders a second; a flash sale sends 1,500 a second for ten seconds. Each worker confirms 100 orders a second. How deep does the queue get, how long does the oldest order wait, and what does adding workers do?

burst.mjs

// A flash sale: 300 orders/s normally, 1,500/s for 10 seconds. Each worker confirms 100 orders/s.
const arrivals = (t) => (t >= 10 && t < 20 ? 1500 : 300);

function simulate(workers, seconds = 160) {
  const capacity = workers * 100;
  const queue = []; // FIFO of [arrivalSecond, count] cohorts
  let depth = 0, peak = 0, maxWait = 0, drainedAt = null;
  const samples = [];
  for (let t = 0; t < seconds; t++) {
    queue.push([t, arrivals(t)]);
    depth += arrivals(t);
    let budget = capacity; // messages the workers can take this second
    while (budget > 0 && queue.length) {
      const head = queue[0];
      const take = Math.min(budget, head[1]);
      head[1] -= take; budget -= take; depth -= take;
      maxWait = Math.max(maxWait, t - head[0]); // whole seconds spent waiting
      if (head[1] === 0) queue.shift();
    }
    peak = Math.max(peak, depth);
    if (t >= 20 && depth === 0 && drainedAt === null) drainedAt = t;
    if ([5, 10, 15, 20, 30, 60, 100, 130].includes(t)) samples.push(`t=${t}s:${depth}`);
  }
  console.log(`${workers} workers (${capacity}/s): peak depth ${peak}, longest wait ${maxWait}s, empty again at t=${drainedAt}s`);
  console.log(`  depth ${samples.join("  ")}`);
}

simulate(4);
simulate(8);
simulate(16);

// The same burst with no queue: a synchronous service with 4 workers' capacity turns away the excess.
const rejected = 10 * (1500 - 400);
console.log(`no queue, 400/s capacity: ${rejected} orders rejected or timed out during the burst`);

Result

4 workers (400/s): peak depth 11000, longest wait 28s, empty again at t=129s
  depth t=5s:0  t=10s:1100  t=15s:6600  t=20s:10900  t=30s:9900  t=60s:6900  t=100s:2900  t=130s:0
8 workers (800/s): peak depth 7000, longest wait 9s, empty again at t=33s
  depth t=5s:0  t=10s:700  t=15s:4200  t=20s:6500  t=30s:1500  t=60s:0  t=100s:0  t=130s:0
16 workers (1600/s): peak depth 0, longest wait 0s, empty again at t=20s
  depth t=5s:0  t=10s:0  t=15s:0  t=20s:0  t=30s:0  t=60s:0  t=100s:0  t=130s:0
no queue, 400/s capacity: 11000 orders rejected or timed out during the burst

What’s happening

Every simulated second, that second's arrivals join the back of the queue as one cohort ([arrivalSecond, count]), and the workers take up to capacity messages from the front. Because the oldest cohort is served first, t - head[0] is how long the message being taken has waited.
With 4 workers (400/s) the normal 300/s never queues (t=5s:0). During the burst the queue grows by 1,500 - 400 = 1,100 every second: 1,100 at t=10, 6,600 at t=15, and a peak of 11,000 at the end of t=19.
After the burst only the spare capacity drains the backlog: 400 - 300 = 100 per second. 11,000 / 100 = 110 seconds, so the queue is empty again at t=129, and an order placed at the end of the burst waited 28 seconds (11,000 messages ahead of it at 400/s is 27.5 s, rounded up by the one-second ticks).
With 8 workers (800/s) the peak is 7,000 and the spare capacity is 800 - 300 = 500/s, so it drains in 14 seconds (empty at t=33) and the longest wait is 9 s. Doubling the workers cut the drain time from 110 s to 14 s, about 8 times, because drain speed is capacity minus arrivals, which went from 100 to 500.
With 16 workers (1,600/s) nothing ever queues, but outside the burst those workers run at 300 / 1,600 = 19% utilisation. That's the choice a queue gives you: pay for peak capacity all year, or pay with a few seconds of delay during the peak.
The last line is the same burst with no queue: a synchronous service with 400/s of capacity has to reject or time out the other 11,000 orders. With the queue nothing was lost, only delayed. That trade is right for confirmation emails and wrong for "was my card accepted?", which the user is waiting on.

AdvancedSizing consumers with Little's law

Little's law says that, on average, the number of things in a system equals the rate they arrive times how long each one stays: L = λ × W. For a worker pool, "things in the system" are busy worker slots, so it tells you how many workers you need before you've written any code.

sizing.mjs

// Sizing a worker pool with Little's law: work in progress = arrival rate × time per job.
const jobsPerSecond = 2_000;  // average arrival rate at peak hour
const secondsPerJob = 0.15;   // a worker spends 150 ms per job (mostly waiting on I/O)
const targetUtilisation = 0.7; // leave headroom so the queue doesn't grow on every wobble
const slotsPerWorker = 20;    // concurrent jobs one worker process handles (async I/O)

const busySlots = jobsPerSecond * secondsPerJob;
const slots = Math.ceil(busySlots / targetUtilisation);
const workers = Math.ceil(slots / slotsPerWorker);
console.log(`busy slots on average: ${busySlots}`);
console.log(`slots at ${targetUtilisation * 100}% utilisation: ${slots} -> ${workers} worker processes`);

// How long does a backlog take to clear? Only the spare capacity drains it.
const capacity = workers * slotsPerWorker / secondsPerJob; // jobs per second
const backlog = 600_000; // e.g. 5 minutes of a downstream outage at 2,000/s
const spare = capacity - jobsPerSecond;
console.log(`capacity ${Math.round(capacity)}/s, spare ${Math.round(spare)}/s`);
console.log(`a ${backlog.toLocaleString("en-GB")} backlog clears in ${Math.round(backlog / spare)} s (${(backlog / spare / 60).toFixed(1)} min)`);

Result

busy slots on average: 300
slots at 70% utilisation: 429 -> 22 worker processes
capacity 2933/s, spare 933/s
a 600,000 backlog clears in 643 s (10.7 min)

What’s happening

2,000 jobs/s × 0.15 s = 300 jobs in progress at any moment, on average. That's the minimum number of concurrent slots that could possibly keep up.
Running at 100% means the queue grows on every small wobble in the arrival rate, so divide by a target utilisation of 70%: 300 / 0.7 = 428.6, rounded up to 429 slots. With 20 concurrent jobs per process (async I/O), that's 22 worker processes.
Capacity is 22 × 20 / 0.15 = 2,933 jobs/s. The spare capacity, 2,933 - 2,000 = 933/s, is the only part that can work off a backlog.
A five-minute downstream outage at 2,000/s leaves 600,000 jobs queued. They clear in 600,000 / 933 = 643 s, about 10.7 minutes. So the recovery time after an outage is set by the headroom you chose, which is the number to quote when someone asks "why not run at 95% utilisation?".

AdvancedThe asynchronous API shape

When work moves onto a queue, the API changes shape. Instead of returning the result, it returns a handle to the work:

API sketch

POST /exports            { "format": "pdf", "range": "2026-Q3" }
  202 Accepted           { "jobId": "exp_81f2", "status": "queued" }
  Location: /exports/exp_81f2

GET /exports/exp_81f2
  200 OK                 { "status": "running", "progress": 0.4 }
  200 OK                 { "status": "done", "url": "https://files.example.com/exp_81f2.pdf" }

What’s happening

POST /exports validates the request, writes a job record, enqueues a message with the job id, and returns 202 Accepted immediately. 202 means "accepted for processing, not finished", which is exactly the promise being made.
The Location header and jobId give the client something to ask about. The job record lives in a database, not only in the queue, because the queue holds the work while the database holds the status users can query.
The client polls GET /exports/exp_81f2, or better, gets told: a WebSocket or SSE push, a webhook for server-to-server clients, or an email with the link for long jobs (see Real-time options: polling, long polling, SSE, WebSockets).
If the client retries the POST because the 202 got lost, it creates a second export. Accept an Idempotency-Key header and return the existing job for a repeated key (see Idempotency keys for APIs).

AdvancedWhen not to use a queue

The caller needs the answer now. Login, a price check, "is this username free?", reading data. Putting a queue in front of a read only adds latency and a second place for things to go wrong.
The overload is permanent. If arrivals exceed capacity on average, a queue grows without limit and every message gets later. Fix capacity or shed load.
The system is small. A table in your existing database with a status column and SELECT … FOR UPDATE SKIP LOCKED is a perfectly good queue for thousands of jobs a minute, and it commits atomically with your business data. A broker is one more cluster to run, secure and monitor.
You'd lose more than you gain in debuggability. A synchronous call has one stack trace; an asynchronous flow needs correlation ids in every message, tracing across the broker, and dashboards for depth and age.

How long to drain?

After an outage a queue holds 36,000 messages. New messages keep arriving at 1,000/s, and you have 12 workers that each handle 100/s. How long until the backlog is gone, and what does adding 3 workers change?

Show answer

180 seconds with 12 workers; 72 seconds with 15.

What’s happening

12 workers handle 12 × 100 = 1,200/s, but 1,000/s of that is needed just to keep up with new arrivals.
The spare capacity is 1,200 - 1,000 = 200/s, so the backlog drains in 36,000 / 200 = 180 s, three minutes.
With 15 workers capacity is 1,500/s and the spare is 500/s: 36,000 / 500 = 72 s.
Adding 25% more workers made the drain 2.5 times faster, because it multiplied the spare capacity by 2.5. That's why a little extra headroom matters so much for recovery time.

#Queues vs logs

Added

"Message broker" covers two different data structures, and the choice between them drives almost everything else.

A queue (RabbitMQ, Amazon SQS, ActiveMQ) tracks the state of every message: ready, delivered but not yet acknowledged, acknowledged. When a consumer acknowledges a message, it's deleted. Consumers on one queue compete: each message goes to one of them. You scale by adding consumers, and the broker can do things per message: redeliver it after a timeout, delay it, prioritise it, count its attempts and dead-letter it.

A log (Apache Kafka, Redis Streams, Amazon Kinesis, RabbitMQ Streams) is an append-only sequence split into partitions. Messages are kept for a retention period whether or not anyone has read them. The broker doesn't track individual messages; each consumer group just stores an offset, "I've processed partition 3 up to position 1,042". Every group reads every message, any group can rewind and replay, and within a group each partition is read by exactly one consumer, so order holds per partition and parallelism is capped by the partition count.

The mental model: a queue is a pile of to-do tickets; a ticket that's been done is thrown away, and whoever is free takes the next one. A log is a ship's logbook: entries are written in order and never torn out, and each reader keeps their own bookmark. A new crew member can read it from page one; two readers never take pages away from each other.

A queue forgets what it has delivered; a log remembers, and each reader keeps its own bookmark.

	Queue	Log
After a message is processed	Deleted	Kept until retention expires (Kafka's default `log.retention.hours` is 168, one week)
Who gets a message	One consumer of the queue	Every consumer group; one consumer per partition inside a group
Replay	No	Yes: move the group's offset back
Ordering	Roughly FIFO, broken by parallel consumers and redelivery (SQS FIFO keeps order per message group)	Strict per partition
Parallelism	Add consumers	Capped by partitions per group
Per-message features	Ack, redelivery timeout, delay, priority, attempt count, dead-lettering	Few: the consumer commits an offset; retries and dead-letter topics are up to you
Typical jobs	Tasks: send this email, resize this image, call this webhook	Facts many systems need: order events, clickstream, change data capture, event sourcing

The model below shows the parallelism difference, which is the one that surprises people in interviews. A topic has 6 partitions with the same traffic each, and a consumer group grows from 1 to 8 members:

groups.mjs

// A log topic with 6 partitions, each receiving the same rate. Inside one consumer group,
// each partition has exactly one owner, so a partition's messages are handled one after another.
const partitions = 6;

function assign(consumers) {
  const owned = Array.from({ length: consumers }, () => []);
  for (let p = 0; p < partitions; p++) owned[p % consumers].push(p); // round-robin assignment
  return owned;
}

for (const n of [1, 2, 4, 6, 8]) {
  const owned = assign(n);
  const idle = owned.filter((ps) => ps.length === 0).length;
  const busiest = Math.max(...owned.map((ps) => ps.length));
  const desc = owned.map((ps, i) => `c${i + 1}:[${ps.join(",")}]`).join(" ");
  // The group keeps up only as fast as its busiest member: speed-up = partitions / busiest.
  console.log(`group of ${n}: ${desc}`);
  console.log(`  idle ${idle}, busiest owns ${busiest} partition${busiest === 1 ? "" : "s"}, speed-up ${(partitions / busiest).toFixed(1)}x`);
}

// A work queue has no partitions: every free consumer pulls the next message.
for (const n of [4, 8]) console.log(`queue with ${n} consumers: speed-up ${n.toFixed(1)}x (no ordering between messages)`);

Result

group of 1: c1:[0,1,2,3,4,5]
  idle 0, busiest owns 6 partitions, speed-up 1.0x
group of 2: c1:[0,2,4] c2:[1,3,5]
  idle 0, busiest owns 3 partitions, speed-up 2.0x
group of 4: c1:[0,4] c2:[1,5] c3:[2] c4:[3]
  idle 0, busiest owns 2 partitions, speed-up 3.0x
group of 6: c1:[0] c2:[1] c3:[2] c4:[3] c5:[4] c6:[5]
  idle 0, busiest owns 1 partition, speed-up 6.0x
group of 8: c1:[0] c2:[1] c3:[2] c4:[3] c5:[4] c6:[5] c7:[] c8:[]
  idle 2, busiest owns 1 partition, speed-up 6.0x
queue with 4 consumers: speed-up 4.0x (no ordering between messages)
queue with 8 consumers: speed-up 8.0x (no ordering between messages)

What’s happening

Inside a group, every partition is owned by exactly one consumer, so each partition's messages are processed one after another. assign gives partition p to consumer p % n (a round-robin assignor; real assignors also try to keep assignments stable across rebalances).
With 1 consumer it owns all 6 partitions. With 2, each owns 3 and the group is 2.0 times as fast.
With 4 consumers the split is uneven: c1 owns partitions 0 and 4, c2 owns 1 and 5, and c3, c4 own one each. The group can only go as fast as its busiest member, which does 2/6 of the work, so the speed-up is 6 / 2 = 3.0x, not 4x. That's why partition counts with many divisors (6, 12, 24, 60) are a common rule of thumb.
With 6 consumers each owns one partition: 6.0x. With 8, two consumers get nothing (idle 2) and the speed-up is still 6.0x. The partition count is the ceiling on a group's parallelism, so you choose it for the consumer count you expect at peak, not today's.
A work queue has no partitions: any free consumer takes the next message, so 4 consumers give 4x and 8 give 8x, up to what the broker itself can serve. The price is that no order between messages is promised. That's the core trade: the log gives you per-key order and replay; the queue gives you elastic parallelism and per-message handling.

AdvancedWhat a log costs: retention and replay

A log keeps everything for its retention period on every replica, and every consumer group reads the full stream. Both are easy to estimate:

retention.mjs

// Disk for a log that keeps every event for its retention period, on every replica.
const eventsPerSecond = 20_000; // average, all producers together
const bytesPerEvent = 1_000;    // ~1 KB JSON after compression (state your assumption)
const retentionDays = 7;        // Kafka's default (log.retention.hours = 168)
const replicationFactor = 3;    // each partition stored on 3 brokers

const perDay = eventsPerSecond * bytesPerEvent * 86_400;
const total = perDay * retentionDays * replicationFactor;
const TB = 1e12;
console.log(`ingest: ${(eventsPerSecond * bytesPerEvent / 1e6).toFixed(0)} MB/s, ${(perDay / TB).toFixed(2)} TB/day`);
console.log(`stored: ${(total / TB).toFixed(1)} TB for ${retentionDays} days x RF ${replicationFactor}`);

// Every consumer group reads the whole stream again. A queue would delete after the one read.
for (const groups of [1, 3, 6]) {
  console.log(`${groups} consumer group(s): ${(eventsPerSecond * bytesPerEvent * groups / 1e6).toFixed(0)} MB/s read from brokers`);
}
// Replay: a new service reading 7 days of history at 200 MB/s.
const replaySeconds = perDay * retentionDays / 200e6;
console.log(`replaying ${retentionDays} days at 200 MB/s takes ${(replaySeconds / 3600).toFixed(1)} h`);

Result

ingest: 20 MB/s, 1.73 TB/day
stored: 36.3 TB for 7 days x RF 3
1 consumer group(s): 20 MB/s read from brokers
3 consumer group(s): 60 MB/s read from brokers
6 consumer group(s): 120 MB/s read from brokers
replaying 7 days at 200 MB/s takes 16.8 h

What’s happening

20,000 events/s at about 1 KB each is 20 MB/s of ingest, and 20 MB/s × 86,400 s = 1.73 TB per day.
Seven days of retention on three replicas: 1.73 × 7 × 3 = 36.3 TB of disk. Retention and replication factor multiply directly, so "keep it for 30 days" is a 4.3x disk bill, which is why long-term history usually moves to object storage (Kafka's tiered storage, or a sink to S3 and a data lake).
Reads scale with the number of consumer groups: each group reads all 20 MB/s, so 6 groups pull 120 MB/s from the brokers. A queue would have delivered each message once and deleted it. Fan-out by consumer groups is cheap in producer work, not in broker bandwidth.
Replay isn't instant: a new service rebuilding its state from 7 days of history at 200 MB/s (an assumed consumer throughput; measure your own) needs 12.1 TB / 200 MB/s ≈ 16.8 hours. If a new projection must be ready in minutes, keep a compacted topic or a snapshot to start from instead of the raw history.

AdvancedProducts at a glance (October 2026)

Product	Shape	Figures checked in October 2026
Amazon SQS standard	Queue, at-least-once, best-effort order	Messages up to 1 MiB; retention 60 s to 14 days, default 4 days; visibility timeout default 30 s, max 12 h; delay up to 15 min (AWS SQS quotas page)
Amazon SQS FIFO	Queue with order per message group	300 API calls/s per action, or 3,000 messages/s with batches of 10; high-throughput mode raises this much higher (the SQS quotas page lists up to 70,000 API calls/s per action, 700,000 messages/s with batching, in N. Virginia, Oregon and Ireland, and far less in smaller Regions; the queue-types page still says 30,000, so check your Region)
RabbitMQ 4.x	Queue broker (quorum queues), plus streams	Quorum queues have a default delivery limit of 20 since 4.0, after which a message is dead-lettered or dropped (RabbitMQ 4.0 release blog)
Apache Kafka 4.x	Partitioned log	ZooKeeper removed in 4.0 (March 2025), KRaft only; default retention 168 h; "Queues for Kafka" share groups (KIP-932) production-ready in 4.2 (Kafka 4.2 upgrade notes)
Redis Streams	Small log with consumer groups	See the Databases course for commands; durability depends on Redis persistence settings

Kafka or a queue?

I decide by what the consumers need. If each message is a job that one worker must do once (send an email, transcode a file), with retries, delays and a dead-letter queue, and no ordering between jobs, a queue like SQS or RabbitMQ fits: competing consumers, ack-then-delete, and I scale by adding workers. If the message is a business fact that several independent systems need (order placed: billing, search, analytics, fraud), or I need per-entity ordering, replay to rebuild state, or a high-throughput stream, a log like Kafka fits: each system is its own consumer group with its own offset, and I key by the entity whose order matters. I'd also mention the costs: a log needs its partition count chosen for future peak parallelism and its retention budgeted (rate × size × days × replicas); a queue can't replay, so a new consumer starts from now.

#Pub/sub and fan-out

Added

Publish/subscribe separates the sender of a message from its receivers. The publisher sends to a topic ("OrderPlaced"); every subscriber to that topic gets its own copy. The publisher doesn't know who the subscribers are or how many there are, so a new team can start reacting to orders without a single change to the order service. That's fan-out: one event in, N independent deliveries out.

It comes in two flavours, and mixing them up is a classic design mistake:

Live (fire-and-forget) pub/sub delivers only to subscribers connected right now: Redis PUBLISH, a WebSocket broadcast, a server-sent event. Nothing is stored, so a subscriber that's down simply misses the message, and nobody gets an error.
Durable fan-out gives each subscriber its own stored copy: Amazon SNS delivering into one SQS queue per subscriber, a RabbitMQ fanout exchange bound to one queue per service, or one Kafka consumer group per service on a topic. A subscriber that's down builds a backlog and catches up later.

The mental model: live pub/sub is a radio broadcast: if your radio was off, you missed it. Durable fan-out is a newsletter with a letterbox per subscriber: every issue is dropped into every letterbox, and someone back from holiday finds the issues waiting, in order.

Durable fan-out: the topic copies each message into every subscriber's own queue.

The model publishes eight orders to three subscribers while the search indexer is down for orders 3 to 6, once with live pub/sub and once with a durable queue per subscriber:

fanout.mjs

// One "OrderPlaced" topic, three subscribers. Search is down while orders 3-6 are published.
const subscribers = ["email", "analytics", "search"];
const isUp = (name, order) => !(name === "search" && order >= 3 && order <= 6);

// (a) Fire-and-forget pub/sub (Redis PUBLISH, a WebSocket broadcast): deliver to whoever is connected now.
const live = Object.fromEntries(subscribers.map((s) => [s, []]));
// (b) Durable fan-out (SNS -> one SQS queue per subscriber, or one Kafka consumer group each).
const queues = Object.fromEntries(subscribers.map((s) => [s, []]));
const handled = Object.fromEntries(subscribers.map((s) => [s, []]));
let searchPeak = 0;

for (let order = 1; order <= 8; order++) {
  for (const s of subscribers) {
    if (isUp(s, order)) live[s].push(order); // nobody stores it for a subscriber that is down
    queues[s].push(order);                   // the broker keeps a copy per subscriber
    while (isUp(s, order) && queues[s].length) handled[s].push(queues[s].shift()); // drain when up
    if (s === "search") searchPeak = Math.max(searchPeak, queues[s].length);
  }
}

for (const s of subscribers) {
  console.log(`${s.padEnd(9)} live pub/sub got [${live[s]}]  durable queue handled [${handled[s]}]`);
}
const lost = subscribers.reduce((n, s) => n + 8 - live[s].length, 0);
console.log(`messages lost with live pub/sub: ${lost}; with durable queues: 0`);
console.log(`search backlog peaked at ${searchPeak} while it was down, then drained in order`);
console.log(`deliveries per published message: ${subscribers.length} (fan-out factor)`);

Result

email     live pub/sub got [1,2,3,4,5,6,7,8]  durable queue handled [1,2,3,4,5,6,7,8]
analytics live pub/sub got [1,2,3,4,5,6,7,8]  durable queue handled [1,2,3,4,5,6,7,8]
search    live pub/sub got [1,2,7,8]  durable queue handled [1,2,3,4,5,6,7,8]
messages lost with live pub/sub: 4; with durable queues: 0
search backlog peaked at 4 while it was down, then drained in order
deliveries per published message: 3 (fan-out factor)

What’s happening

isUp makes the search indexer unavailable while orders 3 to 6 are published. Email and analytics are up the whole time.
Live pub/sub delivers a message only to subscribers that are up at that moment, so search receives [1,2,7,8]. Orders 3 to 6 are gone: 4 messages lost, and no component saw an error. The publisher can't even tell, because it doesn't know who is subscribed.
Durable fan-out pushes a copy into every subscriber's queue on every publish. Search's queue holds orders 3, 4, 5 and 6 while it's down (backlog peaks at 4). When order 7 arrives and search is back, the while loop drains 3, 4, 5, 6, 7 in order, and search ends with all eight.
Email and analytics never noticed search's outage. Each subscriber has its own backlog, so a slow or broken one can't delay the others. If all three services instead read from one shared queue, they'd be competing consumers: each message would reach only one of them, which is the bug people introduce when they confuse a queue with a topic.
Every publish caused 3 deliveries, the fan-out factor. Broker work, network and storage all scale with the number of subscribers, which matters when fan-out is to users rather than services (below).

AdvancedFan-out to services vs fan-out to users

Fan-out to a handful of services is cheap. Fan-out to users, as in a social feed or a notification system, can be enormous, and that's where the design decision gets interesting:

fanout-cost.mjs

// Service fan-out: each event is delivered once per subscribing service.
const ordersPerSecond = 2_000;
const subscribingServices = 6; // email, analytics, search, fraud, loyalty, warehouse
console.log(`service fan-out: ${(ordersPerSecond * subscribingServices).toLocaleString("en-GB")} deliveries/s`);

// User fan-out (a feed): each post is copied into every follower's timeline.
const postsPerSecond = 5_000;
const avgFollowers = 200;
const timelineWrites = postsPerSecond * avgFollowers;
console.log(`feed fan-out on write: ${timelineWrites.toLocaleString("en-GB")} timeline writes/s`);

// One celebrity post with 50 million followers, pushed by a fleet that writes 1M timelines/s.
const followers = 50_000_000;
const fleetWritesPerSecond = 1_000_000;
console.log(`one celebrity post: ${followers.toLocaleString("en-GB")} writes, ${followers / fleetWritesPerSecond} s of the whole fleet`);
console.log(`same post as fan-out on read: 0 extra writes; each of their followers' feed reads does 1 more lookup`);

Result

service fan-out: 12,000 deliveries/s
feed fan-out on write: 1,000,000 timeline writes/s
one celebrity post: 50,000,000 writes, 50 s of the whole fleet
same post as fan-out on read: 0 extra writes; each of their followers' feed reads does 1 more lookup

What’s happening

Six subscribing services on 2,000 orders/s is 12,000 deliveries/s. That's a normal load for SNS→SQS or for six Kafka consumer groups.
A feed that copies every post into each follower's timeline (fan-out on write) turns 5,000 posts/s with 200 followers on average into 1,000,000 timeline writes per second. Reads become cheap (each user's timeline is precomputed), writes are amplified 200 times.
The average hides the tail. One account with 50 million followers produces 50 million writes from a single post: 50 seconds of a fleet that can do a million writes a second, and every follower's timeline is late while it happens.
Fan-out on read stores the post once and merges it in when each follower loads their feed: zero extra writes, one more lookup per feed read. Real systems mix the two (push for normal accounts, pull for celebrities); that design is worked through in Design a news feed.

AdvancedFiltering and routing

Not every subscriber wants every message. Brokers offer three levels of routing:

By topic: publish OrderPlaced and OrderRefunded to separate topics, and subscribe only to what you need. Simple, and the default with Kafka.
By routing key pattern: a RabbitMQ topic exchange routes order.placed.eu to any queue bound with order.*.eu or order.#.
By message content: an SNS subscription filter policy, such as "only orders where country = "IN"", drops non-matching messages before they reach the subscriber's queue, so it never pays to receive and discard them.

For scale: an SNS standard topic allows 12,500,000 subscriptions and a FIFO topic 100, with up to 200 filter policies per topic (AWS General Reference, SNS quotas, checked October 2026).

AdvancedLive pub/sub inside the backend

Live pub/sub isn't wrong; it's the right tool when a missed message is harmless or recoverable. The common case is WebSocket servers: user A's socket is on server 1 and user B's on server 7, so when A sends a chat message, server 1 publishes to a channel like room:42 and every server with a member of room 42 subscribed pushes it down its sockets. If a server misses a message during a blip, its clients reconnect and refetch recent history from the database, which is the durable copy. See Scaling WebSockets. Cache invalidation broadcasts ("drop key product:9 from your local cache") are the other classic use, as long as entries also have a TTL to cover a missed message.

#Delivery guarantees and ordering

Added

Every messaging design has to answer two questions: how many times can a message take effect, and in what order. The honest answers are "possibly more than once" and "in order only where you arranged it".

There are three delivery guarantees, and the difference is just when the consumer acknowledges:

At-most-once: acknowledge (or commit the offset) first, then process. A crash mid-way loses the message. Acceptable for metrics samples or presence pings, never for money.
At-least-once: process first, then acknowledge. A crash after processing but before the ack means the broker delivers it again. This is the default for SQS, RabbitMQ and Kafka consumers, so duplicates are normal, not a bug.
Effectively-once (often sold as "exactly-once"): at-least-once delivery plus a consumer whose effect is idempotent (a dedup table, a unique constraint, a version check), or a transaction that covers both the processing and the offset when everything stays inside one system, such as Kafka's read-process-write transactions. Over a network you can't get exactly-once delivery, because a sender can never tell "the receiver didn't get it" from "the receiver got it and the ack was lost". The mechanics of idempotent consumers are covered in Idempotency and exactly-once.

Ordering is the same story. A single queue with a single consumer is in order, and has no parallelism. The moment you add consumers, partitions or retries, global order is gone. What a system almost always needs instead is per-entity order: the events of one account, one order, one chat conversation must be applied in sequence, while events of different entities can run in parallel. You get that by choosing a partition key (a Kafka message key, an SQS FIFO message group id) equal to the entity whose order matters: all of that entity's messages land in one partition, and one consumer handles that partition in sequence.

The mental model: supermarket checkout lanes. Each customer's items go through one lane in the order they were put on the belt, but which lane finishes first is anyone's guess. You never need the whole shop to check out in arrival order; you need each customer's shopping to stay together in one lane.

Partitioning by user ID: every user's events share one partition and one consumer.

The model runs the same 12 events (four users, three events each) through three consumers twice: once partitioned by user ID, once with any free consumer taking the next message. Processing times are random but seeded, so the output is the same on every run:

ordering.mjs

// Seeded PRNG so the "random" processing times are the same on every run.
function mulberry32(seed) {
  return () => {
    seed = (seed + 0x6d2b79f5) | 0;
    let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}
const fnv1a = (s) => [...s].reduce((h, c) => Math.imul(h ^ c.charCodeAt(0), 16777619) >>> 0, 2166136261);
const partitionOf = (key) => fnv1a(key) % 3;

// 12 events in the order they were produced: users u1-u4, three events each (seq 1, 2, 3).
const users = ["u1", "u2", "u3", "u4"];
const events = [];
for (let seq = 1; seq <= 3; seq++) for (const key of users) events.push({ key, seq, name: `${key}:${seq}` });

function run(label, pickConsumer) {
  const rand = mulberry32(43); // same seed for both runs
  const freeAt = [0, 0, 0]; // three consumers
  const done = events.map((e, i) => {
    const c = pickConsumer(e, freeAt);
    const start = Math.max(freeAt[c], i); // event i reaches the broker at time i
    freeAt[c] = start + 1 + Math.floor(rand() * 8); // processing takes 1-8 time units
    return { ...e, end: freeAt[c] + i / 100 }; // i/100 breaks ties by arrival
  }).sort((a, b) => a.end - b.end);
  const seen = {};
  const broken = done.filter((e) => { const bad = (seen[e.key] ?? 0) > e.seq; seen[e.key] = Math.max(seen[e.key] ?? 0, e.seq); return bad; });
  console.log(`${label}: ${done.map((e) => e.name).join(" ")}`);
  console.log(`  finished out of per-key order: ${broken.map((e) => e.name).join(", ") || "none"}`);
}

console.log(`produced:      ${events.map((e) => e.name).join(" ")}`);
console.log(`partition key: ${users.map((k) => `${k}->p${partitionOf(k)}`).join(" ")}`);
run("keyed        ", (e) => partitionOf(e.key));                          // consumer c owns partition c
run("any consumer ", (e, freeAt) => freeAt.indexOf(Math.min(...freeAt))); // competing consumers

Result

produced:      u1:1 u2:1 u3:1 u4:1 u1:2 u2:2 u3:2 u4:2 u1:3 u2:3 u3:3 u4:3
partition key: u1->p0 u2->p2 u3->p1 u4->p0
keyed        : u2:1 u3:1 u1:1 u3:2 u4:1 u1:2 u2:2 u4:2 u2:3 u3:3 u1:3 u4:3
  finished out of per-key order: none
any consumer : u2:1 u4:1 u1:2 u3:1 u1:1 u3:2 u4:2 u2:3 u2:2 u3:3 u1:3 u4:3
  finished out of per-key order: u1:1, u2:2

What’s happening

The 12 events are produced as u1:1 u2:1 u3:1 u4:1 u1:2 …, and event i reaches the broker at time i. FNV-1a hashing modulo 3 sends u1 and u4 to p0, u3 to p1 and u2 to p2 (the same layout as the diagram). Each event takes 1 to 8 time units to process; both runs reuse seed 43, so event number i gets the same random duration in each.
Keyed: consumer c owns partition c and handles its messages one at a time. u1:1 runs on c0 from 0 to 8. u4:1 is next in the same partition, so it can't start until 8 and finishes at 9, then u1:2 runs 9 to 10. The finishing order u2:1 u3:1 u1:1 u3:2 u4:1 … is not the produced order, so global order is gone, but every user's events finish as 1, 2, 3: "finished out of per-key order: none".
Look at what per-key order cost: u4:1 reached the broker at 3 but waited until 8, 5 units, behind an unrelated slow u1:1 that happened to share its partition. That's head-of-line blocking, the price of ordering, and it's why one hot or slow key can drag down every other key on its partition.
Any consumer: u1:1 goes to c0 and runs 0 to 8. u1:2 arrives at 4, and the first consumer to come free is c1 (it finishes u4:1 at 5), so u1:2 runs 5 to 6 and finishes before u1:1. If those were "deposit 100" and "withdraw 30", the withdrawal hit an empty account. The same happens to u2: u2:2 runs on c1 from 6 to 12 while u2:3 runs on c0 from 9 to 11.
So the rule is: name the entity whose events must be ordered and make it the key. Global order (one partition, one consumer) is rarely needed and costs all parallelism; no key at all gives full parallelism and no order. Per-key order is the useful middle.

AdvancedWhere duplicates and reorders come from

Even with a good key, things go wrong at the edges. These are the causes interviewers probe:

Cause	What happens	Design answer
Producer retries after a lost ack	The broker stored it, the producer didn't hear back, sends it again: a duplicate	Idempotent producer (Kafka's default since 3.0: `enable.idempotence=true` with `acks=all`) or a dedup id on the message
Consumer crashes after the side effect, before the ack	The message is redelivered and the email goes twice	Idempotent consumer: dedup on message id in the same transaction as the effect
Visibility timeout shorter than the work	SQS makes the message visible again while the first consumer is still working: two consumers at once	Set the timeout above the p99 processing time and extend it for long jobs
Consumer group rebalance	A partition moves to another consumer; messages after the last committed offset are processed again	Commit often enough; idempotent consumer
Retry by re-publishing to the end of the queue	Message 1 fails, message 2 succeeds, message 1 succeeds later: reordered	Per-key sequence numbers checked by the consumer (below), or block the key until the retry is done
Adding partitions	Keys map to new partitions while old messages are still in the old ones	Over-provision partitions up front, or migrate to a new topic (below)

A note on Kafka's producer defaults: Kafka 3.0 made enable.idempotence=true and acks=all the producer defaults (KIP-679), but a bug in 3.0.0 and 3.1.0 left idempotence off unless set explicitly; it was fixed in 3.0.1, 3.1.1 and 3.2.0 (KAFKA-13598, Kafka 3.0 upgrade notes). Producer idempotence only removes duplicates between the producer and the broker. Effects outside Kafka (an email, a call to a payment API) still need their own idempotency key.

AdvancedRestoring order with sequence numbers

When reordering can't be prevented (retries, several producers, a queue without groups), the consumer can restore per-key order itself if each event carries a per-key sequence number:

reorder.mjs

// u1's events arrive out of order (as in the competing-consumers run above), then one is redelivered.
// Each event carries its per-key sequence number; the consumer applies them strictly in sequence.
const received = [
  { seq: 2, op: "withdraw 30" },
  { seq: 1, op: "deposit 100" },
  { seq: 3, op: "withdraw 15" },
  { seq: 1, op: "deposit 100" }, // redelivered after a lost ack
];
let nextSeq = 1; // stored with the account in the same transaction as the balance
const held = new Map(); // seq -> event that arrived early

for (const e of received) {
  if (e.seq < nextSeq) { console.log(`seq ${e.seq} arrives: already applied, skipped`); continue; }
  held.set(e.seq, e);
  console.log(`seq ${e.seq} arrives${e.seq > nextSeq ? `: early, held until ${nextSeq} comes` : ""}`);
  while (held.has(nextSeq)) { // apply everything that is now contiguous
    console.log(`  apply seq ${nextSeq}: ${held.get(nextSeq).op}`);
    held.delete(nextSeq++);
  }
}

Result

seq 2 arrives: early, held until 1 comes
seq 1 arrives
  apply seq 1: deposit 100
  apply seq 2: withdraw 30
seq 3 arrives
  apply seq 3: withdraw 15
seq 1 arrives: already applied, skipped

What’s happening

nextSeq is the next sequence the consumer will apply for this key. It's stored with the account, in the same transaction as the balance, so a crash can't leave the two disagreeing.
seq 2 arrives first, as in the competing-consumers run above. It's ahead of nextSeq (1), so it's held, not applied: applying "withdraw 30" before "deposit 100" is exactly the bug from the main example.
seq 1 arrives. It equals nextSeq, so it's applied, and the while loop then finds seq 2 waiting and applies it too; nextSeq is now 3. seq 3 arrives and is applied straight away.
The redelivered seq 1 is below nextSeq, so it's recognised as already applied and skipped. The same counter handles both reordering and duplicates.
The cost: a held event waits for the missing one, so a lost message stalls its key. Real consumers add a timeout ("if seq 1 hasn't come after 60 s, alert or fetch the state from the source"). If events carry the full new state with a version number instead of a delta, the consumer can simply ignore anything older than what it has, with no buffering at all.

AdvancedAdding partitions moves keys

Kafka's default partitioner maps a key with hash(key) % partitionCount, so changing the count changes where most keys go:

repartition.mjs

// What adding partitions does to "key -> partition" (hash mod N, like Kafka's default partitioner).
const fnv1a = (s) => [...s].reduce((h, c) => Math.imul(h ^ c.charCodeAt(0), 16777619) >>> 0, 2166136261);
const keys = Array.from({ length: 10_000 }, (_, i) => `order-${i}`);

for (const [from, to] of [[6, 7], [6, 8], [6, 12]]) {
  const moved = keys.filter((k) => fnv1a(k) % from !== fnv1a(k) % to).length;
  console.log(`${from} -> ${to} partitions: ${(moved / keys.length * 100).toFixed(1)}% of keys now map to a different partition`);
}
// A key that moves has old events still waiting in its old partition and new ones in the new partition:
// two consumers can now process the same order's events at the same time, in either order.

Result

6 -> 7 partitions: 85.5% of keys now map to a different partition
6 -> 8 partitions: 74.9% of keys now map to a different partition
6 -> 12 partitions: 49.7% of keys now map to a different partition

What’s happening

10,000 keys are hashed with the same FNV-1a function as before, and each key's partition is compared under the old and new counts.
Going from 6 to 7 partitions moves 85.5% of keys, close to the theoretical 1 - 1/7 = 85.7%: with modulo hashing, almost every key changes partition when the count changes by one.
6 to 8 moves 74.9% and 6 to 12 moves 49.7%. Doubling is the kindest change, because a key stays put whenever h % 12 is below 6, which is half the keys. It's still half.
For a key that moved, its old events may still be waiting in the old partition while new ones arrive in the new partition, read by a different consumer: for a while two consumers can process the same order's events at the same time. Kafka doesn't move existing data when you add partitions (it can't, offsets would change).
That's why you choose the partition count up front for future peak parallelism, and when you must grow, create a new topic and migrate (dual-write, drain the old topic, switch consumers). Consistent hashing reduces movement in caches; Kafka's partitioner doesn't use it.

Can you guarantee exactly-once processing?

Not exactly-once delivery, no: over a network a sender can't distinguish a lost message from a lost acknowledgement, so it has to choose between possibly losing (at-most-once) and possibly duplicating (at-least-once). What I build is at-least-once delivery plus idempotent effects, which gives exactly-once results. Concretely: every message has a unique id, and the consumer records processed ids in the same database transaction as its side effect, so a redelivery is detected and skipped; for external calls, I pass the message id as the idempotency key. If the whole pipeline is Kafka to Kafka, Kafka transactions can commit the output and the consumer offset atomically. For ordering, I key by the entity whose order matters, accept that there's no global order, and use per-key sequence numbers when retries can reorder.

Which partition key for chat?

A chat service writes every message to a Kafka topic. Should the key be the sender's user id or the conversation id? Why?

Show answer

The conversation id.

What’s happening

The order that matters is the order of messages within a conversation: everyone must see "Are you free?" before "Great, see you at 7".
With the sender's id as key, Alice's and Bob's messages in the same conversation hash to different partitions, read by different consumers at different speeds. Bob's reply can be processed before Alice's question, so the conversation is stored out of order.
With the conversation id as key, every message of that conversation goes to one partition and one consumer, in the order the broker received them.
The trade-off: a huge group chat is one hot key, so one partition (and one consumer) carries all of it. That's acceptable because a single conversation's message rate is small compared with what a partition can take; if not, you'd split by conversation and time bucket and order by a server-assigned sequence number instead.

#Retries, dead-letter queues and poison messages

Added

With at-least-once delivery, a message that isn't acknowledged comes back. That's the feature that turns crashes into delays, but it raises a question: what if it fails every time? Failures come in two kinds:

Transient: the downstream is briefly overloaded, a connection reset, a lock timeout. The same message succeeds a little later. Retry it, with backoff and jitter.
Permanent: the payload is malformed, it references a deleted record, or it triggers a bug. It will fail on every attempt. This is a poison message. Retrying it forever wastes capacity, and in an ordered queue or a partition it blocks everything behind it.

The standard design handles both: retry with exponential backoff (wait longer after each failure) and full jitter (wait a random time between 0 and that cap, so a crowd of failed messages doesn't retry in lockstep), up to a maximum number of attempts. After that, move the message to a dead-letter queue (DLQ): a separate queue where it waits, untouched, for a person or a tool to inspect it, fix the cause and redrive it back. The rest of the traffic keeps flowing. The full-jitter recipe comes from Marc Brooker's AWS Architecture Blog post "Exponential Backoff And Jitter" (2015, updated 2023); the client-side implementation, with timeouts and abort signals, is in the Node.js course.

The mental model is the post office. A letter that can't be delivered today is tried again tomorrow, then in a few days. After the last attempt it goes to the dead-letter office, where someone opens it to find out what's wrong. It doesn't sit at the front of the sorting line, holding up every other letter.

Bounded retries keep transient failures from becoming losses; the DLQ keeps permanent ones from blocking the queue.

The model pushes 1,000 messages through that flow. 2% are poison; the rest call a flaky downstream that fails 30% of attempts. Five attempts at most, base 1 s, cap 60 s, full jitter, all from a seeded generator:

retry-dlq.mjs

function mulberry32(seed) { // seeded PRNG: same output on every run
  return () => {
    seed = (seed + 0x6d2b79f5) | 0;
    let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}
const rand = mulberry32(2026);
const MAX_ATTEMPTS = 5, BASE_S = 1, CAP_S = 60;
const backoff = (attempt) => rand() * Math.min(CAP_S, BASE_S * 2 ** (attempt - 1)); // full jitter

// 1,000 messages: 2% are poison (a malformed payload: fails every time).
// The rest hit a flaky downstream that fails 30% of attempts.
const succeededOn = Array(MAX_ATTEMPTS + 1).fill(0);
const dlq = { poison: 0, unlucky: 0 };
const delays = [];
let example = null;

for (let id = 1; id <= 1000; id++) {
  const poison = rand() < 0.02;
  let waited = 0;
  const log = [];
  for (let attempt = 1; ; attempt++) {
    const ok = !poison && rand() >= 0.3;
    log.push(`#${attempt} at ${waited.toFixed(1)}s ${ok ? "ok" : "fail"}`);
    if (ok) { succeededOn[attempt]++; delays.push(waited); break; }
    if (attempt === MAX_ATTEMPTS) { poison ? dlq.poison++ : dlq.unlucky++; break; } // park it, move on
    waited += backoff(attempt);
  }
  if (!example && log.length === 4) example = `message ${id}: ${log.join(", ")}`;
}

console.log(succeededOn.slice(1).map((n, i) => `ok on attempt ${i + 1}: ${n}`).join("\n"));
console.log(`dead-lettered: ${dlq.poison + dlq.unlucky} (${dlq.poison} poison, ${dlq.unlucky} transient but unlucky 5 times)`);
delays.sort((a, b) => a - b);
console.log(`extra delay for successes: p50 ${delays[delays.length >> 1].toFixed(1)}s, p99 ${delays[Math.floor(delays.length * 0.99)].toFixed(1)}s, max ${delays.at(-1).toFixed(1)}s`);
console.log(example);
const attempts = succeededOn.reduce((s, n, i) => s + n * i, 0) + (dlq.poison + dlq.unlucky) * MAX_ATTEMPTS;
console.log(`attempts made: ${attempts} for 1000 messages (${(attempts / 1000).toFixed(2)}x the load)`);

Result

ok on attempt 1: 678
ok on attempt 2: 206
ok on attempt 3: 67
ok on attempt 4: 20
ok on attempt 5: 8
dead-lettered: 21 (18 poison, 3 transient but unlucky 5 times)
extra delay for successes: p50 0.0s, p99 5.1s, max 11.5s
message 89: #1 at 0.0s fail, #2 at 0.5s fail, #3 at 1.5s fail, #4 at 4.8s ok
attempts made: 1516 for 1000 messages (1.52x the load)

What’s happening

For each message the seeded generator first decides whether it's poison (2%). Then each attempt succeeds if the message isn't poison and a draw is at least 0.3. After failed attempt n, the wait is rand() × min(60, 1 × 2^(n-1)): caps of 1, 2, 4 and 8 seconds before attempts 2 to 5.
678 messages succeed first time (expected about 980 × 0.7 = 686), then 206, 67, 20, 8 on attempts 2 to 5: each attempt rescues roughly 30% as many as the one before, because 30% of each group fails again.
21 messages are dead-lettered: 18 poison (about 2% of 1,000) and 3 healthy messages that were just unlucky five times in a row (0.3^5 = 0.24% of 980 ≈ 2.4). A DLQ isn't only for bad data, so after the downstream recovers, those 3 can simply be redriven.
Delay: the median success waits 0 s (it worked first time), p99 waits 5.1 s, and the worst waits 11.5 s; the ceiling is 1 + 2 + 4 + 8 = 15 s. Message 89 shows the jitter: it retried at 0.5 s (cap 1), at 1.5 s (waited 1.0 of a possible 2) and at 4.8 s (3.3 of a possible 4), and attempt 4 succeeded.
Retries cost load: 1,516 attempts for 1,000 messages, 1.52 times the traffic, and the 18 poison messages alone burned 18 × 5 = 90 attempts for nothing. If the downstream were completely down, every message would make 5 attempts, five times the normal load, aimed at a service that's already failing. That's why retries need a budget and a circuit breaker on top.

AdvancedWhere the waiting happens

Different brokers give you different places to hold a message between attempts:

Broker	How a failed message comes back	Attempt limit and DLQ
Amazon SQS	Not deleted, so it reappears when its visibility timeout expires (default 30 s, max 12 h); the consumer can change the timeout per message to implement backoff	Redrive policy with `maxReceiveCount` moves it to a DLQ
RabbitMQ quorum queue	Consumer rejects or the connection drops; the message is requeued	`delivery-limit` (default 20 since RabbitMQ 4.0), then the dead-letter exchange
Kafka consumer group	It doesn't: the offset is per partition, so the consumer must either retry in place (blocking the partition) or publish the message to a retry topic and commit	A chain like `orders.retry.10s` → `orders.retry.5m` → `orders.dlt`, consumed by delayed consumers
Kafka share group (4.2+)	Per-record acknowledgement; a released record is redelivered	The broker counts delivery attempts per record

The Kafka retry-topic pattern unblocks the partition but gives up per-key order for the retried message: later events for the same key may be processed before it. If order matters, the consumer must also park later messages of that key until the retry succeeds, or the retry has to happen in place with a short, bounded backoff.

AdvancedWhat a poison message costs without a limit

Without an attempt limit, a poison message is redelivered every time its timeout expires, until the queue's retention period deletes it:

poison-cost.mjs

// No delivery limit: a poison message comes back every time its visibility timeout expires,
// until the queue's retention period deletes it (SQS defaults: 30 s timeout, 4 days retention).
const visibilityTimeoutS = 30;
const retentionS = 4 * 86_400;
const receivesEach = retentionS / visibilityTimeoutS;
console.log(`receives per poison message: ${receivesEach.toLocaleString("en-GB")}`);

const poisonPerDay = 50; // a bad deploy writes 50 malformed messages a day
const workerSecondsPerTry = 0.2;
const steadyPoison = poisonPerDay * 4; // after 4 days, each day's batch expires as a new one arrives
console.log(`poison messages in the queue after 4 days: ${steadyPoison}`);
const wasted = steadyPoison / visibilityTimeoutS;
console.log(`wasted receives: ${wasted.toFixed(1)}/s, keeping ${(wasted * workerSecondsPerTry).toFixed(2)} workers busy for nothing`);

Result

receives per poison message: 11,520
poison messages in the queue after 4 days: 200
wasted receives: 6.7/s, keeping 1.33 workers busy for nothing

What’s happening

With SQS's defaults (30 s visibility timeout, 4 days retention) a single poison message is received 4 × 86,400 / 30 = 11,520 times before it expires.
A bad deploy that writes 50 malformed messages a day reaches a steady state of 50 × 4 = 200 poison messages: each day's batch expires as the next one arrives.
200 messages coming back every 30 s is 6.7 wasted receives per second. At 0.2 s each that's 1.33 workers' worth of capacity doing nothing useful, plus a constant stream of errors that buries real ones in the logs.
In a FIFO queue or a Kafka partition it's far worse than that number suggests: one poison message stops its whole message group or partition, so the cost is "every message behind it is stuck", not 0.2 s.

AdvancedRunning a DLQ in production

Alert when the DLQ isn't empty. A DLQ nobody watches is a slow way to lose data.
Keep the evidence: the original message unchanged, plus the error, the attempt count, the consumer version and a timestamp in headers or attributes.
Give the DLQ a longer retention than the source. For SQS standard queues, a message keeps its original enqueue timestamp when it moves to the DLQ, so a DLQ with the same 4-day retention deletes it early; AWS recommends a longer DLQ retention (FIFO queues reset the timestamp) (AWS SQS dead-letter queue docs, checked October 2026).
Redrive deliberately, after the fix is deployed, and at a controlled rate so the backlog doesn't become a burst. Handlers must be idempotent, because a redriven message may have half-succeeded before.
Classify before retrying. A 400 Bad Request or a schema validation error is permanent: send it to the DLQ on the first attempt instead of burning five.

How do you handle a message that keeps failing?

I separate transient from permanent failures. Transient ones get bounded retries with exponential backoff and full jitter, for example five attempts with caps of 1, 2, 4 and 8 seconds, so they spread out instead of hammering a recovering service. Permanent ones (bad payload, validation errors) go to the dead-letter queue immediately. Anything still failing after the last attempt goes to the DLQ too, which keeps the main queue flowing; for an ordered stream like a Kafka partition, that's essential, because one poison message would otherwise block the partition. The DLQ alerts when it's non-empty, keeps the original message plus the error and attempt count, has a longer retention than the source, and after a fix we redrive it at a controlled rate. All of this assumes the consumer is idempotent, because retries and redrives deliver the same message again.

Worst-case delay before the DLQ

Full jitter with base 1 s and a 60 s cap. How long can a message take, at most, to reach the DLQ with 5 attempts? With 10 attempts?

Show answer

At most 15 seconds with 5 attempts and 243 seconds with 10, plus processing time.

What’s happening

Full jitter waits a random time between 0 and the cap, so the worst case is waiting the whole cap every time.
Five attempts have four waits before attempts 2 to 5, with caps 1, 2, 4, 8: the sum is 15 s.
Ten attempts have nine waits: 1, 2, 4, 8, 16, 32, then the cap holds at 60 for the remaining three: 63 + 180 = 243 s, just over four minutes.
The expected wait is half of that (about 7.5 s and 121.5 s), because each jittered wait averages half its cap. Choose the attempt count by how long the downstream's outages usually last: retrying for four minutes rides out a deploy, but not a regional incident.

#Backpressure and load shedding

Added

A queue absorbs a burst. But when work arrives faster than it can be done for longer than a burst, something has to give, and the only question is what you choose:

Buffer it: let the queue grow. Latency grows with it, without limit, and eventually memory or disk runs out.
Backpressure: make the producer slow down to the consumer's pace. A pull-based consumer only takes what it can handle; a push-based broker honours a credit or prefetch limit; an API answers 429 Too Many Requests with Retry-After. The excess waits upstream, where it's visible and often durable.
Shed load: refuse, quickly and cheaply, the work you can't finish in time, so the work you do accept finishes in time. Shed the least important work first.
Add capacity: autoscale. That works, but it takes minutes, so for the minutes in between you still need one of the other three.

Little's law explains why buffering is the trap: latency = queue length ÷ throughput. Throughput is capped at capacity, so in overload every extra queued request is pure added latency. And once requests wait longer than the client's timeout, the server is spending its whole capacity on answers nobody is waiting for.

The mental model is a club with a fire limit. The bouncer at the door turns people away when it's full, and everyone inside can move and get a drink (shedding). Letting everyone in makes the room so packed that nobody gets served and people leave anyway (an unbounded queue). Phoning the promoter to stop sending buses is backpressure.

The model sends a server that can handle 100 requests a second 120 per second for a minute (seeded random arrivals). Clients give up after 1 second. It compares an unbounded queue with two ways of shedding:

shed.mjs

function mulberry32(seed) { // seeded PRNG: same arrivals on every run
  return () => {
    seed = (seed + 0x6d2b79f5) | 0;
    let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}
// A server that handles 100 requests/s (10 ms each, one at a time) receives 120/s for 60 s.
// Clients give up after 1 s: an answer later than that is wasted work.
const SERVICE_S = 0.01, RATE = 120, DURATION_S = 60, CLIENT_TIMEOUT_S = 1;
const rand = mulberry32(1);
const arrivals = [];
for (let t = 0; t < DURATION_S; ) { t += -Math.log(1 - rand()) / RATE; arrivals.push(t); } // Poisson arrivals

function simulate(label, admit) {
  let freeAt = 0; // when the server finishes everything accepted so far
  const latencies = [];
  let rejected = 0;
  for (const t of arrivals) {
    const wait = Math.max(0, freeAt - t); // FIFO: time until the work ahead of us is done
    if (!admit(wait)) { rejected++; continue; } // shed: answer 503 at once, costs ~nothing
    freeAt = Math.max(freeAt, t) + SERVICE_S;
    latencies.push(wait + SERVICE_S);
  }
  latencies.sort((a, b) => a - b);
  const pct = (p) => latencies[Math.floor(latencies.length * p)].toFixed(2);
  const useful = latencies.filter((l) => l <= CLIENT_TIMEOUT_S).length;
  console.log(`${label}: served ${latencies.length}, rejected ${rejected}, p50 ${pct(0.5)}s, p99 ${pct(0.99)}s`);
  console.log(`  answered within the 1 s timeout (goodput): ${useful}, ${(useful / arrivals.length * 100).toFixed(0)}% of arrivals`);
}

console.log(`${arrivals.length} requests arrived (${(arrivals.length / DURATION_S).toFixed(0)}/s) for 100/s of capacity`);
simulate("unbounded queue   ", () => true);
simulate("queue capped at 20", (wait) => wait / SERVICE_S < 20);   // at most 20 waiting = 200 ms
simulate("shed if wait > 1 s", (wait) => wait + SERVICE_S <= CLIENT_TIMEOUT_S);

Result

7301 requests arrived (122/s) for 100/s of capacity
unbounded queue   : served 7301, rejected 0, p50 6.82s, p99 12.85s
  answered within the 1 s timeout (goodput): 477, 7% of arrivals
queue capped at 20: served 6016, rejected 1285, p50 0.19s, p99 0.21s
  answered within the 1 s timeout (goodput): 6016, 82% of arrivals
shed if wait > 1 s: served 6095, rejected 1206, p50 0.98s, p99 1.00s
  answered within the 1 s timeout (goodput): 6095, 83% of arrivals

What’s happening

Arrivals are a seeded Poisson process at 120/s: 7,301 requests in 60 s, about 122/s, against a capacity of 100/s (10 ms each, one at a time). For each arrival, wait is how long until the work already accepted is done; admit(wait) decides whether to take it.
Unbounded queue: everything is admitted. The backlog grows by about 22 requests a second, so by the end about 1,300 are waiting, 13 s of work. p50 latency is 6.82 s and p99 12.85 s. Only 477 requests (7%) came back within the 1 s timeout, nearly all in the first few seconds. The server was 100% busy the whole time, yet 93% of its work was for clients that had already given up. In real life those clients retry, which adds even more load: this is congestion collapse.
Queue capped at 20 (200 ms of work): when 20 are waiting, a new request is rejected at once with a 503, which costs almost nothing. 1,285 are rejected; the rest see p50 0.19 s, p99 0.21 s, and 6,016 (82%) are answered in time. That's essentially the server's whole capacity (100/s × 60 s = 6,000, plus a few in the final second) turned into useful answers.
Shed if the wait would exceed 1 s: admits a request only if it can finish before the client gives up. Goodput is about the same (6,095, 83%) and fewer requests are rejected, but p50 latency is 0.98 s: the queue is permanently a second long, right at the edge. So choose the bound by the latency you want to offer, not by the timeout: a short queue gives the same goodput with far lower latency.
In overload, capacity sets how much useful work you do, and the queue bound sets how long each piece takes. An unbounded queue turns overload into lateness for everyone instead of a fast "no" for a few.

AdvancedBackpressure: slowing the producer

Load shedding protects a server from its clients. Backpressure protects a consumer from its own supply, by letting it pull only what it can handle:

credits.mjs

// A producer that could send 50 msgs/tick feeds a consumer that handles 20/tick, for 10 ticks.
function run(label, sendAllowed) {
  let buffer = 0, sent = 0, handled = 0, peak = 0;
  for (let tick = 1; tick <= 10; tick++) {
    const send = Math.min(50, sendAllowed(buffer)); // the producer asks how much it may send
    buffer += send; sent += send;
    peak = Math.max(peak, buffer);
    const take = Math.min(20, buffer); // the consumer works through its buffer
    buffer -= take; handled += take;
  }
  console.log(`${label}: sent ${sent}, handled ${handled}, consumer buffer peak ${peak}, left ${buffer}`);
}

run("push, no backpressure ", () => Infinity);                     // the broker pushes everything it has
run("credits (prefetch 40) ", (buffer) => Math.max(0, 40 - buffer)); // at most 40 unacked in flight
// With credits the producer is held at the consumer's pace (20/tick) and the excess waits upstream,
// where it is durable and visible (queue depth), instead of in one process's memory.

Result

push, no backpressure : sent 500, handled 200, consumer buffer peak 320, left 300
credits (prefetch 40) : sent 220, handled 200, consumer buffer peak 40, left 20

What’s happening

The producer could send 50 messages per tick; the consumer handles 20. sendAllowed(buffer) is the producer asking how much it may send right now.
Push with no limit: the producer sends 50 every tick regardless. After 10 ticks it has sent 500, the consumer has handled 200, and 300 sit in the consumer's memory. The in-flight buffer peaked at 320, and it keeps growing by 30 a tick until the process runs out of memory, taking all 300 unacknowledged messages down with it.
Credits (prefetch 40): the consumer allows at most 40 unacknowledged messages. The first tick sends 40; after that the producer may only send what the consumer has freed up, 20 a tick. Total sent is 220, the buffer never exceeds 40, and the producer has been slowed to the consumer's pace automatically.
The excess didn't disappear: it's still in the broker, where it's durable, shared among consumers, and visible as queue depth to alert and autoscale on. That's the point of backpressure: keep the backlog where it's safe and measurable.

The same idea appears at every layer: RabbitMQ's prefetch (basic.qos), Kafka consumers that pull (a slow consumer just falls behind, and its lag grows instead of its memory), TCP and HTTP/2 flow-control windows, Node streams' highWaterMark and drain, and 429 with Retry-After at an API.

AdvancedWhat to shed first

Shed at the edge. Rejecting at the load balancer or gateway costs microseconds; rejecting after authentication, parsing and two database calls has already spent the capacity you were trying to save.
Shed by priority. Checkout and login are critical; recommendations, analytics beacons and prefetches are not. Tag requests with a priority and drop the low ones first, so overload degrades the page instead of breaking it (see Graceful degradation and fallbacks).
Shed fairly. Per-tenant or per-client limits stop one noisy customer from taking everyone down (see Rate limiting algorithms).
Drop work that's already stale. A request that has waited past its deadline should be discarded when it reaches the front of the queue, not processed. Pass deadlines along with requests so downstream services can do the same.
Tell clients how to back off. A 503 or 429 with Retry-After and clients that use jittered backoff turn rejection into a smooth reduction in load instead of an immediate retry storm.

Google's SRE book has a chapter on handling overload that covers per-client quotas, criticality levels and client-side throttling in more depth.

The service is at 120% of capacity. What do you do?

First, protect what's working: bound the queues so latency stays inside the clients' timeout, and shed the excess quickly at the edge, lowest-priority traffic first, with a 429 or 503 and a Retry-After. Without that, an unbounded queue makes every request late, the server spends its capacity on requests whose clients have given up, and their retries make it worse. Inside the system, I'd use backpressure between stages, with pull-based consumers and prefetch limits, so backlogs sit in the broker where they're durable and visible rather than in a service's memory. Then I add capacity: autoscale the stateless tier on queue age or backlog per worker, knowing it takes minutes and that the database may be the real limit. Finally I'd check for amplification: retries at several layers and clients without jitter can turn 120% into 300%.

#Event-driven architecture, event sourcing and CQRS

Added

These three names are often said in one breath, but they're separate decisions, and you can take any one without the others.

Event-driven architecture is about how services talk. Instead of the order service calling billing, inventory and email ("do this"), it publishes a fact ("OrderPlaced, order 812, these items, this total") and each interested service reacts on its own. The publisher doesn't know who listens; adding a consumer needs no change to the producer. Events come in two weights. An event notification is thin ("order 812 changed") and consumers call back for the details, which keeps events small but couples consumers to the producer's API. Event-carried state transfer puts the data in the event, so consumers can keep their own local copy and never call back, at the cost of bigger events and duplicated data.

Event sourcing is about how one service stores its data. Instead of storing the current state and overwriting it, you store the sequence of changes (Deposited 100, Withdrew 30) as the source of truth, and current state is computed by replaying them. You get a complete audit trail, the ability to answer "what was the balance on 3 March?", and the freedom to build new views from old history. You pay with events you must be able to read forever, replays that need snapshots, and queries that need separate read models.

CQRS (Command Query Responsibility Segregation) is about separating writes from reads. The write side handles commands, enforces the rules and records changes. One or more read models, each shaped for a particular query (a balance table, a search index, a monthly statement), are built from those changes asynchronously. Reads get fast, denormalised data; the price is that read models lag behind the write side by milliseconds to seconds. CQRS doesn't need event sourcing: a normal database plus change data capture (CDC) feeding Elasticsearch is CQRS too.

The mental model is your bank account. The bank doesn't keep "balance = 55" and overwrite it; it keeps every transaction (the ledger, event sourcing). The balance in the app, the monthly PDF statement and the fraud team's dashboard are all different views computed from those transactions (CQRS read models), and the app's balance can be a few seconds behind a payment you just made.

CQRS with event sourcing: one write model that enforces the rules, many read models shaped for queries.

The model is a tiny event-sourced account with one asynchronous read model:

cqrs.mjs

// Write side: commands are checked against current state, then recorded as events (the log is the truth).
const log = [];
const stateOf = (account) => log
  .filter((e) => e.account === account)
  .reduce((s, e) => ({ balance: s.balance + (e.type === "Deposited" ? e.amount : -e.amount), version: s.version + 1 }), { balance: 0, version: 0 });

function handle(cmd) {
  const s = stateOf(cmd.account);
  if (cmd.type === "Withdraw" && cmd.amount > s.balance) return `rejected ${cmd.type} ${cmd.amount}: balance ${s.balance}`;
  log.push({ seq: log.length + 1, account: cmd.account, type: cmd.type === "Deposit" ? "Deposited" : "Withdrew", amount: cmd.amount });
  return `accepted ${cmd.type} ${cmd.amount} -> event #${log.length}`;
}

// Read side: a projection built from the events, updated asynchronously (it lags the log).
const view = { position: 0, balances: {} };
function project(upTo = log.length) {
  for (; view.position < upTo; view.position++) {
    const e = log[view.position];
    view.balances[e.account] = (view.balances[e.account] ?? 0) + (e.type === "Deposited" ? e.amount : -e.amount);
  }
}

console.log(handle({ account: "acc-1", type: "Deposit", amount: 100 }));
console.log(handle({ account: "acc-1", type: "Withdraw", amount: 30 }));
console.log(handle({ account: "acc-1", type: "Withdraw", amount: 500 }));
console.log(handle({ account: "acc-1", type: "Withdraw", amount: 15 }));
project(1); // the projector has only caught up to event #1 so far
console.log(`read model at #${view.position}: acc-1 = ${view.balances["acc-1"]} (write side says ${stateOf("acc-1").balance})`);
project();
console.log(`read model at #${view.position}: acc-1 = ${view.balances["acc-1"]}`);

// A new screen next month ("list my withdrawals"): build a new projection by replaying the whole log.
const withdrawals = log.filter((e) => e.type === "Withdrew").map((e) => `#${e.seq} ${e.amount}`);
console.log(`new projection from replay: withdrawals ${withdrawals.join(", ")}`);
console.log(`events stored: ${log.length}, state rebuilt from them: ${JSON.stringify(stateOf("acc-1"))}`);

Result

accepted Deposit 100 -> event #1
accepted Withdraw 30 -> event #2
rejected Withdraw 500: balance 70
accepted Withdraw 15 -> event #3
read model at #1: acc-1 = 100 (write side says 55)
read model at #3: acc-1 = 55
new projection from replay: withdrawals #2 30, #3 15
events stored: 3, state rebuilt from them: {"balance":55,"version":3}

What’s happening

On the write side, stateOf("acc-1") computes the current state by folding over that account's events: each Deposited adds, each Withdrew subtracts, and version counts events. There is no stored balance at all. Deposit 100 becomes event #1 and Withdraw 30 event #2.
Withdraw 500 is checked against the folded state (balance 70) and rejected. No event is written. That's the difference between a command (a request, in the imperative, which can be refused) and an event (a fact, in the past tense, which has already happened and can't be).
Withdraw 15 becomes event #3. The projector has only processed event #1, so the read model still says 100 while the write side says 55. That's CQRS's eventual consistency: a user who just withdrew money and refreshes the page sees the old balance.
After project() catches up to #3, the read model says 55. The usual fixes for the stale read: return the new state in the command's response, read from the write side for the user who just wrote, or have the client wait until the read model's position is at least the event number it got back (read-your-writes; see Consistency models).
A month later product wants a "my withdrawals" screen. No migration and no backfill script: a new projection replays the log and produces #2 30, #3 15 from history that was always there. Rebuilding state from the events gives {"balance":55,"version":3}. In a real event store, version is also used for optimistic concurrency: an append succeeds only if the account is still at the version the command read, so two concurrent withdrawals can't both pass the balance check.

AdvancedSnapshots keep replays short

Folding every event on every command is fine for three events and not for three years of them:

snapshot.mjs

// How many events must be read to load one aggregate (an account) before handling a command?
const eventsPerDay = 30;    // a busy business account
const years = 3;
const total = eventsPerDay * 365 * years;
console.log(`events after ${years} years: ${total.toLocaleString("en-GB")}`);

for (const every of [0, 1_000, 100]) {
  // With a snapshot every N events, load the latest snapshot and replay at most N - 1 events after it.
  const worst = every === 0 ? total : every - 1;
  const snapshots = every === 0 ? 0 : Math.floor(total / every);
  console.log(`${every === 0 ? "no snapshots      " : `snapshot every ${String(every).padEnd(4)}`}: read up to ${worst.toLocaleString("en-GB")} events per command, ${snapshots} snapshots stored`);
}

Result

events after 3 years: 32,850
no snapshots      : read up to 32,850 events per command, 0 snapshots stored
snapshot every 1000: read up to 999 events per command, 32 snapshots stored
snapshot every 100 : read up to 99 events per command, 328 snapshots stored

What’s happening

A busy account with 30 events a day has 32,850 events after three years. Without snapshots, every command must read all of them to compute the balance it checks against.
A snapshot stores the folded state at some event number. Loading means reading the latest snapshot, then replaying only the events after it: with a snapshot every 1,000 events that's at most 999 events, and with one every 100 at most 99.
The cost is storage and a background job (32 or 328 snapshots if you keep them all; most systems keep only the latest few). Snapshots are a cache: they can always be thrown away and rebuilt from the events, which stay the source of truth.
If you need snapshots for an aggregate after a week, that's a hint the aggregate is too big (an "account" that's really a whole company's ledger) and should be split.

AdvancedHow services coordinate through events

	Choreography	Orchestration
How it works	Each service reacts to events and emits its own: `OrderPlaced` → payment emits `PaymentTaken` → inventory emits `StockReserved`	A coordinator (saga orchestrator, workflow engine) tells each service what to do next and tracks the state
Strengths	No central component; services stay independent; easy to add a listener	The whole flow is visible in one place; timeouts and compensations are explicit
Weaknesses	Nobody owns the end-to-end flow; hard to answer "where is order 812 stuck?"; cycles between services	The orchestrator is one more service to run, and it knows about everyone
Good fit	Simple flows with few steps and independent side effects	Long, multi-step business processes with compensations (refund if shipping fails)

Both are ways to run a saga, a multi-step process across services with compensating actions instead of a distributed transaction (see Distributed transactions: two-phase commit, sagas and the outbox). Whichever you choose, publishing an event and updating your database must not be two independent writes that can half-fail: use the transactional outbox (write the event to an outbox table in the same transaction, and relay it to the broker afterwards) or CDC on the database's own log.

AdvancedWhen not to use event sourcing

Most CRUD. A user profile or a product catalogue gains little from a history of every edit, and every query becomes a projection to maintain.
When you'll need ad-hoc queries on current state and nobody wants to run projections. Reporting teams expect tables.
When personal data must be deletable. Events are immutable, so the "right to be forgotten" needs a plan up front, typically storing personal fields encrypted with a per-user key and deleting the key ("crypto-shredding").
When the team hasn't done it before and the domain doesn't demand an audit trail. Event sourcing earns its keep in ledgers, payments, inventory, bookings and anything regulators audit.

Would you use event sourcing for an e-commerce site?

For parts of it. I'd event-source the order lifecycle and payments, where the history is the business (placed, paid, partially refunded, shipped) and audits matter, and I'd use CQRS read models for the order-history page and the operations dashboard, accepting a second or so of lag with read-your-writes for the customer who just acted. I would not event-source the product catalogue or user profiles; plain tables with an updated_at and maybe an audit table are simpler. Between services I'd publish domain events through an outbox, mostly event-carried state transfer so consumers don't call back, and I'd use an orchestrated saga for checkout, because payment, stock and shipping need explicit compensations and someone has to own the question "where is this order stuck?".

#Batch vs stream processing

Added

Batch processing runs over a bounded set of data that's already been collected: "all of yesterday's orders", "every log file from the last hour". The job starts, reads everything, writes a result and stops. It's simple to reason about, exact (all the data is there), easy to rerun after a bug, and efficient at huge volume. Its weakness is latency: the answer arrives hours after the events.

Stream processing runs continuously over unbounded data, updating results as each event arrives: fraud scores within a second, a live dashboard, "alert if errors exceed 100 a minute". The hard part is time. Each event has an event time (when it happened, on the device) and a processing time (when it reached you), and networks, retries and offline phones make them differ, so events arrive out of order, and some arrive late. To count "clicks per minute" you group events into windows: tumbling windows (fixed, non-overlapping: 12:00–12:01, 12:01–12:02), sliding windows (fixed length, overlapping: the last 10 minutes, recomputed every minute) or session windows (a burst of activity followed by a gap). A watermark is the processor's estimate that "no more events older than T are coming"; when it passes a window's end, the window is emitted. An event that arrives after that is late, and you choose: drop it, keep windows open a little longer (allowed lateness) and emit a correction, or send it to a side output for a later batch fix.

The mental model is counting votes. Batch is closing the polls and counting every ballot in a hall: exact, but you get the result in the morning. Streaming is the live tally on TV as each station reports: fast, but when postal ballots for a district arrive after you've called it, you either ignore them or announce a correction.

The same 15 events in tumbling and sliding windows. Sliding windows overlap, so an event after the first 5 s falls in two windows and is counted twice.

The model feeds 15 click events, in the order they arrive, to three streaming setups, and then recounts them as a batch job would:

windows.mjs

// Click events as [eventTime in seconds], in the order they ARRIVE (network delays reorder them).
const arrivals = [1, 3, 4, 7, 11, 12, 9, 16, 18, 14, 22, 8, 25, 27, 31];
const MAX_DELAY = 5; // watermark = highest event time seen - 5 s: "nothing older should still be coming"

function stream(label, windowsFor, lateness) {
  const windows = new Map(); // "start-end" -> { start, end, count, fired }
  const out = [];
  let watermark = -Infinity;
  for (const t of arrivals) {
    for (const [start, end] of windowsFor(t)) {
      const key = `${start}-${end}`;
      const w = windows.get(key) ?? windows.set(key, { start, end, count: 0, fired: false }).get(key);
      if (w.fired && watermark >= end + lateness) { out.push(`late ${t} dropped from [${key})`); continue; }
      w.count++;
      if (w.fired) out.push(`[${key}) corrected to ${w.count} (late ${t})`);
    }
    watermark = Math.max(watermark, t - MAX_DELAY);
    for (const w of windows.values()) {
      if (!w.fired && watermark >= w.end) { w.fired = true; out.push(`[${w.start}-${w.end})=${w.count}`); }
    }
  }
  for (const w of windows.values()) if (!w.fired) out.push(`[${w.start}-${w.end})=${w.count} (end of stream)`);
  console.log(`${label}:`);
  for (let i = 0; i < out.length; i += 3) console.log(`  ${out.slice(i, i + 3).join("  ")}`);
}

const tumbling = (t) => [[Math.floor(t / 10) * 10, Math.floor(t / 10) * 10 + 10]]; // 10 s, no overlap
const sliding = (t) => { // 10 s windows starting every 5 s: an event is in two of them (one before t = 5)
  const s = Math.floor(t / 5) * 5;
  return [[s - 5, s + 5], [s, s + 10]].filter(([a]) => a >= 0);
};

stream("tumbling 10s, no allowed lateness", tumbling, 0);
stream("tumbling 10s, 10s allowed lateness", tumbling, 10);
stream("sliding 10s every 5s, no allowed lateness", sliding, 0);

// Batch: wait until all the data is in, sort by event time, count. Exact, but hours later.
const exact = {};
for (const t of arrivals) exact[Math.floor(t / 10) * 10] = (exact[Math.floor(t / 10) * 10] ?? 0) + 1;
console.log(`batch recount (tumbling): ${Object.entries(exact).map(([s, n]) => `[${s}-${+s + 10})=${n}`).join("  ")}`);

Result

tumbling 10s, no allowed lateness:
  [0-10)=5  late 8 dropped from [0-10)  [10-20)=5
  [20-30)=3 (end of stream)  [30-40)=1 (end of stream)
tumbling 10s, 10s allowed lateness:
  [0-10)=5  [0-10) corrected to 6 (late 8)  [10-20)=5
  [20-30)=3 (end of stream)  [30-40)=1 (end of stream)
sliding 10s every 5s, no allowed lateness:
  [0-10)=5  [5-15)=5  late 8 dropped from [0-10)
  late 8 dropped from [5-15)  [10-20)=5  [15-25)=3
  [20-30)=3 (end of stream)  [25-35)=3 (end of stream)  [30-40)=1 (end of stream)
batch recount (tumbling): [0-10)=6  [10-20)=5  [20-30)=3  [30-40)=1

What’s happening

Events arrive out of event-time order: 9 after 11, 12, 14 after 18, and 8 after 22. The watermark is "highest event time seen minus 5 s", a promise that nothing older than that should still be on its way. A window is emitted once the watermark passes its end.
Tumbling, no lateness: after 1, 3, 4, 7, 11, 12 the watermark is 7, so when 9 arrives the window [0, 10) is still open and it's counted. 16 raises the watermark to 11 and [0, 10) is emitted with 5. When 8 arrives the watermark is 17, past 10, so it's dropped. [10, 20) holds 11, 12, 16, 18, 14 and is emitted with 5 when 25 lifts the watermark to 20. The last two windows never see a watermark past their end, so they're flushed when the stream ends.
Tumbling, 10 s allowed lateness: [0, 10) is still emitted with 5 at watermark 11, but its state is kept until the watermark reaches 10 + 10 = 20. 8 arrives at watermark 17, so the window is corrected to 6. Downstream must treat window results as upserts keyed by the window, not as appends, and the processor holds state for every open window for 10 s longer (memory).
Sliding, 10 s every 5 s: each event after the first 5 s falls in two windows, so there are nearly twice as many results: [0, 10) = 5 is emitted when 16 lifts the watermark to 11, as before, and [5, 15) = 5 (7, 11, 12, 9, 14) when 22 lifts it to 17. The late 8 belonged to both and is dropped from both. Sliding windows give smooth "last 10 seconds" numbers at the cost of more state and more output.
The batch recount needs no watermark at all: it waits until all the data is in and counts [0, 10) = 6, the true answer. The stream answered within seconds and was off by one for that window. The watermark delay is the dial: a longer delay drops fewer late events but emits every result later.

AdvancedMapReduce: the batch idea in four steps

MapReduce (from Google's 2004 paper, reimplemented as open source in Hadoop) is the pattern behind most batch systems, Spark included: split the input, map each piece independently, shuffle so all values for a key meet in one place, and reduce each key:

mapreduce.mjs

// Batch job: page views per country from a day of access logs, split across 3 input files.
const inputSplits = [
  ["IN /home", "US /home", "IN /cart"],
  ["GB /home", "IN /home", "US /cart"],
  ["IN /pay", "US /home", "IN /home"],
];
const REDUCERS = 2;
const fnv1a = (s) => [...s].reduce((h, c) => Math.imul(h ^ c.charCodeAt(0), 16777619) >>> 0, 2166136261);

// 1. Map: each mapper reads one split and emits (key, 1) pairs. Mappers don't talk to each other.
const mapped = inputSplits.map((lines) => lines.map((line) => [line.split(" ")[0], 1]));

// 2. Combine (optional): pre-sum inside each mapper so less data crosses the network.
const combined = mapped.map((pairs) => Object.entries(pairs.reduce((acc, [k, v]) => ({ ...acc, [k]: (acc[k] ?? 0) + v }), {})));
console.log(`pairs sent over the network: ${mapped.flat().length} without combiner, ${combined.flat().length} with`);

// 3. Shuffle: every pair for a key goes to the same reducer, chosen by hash(key) % reducers.
const buckets = Array.from({ length: REDUCERS }, () => ({}));
for (const [k, v] of combined.flat()) (buckets[fnv1a(k) % REDUCERS][k] ??= []).push(v);
buckets.forEach((b, i) => console.log(`reducer ${i} receives ${JSON.stringify(b)}`));

// 4. Reduce: sum each key's values. Each reducer writes its own output file.
const output = buckets.map((b) => Object.fromEntries(Object.entries(b).map(([k, vs]) => [k, vs.reduce((a, c) => a + c, 0)])));
console.log(`output: ${output.map((o) => JSON.stringify(o)).join(" + ")}`);

Result

pairs sent over the network: 9 without combiner, 7 with
reducer 0 receives {"IN":[2,1,2],"GB":[1]}
reducer 1 receives {"US":[1,1,1]}
output: {"IN":5,"GB":1} + {"US":3}

What’s happening

The input is three files (splits) of access-log lines. Map runs once per split, in parallel and without coordination, turning each line into a pair (country, 1). Nine lines give nine pairs.
The optional combiner pre-sums inside each mapper: split 1 becomes IN: 2, US: 1 instead of three separate pairs. Pairs crossing the network drop from 9 to 7 here, and by orders of magnitude on real data, because the shuffle is usually the most expensive step.
Shuffle sends every pair for a key to the same reducer, chosen by hash(key) % 2. Reducer 0 receives IN: [2, 1, 2] and GB: [1]; reducer 1 receives US: [1, 1, 1]. It's the same idea as a partition key in Kafka: one key, one place.
Reduce sums each key's list: IN = 5, GB = 1, US = 3, each reducer writing its own output file. If a mapper or reducer machine dies, the framework simply reruns that task on another machine, because tasks are deterministic and their inputs are still on disk. That rerun-ability is what makes batch jobs so robust.
The skew problem is visible even here: reducer 0 got two keys and 6 of the 9 page views, reducer 1 one key and 3. With real data, one huge key (one country, one celebrity, one bot) makes one reducer the straggler that the whole job waits for.

AdvancedLambda and Kappa

When you need both fresh and exact numbers, there are two classic architectures:

Lambda: run a stream path for fast, approximate results and a batch path that recomputes exact results from the raw data later; queries merge the two. Correct and robust, but you write and maintain the same logic twice, in two systems, and they drift apart.
Kappa: use only the streaming path, over a replayable log. To fix a bug or change the logic, deploy a new version of the job, replay the log from the beginning into a new output, and switch over when it has caught up. One codebase, but it needs a log that retains enough history and a stream processor that can replay fast.

Modern stream processors (Apache Flink, Kafka Streams, Spark Structured Streaming) narrow the gap: they keep window state in checkpointed storage, so a crash resumes with exactly-once state, and they handle event time, watermarks and late data directly. In Flink, for example, late events are dropped by default, allowed lateness defaults to zero, and events too late even for that can be collected in a side output (Flink documentation on windows and event time).

Where does the late event go?

A stream counts orders in 1-minute tumbling windows by event time. The watermark is "maximum event time seen minus 30 seconds", and allowed lateness is 0. The processor has just seen an event stamped 12:05:40. An order stamped 12:04:50 arrives now. Is it counted?

Show answer

No, it's late and dropped (or sent to a side output if you configured one).

What’s happening

The watermark is 12:05:40 - 30 s = 12:05:10.
The order belongs to the window [12:04, 12:05). That window was emitted when the watermark passed 12:05:00.
The watermark (12:05:10) is already past the window's end and allowed lateness is 0, so the window's state is gone and the event can't be added.
To count it you'd need allowed lateness of at least 10 s (keep the window until the watermark passes 12:05:10), a longer watermark delay (results come out later for everyone), or a batch job that recounts from the log.

#Scheduled and delayed jobs

Added

Lots of work is "do this at a certain time", not "do this now". Recurring jobs run on a schedule: the nightly report, hourly cleanup, monthly invoices. Delayed jobs run once, later: cancel an unpaid order after 15 minutes, send a reminder at 9:00, retry a webhook in 40 seconds, email "still there?" tomorrow.

On one machine, cron and setTimeout do this. At scale they break in predictable ways: one cron server is a single point of failure (if it's down at 02:00, the report doesn't run); two cron servers for redundancy run every job twice; timers held in a process's memory vanish on a deploy; and nobody can see what's scheduled or what ran. The scalable design keeps schedules durably (a database table, a sorted set, a managed scheduler), has workers claim due jobs with a lease so one owner runs each job at a time, and makes every job idempotent, because "at most once" can't be guaranteed: a worker can do the work and die before it records success.

The mental model is a hotel's wake-up calls. Requests are written in the front-desk book with a time. Whoever is on shift checks the book each minute, makes the calls that are due and ticks them off. If a clerk collapses mid-call, the next clerk sees the unticked entry after a while and makes the call again, so a guest might occasionally get two calls, which is far better than none.

Durable schedules, leased claims and idempotent workers: no single cron box, no lost timers.

The model is the core of that design: a jobs table with leases and fencing tokens, two workers, and one worker that freezes mid-job:

lease.mjs

// A jobs table with leases. In SQL the claim is one statement:
//   UPDATE jobs SET owner=?, lease_until=now()+30s, token=token+1
//   WHERE id = (SELECT id FROM jobs WHERE status='due' AND run_at<=now() AND lease_until<now()
//               ORDER BY run_at LIMIT 1 FOR UPDATE SKIP LOCKED)
const LEASE_S = 30;
const jobs = [
  { id: "nightly-report", runAt: 0, status: "due", owner: null, leaseUntil: -1, token: 0 },
  { id: "send-digest", runAt: 5, status: "due", owner: null, leaseUntil: -1, token: 0 },
];
const log = (now, msg) => console.log(`t=${String(now).padStart(2)}s ${msg}`);

function claim(worker, now) {
  const job = jobs.find((j) => j.status === "due" && j.runAt <= now && j.leaseUntil < now);
  if (!job) return log(now, `${worker} polls: nothing claimable`);
  Object.assign(job, { owner: worker, leaseUntil: now + LEASE_S, token: job.token + 1 });
  log(now, `${worker} claims ${job.id} (token ${job.token}, lease until t=${job.leaseUntil})`);
}
function renew(worker, id, now) {
  const job = jobs.find((j) => j.id === id);
  if (job.owner === worker) { job.leaseUntil = now + LEASE_S; log(now, `${worker} renews ${id} until t=${job.leaseUntil}`); }
}
function complete(worker, id, token, now) {
  const job = jobs.find((j) => j.id === id);
  if (job.token !== token) return log(now, `${worker} tries to finish ${id} with token ${token}: REJECTED, current token is ${job.token}`);
  job.status = "done";
  log(now, `${worker} finishes ${id} (token ${token}) -> done`);
}

claim("A", 0);
claim("B", 1);                       // the only due job is leased: B skips it
claim("B", 5);
complete("B", "send-digest", 1, 8);
renew("A", "nightly-report", 10);    // heartbeat; then A freezes (long GC pause, lost network)
claim("B", 39);                      // lease still valid until t=40
claim("B", 41);                      // lease expired: B takes over with a new token
complete("B", "nightly-report", 2, 50);
complete("A", "nightly-report", 1, 55); // A wakes up and tries to write its stale result

Result

t= 0s A claims nightly-report (token 1, lease until t=30)
t= 1s B polls: nothing claimable
t= 5s B claims send-digest (token 1, lease until t=35)
t= 8s B finishes send-digest (token 1) -> done
t=10s A renews nightly-report until t=40
t=39s B polls: nothing claimable
t=41s B claims nightly-report (token 2, lease until t=71)
t=50s B finishes nightly-report (token 2) -> done
t=55s A tries to finish nightly-report with token 1: REJECTED, current token is 2

What’s happening

Each job row has a runAt, a status, the current owner, a leaseUntil time and a token that goes up by one on every claim. claim takes the first due job whose lease has expired, which in PostgreSQL is a single UPDATE … WHERE id = (SELECT … FOR UPDATE SKIP LOCKED) as in the comment. SKIP LOCKED lets many pollers run at once without blocking on each other's rows.
At t=0, A claims nightly-report (token 1, lease until 30). At t=1, B finds nothing claimable: the report is leased and send-digest isn't due until 5. At t=5 B claims send-digest and finishes it at t=8.
At t=10 A renews its lease (a heartbeat) to 40, then freezes: a long garbage-collection pause, a VM migration, a network partition. A doesn't know it has stopped. At t=39 B polls and still can't take the report, because the lease is valid until 40.
At t=41 the lease has expired, so B claims the report with token 2 and finishes it at t=50. Without leases, the job would be stuck with a dead owner forever; with a lease, the worst case is a delay of one lease length.
At t=55 A wakes up and tries to record its result with token 1. The table's current token is 2, so the write is rejected. That's a fencing token: an owner whose lease has been taken over can't overwrite the new owner's work. Note what it can't prevent: A may already have done side effects (sent half the report) before freezing, which is why the job itself must be idempotent.
The lease length is a trade-off: short leases recover quickly but need frequent heartbeats and risk false takeovers during ordinary pauses; long leases are quiet but leave a job stuck for longer when its worker really dies. Heartbeat at a third of the lease or so, a common rule of thumb.

AdvancedDelay queues

For one-off delays, the job is stored with its due time and something polls for "everything due by now". A sorted set is the classic structure:

delay-queue.mjs

// A delay queue as a sorted set (Redis: ZADD delayed <dueAt> <job>; ZRANGE delayed -inf <now> BYSCORE).
const delayed = []; // kept sorted by dueAt
function schedule(job, dueAt) {
  const i = delayed.findIndex((d) => d.dueAt > dueAt);
  delayed.splice(i === -1 ? delayed.length : i, 0, { job, dueAt });
}
function pollDue(now) { // take everything due; in Redis do the read + remove atomically (Lua or ZPOPMIN loop)
  const n = delayed.findIndex((d) => d.dueAt > now);
  return delayed.splice(0, n === -1 ? delayed.length : n);
}

schedule("cancel unpaid order 17", 900); // 15 min hold
schedule("reminder: dentist", 3_600);
schedule("retry webhook #4", 40);
schedule("send 'still there?' email", 86_400);

for (const now of [30, 60, 1_000, 4_000]) { // a poller that wakes up every so often
  const due = pollDue(now);
  console.log(`t=${now}s: ${due.map((d) => `${d.job} (due ${d.dueAt}, ${now - d.dueAt}s late)`).join("; ") || "nothing due"}`);
}
console.log(`still waiting: ${delayed.map((d) => d.job).join(", ")}`);

// Estimate: reminders cluster on round times.
const perDay = 100_000_000;
const atNine = perDay * 0.1;  // assume 10% are set for exactly 09:00:00
const dispatchPerSecond = 50_000;
console.log(`average ${Math.round(perDay / 86_400).toLocaleString("en-GB")}/s, but ${atNine.toLocaleString("en-GB")} due at 09:00:00`);
console.log(`at ${dispatchPerSecond.toLocaleString("en-GB")}/s the last one leaves ${atNine / dispatchPerSecond} s late`);

Result

t=30s: nothing due
t=60s: retry webhook #4 (due 40, 20s late)
t=1000s: cancel unpaid order 17 (due 900, 100s late)
t=4000s: reminder: dentist (due 3600, 400s late)
still waiting: send 'still there?' email
average 1,157/s, but 10,000,000 due at 09:00:00
at 50,000/s the last one leaves 200 s late

What’s happening

schedule keeps the list sorted by dueAt, like ZADD delayed <dueAt> <job> in Redis, where the score is the due time. pollDue(now) removes and returns everything with dueAt <= now, like ZRANGE delayed -inf <now> BYSCORE (the older spelling is ZRANGEBYSCORE) followed by removing those members. In Redis the read and the remove must be atomic (a Lua script, or ZPOPMIN in a loop that pushes back anything not yet due), or two pollers can take the same job.
At t=30 nothing is due: the earliest job, the webhook retry, is due at 40. At t=60 it comes out 20 s late, because the poller only woke at 60.
At t=1,000 the 15-minute order hold (due at 900) comes out 100 s late, and at t=4,000 the dentist reminder (due at 3,600) 400 s late. The polling interval bounds the precision: poll every second for second-level accuracy, and accept the load of doing so.
The second part is an estimate of the thundering herd that real schedules produce. 100 million reminders a day average 1,157/s, but people pick round times: if 10% are set for exactly 09:00:00, 10 million jobs are due in the same second. At a dispatch rate of 50,000/s the last one goes out 200 s late.
The fixes are design decisions: spread due times with a little jitter where the product allows ("around 9:00"), shard the sorted set by time bucket and job id so many dispatchers work in parallel, and start dispatching a few seconds early, holding each job until its exact time.

Option	Good for	Limits and notes (checked October 2026 where a number is given)
Message delay on the queue	Short delays (retries, holds)	SQS delivery delay is at most 15 minutes; in RabbitMQ, a per-message TTL plus a dead-letter exchange, or the community delayed-message plugin
Database table with an index on `run_at`	Anything durable that must commit with business data	Polling load; claim with `FOR UPDATE SKIP LOCKED`
Redis sorted set	High volume, second-level precision	Durability depends on Redis persistence; atomic claim needed
Managed scheduler (EventBridge Scheduler, Cloud Scheduler)	Many one-off or recurring schedules without running pollers	EventBridge Scheduler defaults to 10 million schedules per Region, adjustable (AWS quotas page)
Timing wheel in memory	Millions of short timers inside one process (connection timeouts, Kafka's delayed operations)	Not durable; rebuild from the source of truth after a restart

AdvancedCron at scale

For recurring jobs the same rules apply, plus a few of their own:

Exactly one scheduler, chosen by leader election (a lease in etcd, ZooKeeper or the database), or a scheduler that's already distributed. Kubernetes CronJobs are the common answer in a cluster: concurrencyPolicy is Allow, Forbid (skip a run while the previous one is still going) or Replace (stop the running one), and the docs warn that the scheduling is approximate, so in certain circumstances two Jobs might be created, or none, and Jobs should be idempotent (Kubernetes CronJob documentation).
Decide what happens to missed runs. If the scheduler was down at 02:00, run once when it comes back (catch-up), run every missed instance, or skip? Kubernetes uses startingDeadlineSeconds for this; your own scheduler needs an explicit policy.
Don't schedule every tenant at midnight. "Daily at 00:00" for 50,000 tenants is a self-inflicted thundering herd. Hash each tenant to a minute in the night.
Store times in UTC and convert at the edges. A job "every day at 02:30 local time" runs zero times or twice on daylight-saving changes in many time zones; decide which you want.
Make the job record its own progress (last processed id or date), so a rerun continues instead of starting again, and two overlapping runs don't double the work.

Design a reminder service: "remind me at time T"

I'd store each reminder durably in a table keyed by id, with due_at and status, partitioned by time bucket so a day's reminders are spread over many shards. Dispatchers, one or more per shard, poll for rows due in the next few seconds, claim them with a lease using FOR UPDATE SKIP LOCKED and a fencing token, and push them onto a work queue; workers send the push or email and mark the row done. Delivery is at-least-once, so the send uses the reminder id as an idempotency key. For precision I'd poll every second and pre-fetch slightly early. The scaling problem is clustering on round times, 10 million at 09:00, so I'd size dispatch for that peak, shard by time and id, and allow a small product-level spread. For short delays like "cancel the order after 15 minutes" I'd use the queue's own delay instead, since SQS supports up to 15 minutes. The full case study is Design a distributed job scheduler.

Lease and a long pause

A worker holds a 30 s lease and renews it every 10 s. Right after a renewal at t=100, it pauses for 45 s. What happens, and what stops it from corrupting the result?

Show answer

The lease expires at t=130, another worker claims the job with a higher token, and the paused worker's late write is rejected by the fencing token.

What’s happening

The renewal at t=100 extends the lease to t=130. The pause lasts until t=145, so no further renewal happens.
From t=130 the lease is expired, and the next poller to look (say at t=131) claims the job, incrementing the token from, say, 7 to 8.
At t=145 the first worker resumes, with no idea that 45 s have passed. If it now writes its result, the table checks the token: 7 is not the current 8, so the write is refused.
The token stops the stale write, but not work the first worker did before the pause, such as an email already sent. That's why the job must also be idempotent, for example by recording "email for job 42 sent" under the job id before or as it sends.