System Design NotesRohit’s interview study guide
Chapter 08

Building Blocks: IDs, Hashing and Probabilistic Structures

The small parts that case studies keep reaching for: unique IDs, short codes, Bloom filters, HyperLogLog, count-min sketches, geohashes, quadtrees, tries and Merkle trees, each as a model you can run and defend.

7 topics
Parts marked Advanced are extra depth. Skip them on a first read or a quick revision.

#Unique ID generation

Added

Every row, message, order and upload needs a name that no other one will ever have. On one database that's easy: the database counts 1, 2, 3… for you. The problem starts when many machines create records at the same time and none of them should have to ask anyone else first. Asking costs a network round trip on every insert, and whatever you ask becomes a single point of failure. So the question an interviewer is really asking with "how do you generate IDs?" is: how do you make names unique without coordination, and what else do you want the name to tell you?

A good mental model is a ticket roll stamped with the time. Give every worker its own roll with its own colour (the machine ID). Each ticket it tears off is stamped with the current millisecond and a running number for that millisecond. Two workers can never print the same ticket, because their colours differ; one worker can never print the same ticket twice, because either the time or the running number has moved on. Sort the tickets and you get them back roughly in the order they were printed. That's Snowflake. A UUID v4 is the opposite approach: no roll, no colour, just a number so large and so random that a repeat is never going to happen in practice.

Here are the choices you'll be expected to compare:

ApproachSizeSortable by timeNeeds coordinationWhat it leaks
Database auto-increment8 bytes (BIGINT)YesEvery insert goes to the one database that owns the counterVolume: ID 1,000,000 says you have a million rows
Ticket server (Flickr, 2010)8 bytesYesOne round trip to a small counter database per ID (or per batch)Volume
UUID v416 bytesNoNoneNothing: 122 random bits
UUID v716 bytesYes (to the millisecond)NoneCreation time to the millisecond
Snowflake8 bytesYes (to the millisecond)Only to hand out machine IDs onceCreation time, which machine, how busy it was that millisecond
Snowflake · 64 bits (one bit = 10 px) timestamp: ms since custom epoch DC worker sequence 0 41 bits ≈ 69.7 years 5 + 5 bits 12 bits: 4,096 per ms UUID v7 · 128 bits (one bit = 5 px) unix_ts_ms 7 rand_a rand_b 48 bits of Unix ms ver 12 var 62 bits · 74 random bits in all UUID v4 · 128 bits random 4 random random · 122 random bits in all
Snowflake packs time, place and a counter into 64 bits. UUID v7 puts time first and fills the rest with randomness. UUID v4 is all randomness.

The layouts above are the published ones. Twitter announced Snowflake in 2010: one unused sign bit, 41 bits of milliseconds since its own epoch (1288834974657, which is 4 November 2010), 10 bits of machine ID (the original Scala code split it into 5 bits of datacenter and 5 of worker) and a 12-bit sequence. UUID v7 and v4 are defined in RFC 9562 (May 2024, which replaced RFC 4122): v7 is a 48-bit big-endian Unix millisecond timestamp, the version nibble 0111, 12 bits called rand_a, the variant bits 10, and 62 bits called rand_b. The RFC says implementations "SHOULD utilize UUIDv7 instead of UUIDv1 and UUIDv6 if possible".

Here is a Snowflake generator. The clock is injected, so the run is the same every time and you can freeze time or step it backwards on purpose:

snowflake.mjs
// A Snowflake generator: 1 unused bit | 41 bits ms since epoch | 5 datacenter | 5 worker | 12 sequence.
const EPOCH = 1288834974657n; // Twitter's epoch: 2010-11-04T01:42:54.657Z

class Snowflake {
  constructor(datacenter, worker, clock) {
    this.dc = BigInt(datacenter);
    this.worker = BigInt(worker);
    this.clock = clock; // injected so the run is reproducible
    this.lastMs = -1n;
    this.seq = 0n;
  }
  next() {
    let ms = BigInt(this.clock.now());
    if (ms < this.lastMs) throw new Error(`clock moved back ${this.lastMs - ms} ms; refusing to issue IDs`);
    if (ms === this.lastMs) {
      this.seq = (this.seq + 1n) & 0xfffn; // 12 bits: 0..4095
      if (this.seq === 0n) {
        // 4096 IDs already issued this millisecond: wait for the next one
        while (ms <= this.lastMs) ms = BigInt(this.clock.waitTick());
      }
    } else this.seq = 0n;
    this.lastMs = ms;
    return ((ms - EPOCH) << 22n) | (this.dc << 17n) | (this.worker << 12n) | this.seq;
  }
}

const decode = (id) => ({
  time: new Date(Number((id >> 22n) + EPOCH)).toISOString(),
  datacenter: Number((id >> 17n) & 31n),
  worker: Number((id >> 12n) & 31n),
  sequence: Number(id & 0xfffn),
});

// Fake clock: frozen at one millisecond until someone waits.
let t = Date.UTC(2026, 9, 24, 12, 0, 0, 0);
const clock = { now: () => t, waitTick: () => ++t };
const gen = new Snowflake(3, 17, clock);

const ids = Array.from({ length: 5000 }, () => gen.next());
const show = (label, id) => console.log(label, String(id), JSON.stringify(decode(id)));
show("id 1:   ", ids[0]);
show("id 4096:", ids[4095]);
show("id 4097:", ids[4096]);
console.log("all unique:", new Set(ids).size === ids.length);
console.log("sorted by creation:", ids.every((id, i) => i === 0 || id > ids[i - 1]));
console.log("digits:", ids[0].toString().length, "| fits in a signed 64-bit long:", ids[0] < 2n ** 63n);

t -= 5; // NTP steps the clock back 5 ms
try {
  gen.next();
} catch (e) {
  console.log("after clock step back:", e.message);
}
const yearsOf41Bits = Number(2n ** 41n) / (365.25 * 24 * 3600 * 1000);
console.log("41-bit ms lasts", yearsOf41Bits.toFixed(1), "years -> runs out in", new Date(Number(EPOCH) + 2 ** 41).getUTCFullYear());
Result
id 1:    2113963715589509120 {"time":"2026-10-24T12:00:00.000Z","datacenter":3,"worker":17,"sequence":0}
id 4096: 2113963715589513215 {"time":"2026-10-24T12:00:00.000Z","datacenter":3,"worker":17,"sequence":4095}
id 4097: 2113963715593703424 {"time":"2026-10-24T12:00:00.001Z","datacenter":3,"worker":17,"sequence":0}
all unique: true
sorted by creation: true
digits: 19 | fits in a signed 64-bit long: true
after clock step back: clock moved back 5 ms; refusing to issue IDs
41-bit ms lasts 69.7 years -> runs out in 2080
What’s happening
  1. The first call reads the fake clock: 2026-10-24T12:00:00.000Z, which is 1792843200000 ms since 1970. Subtracting Twitter's epoch leaves 504008225343 ms; shifting that left by 22 bits puts it in the top 41 bits, and the datacenter (3 << 17) and worker (17 << 12) are ORed in below it. The sequence is 0, so the first ID is 2113963715589509120.
  2. The clock is frozen, so calls 2 to 4096 all see the same millisecond. Each one takes the ms === this.lastMs branch and bumps seq: 0 → 1 → … → 4095. The IDs differ only in their last 12 bits, which is why ID 4096 is exactly 4095 more than ID 1 (…509120 → …513215).
  3. Call 4097 would need sequence 4096, which doesn't fit in 12 bits: (4095 + 1) & 0xfff wraps to 0. Reusing 0 in the same millisecond would duplicate ID 1, so the generator waits for the next millisecond (waitTick) and issues …593703424 stamped .001. That wait is the throughput limit: one worker can issue 4,096 IDs per millisecond, about 4.1 million per second, and then it stalls.
  4. All 5,000 IDs are unique and strictly increasing, because each one is "later time, or same time and bigger sequence". That's the property that makes Snowflakes good primary keys: new rows always land at the right-hand end of the index.
  5. Then the clock steps back 5 ms, as it can when NTP corrects a fast clock. If the generator carried on, it would re-enter milliseconds it already used and could repeat an ID it has already handed out. This version refuses and throws; the alternatives are below.
  6. 41 bits of milliseconds is 2^41 / (365.25 × 86,400,000) = 69.7 years, so Twitter's IDs run out in 2080. A custom epoch exists precisely to spend those 69 years from your launch date rather than from 1970 (counting from 1970, a 41-bit timestamp would already have run out in 2039).
AdvancedWhen the clock goes backwards

This is the question interviewers probe once you've drawn the bit layout, because it's where Snowflake depends on something it can't control. Wall clocks on servers are corrected by NTP, and a correction can step the clock backwards (see Why distributed systems are hard). Your options:

  • Refuse until the clock catches up (what the model does, and what Twitter's original did: it threw an exception). Fine for a step of a few milliseconds; a step of a minute means a minute of errors from that worker.
  • Wait it out if the step is small: sleep lastMs - now milliseconds and carry on. Cap the wait (say 10 ms) and refuse beyond that.
  • Keep your own monotonic clock: never let ms go below lastMs; if the wall clock is behind, keep issuing from lastMs with the sequence, borrowing from the future. IDs stay unique and ordered; the timestamp inside them is slightly wrong for a while. This is the same idea as a hybrid logical clock.
  • Run the clock with slewing, not stepping: configure NTP or chrony to slew (speed up or slow down gradually) so time never goes backwards on a running machine. This removes most of the problem at the source.
AdvancedUUID v4 vs UUID v7

UUIDs need no machine IDs at all. A v4 is 122 random bits, and the chance of two ever colliding is negligible (the birthday maths below shows why: with 122 bits, you'd need around 2.7 × 10^18 of them for even a 50% chance of a single repeat). The cost is that they are 16 bytes and completely unordered, and that matters for databases. A v7 keeps 74 random bits for uniqueness and spends the first 48 on the time, so it sorts by creation time like a Snowflake while needing no coordination like a v4.

The model builds both by hand from the RFC layout (with a seeded random source so the output is reproducible), then checks Node's own generators:

uuid-v7.mjs
import { randomUUID, randomUUIDv7 } from "node:crypto";

// Seeded PRNG (mulberry32) so the "random" bytes are the same on every run.
const mulberry32 = (a) => () => {
  a = (a + 0x6d2b79f5) | 0;
  let t = Math.imul(a ^ (a >>> 15), 1 | a);
  t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
  return ((t ^ (t >>> 14)) >>> 0) / 2 ** 32;
};
const rand = mulberry32(42);
const randomBytes = (n) => Uint8Array.from({ length: n }, () => Math.floor(rand() * 256));
const hex = (b) => [...b].map((x) => x.toString(16).padStart(2, "0")).join("");
const format = (b) => hex(b).replace(/^(.{8})(.{4})(.{4})(.{4})(.{12})$/, "$1-$2-$3-$4-$5");

// RFC 9562, section 5.7: 48-bit unix_ts_ms | ver=0111 | 12-bit rand_a | var=10 | 62-bit rand_b
function uuidv7(ms) {
  const b = randomBytes(16);
  for (let i = 0; i < 6; i++) b[i] = Number((BigInt(ms) >> BigInt(8 * (5 - i))) & 0xffn);
  b[6] = (b[6] & 0x0f) | 0x70; // version 7 in the top 4 bits of octet 6
  b[8] = (b[8] & 0x3f) | 0x80; // variant 10 in the top 2 bits of octet 8
  return format(b);
}
// RFC 9562, section 5.4: 122 random bits, then the same version (0100) and variant bits
function uuidv4() {
  const b = randomBytes(16);
  b[6] = (b[6] & 0x0f) | 0x40;
  b[8] = (b[8] & 0x3f) | 0x80;
  return format(b);
}

const start = Date.UTC(2026, 9, 24, 12, 0, 0);
const v7 = [0, 1, 1, 2, 5].map((dt) => uuidv7(start + dt)); // created in this order
const v4 = Array.from({ length: 5 }, uuidv4);
console.log("v7, in creation order:");
v7.forEach((u) => console.log(" ", u));
console.log("v7 sorted as strings == creation order?", JSON.stringify([...v7].sort()) === JSON.stringify(v7));
console.log("v4, in creation order:");
v4.forEach((u) => console.log(" ", u));
console.log("v4 sorted as strings == creation order?", JSON.stringify([...v4].sort()) === JSON.stringify(v4));

const ms = parseInt(v7[0].replace(/-/g, "").slice(0, 12), 16); // the first 48 bits
console.log("timestamp read back from v7[0]:", new Date(ms).toISOString());
console.log("version nibble:", v7[0][14], "| variant bits of", v7[0][19], "=", parseInt(v7[0][19], 16).toString(2).slice(0, 2));

// 100 IDs inside ONE millisecond: the timestamp ties, so order falls to the random bits.
const burst = Array.from({ length: 100 }, () => uuidv7(start + 9));
const outOfOrder = burst.filter((u, i) => i > 0 && u < burst[i - 1]).length;
console.log("same-millisecond burst: out of order", outOfOrder, "of 99 neighbours");

// Node's own generators (randomUUIDv7 needs Node 26.1+ or 24.16+): only print what is stable between runs.
const a = randomUUIDv7(), b = randomUUID();
console.log("node randomUUIDv7 version:", a[14], "| node randomUUID version:", b[14]);
console.log("random bits: v4 =", 128 - 4 - 2, "| v7 =", 128 - 48 - 4 - 2);
Result
v7, in creation order:
  01a16dc8-3600-759f-9d78-3fe1be4e3280
  01a16dc8-3601-7708-840f-2fc887062cd7
  01a16dc8-3601-7e98-bea5-354dbddb8234
  01a16dc8-3602-7d16-b16e-f1231c045c7b
  01a16dc8-3605-742d-85d2-1efe90ea6f94
v7 sorted as strings == creation order? true
v4, in creation order:
  569852ba-4b73-47b1-bee3-6e8b4b19b250
  c9e71779-d221-492d-b5b3-396929a75534
  a0e353ca-6421-4d5e-82bc-a2e0e8359a7d
  2f0cb4b5-4a46-4607-b8fe-92fd292b8f4a
  656f6baf-6656-4553-be9c-8b963c534c52
v4 sorted as strings == creation order? false
timestamp read back from v7[0]: 2026-10-24T12:00:00.000Z
version nibble: 7 | variant bits of 9 = 10
same-millisecond burst: out of order 45 of 99 neighbours
node randomUUIDv7 version: 7 | node randomUUID version: 4
random bits: v4 = 122 | v7 = 74
What’s happening
  1. uuidv7(ms) starts from 16 random bytes, then overwrites bytes 0–5 with the timestamp, most significant byte first. 1792843200000 in hex is 01a16dc83600, which is exactly the first 12 hex digits of the first v7 (01a16dc8-3600). Byte 6 gets 0111 in its top nibble, so the 13th hex digit is always 7; byte 8 gets 10 in its top two bits, so the 17th hex digit is always 8, 9, a or b.
  2. The five v7s were made 0, 1, 1, 2 and 5 ms after the start, and their prefixes go …3600, …3601, …3601, …3602, …3605. Because the time sits in the most significant bits and the hex is fixed-width, plain string sorting gives creation order: true. The v4s have no time in them, so sorting scrambles them: false.
  3. The timestamp can be read back from the first 48 bits (2026-10-24T12:00:00.000Z). That's a feature for debugging and for range queries ("rows created today" is a key range), and a leak if creation time is sensitive.
  4. The two IDs made in the same millisecond (…3601-7708 and …3601-7e98) happen to be in order, but that was luck: in the burst of 100 IDs in one millisecond, 45 of 99 neighbours are out of order, as you'd expect from coin flips. Within a millisecond, a plain v7 is only as ordered as its random bits.
  5. Node 26 has both generators built in: crypto.randomUUID() gives version 4 and crypto.randomUUIDv7() gives version 7 (the release notes list it in Node 26.1.0 on 7 May 2026, and backported to the 24.x LTS line in 24.16.0 on 21 May 2026). The last line is the trade: v7 gives up 48 random bits (122 → 74) to buy time ordering; 74 random bits per millisecond is still far more than enough to avoid collisions.

RFC 9562 (section 6.2) knows about that same-millisecond disorder and offers three fixes: a counter in rand_a that increments within the millisecond (method 1), treating the random bits as a counter that grows by a random step (method 2), or putting a sub-millisecond clock fraction in the 12 bits of rand_a (method 3). Implementations differ, so check yours:

  • PostgreSQL 18 (released 25 September 2025) added uuidv7(). Its source uses method 3, with 12 bits of sub-millisecond fraction, and guarantees each value is greater than the previous one within the same backend (connection). Running uuidv7() 100,000 times in one query on PostgreSQL 18.6 gave 0 out-of-order neighbours.
  • Node's crypto.randomUUIDv7() is not monotonic within a millisecond. 200,000 calls in a tight loop on Node 26.7 gave 100,138 out-of-order neighbours (about half, as with the model's burst).

That matters less than it sounds. Within one millisecond the order is meaningless anyway, and B-tree locality only needs IDs to be roughly increasing. It matters if you use the ID itself as a cursor or a "happened before" signal. Then you need a monotonic generator, and even that only orders IDs from one generator, never across machines.

AdvancedWhy random keys hurt a B-tree

This is the strongest argument for v7 or Snowflake over v4, so it's worth seeing in numbers. A B-tree index keeps keys sorted in fixed-size pages. A new time-ordered key always goes into the rightmost page, which is hot in memory. A random key goes to a random page, which is probably not in memory, and that page is often full, so it splits. The model inserts 200,000 keys into the leaf level of a B-tree (100 keys per page) with a 64-page buffer cache, and counts page reads and how full the pages end up:

index-locality.mjs
// Insert 200,000 keys into a B-tree's leaf level (100 keys per page) with a 64-page buffer cache.
// Compare random keys (UUID v4) with time-ordered keys (UUID v7, Snowflake, auto-increment).
const mulberry32 = (a) => () => {
  a = (a + 0x6d2b79f5) | 0;
  let t = Math.imul(a ^ (a >>> 15), 1 | a);
  t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
  return ((t ^ (t >>> 14)) >>> 0) / 2 ** 32;
};
const PAGE = 100, CACHE = 64, N = 200_000;

function run(nextKey) {
  const pages = [[]]; // leaf pages in key order; each holds sorted keys
  const cache = new Map(); // LRU of page objects
  let misses = 0;
  const touch = (p) => {
    if (cache.has(p)) cache.delete(p); else misses++; // a miss = a page read from disk
    cache.set(p, true);
    if (cache.size > CACHE) cache.delete(cache.keys().next().value);
  };
  for (let i = 0; i < N; i++) {
    const key = nextKey(i);
    let lo = 0, hi = pages.length - 1; // last page whose first key <= key
    while (lo < hi) { const mid = (lo + hi + 1) >> 1; if (pages[mid][0] <= key) lo = mid; else hi = mid - 1; }
    const page = pages[lo];
    touch(page);
    let j = page.length; while (j > 0 && page[j - 1] > key) j--;
    page.splice(j, 0, key);
    if (page.length > PAGE) {
      // Appending at the far right: start a fresh page and leave this one full (what B-trees do).
      // Anywhere else: split the page in half.
      const rightmostAppend = lo === pages.length - 1 && j === page.length - 1;
      const cut = rightmostAppend ? PAGE : PAGE / 2;
      pages.splice(lo + 1, 0, page.splice(cut));
    }
  }
  const fill = N / (pages.length * PAGE);
  return `${pages.length} pages, ${(fill * 100).toFixed(0)}% full, ${misses} page reads (${((misses / N) * 100).toFixed(1)}% of inserts)`;
}

const rand = mulberry32(7);
console.log("random (v4)       :", run(() => rand()));
console.log("time-ordered (v7) :", run((i) => i + rand() * 0.5)); // increasing, a little jitter
Result
random (v4)       : 2880 pages, 69% full, 178014 page reads (89.0% of inserts)
time-ordered (v7) : 2000 pages, 100% full, 2000 page reads (1.0% of inserts)
What’s happening
  1. Each insert finds its leaf page by binary search on the pages' first keys, "reads" the page through the LRU cache (a miss counts as a disk read), and inserts the key in sorted position. A page that overflows 100 keys splits.
  2. With random keys, the target page is anywhere among thousands, and only 64 fit in the cache, so 89% of inserts need a page read: 178,014 reads for 200,000 inserts. On a real database each of those is a random I/O, plus write-ahead log and checkpoint work for every page dirtied.
  3. Random inserts also split pages in the middle, leaving two half-full pages that fill up slowly. The index ends at 69% full (2,880 pages), the classic figure for random B-tree inserts: about 44% more pages, and more memory to cache the same data.
  4. With increasing keys, every insert lands on the rightmost page. When it fills, a B-tree starts a new page and leaves the old one full instead of splitting it in half. So there are exactly 2,000 page reads (one per new page), and 100% fill with 2,000 pages. The model simplifies here: PostgreSQL's B-trees leave 10% free on that rightmost split (the default leaf fillfactor is 90), so a real index ends nearer 90% full.
  5. This is why "UUID v4 primary keys are slow" is a real complaint once the index outgrows memory, and why v7 fixes it without giving up coordination-free generation.
AdvancedKey size in the index

Size matters too, especially on MySQL, where the MySQL 8.4 manual says "each record in a secondary index contains the primary key columns for the row". An estimate of the key bytes alone:

id-sizes.mjs
// What the choice of key type costs in index space, before page overhead.
// Assumptions: 1 billion rows; the primary key appears in the primary index and is copied into
// each of 3 secondary indexes (InnoDB secondary indexes store the primary key) and into 2 tables that reference it.
const rows = 1e9, copies = 1 + 3 + 2;
const types = { "BIGINT (Snowflake, auto-increment)": 8, "UUID as BINARY(16) / Postgres uuid": 16, "UUID as CHAR(36) text": 36 };
for (const [name, bytes] of Object.entries(types)) {
  const gib = (rows * copies * bytes) / 2 ** 30;
  console.log(`${name.padEnd(36)} ${String(bytes).padStart(2)} B x ${copies} copies -> ${gib.toFixed(0).padStart(4)} GiB of key bytes`);
}
Result
BIGINT (Snowflake, auto-increment)    8 B x 6 copies ->   45 GiB of key bytes
UUID as BINARY(16) / Postgres uuid   16 B x 6 copies ->   89 GiB of key bytes
UUID as CHAR(36) text                36 B x 6 copies ->  201 GiB of key bytes
What’s happening
  1. The assumptions are in the comment: a billion rows, and the key stored six times (primary index, three secondary indexes that carry the primary key, two foreign keys elsewhere).
  2. A 64-bit key costs 45 GiB across those copies; a binary UUID doubles it to 89 GiB. That's a real difference, but rarely the deciding one on its own.
  3. Storing UUIDs as 36-character text costs 201 GiB, 4.5 times the BIGINT. That's a mistake, not a trade-off: store them as PostgreSQL's native uuid type or MySQL BINARY(16).
Advanced64-bit IDs in JavaScript clients
id-precision.mjs
// A 64-bit ID sent as a JSON number loses digits in JavaScript: Number has 53 bits of integer precision.
const body = '{"id": 2113963715589509121, "id_str": "2113963715589509121"}';
const parsed = JSON.parse(body);
console.log("as number:", parsed.id, "| exact?", String(parsed.id) === parsed.id_str);
console.log("as string:", parsed.id_str, "-> BigInt", BigInt(parsed.id_str));
console.log("largest safe integer:", Number.MAX_SAFE_INTEGER, `(2^53 - 1, ${String(Number.MAX_SAFE_INTEGER).length} digits)`);
console.log("neighbours collapse:", 2113963715589509121 === 2113963715589509120);

// A recent JavaScript addition (TC39 "JSON.parse source text access"): the reviver sees the raw text.
const exact = JSON.parse(body, (key, value, ctx) => (key === "id" ? BigInt(ctx.source) : value));
console.log("reviver with source:", exact.id);
Result
as number: 2113963715589509000 | exact? false
as string: 2113963715589509121 -> BigInt 2113963715589509121n
largest safe integer: 9007199254740991 (2^53 - 1, 16 digits)
neighbours collapse: true
reviver with source: 2113963715589509121n
What’s happening
  1. JSON.parse turns 2113963715589509121 into the nearest double, which prints as 2113963715589509000. The last digits are gone, and nothing throws.
  2. At this size, doubles are 256 apart, so two different IDs, …121 and …120, become the same number: true. A client could merge two users or fetch the wrong tweet.
  3. Strings survive intact and convert to BigInt when you need arithmetic. The rule for an API: send 64-bit IDs as strings. The newer JSON.parse reviver context (ctx.source, which works on the Node 26 used here) can rescue a number field, but you can't count on every client having it.
AdvancedTicket servers and database sequences

Before Snowflake, Flickr's 2010 post "Ticket Servers: Distributed Unique Primary Keys on the Cheap" described the simplest coordinated scheme: a tiny MySQL database whose only job is a table with one row and an auto-increment column. Each request does REPLACE INTO Tickets64 (stub) VALUES ('a') and reads LAST_INSERT_ID(). For availability they ran two ticket servers, one issuing odd numbers and one even (write-ups describe auto-increment-increment = 2 with offsets 1 and 2). It works, it's 64 bits and roughly ordered, but every ID costs a round trip, and you have a database to run just for counting. The common improvement is to fetch a block (+1000 at a time) and hand IDs out from memory: one round trip per thousand IDs, at the cost of gaps when a server restarts with an unused block.

Instagram's scheme (from its engineering post "Sharding & IDs at Instagram") is a third variant worth knowing because it reuses the database you already have: 41 bits of milliseconds, 13 bits of logical shard ID and 10 bits of a per-shard sequence (mod 1024), generated by a PL/pgSQL function inside each PostgreSQL shard. The shard ID inside the ID means you can find a row's shard from its ID alone. Discord's IDs are Snowflakes too, with its own epoch (the first second of 2015).

How many IDs can a Snowflake deployment issue per second?

With the standard layout (10 bits of machine ID, 12 bits of sequence), what's the maximum rate for one worker, and for the whole deployment?

Show answer

One worker: 4,096,000 per second. The whole deployment: about 4.19 billion per second.

What’s happening
  1. The sequence has 12 bits, so one worker can issue 2^12 = 4,096 IDs in one millisecond before it must wait for the clock to tick.
  2. There are 1,000 milliseconds in a second, so one worker tops out at 4,096 × 1,000 = 4,096,000 IDs per second.
  3. The machine field has 10 bits, so at most 2^10 = 1,024 workers can run at once without sharing a number.
  4. 4,096,000 × 1,024 = 4,194,304,000 IDs per second in total. You'd run out of CPU long before that; the real limits are per-worker bursts and the number of worker IDs. If you need more workers, take bits from the sequence (fewer IDs per millisecond each), and if you need more years, take bits from the machine field. The layout is a budget you can re-split.

#Base62 and short codes

Added

A URL shortener, a paste site, an invite link and a voucher all need the same thing: a short, URL-safe string that maps to one record. The usual answer is base62: write a number using the 62 characters 0–9, A–Z and a–z. They're safe in a URL path without escaping, and they pack about 5.95 bits into each character (log2 62), against 3.3 bits for a decimal digit. So the question splits in two: how do you turn a number into a code (base62, which is simple), and where does the number come from (a counter, randomness or a hash, which is where the design is).

The mental model is an odometer with 62 symbols on each wheel. Adding one turns the rightmost wheel; after z it rolls over to 0 and turns the next wheel, exactly like 9 → 10 in decimal. Seven wheels give 62^7, about 3.5 trillion positions. Encoding is "which position is each wheel at?" (repeated division by 62), and decoding is reading the wheels back (multiply by 62 and add, wheel by wheel).

base62.mjs
const ALPHABET = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz";
const BASE = 62n;

function encode(n) {
  n = BigInt(n);
  if (n === 0n) return ALPHABET[0];
  let out = "";
  while (n > 0n) {
    out = ALPHABET[Number(n % BASE)] + out; // least significant digit first, so prepend
    n /= BASE; // BigInt division truncates
  }
  return out;
}

function decode(s) {
  let n = 0n;
  for (const ch of s) {
    const d = ALPHABET.indexOf(ch);
    if (d < 0) throw new Error(`not base62: ${JSON.stringify(ch)}`);
    n = n * BASE + BigInt(d);
  }
  return n;
}

for (const n of [0, 61, 62, 3843, 3844, 1_000_000_000, 125_000_000_000]) {
  const code = encode(n);
  console.log(String(n).padStart(12), "->", code.padEnd(7), "->", String(decode(code)));
}

for (let len = 6; len <= 8; len++) console.log(`${len} chars hold`, (62n ** BigInt(len)).toLocaleString("en-US"), "codes");

try {
  decode("abc-1");
} catch (e) {
  console.log("decode('abc-1'):", e.message);
}
Result
           0 -> 0       -> 0
          61 -> z       -> 61
          62 -> 10      -> 62
        3843 -> zz      -> 3843
        3844 -> 100     -> 3844
  1000000000 -> 15ftgG  -> 1000000000
125000000000 -> 2CRTGCG -> 125000000000
6 chars hold 56,800,235,584 codes
7 chars hold 3,521,614,606,208 codes
8 chars hold 218,340,105,584,896 codes
decode('abc-1'): not base62: "-"
What’s happening
  1. encode(61): 61 % 62 is 61, the last character of the alphabet, z; 61 / 62 truncates to 0, so the loop stops. One wheel, at its last position.
  2. encode(62): 62 % 62 = 0 gives 0, then 62 / 62 = 1 gives 1, and because each new digit is prepended, the result is 10, the odometer rolling over. Likewise 3843 (61 × 62 + 61) is zz and 3844 is 100.
  3. A billion needs only 6 characters (15ftgG) and 125 billion needs 7 (2CRTGCG). decode reverses it: for 15ftgG it computes ((((1 × 62 + 5) × 62 + 15) × 62 + …, getting the exact number back each time.
  4. The capacity lines are the numbers to remember: 6 characters hold 56.8 billion, 7 hold 3.52 trillion, 8 hold 218 trillion. A shortener creating 100 million links a month uses 1.2 billion a year, so 7 characters last thousands of years if you count, but (as shown below) not if you pick at random without checking.
  5. BigInt is used because the codes for large numbers go past 2^53, and Number division would start rounding. decode rejects anything outside the alphabet (- here) instead of silently mapping it to -1.
AdvancedWhere the number comes from
Counter next ID range per server permute optional, keyed base62 c9Zl8kL never collides · one write · needs a counter Random 7 random chars secure RNG INSERT … unique index on code the database decides who got it code taken: retry Hash SHA-256(URL) same URL, same code INSERT … clash with another URL? add a salt and hash again code
A counter makes uniqueness free; random and hashed codes rely on a unique index to catch the clashes the birthday maths promises.

There are three families, and the design decision is between them:

Source of the numberCollisionsGuessableSame URL twiceCost per new code
Counter (database sequence, ticket server, Snowflake, a range handed to each server)NeverYes, unless permutedTwo codes (unless you look the URL up first)One write
Random 7 charactersYes, and more as the table fillsNo (with a secure RNG)Two codesOne write, plus a retry on a clash
Hash of the URL, truncatedYes, same maths as randomNo (but anyone can compute the code for a known URL)Same code, free deduplicationOne write, plus a re-hash on a clash

The model below shows what each looks like for three consecutive IDs, including the trap in the "just multiply it" scramble and a proper keyed permutation:

short-codes.mjs
import { createHash } from "node:crypto";

const ALPHABET = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz";
const encode = (n, len = 7) => {
  let out = "";
  for (let i = 0; i < len; i++, n /= 62n) out = ALPHABET[Number(n % 62n)] + out; // fixed width
  return out;
};
const decode = (s) => [...s].reduce((n, ch) => n * 62n + BigInt(ALPHABET.indexOf(ch)), 0n);
const SPACE = 62n ** 7n; // 3.5 trillion 7-char codes
const ids = [1_000_000_000n, 1_000_000_001n, 1_000_000_002n];

// 1. Plain counter: short and collision-free, but it leaks volume and is easy to enumerate.
console.log("counter:    ", ids.map((n) => encode(n)).join(" "));

// 2. Naive scramble: multiply by a constant mod 62^7. A bijection, but the low digits still count up.
console.log("multiply:   ", ids.map((n) => encode((n * 2_654_435_761n) % SPACE)).join(" "));

// 3. Feistel permutation on 42 bits (2^42 > 62^7), "cycle-walking" until the result fits in 62^7.
const KEYS = [0x2f1a3, 0x1b7e9, 0x0c4d5, 0x3a961]; // the secret
const MASK = (1n << 21n) - 1n;
const round = (half, key) => BigInt(Math.imul(Number(half) ^ key, 0x45d9f3b) >>> 11) & MASK;
const feistel = (x, keys) => {
  let L = x >> 21n, R = x & MASK;
  for (const k of keys) [L, R] = [R, L ^ round(R, k)];
  return (R << 21n) | L; // swap halves at the end so the same code with reversed keys inverts it
};
const permute = (x, keys) => { do x = feistel(x, keys); while (x >= SPACE); return x; };
const toCode = (id) => encode(permute(id, KEYS));
const toId = (code) => permute(decode(code), [...KEYS].reverse());
const codes = ids.map(toCode);
console.log("feistel:    ", codes.join(" "), "-> ids", codes.map(toId).join(", "));

const sample = Array.from({ length: 100_000 }, (_, i) => toCode(BigInt(i)));
console.log("feistel on ids 0..99,999: distinct codes", new Set(sample).size, "| all decode back:", sample.every((c, i) => toId(c) === BigInt(i)));

// 4. Hash of the URL: the same URL always gets the same code, but two URLs can share one.
const hashCode = (url) => encode(BigInt("0x" + createHash("sha256").update(url).digest("hex")) % SPACE);
console.log("hash:       ", hashCode("https://example.com/a"), hashCode("https://example.com/a"), hashCode("https://example.com/b"));
Result
counter:     015ftgG 015ftgH 015ftgI
multiply:    K8B5lgG KB4jVgH KDyNFgI
feistel:     c9Zl8kL nvw5BpY kk2mpWe -> ids 1000000000, 1000000001, 1000000002
feistel on ids 0..99,999: distinct codes 100000 | all decode back: true
hash:        4B9m7rA 4B9m7rA pv1bEGR
What’s happening
  1. Counter: IDs 1,000,000,000 to 1,000,000,002 become 015ftgG, 015ftgH, 015ftgI (padded to 7 characters). Unique by construction, but anyone can see you've made about a billion links, and anyone can walk 015ftgJ, 015ftgK… to scrape every link, including ones people assumed were private.
  2. Multiply mod 62^7: multiplying by a constant that shares no factor with 62^7 (it's odd and not a multiple of 31) is a bijection, so it can't collide. But look at the endings: …gG, …gH, …gI. The last base62 digit of (id × A) mod 62^7 depends only on id mod 62, so the low digits still count up in step with the ID, and the high digits drift slowly too. It looks scrambled at a glance and isn't.
  3. Feistel: the 42-bit number is split into two 21-bit halves, and each of four rounds swaps them and mixes one half into the other with a keyed function. Every round is reversible, so the whole thing is a permutation of all 2^42 values. Since 2^42 (4.4 trillion) is a bit bigger than 62^7 (3.5 trillion), a result that falls outside the code space is fed through again (cycle-walking) until it lands inside, which takes about 1.25 tries on average. Neighbouring IDs give unrelated codes (c9Zl8kL nvw5BpY kk2mpWe).
  4. Running the same rounds with the keys reversed undoes the permutation, so toId gets the original IDs back, and the check over 100,000 IDs confirms there are 100,000 distinct codes that all decode correctly. You get counter uniqueness with random-looking codes, and you can even skip storing the code, because it's computable from the ID.
  5. Hash: https://example.com/a gives 4B9m7rA both times, which is free deduplication (the same long URL shortens to the same code). But a hash squeezed into 42 bits behaves exactly like a random draw, so two different URLs will eventually share a code, and the collision maths below applies in full.
AdvancedCollision maths for random and hashed codes

This is the birthday problem: in a room of 23 people, two probably share a birthday, even though there are 365 days. Collisions arrive much sooner than "n reaches N" because every new item can collide with any of the ones already there. For n codes drawn uniformly from N, the expected number of colliding pairs is about n²/2N, and the chance of at least one collision is about 1 − e^(−n²/2N). It reaches 50% at about √(2N ln 2), roughly 1.18√N.

collisions.mjs
// Birthday maths for random (or hashed) short codes: n codes drawn uniformly from N possible.
const fmt = (x) => (x >= 1e7 ? x.toExponential(2) : x.toLocaleString("en-US", { maximumFractionDigits: x < 10 ? 3 : 0 }));

function report(chars, n) {
  const N = 62 ** chars;
  const pairs = (n * (n - 1)) / (2 * N); // expected number of colliding pairs
  const pAny = -Math.expm1(-pairs); // P(at least one collision) ~ 1 - e^(-n^2 / 2N)
  const pNext = n / N; // chance the next new code hits one already taken
  const half = Math.sqrt(2 * N * Math.LN2); // n at which P(any collision) = 50%
  console.log(`${chars} chars, ${fmt(n)} codes in a space of ${fmt(N)}:`);
  console.log(`  expected colliding pairs ${fmt(pairs)} | P(at least one) ${pAny.toFixed(6)}`);
  console.log(`  the next new code is already taken: 1 in ${fmt(Math.round(1 / pNext))} | P(any) reaches 50% at ${fmt(Math.round(half))} codes`);
}
report(7, 1e9);
report(8, 1e9);
report(7, 1e6);

// Check the formula by simulation (seeded): 100,000 random 4-char codes, count draws that hit a taken code.
const mulberry32 = (a) => () => {
  a = (a + 0x6d2b79f5) | 0;
  let t = Math.imul(a ^ (a >>> 15), 1 | a);
  t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
  return ((t ^ (t >>> 14)) >>> 0) / 2 ** 32;
};
const rand = mulberry32(2026);
const N4 = 62 ** 4, n = 100_000, seen = new Set();
let hits = 0;
for (let i = 0; i < n; i++) {
  const code = Math.floor(rand() * N4);
  if (seen.has(code)) hits++; else seen.add(code);
}
console.log(`simulated 4 chars: ${hits} collisions | formula n^2/2N = ${((n * n) / (2 * N4)).toFixed(1)}`);
Result
7 chars, 1.00e+9 codes in a space of 3.52e+12:
  expected colliding pairs 141,980 | P(at least one) 1.000000
  the next new code is already taken: 1 in 3,522 | P(any) reaches 50% at 2,209,524 codes
8 chars, 1.00e+9 codes in a space of 2.18e+14:
  expected colliding pairs 2,290 | P(at least one) 1.000000
  the next new code is already taken: 1 in 218,340 | P(any) reaches 50% at 1.74e+7 codes
7 chars, 1,000,000 codes in a space of 3.52e+12:
  expected colliding pairs 0.142 | P(at least one) 0.132362
  the next new code is already taken: 1 in 3,521,615 | P(any) reaches 50% at 2,209,524 codes
simulated 4 chars: 347 collisions | formula n^2/2N = 338.4
What’s happening
  1. A billion random 7-character codes: the space is 3.52 trillion, so it's only 0.03% full, yet the expected number of colliding pairs is 1e9² / (2 × 3.52e12) ≈ 141,980. A collision is certain (P = 1.000000). In fact there's a 50% chance of one after only 2.2 million codes, about √N.
  2. That's why "pick a random code" must always be "pick, then insert with a unique constraint, and retry on a clash". What the retry costs is the second number: at a billion codes, a new code is already taken 1 time in 3,522. The retry is cheap and rare, but it has to exist.
  3. Eight characters shrink the pairs to 2,290 and make a clash 1 in 218,340, which is still certain to happen somewhere. Adding a character buys a 62× smaller clash rate, never zero.
  4. A million codes in the 7-character space: 0.142 expected pairs, a 13% chance of at least one. Small systems get bitten too, just rarely enough that the bug survives testing.
  5. The simulation checks the formula. 100,000 random 4-character codes (14.8 million possible) hit an already-taken code 347 times, against 338.4 predicted. Random variation of about ±18 (the square root of 338) is expected, so the formula holds.
AdvancedPractical details for short codes

Other things worth saying about short codes:

  • Fixed or variable length: counter codes grow (zz → 100), which is fine, but a fixed width (pad to 7) makes routing and validation simpler and hides how young the system is.
  • Reserved and unlucky codes: random codes will eventually spell words. Filter generated codes against a block list, and reserve paths your app uses (api, login).
  • Custom aliases ("example.com/summer-sale") share the same table, so they go through the same unique index: first writer wins, and the loser gets a "taken" error.
  • Counters without a hot spot: give each app server a range of IDs (1,000 or 1,000,000 at a time) from a small coordinator, as with the ticket server above. That's one coordinator call per range, not per link, and losing a server only wastes the rest of its range.

The full design, with the read path, caching and analytics, is in Design a URL shortener.

How many characters for 100 billion links?

You expect to create 100 billion short links over the system's lifetime, using a counter. What's the minimum code length in base62? And if you used random codes instead, would that length be enough?

Show answer

7 characters for a counter. Random codes would need a unique index and retries at any length, and 7 would be uncomfortably tight.

What’s happening
  1. 6 characters hold 62^6 = 56.8 billion, which is less than 100 billion, so 6 is too short.
  2. 7 characters hold 3.52 trillion, comfortably above 100 billion, so a counter needs 7 (the counter reaches 100,000,000,000, which encodes to 7 characters).
  3. With random codes, the table would end 100e9 / 3.52e12 ≈ 2.8% full, so near the end about 1 in 35 new codes would clash and need a retry, and the expected number of colliding pairs (n²/2N) would be about 1.4 billion. The retry loop would cope, but 8 characters (62 times the space) bring it down to 1 in 2,183.
  4. So the length depends on the source: a counter needs only enough codes to count to n, while random codes need enough room to keep clashes rare at the end of the system's life.

#Bloom filters

Added

A Bloom filter answers one question, "have I seen this before?", using a small fraction of the memory a real set would need, at the price of sometimes saying "yes" when the answer is no. It never says "no" wrongly. So it's a "definitely not, or maybe" structure: when it says no, you skip the expensive check (a disk read, a database query, a network call); when it says maybe, you do the check. Since most lookups in the places it's used are for things that aren't there, the filter saves most of the expensive work.

The mental model is a wall of light switches and a few fingerprints per guest. When a guest arrives, you run their name through k different hash functions, get k switch numbers, and flip those switches on. To ask "was Carol here?", you hash her name the same way and look at her k switches. If any is off, Carol was never here, because her arrival would have turned it on. If all are on, she probably was, but other guests may have turned those switches on between them. Switches are only ever turned on, which is why there are no false negatives and also why you can't remove anyone.

add alice add bob 0 0 1 1 0 2 1 3 0 4 0 5 1 6 0 7 0 8 0 9 0 10 1 11 0 12 0 13 1 14 0 15 check carol maybe (false positive) check dave no: bit 8 is 0
Adding sets k bits; checking reads them. A zero anywhere is a definite no; all ones is only a maybe, because other items may have set them.

Here's a complete filter. It uses one 32-bit hash with two seeds and derives the k positions by double hashing (position i = h1 + i·h2, which Kirsch and Mitzenmacher showed performs as well as k independent hashes). It adds 10,000 items to 100,000 bits, then measures the false-positive rate on 200,000 items that were never added, for several values of k, next to the textbook formula (1 − e^(−kn/m))^k:

bloom.mjs
// A Bloom filter: m bits, k hash functions. Measured false-positive rate vs (1 - e^(-kn/m))^k.
function fnv1a(str, seed) { // 32-bit FNV-1a with a murmur-style finaliser
  let h = 0x811c9dc5 ^ seed;
  for (let i = 0; i < str.length; i++) h = Math.imul(h ^ str.charCodeAt(i), 0x01000193);
  h ^= h >>> 16; h = Math.imul(h, 0x85ebca6b); h ^= h >>> 13; h = Math.imul(h, 0xc2b2ae35);
  return (h ^ (h >>> 16)) >>> 0;
}

class BloomFilter {
  constructor(m, k) {
    this.m = m;
    this.k = k;
    this.bits = new Uint8Array(Math.ceil(m / 8));
  }
  *positions(item) { // double hashing: g_i = h1 + i*h2 (Kirsch & Mitzenmacher)
    const h1 = fnv1a(item, 0), h2 = (fnv1a(item, 0x9e3779b9) | 1) >>> 0;
    for (let i = 0; i < this.k; i++) yield (h1 + i * h2) % this.m;
  }
  add(item) {
    for (const p of this.positions(item)) this.bits[p >> 3] |= 1 << (p & 7);
  }
  mightContain(item) {
    for (const p of this.positions(item)) if (!(this.bits[p >> 3] & (1 << (p & 7)))) return false; // definitely not
    return true; // probably yes
  }
}

const n = 10_000, m = 100_000; // 10 bits per item
console.log(`n = ${n} items, m = ${m} bits (${m / n} bits per item), optimal k = (m/n) ln 2 = ${((m / n) * Math.LN2).toFixed(2)}`);
console.log(" k | measured FP | formula  | false negatives");
for (const k of [1, 2, 3, 5, 7, 10, 14]) {
  const bf = new BloomFilter(m, k);
  for (let i = 0; i < n; i++) bf.add(`user:${i}`);
  let fn = 0;
  for (let i = 0; i < n; i++) if (!bf.mightContain(`user:${i}`)) fn++;
  let fp = 0;
  const probes = 200_000; // items that were never added
  for (let i = 0; i < probes; i++) if (bf.mightContain(`other:${i}`)) fp++;
  const formula = (1 - Math.exp((-k * n) / m)) ** k;
  console.log(`${String(k).padStart(2)} | ${((fp / probes) * 100).toFixed(3).padStart(9)}%  | ${(formula * 100).toFixed(3).padStart(6)}% | ${fn}`);
}
Result
n = 10000 items, m = 100000 bits (10 bits per item), optimal k = (m/n) ln 2 = 6.93
 k | measured FP | formula  | false negatives
 1 |     9.600%  |  9.516% | 0
 2 |     3.242%  |  3.286% | 0
 3 |     1.749%  |  1.741% | 0
 5 |     0.940%  |  0.943% | 0
 7 |     0.808%  |  0.819% | 0
10 |     1.059%  |  1.019% | 0
14 |     1.935%  |  1.898% | 0
What’s happening
  1. positions(item) computes two hashes of the string and yields h1 + i·h2 mod m for i = 0…k−1. add sets those bits (bit p lives in byte p >> 3, at bit p & 7); mightContain returns false at the first zero bit it finds, so a "no" usually costs one or two memory reads.
  2. With k = 1, each item sets one bit, so after 10,000 items about 1 − e^(−0.1) ≈ 9.5% of the bits are set, and a stranger's single bit is set 9.6% of the time (measured) against 9.516% (formula).
  3. More hash functions make a false positive need more bits to line up, so the rate falls (k = 3: 1.75%, k = 5: 0.94%), until k = 7, the optimum, where it bottoms out at 0.81%. The formula for the optimum is k = (m/n) ln 2 = 6.93. At the optimum, about half the bits are set.
  4. Past the optimum, each item sets so many bits that the array fills up: k = 14 is back up to 1.9%. More hashing is not always better.
  5. The measured column tracks the formula to within a few hundredths of a percent at every k, and the false negatives are 0 every time, as they must be: every bit an added item needs was set when it was added, and nothing ever clears a bit.
AdvancedThe bug that gave this filter false negatives
AdvancedSizing a Bloom filter

You size a Bloom filter from two numbers: how many items n it must hold, and the false-positive rate p you can tolerate. Then m = −n ln p / (ln 2)² bits and k = (m/n) ln 2. The useful facts fall out of that: about 9.6 bits per item for 1%, and each extra 4.8 bits per item cuts the rate by 10×, whatever the items are. A URL, a username or a 1 KB document all cost the same 9.6 bits.

bloom-size.mjs
// Sizing a Bloom filter: m = -n ln(p) / (ln 2)^2 bits, k = (m/n) ln 2 hash functions.
const size = (n, p) => {
  const m = Math.ceil((-n * Math.log(p)) / Math.LN2 ** 2);
  const k = Math.round((m / n) * Math.LN2);
  return { bitsPerItem: (m / n).toFixed(2), k, memory: m / 8 };
};
const human = (bytes) => (bytes >= 2 ** 30 ? `${(bytes / 2 ** 30).toFixed(2)} GiB` : `${(bytes / 2 ** 20).toFixed(1)} MiB`);

for (const [what, n] of [["URLs a crawler has seen", 1e9], ["usernames taken", 5e8], ["keys in one SSTable", 1e7]]) {
  for (const p of [0.1, 0.01, 0.001]) {
    const s = size(n, p);
    console.log(`${what.padEnd(23)} n=${n.toExponential(0)} p=${String(p).padEnd(5)} -> ${s.bitsPerItem} bits/item, k=${s.k}, ${human(s.memory)}`);
  }
}
const avgUrl = 80; // bytes per URL if you stored them in a hash set instead (rule of thumb, before overhead)
console.log(`exact set of 1e9 URLs at ${avgUrl} bytes each: ${human(1e9 * avgUrl)} before any hash-table overhead`);
Result
URLs a crawler has seen n=1e+9 p=0.1   -> 4.79 bits/item, k=3, 571.3 MiB
URLs a crawler has seen n=1e+9 p=0.01  -> 9.59 bits/item, k=7, 1.12 GiB
URLs a crawler has seen n=1e+9 p=0.001 -> 14.38 bits/item, k=10, 1.67 GiB
usernames taken         n=5e+8 p=0.1   -> 4.79 bits/item, k=3, 285.7 MiB
usernames taken         n=5e+8 p=0.01  -> 9.59 bits/item, k=7, 571.3 MiB
usernames taken         n=5e+8 p=0.001 -> 14.38 bits/item, k=10, 857.0 MiB
keys in one SSTable     n=1e+7 p=0.1   -> 4.79 bits/item, k=3, 5.7 MiB
keys in one SSTable     n=1e+7 p=0.01  -> 9.59 bits/item, k=7, 11.4 MiB
keys in one SSTable     n=1e+7 p=0.001 -> 14.38 bits/item, k=10, 17.1 MiB
exact set of 1e9 URLs at 80 bytes each: 74.51 GiB before any hash-table overhead
What’s happening
  1. size(n, p) applies the two formulas. For p = 0.01, −ln(0.01) / (ln 2)² = 4.605 / 0.4805 = 9.59 bits per item, with k = 7. Notice that the bits per item depend only on p: the same three numbers (4.79, 9.59, 14.38) repeat for every n.
  2. A crawler that remembers a billion URLs at 1% needs 1.12 GiB, which fits in one server's memory. Storing the URLs themselves, at a rule-of-thumb 80 bytes each, would be 74.5 GiB before any hash-table overhead. That 66× saving is the whole point.
  3. Each 10× improvement in p costs the same 4.8 bits per item: 0.1 → 0.01 → 0.001 is 571 MiB → 1.12 GiB → 1.67 GiB. Pick p by asking what a false positive costs. For a crawler, 1% means wrongly skipping 1% of new URLs. For an SSTable, 1% means an occasional wasted disk read.
  4. For a single SSTable of 10 million keys, the filter is 11.4 MiB at 1%. A database keeps one per file in memory, which is why Cassandra lets you tune it per table (below).
AdvancedWhere Bloom filters earn their place
  • LSM-tree storage engines (Cassandra, RocksDB, HBase, ScyllaDB). A read may have to check many immutable SSTable files on disk. A per-file Bloom filter in memory answers "this key is definitely not in this file" without touching the disk. Apache Cassandra's documentation gives bloom_filter_fp_chance defaults of 0.01 for most compaction strategies and 0.1 for LeveledCompactionStrategy (Cassandra 4.0 docs, checked 2026), the trade being memory against wasted reads.
  • "Have we seen this URL?" in a web crawler (see Design a web crawler). A 1% false positive means occasionally skipping a page you'd never fetched, which a crawler can live with. A false negative would mean fetching pages forever, which is why the guarantee runs the right way round.
  • Shielding a database from lookups for things that don't exist: "is this username taken?", "is this coupon code real?", or keys that aren't in a cache. When bots probe random keys, every miss would go through to the database. A filter of all real keys rejects 99% of the bogus ones in memory.
  • Avoiding a network call: a client or edge node holds a filter of "keys that have something" (blocked URLs, documents with comments) and only calls the server on a maybe.

Redis 8 ships Bloom filters in the core server (RedisBloom's commands became part of Redis Open Source in 8.0, May 2025, along with cuckoo filters, count-min sketch, top-k and t-digest). On Redis 8.10:

redis-cli (Redis 8.10)
> BF.RESERVE seen:urls 0.01 1000000
OK
> BF.ADD seen:urls https://example.com/a
(integer) 1
> BF.EXISTS seen:urls https://example.com/a
(integer) 1
> BF.EXISTS seen:urls https://example.com/zzz
(integer) 0
> BF.INFO seen:urls
Capacity: 1000000 | Size: 1378568 | Number of filters: 1 | Number of items inserted: 1 | Expansion rate: 2

BF.RESERVE takes the error rate and the capacity, exactly the n and p above. The filter for a million items at 1% is 1,378,568 bytes, about 11 bits per item, somewhat more than the formula's 9.6 (the layout is Redis's own). Redis also answers the "what if I add more than I planned" question: with the default expansion rate of 2, a full filter gets a second, bigger sub-filter stacked on it (a scalable Bloom filter), and lookups check all of them. That keeps the error rate near the target, at the cost of more lookups and memory. BF.INFO printed each field on its own line; it's folded onto one line here.

AdvancedWhat a Bloom filter can't do
  • Delete. Clearing a bit might also "remove" every other item that shares it, and that would create false negatives. A counting Bloom filter keeps a small counter per position instead of a bit (typically 4 bits, so 4× the memory). A cuckoo filter stores short fingerprints in a hash table, supports deletion, and is often smaller than a Bloom filter at low false-positive rates.
  • Grow gracefully. Past its planned n, a plain filter's false-positive rate climbs steeply, because more bits are set. Size it for the end of its life, rebuild it periodically, or use a scalable one.
  • Tell you what's in it. You can't list or count the members (although you can estimate the count from the fraction of bits set). It only answers membership.
  • Be trusted on "yes". Every "maybe" must be confirmed against the real data if a wrong yes matters.
Does doubling the bits halve the false-positive rate?

A filter with 10 bits per item and the best k has about a 0.8% false-positive rate. If you double it to 20 bits per item (and re-pick k), roughly what happens to the rate?

Show answer

It falls to about 0.0067%, more than 100× better, not 2×.

What’s happening
  1. With the optimal k, the false-positive rate is p = e^(−(m/n)(ln 2)²) ≈ 0.6185^(m/n), so p falls exponentially in bits per item.
  2. At 10 bits per item: 0.6185^10 ≈ 0.0082, the 0.8% the model measured.
  3. At 20 bits per item: 0.6185^20 ≈ 0.000067, which is 0.0067%. The optimal k doubles too, to about 14.
  4. That's the same fact as "4.8 bits per 10×": 10 extra bits is just over two factors of 10 (100× and a bit more). Memory buys accuracy cheaply. The things that limit you are the extra hash computations and memory reads per lookup as k grows.

#HyperLogLog and count-min sketch

Added

Two questions come up in every analytics-flavoured design: "how many different things?" (unique visitors today, distinct search queries, distinct IPs hitting an endpoint) and "how often does each thing occur?" (which hashtags are trending, which users send the most requests, which cache keys are hot). Answered exactly, both need memory proportional to the number of distinct items: a set of every visitor ID, a map from every query to its count. At hundreds of millions of items per day, per page or per region, that's too much. HyperLogLog answers the first question and the count-min sketch the second, each in a fixed, small amount of memory, by accepting a small, known error.

The mental model for HyperLogLog is coin flipping. If someone tells you the longest run of heads they got in a row was 10, you'd guess they flipped about 2^10 = 1,000 times, because a run of 10 is a 1-in-1,024 event. Hash every item and look at how many leading zero bits the hash starts with: a hash starting with 10 zeros is a 1-in-1,024 event, so seeing one suggests about a thousand distinct items. Duplicates hash identically, so they never change the answer. One such observation is very noisy, so HyperLogLog splits the items into thousands of buckets (registers), keeps the longest run seen in each, and averages them.

The mental model for count-min is several tally sheets with too few rows. Each sheet has a few thousand lines, and every item is tallied on one line of each sheet, chosen by that sheet's hash function. Lines are shared, so a tally may include other items' counts, but it can never be lower than the item's true count. Different sheets mix the item with different neighbours, so you take the smallest of its tallies as the estimate.

user-123 hash₁: first 14 bits → register 9,437 of 16,384 hash₂: 0001 0110 1… first 1 at position 4 → rank 4 2 9434 0 9435 5 9436 3→4 9437 1 9438 2 9439 16,384 registers, 6 bits each keep the max estimate = α · m² / Σ 2^(−register) a harmonic mean over all 16,384 registers; error ≈ 1.04 / √m = 0.81%
HyperLogLog keeps, per register, the rarest-looking hash it has seen. Duplicates change nothing, so the registers reflect distinct items only.

Here's a HyperLogLog with 2^14 = 16,384 registers, the same number Redis uses. It counts up to 5 million distinct users (each added twice, to show duplicates are free), then merges two days of visitors:

hyperloglog.mjs
// HyperLogLog with 2^14 registers (the same register count as Redis): estimate distinct items.
function hash32(str, seed) { // FNV-1a + murmur3 finaliser
  let h = 0x811c9dc5 ^ seed;
  for (let i = 0; i < str.length; i++) h = Math.imul(h ^ str.charCodeAt(i), 0x01000193);
  h ^= h >>> 16; h = Math.imul(h, 0x85ebca6b); h ^= h >>> 13; h = Math.imul(h, 0xc2b2ae35);
  return (h ^ (h >>> 16)) >>> 0;
}

class HyperLogLog {
  constructor(p = 14) {
    this.p = p;
    this.m = 1 << p;
    this.registers = new Uint8Array(this.m);
  }
  add(item) {
    const index = hash32(item, 1) >>> (32 - this.p); // first p bits pick a register
    const rank = Math.clz32(hash32(item, 2)) + 1; // position of the first 1-bit in a second hash
    if (rank > this.registers[index]) this.registers[index] = rank; // keep the max ever seen
  }
  count() {
    const { m } = this;
    let sum = 0, zeros = 0;
    for (const r of this.registers) { sum += 2 ** -r; if (r === 0) zeros++; }
    const alpha = 0.7213 / (1 + 1.079 / m);
    const raw = (alpha * m * m) / sum; // harmonic mean of 2^register, scaled
    if (raw <= 2.5 * m && zeros > 0) return m * Math.log(m / zeros); // small range: linear counting
    return raw;
  }
  merge(other) {
    for (let i = 0; i < this.m; i++) this.registers[i] = Math.max(this.registers[i], other.registers[i]);
  }
}

console.log("standard error 1.04/sqrt(m) =", ((1.04 / Math.sqrt(1 << 14)) * 100).toFixed(2) + "%", "| registers:", 1 << 14);
console.log("    true distinct |      estimate |  error");
const hll = new HyperLogLog();
let distinct = 0;
for (const target of [1_000, 10_000, 100_000, 1_000_000, 5_000_000]) {
  for (; distinct < target; distinct++) {
    hll.add(`user-${distinct}`);
    hll.add(`user-${distinct}`); // duplicates never change a register
  }
  const est = hll.count();
  console.log(`${distinct.toLocaleString("en-US").padStart(17)} | ${Math.round(est).toLocaleString("en-US").padStart(13)} | ${(((est - distinct) / distinct) * 100).toFixed(2).padStart(5)}%`);
}

// Union: monday and tuesday visitors overlap by 50,000 users.
const monday = new HyperLogLog(), tuesday = new HyperLogLog();
for (let i = 0; i < 100_000; i++) monday.add(`v-${i}`);
for (let i = 50_000; i < 150_000; i++) tuesday.add(`v-${i}`);
monday.merge(tuesday);
console.log("union of two days (true 150,000):", Math.round(monday.count()).toLocaleString("en-US"));
console.log("memory: 16384 registers x 6 bits =", (16384 * 6) / 8 / 1024, "KiB");
Result
standard error 1.04/sqrt(m) = 0.81% | registers: 16384
    true distinct |      estimate |  error
            1,000 |         1,008 |  0.84%
           10,000 |         9,962 | -0.38%
          100,000 |        98,994 | -1.01%
        1,000,000 |     1,003,128 |  0.31%
        5,000,000 |     5,019,983 |  0.40%
union of two days (true 150,000): 149,692
memory: 16384 registers x 6 bits = 12 KiB
What’s happening
  1. add hashes the item twice. The top 14 bits of the first hash choose a register (0 to 16,383); Math.clz32 counts the leading zeros of the second hash, and the rank is that count plus one. The register keeps the maximum rank it has seen. Adding user-0 a second time produces exactly the same register and rank, so the duplicate line in the loop changes nothing: HyperLogLog counts distinct items by construction.
  2. count combines the registers with a harmonic mean of 2^rank, which stops one unlucky register with a huge rank from dominating, scaled by the bias-correction constant α. With few items, most registers are still 0, and the raw formula is biased, so it switches to linear counting (m · ln(m / empty registers)), which is very accurate while many registers are empty.
  3. The errors are 0.84%, −0.38%, −1.01%, 0.31% and 0.40%, all around the theoretical standard error 1.04/√16,384 = 0.81%. That's a standard deviation, not a bound, so about a third of estimates land further out than 0.81%, like the −1.01% at 100,000. The error doesn't grow with the count: 5,000,000 is estimated as 5,019,983.
  4. The memory is the same 16,384 registers whether you've seen 1,000 items or 5 million: 6 bits each is 12 KiB. For comparison, a JavaScript Set of those 5 million user-N strings took 313 MiB of heap on Node 26.7, about 26,000 times more.
  5. merge takes the register-wise maximum. Monday saw v-0…v-99,999 and Tuesday v-50,000…v-149,999; the merged sketch estimates 149,692 against the true union of 150,000 (−0.2%). Merging is exact in the sense that it gives the same registers as adding both days' items to one sketch, which is what makes HyperLogLog so useful in practice: keep one sketch per hour, and combine any range of hours into a daily, weekly or monthly unique count without re-reading the raw data.
AdvancedHyperLogLog in practice

The Redis documentation describes its HyperLogLog as using up to 12 KB with a standard error of 0.81% (redis.io, checked October 2026). Small sketches are stored in a compact sparse form and grow to the full 12 KB dense form as they fill. On Redis 8.10:

redis-cli (Redis 8.10)
> PFADD visitors:2026-10-24 alice bob carol alice
(integer) 1
> PFCOUNT visitors:2026-10-24
(integer) 3
> MEMORY USAGE visitors:2026-10-24
(integer) 86
(after PFADD of 100,000 distinct ids to another key)
> PFCOUNT big
(integer) 100420
> MEMORY USAGE big
(integer) 14358

The three-visitor key takes 86 bytes in the sparse form; with 100,000 distinct IDs the estimate is 100,420 (+0.42%) and the key takes 14,358 bytes: the 12 KiB dense registers plus Redis's per-key overhead. PFMERGE does the register-wise maximum from the model, and PFCOUNT key1 key2 counts the union on the fly.

What HyperLogLog is good for, and what it isn't:

  • Good: unique visitors per page per day, distinct users per feature, distinct IPs per endpoint (an anomaly signal), distinct values per column in a query planner. Anywhere you need many counters, each over many items, and 1% error is fine.
  • Unions are easy, intersections aren't. |A ∩ B| = |A| + |B| − |A ∪ B| works on paper, but the errors of three estimates add up while the answer may be small. If A and B are 10 million each and overlap by 10,000, the ±1% errors (about 100,000) swamp the overlap. Use exact sets, or another kind of sketch, for overlaps.
  • Not for small exact numbers that people will check (billing, "you have 3 followers"). 0.81% of a small number is invisible, but a wrong count that a user can verify is a bug report.
  • Not for lists. It can't tell you which users visited, only how many.
AdvancedCount-min sketch for frequencies
q42 h1 25 14 83 61 7 40 14 5 14 12 h2 14 3 5 9 61 25 25 91 5 61 h3 61 5 18 7 87 25 7 61 18 3 h4 12 7 14 3 9 3 9 5 140 14 estimate min = 83 add: +1 in one counter per row · query: the smallest of those counters
Every row over-counts by however much its collisions add; the row with the fewest collisions gives the best estimate.

A count-min sketch with error bound ε and failure probability δ uses d = ⌈ln(1/δ)⌉ rows of w = ⌈e/ε⌉ counters (Cormode and Muthukrishnan, 2005). The guarantee: the estimate is never below the true count, and with probability at least 1 − δ it's at most ε·N above it, where N is the total number of events. The model feeds it a million search queries drawn from 100,000 terms with a Zipf-like skew (a few terms are very popular, most are rare, as real query logs are), and keeps a small top-5 list beside it to find the heavy hitters:

count-min.mjs
// Count-min sketch (d rows x w counters) + a small top-k list for heavy hitters, on a skewed stream.
function hash32(str, seed) {
  let h = 0x811c9dc5 ^ seed;
  for (let i = 0; i < str.length; i++) h = Math.imul(h ^ str.charCodeAt(i), 0x01000193);
  h ^= h >>> 16; h = Math.imul(h, 0x85ebca6b); h ^= h >>> 13; h = Math.imul(h, 0xc2b2ae35);
  return (h ^ (h >>> 16)) >>> 0;
}
const mulberry32 = (a) => () => {
  a = (a + 0x6d2b79f5) | 0;
  let t = Math.imul(a ^ (a >>> 15), 1 | a);
  t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
  return ((t ^ (t >>> 14)) >>> 0) / 2 ** 32;
};

class CountMin {
  constructor(epsilon, delta) {
    this.w = Math.ceil(Math.E / epsilon); // error <= epsilon * N ...
    this.d = Math.ceil(Math.log(1 / delta)); // ... with probability >= 1 - delta
    this.rows = Array.from({ length: this.d }, () => new Uint32Array(this.w));
  }
  add(item, c = 1) {
    for (let r = 0; r < this.d; r++) this.rows[r][hash32(item, r + 1) % this.w] += c;
  }
  estimate(item) { // every row over-counts (collisions only add), so the smallest is closest
    let min = Infinity;
    for (let r = 0; r < this.d; r++) min = Math.min(min, this.rows[r][hash32(item, r + 1) % this.w]);
    return min;
  }
}

// Stream: 1,000,000 search queries over 100,000 distinct terms, Zipf-like (term i has weight 1/i).
const rand = mulberry32(99), TERMS = 100_000, N = 1_000_000;
const cdf = []; let total = 0;
for (let i = 1; i <= TERMS; i++) cdf.push((total += 1 / i));
const pick = () => { const x = rand() * total; let lo = 0, hi = TERMS - 1; while (lo < hi) { const mid = (lo + hi) >> 1; if (cdf[mid] < x) lo = mid + 1; else hi = mid; } return `q${lo + 1}`; };

const cms = new CountMin(0.001, 0.01), exact = new Map(), K = 5;
let top = []; // [term, estimate] kept sorted, at most K long
for (let i = 0; i < N; i++) {
  const term = pick();
  cms.add(term);
  exact.set(term, (exact.get(term) || 0) + 1);
  const est = cms.estimate(term);
  top = top.filter(([t]) => t !== term);
  if (top.length < K || est > top[K - 1][1]) top = [...top, [term, est]].sort((a, b) => b[1] - a[1]).slice(0, K);
}
console.log(`sketch: ${cms.d} rows x ${cms.w} counters = ${(cms.d * cms.w * 4 / 1024).toFixed(0)} KiB | bound eps*N = ${0.001 * N}`);
console.log("top-5 by sketch:", top.map(([t, e]) => `${t}=${e} (true ${exact.get(t)})`).join(", "));
for (const t of ["q10", "q1000", "q50000"]) console.log(`${t.padEnd(6)} true ${String(exact.get(t) ?? 0).padStart(5)} | estimate ${cms.estimate(t)}`);
let worst = 0, sum = 0, under = 0;
for (const [t, c] of exact) { const err = cms.estimate(t) - c; worst = Math.max(worst, err); sum += err; if (err < 0) under++; }
console.log(`over all ${exact.size} terms seen: average over-count ${(sum / exact.size).toFixed(2)}, worst ${worst}, under-counted ${under}`);
console.log(`exact Map would hold ${exact.size} keys; the sketch stays ${cms.d * cms.w} counters whatever the stream`);
Result
sketch: 5 rows x 2719 counters = 53 KiB | bound eps*N = 1000
top-5 by sketch: q1=82993 (true 82876), q2=41357 (true 41228), q3=27697 (true 27597), q4=20496 (true 20389), q5=16783 (true 16695)
q10    true  8307 | estimate 8398
q1000  true    79 | estimate 136
q50000 true     0 | estimate 56
over all 80786 terms seen: average over-count 98.07, worst 500, under-counted 0
exact Map would hold 80786 keys; the sketch stays 13595 counters whatever the stream
What’s happening
  1. new CountMin(0.001, 0.01) asks for an error of at most 0.1% of N with 99% probability: w = ⌈e / 0.001⌉ = 2,719 counters per row and d = ⌈ln 100⌉ = 5 rows, 13,595 counters in all (53 KiB as 32-bit counters). With N = 1,000,000 events, the bound is ε·N = 1,000.
  2. Each event increments one counter per row, and estimate reads the same five counters and returns the smallest. For the top term q1, the estimate is 82,993 against a true 82,876: off by 117, which is 0.14% of its count. For heavy hitters, the collisions are noise on top of a big number.
  3. The top-5 list is maintained as the stream goes: after each event, the term's current estimate is compared with the smallest entry in the list. Because popular terms have large, accurate estimates, the list ends with exactly q1…q5 in the right order. This "sketch + small heap" is the standard way to find heavy hitters in one pass with bounded memory.
  4. Rare terms are where the error shows. q1000 really occurred 79 times but is estimated at 136, 72% too high, and q50000, which never occurred at all, is estimated at 56. The absolute error is bounded (the worst over all 80,786 seen terms is 500, well inside the 1,000 bound; the average is 98), but relative to a small count that's huge.
  5. No term is ever under-counted (under-counted 0): every counter a term touches includes all of that term's own increments, plus possibly others. And the sketch's size is fixed in advance, while the exact Map grew to 80,786 keys and would keep growing with the vocabulary.

Where count-min (plus a heap) is the right tool:

  • Trending topics and top-k queries: count hashtags or queries per time window, keep the top 100. The error only matters near the cut-off, and there it's a small fraction of large counts. The search autocomplete design uses exactly this to decide which queries are popular enough to suggest.
  • Hot keys: find the cache or database keys taking a disproportionate share of traffic, then replicate or locally cache those (see Hot keys and cache stampedes).
  • Heavy users in rate limiting and abuse detection: "which API keys or IPs sent the most requests this minute", without a counter per key. A sketch can't decide whether to block a particular user (a light user might share counters with a heavy one), but it points at candidates for an exact check.
  • Network monitoring: per-flow byte counts at line rate in routers, where memory per packet is tiny.

Redis 8 has both as built-ins: CMS.INITBYPROB (create from ε and δ) with CMS.INCRBY and CMS.QUERY, and a separate TOPK.* family that keeps the top k for you.

Why can't HyperLogLog be fooled by duplicates?

Your analytics pipeline delivers each page-view event at least once, so retries sometimes send the same (user, page, day) event two or three times. A count-min sketch and a HyperLogLog both consume the stream. Which one's answer is affected, and why?

Show answer

The count-min sketch is inflated by duplicates; the HyperLogLog isn't.

What’s happening
  1. HyperLogLog's update is register = max(register, rank). A duplicate event hashes to the same register with the same rank, and max(x, x) is x, so the second and third copies change nothing. Taking a maximum is idempotent.
  2. Count-min's update is counter += 1 in each row. A duplicate adds another 1 to the same counters, so the estimated frequency counts deliveries, not events. Addition isn't idempotent.
  3. So HyperLogLog is naturally safe with at-least-once delivery (see Idempotency and exactly-once), and merging sketches from replicas or overlapping batches is safe for the same reason. For count-min you need deduplication upstream (an event ID with a short-lived seen-set or Bloom filter) if retries are common.

#Geohashing and quadtrees

Added

"Find restaurants within 2 km" and "find the nearest available driver" are range queries in two dimensions, and ordinary database indexes are one-dimensional: a B-tree or a sorted set can answer "keys between X and Y" quickly, but an index on latitude plus an index on longitude can only narrow down one of them at a time. Scanning a strip of latitude around the whole planet to then filter by longitude is hopeless at scale. Spatial indexes solve this by mapping 2D positions to 1D keys so that nearby places usually get nearby keys, or by building a tree that divides space where the data is.

Geohash (Gustavo Niemeyer, 2008, a base32 form of an older idea, the Morton or Z-order curve) is the first approach. The mental model is a game of "left or right? top or bottom?". Halve the world's longitude range and write 1 if the point is in the east half, 0 if west; halve the latitude range and do the same for north and south; keep alternating. Every bit halves the box the point is known to be in. Group the bits five at a time and write each group as a base32 character. The result has a lovely property: a geohash's prefixes are the boxes that contain it. u09tunq is inside u09tun, which is inside u09t, which is inside u. "Everything in this box" becomes "every key starting with this prefix", which any sorted index can do.

Quadtrees take the second approach: start with one square, and whenever a square holds too many points, split it into four quarters. Dense areas (downtown) end up with tiny cells and sparse areas (farmland) with huge ones, so every leaf holds a similar number of points.

One cell → 32 children (8 × 4) b c f g u v y z 8 9 d e s t w x 2 3 6 7 k m q r 0 1 4 5 h j n p longitude bits pick the column, latitude bits the row; the next level repeats inside (4 × 8, then 8 × 4 …) Search = the cell + its 8 neighbours u09tunt u09tunw u09tunx u09tunm u09tunq u09tunr u09tunj u09tunn u09tunp a point near the north-east corner: its search circle spills into u09tunw, u09tunx and u09tunr, so query all 9
Each geohash character picks one of 32 sub-cells; a nearby search must look in the neighbours too, because closeness on the map doesn't guarantee a shared prefix.

Here's a geohash encoder, a decoder that returns a cell's bounding box, and a neighbour finder, tried on the Eiffel Tower and the Louvre (about 3.2 km apart):

geohash.mjs
// Geohash: interleave longitude and latitude bisection bits, 5 bits per base32 character.
const BASE32 = "0123456789bcdefghjkmnpqrstuvwxyz"; // no a, i, l, o

function encode(lat, lon, precision) {
  let latR = [-90, 90], lonR = [-180, 180], even = true, bits = 0, n = 0, out = "";
  while (out.length < precision) {
    const range = even ? lonR : latR, value = even ? lon : lat; // even bits: longitude
    const mid = (range[0] + range[1]) / 2;
    if (value >= mid) { bits = (bits << 1) | 1; range[0] = mid; } else { bits <<= 1; range[1] = mid; }
    even = !even;
    if (++n === 5) { out += BASE32[bits]; bits = 0; n = 0; }
  }
  return out;
}

function bounds(hash) { // the cell a geohash names
  let latR = [-90, 90], lonR = [-180, 180], even = true;
  for (const ch of hash) {
    const v = BASE32.indexOf(ch);
    for (let b = 4; b >= 0; b--) {
      const range = even ? lonR : latR, mid = (range[0] + range[1]) / 2;
      if ((v >> b) & 1) range[0] = mid; else range[1] = mid;
      even = !even;
    }
  }
  return { latR, lonR };
}

function neighbours(hash) { // step one cell in each direction from the centre and re-encode
  const { latR, lonR } = bounds(hash);
  const dLat = latR[1] - latR[0], dLon = lonR[1] - lonR[0];
  const cLat = (latR[0] + latR[1]) / 2, cLon = (lonR[0] + lonR[1]) / 2;
  const out = {};
  for (const [name, dy, dx] of [["n", 1, 0], ["ne", 1, 1], ["e", 0, 1], ["se", -1, 1], ["s", -1, 0], ["sw", -1, -1], ["w", 0, -1], ["nw", 1, -1]]) {
    const lon = ((cLon + dx * dLon + 540) % 360) - 180; // wrap across the antimeridian
    out[name] = encode(cLat + dy * dLat, lon, hash.length); // (cells at the poles need a clamp; omitted)
  }
  return out;
}

const eiffel = [48.85826, 2.29451], louvre = [48.86061, 2.33764];
for (const p of [1, 4, 6, 7, 9]) console.log(`precision ${p}: Eiffel ${encode(...eiffel, p).padEnd(9)} Louvre ${encode(...louvre, p)}`);
console.log("cell u09tunq:", JSON.stringify(bounds("u09tunq")));
console.log("neighbours of u09tunq:", JSON.stringify(neighbours("u09tunq")));

// Cell size at the equator for each length (1 degree of latitude ~ 111.32 km)
for (const p of [4, 5, 6, 7, 8]) {
  const { latR, lonR } = bounds("s".repeat(p)); // any cell of that length has the same size in degrees
  const h = (latR[1] - latR[0]) * 111.32, w = (lonR[1] - lonR[0]) * 111.32;
  console.log(`length ${p}: ${w >= 1 ? w.toFixed(1) + " km" : (w * 1000).toFixed(0) + " m"} x ${h >= 1 ? h.toFixed(1) + " km" : (h * 1000).toFixed(0) + " m"}`);
}

// The edge problem: two points 22 m apart either side of a cell boundary share no prefix at all.
console.log("lat 0.0001 vs -0.0001 at lon 0:", encode(0.0001, 0.0001, 6), encode(-0.0001, 0.0001, 6));
Result
precision 1: Eiffel u         Louvre u
precision 4: Eiffel u09t      Louvre u09t
precision 6: Eiffel u09tun    Louvre u09tvn
precision 7: Eiffel u09tunq   Louvre u09tvny
precision 9: Eiffel u09tunqgc Louvre u09tvny1w
cell u09tunq: {"latR":[48.857574462890625,48.85894775390625],"lonR":[2.29339599609375,2.294769287109375]}
neighbours of u09tunq: {"n":"u09tunw","ne":"u09tunx","e":"u09tunr","se":"u09tunp","s":"u09tunn","sw":"u09tunj","w":"u09tunm","nw":"u09tunt"}
length 4: 39.1 km x 19.6 km
length 5: 4.9 km x 4.9 km
length 6: 1.2 km x 611 m
length 7: 153 m x 153 m
length 8: 38 m x 19 m
lat 0.0001 vs -0.0001 at lon 0: s00000 kpbpbp
What’s happening
  1. encode keeps a latitude range and a longitude range and alternates between them, starting with longitude. For the Eiffel Tower, the first bit asks "is 2.29451 ≥ 0 (the middle of −180…180)?" Yes, so 1, and the range becomes 0…180. Five bits later the character is u, a cell covering most of Europe. Every five bits narrow it further: u09t (a box about 26 km × 20 km around Paris), u09tun, u09tunq, u09tunqgc.
  2. The two landmarks share the prefix u09t and then split at the 5th character (u vs v). A shared prefix proves closeness: both are inside the same u09t box. The reverse doesn't hold, as point 5 shows.
  3. bounds("u09tunq") replays the bits to recover the cell: latitude 48.85757…48.85895 and longitude 2.29340…2.29477, a box of about 153 m × 153 m. That's the precision table below it: length 5 ≈ 4.9 km, 6 ≈ 1.2 km × 0.6 km, 7 ≈ 153 m, 8 ≈ 38 m × 19 m at the equator. (East-west widths shrink with the cosine of the latitude; at Paris's 49°, a length-7 cell is about 100 m wide.)
  4. neighbours steps one cell-width from the centre in each of the 8 directions and re-encodes. The neighbours of u09tunq all share the u09tun prefix here, but that's luck: for a cell on the edge of its parent, some neighbours have different parents. The longitude is wrapped so that stepping east from 180° comes back at −180°. Cells at the poles need clamping, which this short version omits.
  5. The edge problem in one line: two points 22 m apart on either side of the equator are s00000 and kpbpbp, with no common prefix at all, because the very first latitude bit differs. That's why a nearby search never queries just one prefix: it queries the point's cell and its 8 neighbours, then filters the candidates by true distance.
AdvancedRunning a nearby search with geohashes

The recipe, which is also how Design nearby places search works:

  1. Pick the precision from the radius. Choose the length whose cells are at least as big as the search radius, so the circle can't reach past the 3 × 3 block: for 1 km, length 5 (4.9 km cells); for 100 m, length 7 (153 m cells). Those are sizes at the equator: cells get narrower east–west by cos(latitude), so a length-7 cell is only about 100 m wide in Paris. Check the width at your latitude and use one character fewer if the radius is close to it.
  2. Store each place with its geohash in an indexed column (or as a sorted-set score), so "prefix = X" is an index range scan.
  3. Query 9 prefixes (the cell and its 8 neighbours), then compute the real distance for each candidate (haversine) and sort. The prefix query is only a coarse filter.
  4. Handle density. A fixed precision gives a fixed cell size, so a downtown cell might hold 10,000 places and a rural one none. Either try a longer prefix first and fall back to shorter ones until you have enough results, or switch to a quadtree, which adapts by itself.

Redis's GEO commands are exactly this. GEOADD stores each member in an ordinary sorted set whose score is a 52-bit interleaved geohash (a 52-bit integer fits exactly in the double-precision score), and GEOSEARCH computes the score ranges covering the search area and filters by distance. The Redis docs limit latitude to ±85.05112878°, the range of the Web Mercator projection. On Redis 8.10, with the same two landmarks:

redis-cli (Redis 8.10)
> GEOADD places 2.29451 48.85826 eiffel 2.33764 48.86061 louvre
(integer) 2
> GEOHASH places eiffel louvre
1) "u09tunqgcz0"
2) "u09tvny1wv0"
> GEOSEARCH places FROMLONLAT 2.2945 48.8583 BYRADIUS 4 km ASC WITHDIST
1) 1) "eiffel"
   2) "0.0045"
2) 1) "louvre"
   2) "3.1672"

Redis's 11-character GEOHASH strings start with the same u09tunqgc and u09tvny1w the model computed. Note that Redis takes longitude first, the opposite order from the usual "lat, lon", which is a classic source of points landing in the ocean.

AdvancedQuadtrees: cells that adapt to density
root NW NE SW SE NW NE SW SE NW NE SW SE a leaf splits when it holds more than its capacity
A quadtree splits only where points crowd together, so every leaf holds a similar number of points whatever the density. (The map shows one more split inside the cluster than the tree on the right draws.)

The model puts 100,000 places on a 100 km × 100 km map, 70% of them packed into a 10 km downtown, and splits any square holding more than 100. Then it runs a 1 km radius search in two places and checks it against brute force:

quadtree.mjs
// A point quadtree: a square splits into 4 children once it holds more than CAPACITY points.
const mulberry32 = (a) => () => {
  a = (a + 0x6d2b79f5) | 0;
  let t = Math.imul(a ^ (a >>> 15), 1 | a);
  t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
  return ((t ^ (t >>> 14)) >>> 0) / 2 ** 32;
};
const CAPACITY = 100;

class Quad {
  constructor(x, y, size, depth = 0) { Object.assign(this, { x, y, size, depth, points: [], kids: null }); }
  insert(p) {
    if (this.kids) return this.kids[this.childIndex(p)].insert(p);
    this.points.push(p);
    if (this.points.length > CAPACITY && this.size > 1e-6) { // split: hand every point to a child
      const h = this.size / 2;
      this.kids = [[0, 0], [h, 0], [0, h], [h, h]].map(([dx, dy]) => new Quad(this.x + dx, this.y + dy, h, this.depth + 1));
      for (const q of this.points) this.kids[this.childIndex(q)].insert(q);
      this.points = [];
    }
  }
  childIndex([px, py]) { const h = this.size / 2; return (px >= this.x + h ? 1 : 0) + (py >= this.y + h ? 2 : 0); }
  search(cx, cy, r, out, stats) { // all points within r of (cx, cy)
    const nx = Math.max(this.x, Math.min(cx, this.x + this.size)), ny = Math.max(this.y, Math.min(cy, this.y + this.size));
    if ((nx - cx) ** 2 + (ny - cy) ** 2 > r * r) return; // the circle misses this square: skip it all
    stats.nodes++;
    if (this.kids) for (const k of this.kids) k.search(cx, cy, r, out, stats);
    else for (const p of this.points) { stats.points++; if ((p[0] - cx) ** 2 + (p[1] - cy) ** 2 <= r * r) out.push(p); }
  }
  leaves(acc = []) { if (this.kids) this.kids.forEach((k) => k.leaves(acc)); else acc.push(this); return acc; }
}

// 100,000 places on a 100 km x 100 km map: 70% packed into a 10 km "downtown", the rest spread out.
const rand = mulberry32(5), root = new Quad(0, 0, 100), places = [];
for (let i = 0; i < 100_000; i++) {
  const p = rand() < 0.7 ? [40 + rand() * 10, 40 + rand() * 10] : [rand() * 100, rand() * 100];
  places.push(p);
  root.insert(p);
}
const leaves = root.leaves();
const byDepth = {};
for (const l of leaves) byDepth[l.depth] = (byDepth[l.depth] || 0) + 1;
console.log("leaves:", leaves.length, "| leaves per depth:", JSON.stringify(byDepth));
const cell = (x, y) => leaves.find((l) => x >= l.x && x < l.x + l.size && y >= l.y && y < l.y + l.size);
console.log("leaf downtown is", cell(45, 45).size.toFixed(3), "km wide; leaf in the suburbs is", cell(5, 5).size.toFixed(3), "km wide");

for (const [name, cx, cy] of [["downtown", 45, 45], ["suburbs", 5, 5]]) {
  const out = [], stats = { nodes: 0, points: 0 };
  root.search(cx, cy, 1, out, stats); // radius 1 km
  const brute = places.filter((p) => (p[0] - cx) ** 2 + (p[1] - cy) ** 2 <= 1).length;
  console.log(`${name}: ${out.length} places within 1 km (brute force ${brute}); checked ${stats.points} points in ${stats.nodes} nodes instead of 100,000`);
}
Result
leaves: 3052 | leaves per depth: {"4":20,"5":928,"6":15,"7":27,"8":214,"9":1848}
leaf downtown is 0.195 km wide; leaf in the suburbs is 3.125 km wide
downtown: 2210 places within 1 km (brute force 2210); checked 2937 points in 122 nodes instead of 100,000
suburbs: 12 places within 1 km (brute force 12); checked 25 points in 6 nodes instead of 100,000
What’s happening
  1. insert walks down to the leaf containing the point and appends it. When a leaf goes over 100 points, it creates four children of half the size and pushes its points down into them, so data lives only in leaves. The size check (> 1e-6) stops infinite splitting if 101 points share one exact location.
  2. The result has 3,052 leaves at very different depths. The sparse suburbs stop at depth 4–5 (leaves 6.25 or 3.125 km wide), while downtown goes to depth 8–9: a downtown leaf is 0.195 km wide and a suburban one 3.125 km, 16 times wider. That's the adaptivity a fixed-length geohash lacks.
  3. search prunes: if the circle doesn't touch a square (the nearest point of the square is further than r), the whole subtree is skipped. Downtown, the 1 km search visits 122 nodes and checks 2,937 points to find 2,210; in the suburbs it visits 6 nodes and checks 25 points to find 12. Brute force agrees both times, and checks all 100,000.
  4. The work tracks the size of the answer, not the size of the map. That's the property you want from any spatial index, and why quadtrees are a classic choice for "nearest drivers" or "places near me".
AdvancedGeohash or quadtree?
Geohash (or S2/H3 cell IDs)Quadtree
Where it livesA column or sorted-set score in an existing store (MySQL, PostgreSQL, Redis, DynamoDB)Usually in memory, in your own service
Adapts to densityNo, the cell size is fixed per precisionYes, cells split where points are dense
Updates (moving drivers)Cheap: rewrite one keyCheap for a single point, but rebalancing and replicating the tree across servers is your problem
ShardingEasy: shard by prefixHarder: shard by subtree, rebuild on restart
Edge effectsMust query 9 cellsSearch visits whichever cells the circle overlaps

Two other systems are worth naming in an interview. Google's S2 maps the sphere onto a cube and orders cells along a Hilbert curve, which keeps neighbours closer in key order than geohash's Z-order and avoids the distortion near the poles. Uber's H3 (open-sourced 2018) uses hexagons with 64-bit cell IDs; a hexagon's six neighbours are all the same distance from its centre, which suits movement and pricing models. For most interview designs, "geohash in Redis or the database, or a quadtree in memory if density varies wildly" is a complete answer, and the trade-offs in the table are what the interviewer wants to hear.

Same prefix, same neighbourhood?

Two geohashes u09tunq and u09tunr share a 6-character prefix. Two others, s00000 and kpbpbp, share none. Which statements are always true?

Show answer

Sharing a prefix guarantees the points are close (both are in the same u09tun cell, about 1.2 km × 0.6 km). Sharing no prefix guarantees nothing: s00000 and kpbpbp were 22 m apart.

What’s happening
  1. Each character narrows the cell, so two hashes sharing 6 characters lie in the same length-6 cell, and their distance is at most that cell's diagonal (about 1.4 km near the equator). That direction always holds.
  2. The converse fails at cell boundaries. The boundary between s and k is the equator itself: the very first latitude bit is 1 north of it and 0 south of it. So points a hair apart get different first characters, and nothing after that can be shared.
  3. The same thing happens at every level, at every cell edge. So a prefix match is a sufficient test for closeness but not a necessary one, and a correct search must add the neighbouring cells.

#Merkle trees

Added

Two replicas are supposed to hold the same data, but they drift apart: a write reached one replica but not the other, a disk returned a corrupted block, a node was down during an update. To repair them, you first need to know what differs. Sending every row from one replica to the other to compare would move the whole dataset across the network to find perhaps a handful of differences. A Merkle tree (Ralph Merkle, 1979) finds the differences while exchanging only a few hashes.

Build it by hashing each block of data (a block can be a key range, a file chunk or a single record), then hashing pairs of those hashes, then pairs of those, until one root hash is left. The root summarises everything: change one byte anywhere and the root changes. The mental model is comparing two copies of a long book by checksums. You compare the checksum of the whole book; if it matches, you're done. If it doesn't, you compare the two halves and keep only the half that differs, then its halves, and so on, until you've cornered the one page that's wrong. With n pages, that takes about log₂ n rounds instead of reading n pages.

block 0 block 1 block 2 block 3 block 4 block 5 block 6 block 7 H1.0 H1.1 H1.2 H1.3 H2.0 H2.1 root hashes differ: descend compared, equal: skip the subtree never looked at
Comparing top-down, the replicas only descend where hashes disagree; equal subtrees are skipped whole, however much data they cover.

The model builds a tree over 16 blocks on each of two replicas, changes one block on replica B, and walks both trees top-down to find it:

merkle.mjs
// Two replicas each build a Merkle tree over 16 blocks; comparing top-down finds the block that differs.
import { createHash } from "node:crypto";
const sha = (s) => createHash("sha256").update(s).digest("hex").slice(0, 8); // 8 hex chars is enough to read

function build(blocks) { // levels[0] = leaf hashes ... levels[last] = [root]
  const levels = [blocks.map(sha)];
  while (levels.at(-1).length > 1) {
    const below = levels.at(-1), up = [];
    for (let i = 0; i < below.length; i += 2) up.push(sha(below[i] + (below[i + 1] ?? below[i]))); // parent = H(left + right)
    levels.push(up);
  }
  return levels;
}

function diff(a, b) { // walk down only where the hashes disagree
  let compared = 0, suspects = [0];
  for (let level = a.length - 1; level >= 0; level--) {
    const next = [];
    for (const i of suspects) {
      compared++;
      if (a[level][i] === b[level][i]) continue; // whole subtree identical: skip it
      if (level === 0) next.push(i);
      else next.push(2 * i, 2 * i + 1); // children of node i
    }
    suspects = next.filter((i) => level === 0 || i < a[level - 1].length);
  }
  return { blocks: suspects, compared };
}

const replicaA = Array.from({ length: 16 }, (_, i) => `rows ${i * 1000}-${i * 1000 + 999}`);
const replicaB = [...replicaA];
replicaB[11] = "rows 11000-11999 (one row updated)"; // a write that reached A but not B

const ta = build(replicaA), tb = build(replicaB);
console.log("root A", ta.at(-1)[0], "| root B", tb.at(-1)[0], "->", ta.at(-1)[0] === tb.at(-1)[0] ? "in sync" : "differ");
console.log("level sizes:", ta.map((l) => l.length).join(" -> "));
console.log("one block differs:", JSON.stringify(diff(ta, tb)));

replicaB[2] = "rows 2000-2999 (another change)";
console.log("two blocks differ:", JSON.stringify(diff(ta, build(replicaB))));
console.log("identical replicas:", JSON.stringify(diff(ta, build(replicaA))));

// The same walk on a big keyspace: 2^20 leaf ranges, one bad range.
const depth = 20, leaves = 2 ** depth;
console.log(`2^20 leaves: one difference costs ${1 + 2 * depth} hash comparisons, instead of ${leaves.toLocaleString("en-US")} leaf hashes or every row`);
Result
root A 2674ab49 | root B 7f16dd97 -> differ
level sizes: 16 -> 8 -> 4 -> 2 -> 1
one block differs: {"blocks":[11],"compared":9}
two blocks differ: {"blocks":[2,11],"compared":15}
identical replicas: {"blocks":[],"compared":1}
2^20 leaves: one difference costs 41 hash comparisons, instead of 1,048,576 leaf hashes or every row
What’s happening
  1. build hashes each block into a leaf (levels[0], 16 hashes), then hashes each pair of adjacent hashes into a parent: 16 → 8 → 4 → 2 → 1. Hashes are shortened to 8 hex characters only so they're readable; a real tree keeps the whole SHA-256.
  2. Block 11 on replica B has a row update that A doesn't have (or vice versa), so B's leaf 11 differs, and so does every hash on the path above it, up to the root: 2674ab49 vs 7f16dd97. One comparison of roots tells you whether anything differs.
  3. diff starts with the root as the only suspect. At each level, it compares the suspects; equal ones are dropped with their whole subtree, and unequal ones put their two children on the list for the next level down. For one bad block that's 1 (root) + 2 + 2 + 2 + 2 = 9 comparisons, ending at [11]. Only block 11 then needs to be transferred and reconciled.
  4. With two bad blocks (2 and 11), the paths separate at the root's children, so the walk compares 15 hashes and finds both. The cost grows with the number of differences times the depth, not with the size of the data. Identical replicas cost exactly 1 comparison.
  5. For 2^20 leaf ranges (about a million), one difference costs 1 + 2 × 20 = 41 hash comparisons, against exchanging 1,048,576 leaf hashes, or all the rows. In a real protocol, the comparisons happen in rounds over the network (send the hashes at a level, get back which ones differ), so it's about 20 round trips of small messages.
AdvancedWhere Merkle trees are used
  • Anti-entropy repair in leaderless databases. Amazon's Dynamo paper (2007) used Merkle trees per key range to synchronise replicas in the background, and Cassandra's nodetool repair does the same: each replica builds a tree over a token range (a "validation compaction" that reads the data), the trees are compared, and only the mismatched ranges are streamed. DataStax's documentation for Cassandra 2.0 and 3.x describes the tree as having a depth of 15, so about 32,000 leaves per range; current Apache Cassandra sizes each tree from the number of partitions in the range, within a memory cap (repair_session_space in cassandra.yaml, by default 1/16 of the heap, checked October 2026). Either way, the trade is worth explaining: a leaf covers many partitions, so one bad row makes the whole leaf range get streamed (over-streaming), and building the trees costs a lot of disk I/O, which is why repair is scheduled, not continuous. See Replication and Design a distributed key-value store.
  • File sync and deduplication. Split files into chunks and hash them: a client and server compare hashes to find which chunks changed, and upload only those (see Design file storage and sync).
  • Git. Every file (blob) and directory (tree) is stored under the hash of its content, and a commit points to the root tree's hash. Two repositories with the same root hash have identical snapshots, and fetching transfers only objects the other side lacks.
  • Proofs to someone who only knows the root: Certificate Transparency logs, Bitcoin's block headers and many blockchains publish a root, and anyone can prove that one item is in the tree with a short list of sibling hashes. That's the inclusion proof below.
AdvancedInclusion proofs

An inclusion proof turns the same tree around: a server holds all the data, a client holds only the 32-byte root (from a trusted source), and the server proves a block belongs to the tree by sending just the sibling hash at each level:

merkle-proof.mjs
// An inclusion proof: show that one block belongs to a tree when the verifier knows only the root.
import { createHash } from "node:crypto";
const sha = (s) => createHash("sha256").update(s).digest("hex").slice(0, 8);

const blocks = Array.from({ length: 16 }, (_, i) => `block ${i}`);
const levels = [blocks.map(sha)];
while (levels.at(-1).length > 1) {
  const below = levels.at(-1);
  levels.push(below.flatMap((h, i) => (i % 2 ? [] : [sha(h + below[i + 1])])));
}
const root = levels.at(-1)[0];

function prove(index) { // the sibling hash at every level, bottom to top
  const proof = [];
  for (let level = 0; level < levels.length - 1; level++, index >>= 1) {
    const sibling = index ^ 1; // flip the last bit: left <-> right neighbour
    proof.push({ side: sibling < index ? "left" : "right", hash: levels[level][sibling] });
  }
  return proof;
}
function verify(block, proof, expectedRoot) {
  let h = sha(block);
  for (const { side, hash } of proof) h = side === "left" ? sha(hash + h) : sha(h + hash);
  return h === expectedRoot;
}

const proof = prove(11);
console.log("root:", root);
console.log("proof for block 11:", proof.map((p) => `${p.side}:${p.hash}`).join(" "));
console.log("verify 'block 11':", verify("block 11", proof, root));
console.log("verify 'block 11 (tampered)':", verify("block 11 (tampered)", proof, root));
console.log("proof size for 16 leaves:", proof.length, "hashes | for 1 billion leaves:", Math.ceil(Math.log2(1e9)), "hashes");
Result
root: 761f25c7
proof for block 11: left:575f32b0 left:7550211e right:85090000 left:28137c53
verify 'block 11': true
verify 'block 11 (tampered)': false
proof size for 16 leaves: 4 hashes | for 1 billion leaves: 30 hashes
What’s happening
  1. prove(11) collects one sibling per level. 11 is 1011 in binary; at the leaves its sibling is 10 (11 ^ 1), on the left. Moving up, the index becomes 5 (sibling 4, left), then 2 (sibling 3, right), then 1 (sibling 0, left). That's 4 hashes for 16 leaves.
  2. verify starts from the hash of the block it was given and folds in each sibling on the side the proof says, recomputing the path to the root. For the real block 11 it arrives at 761f25c7, the trusted root: true.
  3. For a tampered block, the first hash is different, so every hash on the way up is different and the final value can't match the root: false. Forging a block that matches would mean finding a SHA-256 collision.
  4. The proof has log₂ n hashes: 4 for 16 leaves, 30 for a billion. The client checks membership in a billion-item dataset with 30 hashes and no trust in the server, which is why append-only logs (Certificate Transparency) and blockchains are built on Merkle trees. One detail this model skips: real logs hash leaves and internal nodes differently (RFC 6962 prefixes them with 0x00 and 0x01), so an internal node can't be passed off as a leaf in a forged proof.
How much does a mismatch cost?

Two replicas each have a Merkle tree with 2^15 = 32,768 leaves (the depth 15 that DataStax's Cassandra 2.0–3.x docs give). Exactly 3 leaf ranges differ. At most how many hash comparisons does the top-down walk need?

Show answer

At most 1 + 2 × 3 × 15 = 91 comparisons, and fewer when the three paths share their upper levels.

What’s happening
  1. The root is compared once. From then on, each differing node puts its 2 children on the list for the next level, so each level costs 2 comparisons per differing node at the level above.
  2. A single bad leaf makes one differing node per level, so it costs 2 comparisons at each of the 15 levels below the root: 30.
  3. Three bad leaves make at most 3 differing nodes per level, so at most 3 × 30 = 90, plus the root: 91. Where the paths share ancestors near the top (all three are under the root, and two may share a child of the root), the shared nodes are compared once, so the real count is a little lower.
  4. Compare that with 32,768 leaf hashes for a flat comparison: Merkle trees make the cost proportional to differences × depth, which is tiny when replicas are mostly in sync, the normal case.

Written fresh for interviews (there was no Google Doc for this one). Runnable models were run on Node 26; figures and limits were checked against sources in October 2026.

RohitDownloads · All chapters

Sync progress across devices

Type the same private phrase on your Mac and your phone, and your Learned ticks follow you between them — on every study site.

The phrase never leaves this device: only a fingerprint of it is sent, and the server stores a fingerprint of that. Anyone who knows the phrase could see or change your ticks, so pick something you don’t use elsewhere.

Sync is on in this browser.

Sync ID

This ID must be the same on every device. If another device shows a different one, its phrase is different (capital letters count): tap “Turn off here” on it and type the phrase again exactly.