Hard system-design scaling

Your write buffer lives in RAM. How do you size it, and what happens if the consumer falls behind?

The 30 second answer

A buffer in RAM has to be capped, so lag becomes a time budget. At a million views a second, a cap of about 300 million entries is roughly five minutes of headroom and about 36 GB of Redis memory. If the consumer falls behind longer than that, the stream gets trimmed and views are lost. So I size the cap from rate times tolerated outage, and make the consumer drain faster than peak.

Why it must be capped. The server has 96 GB of RAM. An unbounded stream grows until Redis runs out of memory, and then you lose far more than a few minutes of data.

The math from the video. At 1 million views per second, capping the queue at 300 million records gives the janitor about five minutes of breathing room, and 300 million entries use about 36 GB of Redis memory. With the janitor stopped, Redis memory grew from 14 GB to about 20–21 GB within roughly a minute of load. Later in the video the cap is described per stream: around 6 million entries each, with the speaker citing 30 to 36 GB of RAM for Redis in total. Doubling it to 12 million per stream gave about ten minutes of headroom, with memory allowed to reach about 70 GB.

What the cap means in Redis. XADD with MAXLEN (or XTRIM) caps a stream by evicting the oldest entries. That's what keeps memory bounded, and it's also exactly how unconsumed views get lost.

The rule that follows. The consumer must drain faster than the peak produce rate, not just keep up on average. Short bursts above the drain rate are fine if the buffer covers them; sustained load is not. In the final test, the janitor flushed about 800,000 rows per second against 1 million arriving, falling about 200,000 behind every second (next questions).

What I'd monitor. Stream length against the cap, consumer lag in time, Redis memory, and the drain rate against the arrival rate. Alert on the trend before the cap is reached.

Alternate interview scenarios

"Can the consumer go down for maintenance?"

For a few seconds or minutes, yes, as long as it stays inside the buffer's time budget. You can also run a second consumer on another machine in the same consumer group. The janitor itself used three of the server's cores, and with it stopped the server got close to 2 million requests per second, which is an argument for moving it to its own machine.

"Would you turn on Redis persistence?"

The test ran with persistence off. Turning on RDB snapshots or the append-only file protects the buffer across a Redis restart, but costs disk throughput. The speaker says you'd likely need more disk throughput to enable it.

Sources (5)
  1. Same video, Moving to Our Production App (1:07:47)youtube.com
  2. Same video, Battle-Testing Weer and Cpeak, Now in Rust (1:55:58)youtube.com
  3. Same video, Final Gigantic Test (2:18:07)youtube.com
  4. Redis docs: XADD (capped streams with MAXLEN)redis.io
  5. Redis docs: Persistence (RDB and AOF)redis.io