Hard system-design scaling

Walk through the final 30-minute test. How do you prove nothing was lost?

The 30 second answer

I count every side effect independently. The first thirty-minute run averaged about a million per second, close to two billion requests, but the views table had 1.8 billion rows against 1.9 billion requests, and logs were dropped too. The consumer drained only about 800,000 a second, so the capped stream trimmed data. After doubling the cap and the log buffer, requests, rows and log lines matched exactly.

Run 1. 1,800 seconds at an average of about 1 million requests per second, about 300 GB moved, and close to 2 billion requests. The janitor kept draining after the test ended, which went well past the five-minute budget. Then the counts:

  • views rows: 1.8 billion, against 1.9 billion requests handled. Data was lost.
  • Log lines: also short, because of the logger's buffer overflow.
  • Counting took a long time: the exact SELECT count(*) and wc -l on the 175 GB log finished after about another 10 minutes, with the database reading close to half a gigabyte per second from disk.
  • An approximate row count returned about 1.1 billion while the real number was 1.8 billion. The speaker's point: approximations are much faster, and how exact a user-facing number must be is a business decision.

Root cause. Requests arrived at 1 million per second, but the janitor flushed about 800,000 per second, falling behind about 200,000 per second. Past the cap, Redis trimmed the stream. The database was about half idle, so it wasn't the limit. The speaker first suspected the database pool size and considered raising it to 32, but that was only a hypothesis.

Changes for run 2. Per-stream cap from about 6 million to 12 million entries (about ten minutes of headroom), and the logger buffer fix.

Run 2. Again about 1 million per second and close to 2 billion requests. The janitor finished with about a minute to spare before the ten-minute mark. Request count, log lines and views rows matched exactly, once the 2,400 connections opened by the load test are accounted for, and no dropped-log messages appeared.

The real fix, found later. Four more threads for the janitor let it drain about 1.2 million per second, so it never fell behind in a later test that wasn't recorded. A bigger buffer only buys time; drain capacity above the peak rate removes the problem.

Alternate interview scenarios

"How do you verify a bulk load of billions of rows without a slow COUNT?"

Compare per-batch counts recorded by the consumer with per-interval request counts, or count per partition. Save the full exact count for an offline audit, as the speaker did, and expect it to be slow.

Sources (2)
  1. Same video, Final Gigantic Test (2:18:07)youtube.com
  2. PostgreSQL wiki: Count estimatewiki.postgresql.org