Scaling a real URL shortener to 1 million HTTP requests per second 13 questions in 2 chapters
The scenario: one redirect path, scaled step by step
1.1 Design and scale a real URL shortener's redirect to 1M HTTP requests/s and about 2B Postgres rows every 30 minutes. Where do you start? Hard 4 min
I'd start by pinning down what one redirect must do, because that's what scales. Here every click reads the link, appends a log line to disk and inserts a view row for paid analytics, and nothing may be lost. Then I fix the budget: one big app server, one load generator and a 16-core Postgres box. From there I load test, find the hardware bottleneck, fix it, and repeat.
Read the full answer1.2 How do you load test this honestly and find the real bottleneck? Medium 3 min
I generate load from a separate machine, and I check that machine too, so I don't benchmark my own load generator. Concurrency is connections times pipelining. While the test runs, I watch CPU, memory, disk and network on the tester, the server and the database. The bottleneck is always one hardware resource on one machine. If nothing is saturated, I'm simply not sending enough traffic.
Read the full answer1.3 The database CPU is pinned at 5,000 requests/s. What's the first fix, and does switching from Node.js to Go help? Easy 2 min
The first fix is an index on the short code. Without it, every redirect scans the links table, and that's what pinned the database CPU. A unique index took the lookup to a fraction of the time and lifted throughput about ten times, to roughly 50,000 per second. Switching to Go only gave about 60,000 against 53,000, because the app was I/O-bound, not CPU-bound.
Read the full answer1.4 At 50,000 requests/s the disk is mostly idle. Why is the write path still the bottleneck? Hard 2 min
Because every redirect runs its own insert transaction, and each commit waits for durable storage over the network, about a millisecond. The disk had plenty of IOPS and bandwidth left, so a faster SSD wouldn't help; the limit was round trips and commit latency per row. The fix is batching, but not inside the web processes, because a crash or deploy would lose the batch.
Read the full answer1.5 How do you batch the view writes without losing data if a process crashes? Hard 3 min
I put a stream between the web tier and Postgres. Every web process appends the view to a Redis stream, which is a fast in-memory write. A separate consumer, the janitor, reads in batches through a consumer group, writes each batch in one transaction, and only then acknowledges it. Unacknowledged entries stay pending and can be retried. That roughly tripled throughput, until a single Redis instance hit 100% CPU.
Read the full answer1.6 Redis is no longer the limit, but Postgres CPU is. How does caching the links get you to half a million? Medium 2 min
I cache code-to-URL lookups in Redis with cache-aside: check Redis, fall back to Postgres on a miss, then store the result. On edit or delete I simply delete the key instead of updating it, because a Postgres re-fetch is cheap. That left the database about 70% idle and took throughput to roughly half a million per second, after a short warm-up while the cache filled.
Read the full answer1.7 Your write buffer lives in RAM. How do you size it, and what happens if the consumer falls behind? Hard 3 min
A buffer in RAM has to be capped, so lag becomes a time budget. At a million views a second, a cap of about 300 million entries is roughly five minutes of headroom and about 36 GB of Redis memory. If the consumer falls behind longer than that, the stream gets trimmed and views are lost. So I size the cap from rate times tolerated outage, and make the consumer drain faster than peak.
Read the full answer1.8 The app server is out of CPU at half a million. Do you scale up, scale out, or rewrite? Hard 3 min
With a fixed budget, I'd look at what each request costs in CPU and memory. In the video, Node.js used all 48 cores and about 10 to 12 GB of RAM at half a million. The rewrite in a native language reached about 1.2 million with roughly 10% CPU idle and half a gigabyte of RAM. Without a fixed budget, adding servers behind a load balancer is usually cheaper than a rewrite.
Read the full answer1.9 What broke in the Postgres schema and insert path only at a million rows per second? Hard 3 min
Things that are fine at small scale. A 32-bit ID would run out with billions of rows, so it became 64-bit. The foreign key from views to links halved throughput, so it was removed and deletes are cleaned up in code. The views table has no primary key, the visitor hash was shortened, and batched multi-row inserts capped near 700,000 rows a second until he switched to COPY.
Read the full answer1.10 How do you keep request logging from becoming the bottleneck, or silently dropping lines? Medium 2 min
At a million requests a second the log grew about 6 GB a minute and reached 175 GB in thirty minutes. The framework's async logger dropped lines when its buffer overflowed, rather than slowing requests down. So I'd size the buffer, flush on an interval, and alert on dropped lines. In the video, a 4 MB buffer flushed every two seconds fixed it, at the cost of a two-second delay.
Read the full answer1.11 Walk through the final 30-minute test. How do you prove nothing was lost? Hard 3 min
I count every side effect independently. The first thirty-minute run averaged about a million per second, close to two billion requests, but the views table had 1.8 billion rows against 1.9 billion requests, and logs were dropped too. The consumer drained only about 800,000 a second, so the capped stream trimmed data. After doubling the cap and the log buffer, requests, rows and log lines matched exactly.
Read the full answer1.12 The database is a single EC2 box at high load. What would you change for durability before production? Medium 2 min
I wouldn't run the database near its limit, and I'd make the data survive the machine. One instance's disk is one place, so I'd ship write-ahead logs continuously to object storage like S3, which spreads copies across data centers, plus base backups and a replica. I'd keep CPU headroom on every tier, enable Redis persistence for the buffer, and fix the discarded-pending-entries bug.
Read the full answer1.13 What should a senior candidate say about cost, and when is this scale overkill? Medium 2 min
I'd say the hardware is the easy part; at this scale egress dominates the bill. The video estimates about $100,000 a month on a cloud provider, mostly for 4.8 petabytes sent to clients, and argues on-premises servers are cheaper at that volume. A real setup also needs at least four servers, a load balancer, a separate Redis machine and backups. Most products never get close, so I'd design for measured traffic.
Read the full answer
Sources
- Scaling to 1 Million HTTP Requests/s, with Real Business Logic! (YouTube)
- autocannon README
- PostgreSQL docs: Populating a Database
- PostgreSQL docs: COPY
- PostgreSQL docs: Asynchronous commit
- PostgreSQL docs: Continuous archiving and point-in-time recovery
- PostgreSQL docs: Numeric types
- PostgreSQL wiki: Count estimate
- Redis docs: Streams
- Redis docs: XADD
- Redis docs: Persistence
- AWS docs: Spot Instance interruption notices
- Microsoft Azure Architecture Center: Cache-Aside pattern