Medium system-design scaling

How do you load test this honestly and find the real bottleneck?

The 30 second answer

I generate load from a separate machine, and I check that machine too, so I don't benchmark my own load generator. Concurrency is connections times pipelining. While the test runs, I watch CPU, memory, disk and network on the tester, the server and the database. The bottleneck is always one hardware resource on one machine. If nothing is saturated, I'm simply not sending enough traffic.

The tool. The video uses autocannon. Three flags matter: -c (TCP connections), -p (pipelined requests per connection) and -w (worker threads on the load generator). In-flight requests at any moment are c × p; with -c 6 -p 2 the server is always handling 12 concurrent requests. The main runs use 2,400 connections and 48 workers.

Throughput versus latency. Autocannon reports both. The speaker stresses that they can trade off: batching raises throughput but makes each request wait longer.

Reading CPU numbers. One fully busy core can show as 100% of one core (so 48 busy cores show as 4,800%) or as a share of all cores. Know which method your tool uses before you call a machine saturated.

The method he uses at every step. Run the test and watch CPU, RAM, disk IOPS and throughput, and network on all three machines. The bottleneck is a hardware component on a specific machine. That includes the tester.

The mistake that shows why. The first naive run averaged only about 1,000 requests per second (25,000 requests in total). The server, the tester and the network were all nearly idle: about 3 GB of RAM used, 200 KB/s of disk writes, 400 KB/s of network. The cause was a typo in the load script that fell back to five connections. When nothing is saturated, the load generator is the problem. With 2,400 connections and 48 workers, the database CPU hit 100% and throughput settled at about 5,000 requests per second.

Verify the results, not only the tool output. After each run he counts the views rows and the log lines and checks that they match the requests autocannon sent. A load test that silently loses writes is a wrong result.

Alternate interview scenarios

"The load test shows low throughput but every machine is idle. What's going on?"

Not enough concurrency is reaching the server: too few connections, a client-side limit, or a bug in the test script. Check the tester's own CPU and network and the actual connection count before tuning the server.

"Which numbers would you report from the load test?"

Average and percentile latency, max latency, requests per second with its spread, error and status-code counts, and resource usage on each machine. Then an independent count of side effects such as rows and log lines.

Sources (4)
  1. Same video, CPU Utilization & Threads (3:47)youtube.com
  2. Same video, Introduction to AutoCannon (11:57)youtube.com
  3. Same video, Taking the Hardware for a Test Drive (37:49)youtube.com
  4. autocannon README (connections, pipelining, workers)github.com