Medium system-design scaling

The database is a single EC2 box at high load. What would you change for durability before production?

The 30 second answer

I wouldn't run the database near its limit, and I'd make the data survive the machine. One instance's disk is one place, so I'd ship write-ahead logs continuously to object storage like S3, which spreads copies across data centers, plus base backups and a replica. I'd keep CPU headroom on every tier, enable Redis persistence for the buffer, and fix the discarded-pending-entries bug.

What the speaker says. Don't push the database to its absolute limit, because you also need room for redundancy. Instance storage is a single location, so stream the write-ahead logs to S3 or another store that keeps copies in at least three data centers. Running your own Postgres is fine, he says, as long as you handle backups and updates. He also values the server's remaining 10% CPU: it is enough to keep serving logins, edits and other traffic during the peak.

What the test deliberately left out. Redis persistence was off. Spot instances were used. The janitor discarded pending entries after a restart. Each is fine for a benchmark and must change for production.

How I'd make it concrete. PostgreSQL continuous archiving: set wal_level to replica or higher, turn on archive_mode, and ship each WAL segment with archive_command or an archive library, plus regular base backups for point-in-time recovery. Add a streaming replica for failover. Note that WAL archiving and replication add write work on exactly the table that's growing 8 GB a minute, which is one reason to keep headroom.

Alternate interview scenarios

"Why not a managed database?"

Managed Postgres handles backups, failover and upgrades for you, at a higher price and with less control over tuning. The speaker argues self-managed is simpler than people think; a senior answer weighs the team's on-call capacity against the cost.

Sources (3)
  1. Same video, Battle-Testing Weer and Cpeak, Now in Rust (1:55:58)youtube.com
  2. PostgreSQL docs: Continuous archiving and point-in-time recoverypostgresql.org
  3. Redis docs: Persistence (RDB and AOF)redis.io