Hard java backend

Your API p99 went from 200 ms to 2 s. What do you do?

The 30 second answer

First I ask what changed: a deploy, a traffic spike, or a data change. I check whether it is all endpoints or one, all pods or one. Then I open slow traces to see whether time is in the database, our code, or a downstream call. If a deploy caused it, I roll back first and investigate after, because users come before root cause.

First, what changed? A deploy, a traffic spike, a data change. I check dashboards: is it all endpoints or one, all pods or one. Then I open slow traces to see if time is in the database, our code, or a downstream call. If it's the database, I check pg_stat_statements and EXPLAIN ANALYZE for a new slow query or a changed plan, plus locks and connection pool wait time. If a deploy caused it, I roll back first, then investigate.

Sources (3)
  1. OpenTelemetry: Traces
  2. PostgreSQL 17 docs: pg_stat_statements
  3. PostgreSQL 17 docs: Using EXPLAIN