The 30 second answer
I keep dashboards for the four golden signals: latency, traffic, errors and saturation, with latency shown as percentiles. Alerts fire on symptoms users feel, like p99 above the target for several minutes, not on every CPU blip. A performance check in CI or before release catches regressions before production, so the latency win does not slowly disappear.
The Google SRE book's four golden signals are latency, traffic, errors and saturation. Alert on symptoms such as p99 latency and error rate, not on causes.
Don't optimize without measuring, and prove each gain with before and after numbers.