Cache check in the hot path: Redis did what Postgres did

Ten days after we cached country labels, the board went from ~1.7s back to unusable. The trigger was a reasonable feature: countries added through MCP should show up in the panel without a restart. The panel and the MCP server are separate processes. A process-local cache does not see another process’s write.

We made the cache generation-aware. On every read it asked Redis whether the generation had moved. That check ran inside country_label(), which runs per company on every board load. Same shape as 678 Postgres round-trips. Redis instead of Postgres.

What we saw

With Redis on (the real deploy), overview and preview climbed into seconds. With Redis unset, nothing happened. Tests were green. The expensive branch was invisible in the environments where we looked.

Board back to ~1.3–1.7s. Cross-process propagation lags at most 5 seconds.

A cache hit that was not a hit

Postgres stays the source of truth. Writes also bump a Redis generation counter. Any process can reload when the counter moves. That design is fine. The read side was not:

# ran on every all_country_labels() / country_label() call
def _countries_generation_is_current() -> bool:
    if not countries_use_redis():        # Redis PING
        return True
    current = get_countries_generation() # Redis GET
    return _countries_cache_generation == current

Two round-trips on every “hit.” For ~100 companies that is hundreds of PINGs and GETs per board load. The in-memory cache from the previous incident was still there. It never got to matter.

if not countries_use_redis(): return True short-circuits when REDIS_URL is unset. Local dev and the unit tests took that path. Production did not.

What we shipped

Cross-process propagation does not need to be instantaneous. A new country appearing a few seconds later is acceptable. Throttle the Redis check to at most once every 5 seconds. Every other call compares an in-memory timestamp and returns. Local writes still clear the cache immediately. Other processes converge within 5s.

if now - _checked_at < 5.0:
    return True   # hot path: zero I/O

The test that would have caught it is not “does the new country show up.” That passes with the bug. The test is: 500 calls to country_label() inside the TTL cause zero extra Redis reads.

The rule

A cache read that does I/O is not a cache. If a path behaves differently with Redis on, test it with Redis on. Instant cross-process consistency is a product choice. Here it was not worth a per-call network round trip. Five seconds of staleness removed the entire problem.