A Kafka consumer's lag is huge but the pod looks healthy. What do you do

Process health and consumer health are not the same thing. A liveness probe that only checks whether the JVM answers HTTP will stay green while the consumer has been evicted from its group and is no longer fetching.

The picture I look for is **lag climbing and records-consumed-rate at zero** (or stuck at a very low value) on a process that still looks up. Common mechanisms, all of which sit in the client configs and the consumer Javadoc:

• The listener blocked for longer than max.poll.interval.ms (default five minutes in Kafka 4.4). The group coordinator then considers the member failed and reassigns its partitions. If the poll loop never recovers, this instance becomes a zombie: the pod is up, the group has moved on.
• Heartbeats stop. On the classic protocol, session.timeout.ms defaults to 45 seconds and heartbeat.interval.ms to 3 seconds. A blocked or dead thread that cannot heartbeat is removed from the group.
• Offset commits time out, or a call in the listener never returns, so poll is not invoked in time. Spring Kafka can only recover if the container's poll loop keeps running.

Fixes, in order:

1. Confirm with per-partition lag, records-consumed-rate, and rebalance or expired-member logs, not with kubectl get pods.
2. Keep handler work under max.poll.interval.ms. Reduce max.poll.records so each poll is smaller. Do not hand work to an unbounded worker pool and then wait forever to commit; if I use a pool, it is bounded and the commit still happens in a controlled way.
3. Subscribe to container events such as ConsumerStoppedEvent and restart the listener container.
4. Point the Kubernetes liveness or a custom health indicator at consumer progress: lag growing and no records consumed for a defined period means unhealthy, so the pod is recycled. A static group.instance.id reduces extra rebalances when that restart is short.
5. Alert on lag and consumption rate, not only on pod restarts.

This is the "zombie consumer" interview story. The idle-CPU lag investigation on the senior Java page is a close cousin; here the distinctive clue is that the process looks fine while the consumer is not actually in the group.