How do you handle growing consumer lag when producers outpace consumers

Lag is how far the consumer's committed position sits behind the log end offset. Kafka consumers pull, so lag is also the backpressure signal: a slow group simply falls behind instead of being flooded.

I first look at where the lag is growing, with kafka-consumer-groups.sh --describe and the client's records-lag-max metric.

Every partition is behind, and the consumers are busy. The group cannot keep up with the produce rate. I shorten the work per record, cut synchronous downstream calls, or raise throughput inside the current partition count, for example by processing independent keys in parallel. I keep max.poll.records (default 500 in Kafka 4.4) and processing time inside max.poll.interval.ms (default five minutes) so the member is not evicted while it works. If I still need more parallelism, I add partitions and then add consumers. A group assigns each partition to only one member, so starting ten consumers on a three-partition topic leaves seven idle. Adding partitions changes key placement, so I treat that as a planned change, not a live toggle.

Every partition is behind, but CPU and the database look idle. The consumer is waiting, not working. Typical causes are a blocked poll loop, a session that is rebalancing, tiny fetch sizes, or a downstream call that is slow but not visibly burning CPU. That investigation is also answered in Kafka consumer lag keeps rising, but CPU and DB look normal. Why?.

Only one partition is behind. This is a skewed key or a stuck record, not a missing consumer. More members will not help. I fix the key or park the poison record.

For a temporary downstream outage I pause the assigned partitions with KafkaConsumer.pause and resume them later, rather than buffering an unbounded in-memory queue. The KafkaConsumer Javadoc documents this flow-control API. I alert on lag and on the rate at which lag is growing, not only on a static threshold.