Hard java spring

Kafka consumer lag keeps rising, but CPU and DB look normal. Why?

The 30 second answer

When CPU and the database look fine, the consumer is usually waiting rather than working. Common causes are a hot partition from a skewed key, rebalances because processing takes longer than the max poll interval, slow synchronous calls to another service, or tiny fetch batches. I check lag per partition with the consumer groups tool and the consumer lag metrics before changing anything.

If nothing is busy, the consumer is waiting, not working. Check: fewer partitions than consumers you need (one partition goes to only one consumer in a group); a hot partition from a skewed key; rebalances caused by processing slower than max.poll.interval.ms; slow synchronous calls to another service; or small fetch batches. I check lag per partition with kafka-consumer-groups.sh --describe and the consumer's lag metrics. Follow-up: How do you scale past the partition count? (Add partitions, which changes key placement, or process records in parallel inside the consumer with per-key ordering.) Editor's note: the list of causes comes from experience; the tools and config names come from the Kafka docs.

Sources (2)
  1. Kafka consumer configs (max.poll.interval.ms, fetch settings)
  2. Kafka: Basic operations (consumer groups)