Why can one hot partition stall a consumer group, and how do you fix it

The default partitioner hashes the record key. If one key is much hotter than the rest, one partition receives most of the traffic. In a consumer group that partition is assigned to a single member, so that member's lag grows while the other members have little to do. Adding consumers does not spread a single partition's load.

I confirm it with per-partition lag, not with a single group-level number. One large value next to many near-zero values is the signature. The produce rate per partition, if I have it, tells the same story.

The fix is to change how records are keyed, in a way that matches the job:

• If I only need a count or a similar commutative aggregate, I avoid the hot natural key. I key randomly or with a salt such as productId#0..N, pre-aggregate per partition, then merge in a second stage by the real id.
• If I must preserve per-entity order, I cannot randomly rekey that entity's events. I can split only the hot entities (salting a known-hot id and reordering in a later step), or I can scale the work inside that partition with care, still serialising events for the same key.
• A producer-side combiner that batches obvious repeats, for example local counts over a short window, reduces the volume before it hits the hot partition.

For a trending-search or top-K use case, an approximate structure such as a count-min sketch plus a bounded heap is usually enough; exact counts of every term are rarely required. I would say that approximation is a product choice, not a Kafka requirement.

Editor's note: Adding partitions helps only if the new keying actually spreads the hot traffic. Adding partitions while still keying on the hot id just moves the same skew onto a new partition number.