What happens to in-flight work when a new consumer joins and partitions are reassigned

A new member joining, a member leaving, or a member missing a poll or a heartbeat all trigger a rebalance. What happens to work already in flight depends on the protocol and the assignor.

Classic protocol, eager assignor. This is still the starting point. In the Kafka 4.4 client, group.protocol defaults to classic, and partition.assignment.strategy defaults to [RangeAssignor, CooperativeStickyAssignor], which means Range is used until you remove it from the list. Range is eager. The ConsumerRebalanceListener Javadoc says that in eager rebalancing, onPartitionsRevoked is called with the full set of assigned partitions. Every member stops fetching, revokes everything, and waits for a new assignment. In-flight records that have been processed but not committed will be delivered again to whoever receives that partition, unless you commit in onPartitionsRevoked. The Javadoc recommends committing offsets in that callback "to prevent duplicate data." Processing on partitions that will be assigned back to the same member still pauses for the duration of the rebalance. That is the stop-the-world behaviour KIP-429 was written to avoid.

Classic protocol, cooperative sticky. CooperativeStickyAssignor (KIP-429) revokes only the partitions that must actually move. onPartitionsRevoked is invoked with that possibly-empty set, and only then, if it is non-empty. Partitions that stay with the member can keep being processed. The Kafka 4.4 consumer configs note that the default list is there so you can upgrade to cooperative sticky with a single rolling bounce that removes RangeAssignor. I do not mix eager and cooperative members in the same group during that bounce beyond what the upgrade procedure describes.

Static membership. KIP-345 lets me set group.instance.id to a stable, unique id per instance. The member is then static: a brief restart can rejoin with the same id inside the session timeout and keep its partitions, instead of causing a group-wide rebalance. The Kafka 4.4 default for group.instance.id is null (dynamic membership). This helps planned restarts and rolling deploys; it does not help if the process stays down longer than the session timeout.

KIP-848, the next-generation protocol. This is generally available from Kafka 4.0. The 4.0 release announcement and the current "Consumer Rebalance Protocol" page say the broker supports it automatically, but the client must opt in with group.protocol=consumer. The client default remains classic in Kafka 4.4. The new protocol is incremental and broker-coordinated; it no longer uses a global synchronization barrier. Heartbeat and session timeout move to the broker (group.consumer.heartbeat.interval.ms, group.consumer.session.timeout.ms). When it is on, the following classic client settings are not used: heartbeat.interval.ms, session.timeout.ms, partition.assignment.strategy, and enforceRebalance(). onPartitionsRevoked then behaves like cooperative mode: it is called with the partitions to revoke only when that set is non-empty. I would mention this in an interview as the direction of travel, and I would say it is an opt-in on the client, not a silent default.

Lost partitions. If the member is evicted (for example the session timeout expires) without a clean revoke, onPartitionsLost runs instead. Those partitions may already have been assigned elsewhere, so this is the wrong moment to commit offsets. The Javadoc's default is to call onPartitionsRevoked; I override it so I only clean up, I do not commit.