Kafka consumer crashes after processing but before committing the offset

The new owner of the partition, after a restart or a rebalance, resumes from the last committed offset. The record that was processed but not committed is delivered again. The Kafka design docs call this at-least-once consumption: process first, then save the position. The opposite order, commit then process, is at-most-once: a crash after the commit can skip the record entirely.

The default consumer setting enable.auto.commit is true in the Kafka 4.4 client configs, with auto.commit.interval.ms at five seconds. That interval is not tied to your handler finishing, so a crash can still land between a side effect and a commit, or can commit a record you have not finished. For anything with side effects I turn auto-commit off and commit after a successful write, either per record or after a batch I can defend.

The handler must therefore be idempotent. The usual pattern is a processed_events table keyed by a stable event id, written in the same database transaction as the business update. A replay hits the unique constraint and is skipped. If the write is naturally idempotent, an upsert or a monotonic version check on the aggregate is enough.

Exactly-once in Kafka is narrower than it sounds. The design docs describe it for a read-process-write pipeline inside Kafka: a transactional producer writes the output records and the consumer offset in one transaction, and downstream consumers use isolation.level=read_committed. "When writing to an external system," the same page says, you still have to coordinate the consumer's position with what you actually stored. A payment API, an email, or a row in a database that is not part of that transaction remains at-least-once plus idempotent effects.

A shorter scenario form of this question is also answered in Kafka consumer crashes after processing but before committing the offset.