What is the difference between at-most-once, at-least-once and exactly-once

The Kafka design docs define the three guarantees in terms of what happens when a producer or a consumer fails.

At-most-once. Messages may be lost and are never redelivered. On the consumer, that means saving the position and then processing. If the process dies after the commit, the record is skipped. On the producer, it means not retrying, or using acks=0. I only choose this when losing a record is acceptable, for example a metric sample.

At-least-once. Messages are never lost but may be redelivered. The consumer processes and then commits. The design docs state that Kafka "guarantees at-least-once delivery by default." A crash after the side effect and before the commit, or a producer retry after a lost acknowledgement, can produce a duplicate. The practical answer is an idempotent consumer, and, since Kafka 3.0, the idempotent producer which is on by default so producer retries do not duplicate a record inside a partition.

Exactly-once. "Each message is processed once and only once," but the fine print matters. Since 0.11.0.0 the transactional producer can write to several partitions atomically, and the consumer's committed offset can be included in the same transaction. Consumers that should not see aborted transactions must set isolation.level=read_committed. The Kafka 4.4 default is still read_uncommitted. The transactional producer also needs a transactional.id; without it the producer is only idempotent, not transactional. Auto-commit must be off (enable.auto.commit=false).

The docs are explicit about the boundary: this gives exactly-once when you are reading, processing and writing on Kafka topics, as Kafka Streams does. A write to an external system still needs cooperation with that system, typically by storing the offset next to the output, or by making the output idempotent. I say that out loud in an interview. Otherwise it sounds as if turning transactions on will make a payment API safe, and it will not.