How do you make processing idempotent when the same event arrives twice

Start by saying where the second copy came from, because the fix is different.

Producer retries. If the producer times out after the broker has already appended the record, a retry can write it again. Since Kafka 0.11 the idempotent producer assigns a producer id and a sequence number so those retries do not create a second log entry on that partition. In the Kafka 4.4 producer configs, enable.idempotence defaults to true when no conflicting settings are present, acks defaults to all, and max.in.flight.requests.per.connection defaults to 5, which is the maximum the idempotent producer allows while still preserving order. That default was introduced in Kafka 3.0 by KIP-679. In 3.0.0 and 3.1.0 a bug, KAFKA-13598, left the idempotence default unapplied unless you set it yourself; it was fixed in 3.0.1, 3.1.1 and 3.2.0. I still set enable.idempotence=true explicitly in application config so a conflicting acks or retries value cannot silently turn it off.

Consumer redelivery. This is the case the question is usually about. After a crash, a rebalance, or a commit that never reached the broker, the group restarts from the last committed offset and the handler sees the same record again. The idempotent producer does not help here. I put a unique event id on the payload (or use a unique business key such as orderId + eventType + version) and persist it with the side effect:

INSERT INTO processed_events (event_id, processed_at)
VALUES (:eventId, now())
ON CONFLICT (event_id) DO NOTHING;

If the insert is skipped, I skip the business write as well. Both statements must share one database transaction so I cannot apply the payment and then fail to record the id, or the reverse. An inbox table is the same idea with a pending state, useful when the side effect is itself remote.

If the write is naturally idempotent, I can skip the extra table: INSERT ... ON CONFLICT on the business key, or UPDATE ... WHERE version < :incoming so an older replay cannot overwrite a newer state. I do not use "check then insert" without a unique constraint; two threads will both pass the check.

Editor's note: Kafka transactions can fold a Kafka-to-Kafka write and the offset commit into one transaction. They do not make a card charge or an HTTP call exactly-once. For those I still want an idempotency key on the downstream API.