The 30 second answer
Kafka only preserves order within a single partition. I use the entity id, such as the order id, as the message key so every event for that entity lands on the same partition and is consumed in offset order. I keep one consumer in the group on that partition, and I leave producer idempotence on so retries cannot reorder in-flight batches. I do not add partitions later without a plan, because that changes the key-to-partition map.
The Kafka introduction is unambiguous: a topic is a partitioned log, and "there is no global ordering across partitions." Order is the offset order inside one partition. If OrderCreated and OrderPaid for the same order are hashed to different partitions, a consumer can see payment first.
The fix starts at the producer. I set the record key to the entity id (the order id, the SKU, the account id) so the default partitioner sends every event for that entity to the same partition. One consumer in a group is assigned that partition, so it sees the records in order.
Two producer settings can still break that order. If enable.idempotence is false and max.in.flight.requests.per.connection is greater than one, a retried batch can land after a later batch that already succeeded. The Kafka 4.4 producer docs spell this out. With idempotence enabled, which is the default when settings do not conflict, ordering is preserved and max.in.flight.requests.per.connection may be at most 5. I leave those defaults alone unless I have a measured reason not to.
On the consumer, I do not process records from the same partition on many threads unless I pin all records with the same key to the same worker. Otherwise two workers can apply OrderPaid before OrderCreated. I also treat non-blocking retry topics with care: a failed record that is forwarded to a retry topic no longer blocks the partition, but a later event for the same key can be processed first. If order matters, I retry in place, or I retry in a way that still keys by the entity.
Adding partitions later changes the hash of every key, so new events for an existing entity can land on a different partition and appear "out of sequence" relative to the old ones. If I must expand, I plan a migration, or I accept that only events written after the change are comparable on the new assignment.
Consumers can also apply a version or event-time check and drop a stale event, which is how an upsert of "latest state per key" stays correct under retries. That is a safety net, not a substitute for keying correctly.