The 30 second answer
A poison record fails every time, so a blocking retry loop stalls the whole partition and lag grows behind it. I treat deserialization and validation errors as non-retryable and publish them to a dead-letter topic, then commit and continue. Spring Kafka's ErrorHandlingDeserializer catches corrupt payloads before the listener, and DefaultErrorHandler with DeadLetterPublishingRecoverer can park the record. I alert on dead-letter volume so the parked events are actually fixed.
Kafka consumers in a group read a partition in offset order. If I retry the same record indefinitely in the listener thread, I never reach the records behind it. Lag on that partition grows, and every later event for keys on that partition is delayed. That is a poison pill.
The first split is retryable versus not. A database outage or a 503 from a dependency is transient. A payload that will never deserialize, or that fails a schema or validation rule, will not improve with time. The second class must leave the partition.
In Spring Kafka, deserialization errors happen before the listener runs, so a try/catch in the handler never sees them. ErrorHandlingDeserializer wraps the real deserializer, records the failure on the headers, and lets the error handler take over. DefaultErrorHandler, which replaced SeekToCurrentErrorHandler, can then give up and call a DeadLetterPublishingRecoverer, which publishes the record to a dead-letter topic. I register deterministic exceptions with addNotRetryableExceptions(...) so they skip retries and go straight to the recoverer. After recovery, the consumer can commit and continue with the next offset.
The trade-off is the same one any cursor-based log has: skipping preserves flow and risks a gap; stopping preserves order and stalls everything behind the bad record. Parking the record on a dead-letter topic, with enough headers (original topic, partition, offset, exception) to replay it, is how I skip without forgetting. I alert on the size or rate of that topic. A dead-letter topic that nobody reads is just a quieter form of data loss.
If the failure is transient, I do not use this path on the first attempt. That is the next question.