How do you keep compatibility when a producer changes the event schema while older consumers are still running

This is a contract problem, not a Kafka-broker problem. The broker stores bytes. Compatibility lives in the schema, and in whether old and new code can both read those bytes.

Confluent Schema Registry (and compatible registries) enforce a compatibility mode when a new schema is registered:

• BACKWARD (the Confluent default): a new consumer can read data written with the previous schema. The safe rollout is to upgrade consumers first, then the producer. That does not help when older consumers are still running and the producer has already changed.
• FORWARD: an old consumer can read data written with the new schema. The safe rollout is to upgrade producers first. This is the mode that matches the question as asked.
• FULL: both of the above. FULL_TRANSITIVE and BACKWARD_TRANSITIVE / FORWARD_TRANSITIVE check against every previously registered version, not only the last one.

The changes that usually stay compatible are adding a field with a default (old consumers ignore it, or they use the default when the producer has not started writing it, depending on the direction) and adding a new optional member. The changes that break consumers are removing a field they still read, renaming a field, changing a type, or reusing a field id or name for a different meaning.

When I cannot control consumer upgrade timing, I choose FULL or FULL_TRANSITIVE, I keep producers on additive changes, and I roll the producer out gradually so a mistake is limited. For a truly breaking change I run two event types or two subjects in parallel, migrate consumers, and only then stop writing the old shape.

The same schema-registry answer, in a slightly more general wording, is also on How do you evolve Kafka message schemas with many consumer versions?.