The 30 second answer
I use a schema registry with a compatibility mode. Backward, the default, means new consumers can read old data, so consumers upgrade first. Forward means old consumers can read new data, so producers upgrade first. Full means both, and the transitive variants check against all past versions. Safe changes are adding optional fields with defaults. I never rename or reuse a field.
Use Schema Registry with a compatibility mode. The default is BACKWARD: new consumers can read old data, so upgrade consumers first. FORWARD means old consumers can read new data, so upgrade producers first. FULL means both. The _TRANSITIVE variants check against all past versions, not only the last. Safe changes are adding optional fields with defaults; never rename or reuse a field. Follow-up: Which mode when you can't control consumer upgrade timing? (FULL or FULL_TRANSITIVE.)