Medium system-design api

How do you evolve a Protocol Buffers message without breaking old clients?

The 30 second answer

I treat field numbers as immutable identifiers. Adding a field is safe: old binaries ignore it, and new binaries see a default when old senders omit it. Removing a field is safe only if I reserve its number, and preferably its name, so nobody reuses it. Changing a field number, or recycling a deleted number, is a breaking wire change. I use the binary encoding for compatibility-sensitive interchange, because the JSON mapping is stricter.

Protobuf's binary wire format is designed so many schema changes are backward- and forward-compatible when field numbers are treated as immutable identifiers. The proto3 language guide splits changes into wire-safe, wire-unsafe, and conditionally compatible.

Binary wire-safe:

  • Adding new fields is safe. Old binaries ignore unknown fields; new binaries see default values when old senders omit the field. Proto3 preserves unknown fields and writes them back out.
  • Removing a field is safe if its number is never reused. Reserve the deleted number, and preferably the name, for JSON and text formats:
message Foo {
  reserved 2, 15, 9 to 11;
  reserved "foo", "bar";
}
  • Adding values to an enum is safe, with the usual caution that exhaustive switches in application code may need updating.

Not safe: changing a field's number; reusing a deleted number; moving fields into an existing oneof. Reusing a number makes decoding ambiguous and can corrupt data, because the wire format has no way to tell that the definition changed.

Conditionally compatible means the same bytes may parse before and after the change, but the change can be lossy. Examples include widening int32 to int64 if a value later exceeds what the old type can hold; string and bytes when the bytes are valid UTF-8; and enum with integer types. These need a carefully managed rollout, and they are a poor idea if you cannot control every reader and writer.

JSON encoding of protobuf has a narrower set of safe changes than the binary wire format. The official guide treats the binary format as the preferred form for compatibility-sensitive interchange.

Alternate interview scenarios

"Can I renumber fields so the numbers look tidy?"

No. Changing a field number is equivalent to deleting the field and adding a new one. Old senders will still write the old number, and new readers will not see the value where they expect it. If I no longer need a field, I delete it and reserve the number.

"We serialise protobuf as JSON to a mobile app. Are the same rules enough?"

Not always. The binary-safe changes above are the baseline, but the JSON mapping is stricter, and reserved names matter more because the field name appears in the payload. I would check the official JSON mapping before calling a change compatible for those clients.

Sources (2)
  1. Protocol Buffers proto3: Updating a message typeprotobuf.dev
  2. Protocol Buffers proto3: Unknown fieldsprotobuf.dev