The 30 second answer
A rolling upgrade is Elastic's usual path, but on a large live catalog it means a long mixed-version window with relocating shards, and if it fails, recovery is a snapshot restore plus replaying data on a cluster customers depend on. A parallel cluster lets you restore a snapshot, shadow writes and queries, compare, and shift traffic gradually, with rollback by routing. Zalando's public upgrade post describes this choice.
Zalando's engineering blog describes how its search department moved Elasticsearch from 7.17 to 8.x. It is a strong source for questions about upgrades, data synchronisation and client migration.
What Elastic recommends. To reach 8.x from an older 7.x you first upgrade to 7.17, use the Upgrade Assistant to resolve critical issues, and then do a rolling upgrade (data nodes first, master-eligible nodes last). The alternative is a new cluster and copying data across with reindexing.
Why the post chose a parallel cluster. The clusters were large, holding "some terabytes" across a multi-cluster deployment, so a rolling upgrade meant a long degraded window with mixed versions and relocating shards. The real fear was failure: data loss would force a restore from snapshots and a reset of the input streams for every index, during which the catalog would be unavailable or stale. With a second cluster, cutover and rollback become routing changes, and both clusters can be compared side by side.
The procedure the post lists, per cluster.
- Deploy the 8.x cluster and its monitoring, and create index templates.
- Restore the latest snapshot.
- Shadow the intake traffic, the bulk and write-alias requests, to the new cluster, using the Skipper ingress's traffic-teeing filters. Template and mapping requests were shadowed too, because templates live in code and are applied on application restart.
- Close the gap between snapshot time and the moment shadowing began by resetting the data streams to just before the snapshot.
- Shadow query traffic gradually and compare latency, errors and index sizes on dashboards.
- Shift live traffic in percentages, then clean up the old cluster.
Client migration. The main search application was Scala code using the Java High Level REST Client, deprecated since 7.15. The team chose the official Elasticsearch Java API Client over a popular community Scala client, mainly because the official one follows Elastic's releases and already had a DSL for kNN, which was the feature they were upgrading for. They started with the old client in REST API compatibility mode so clusters could move first, and tested three combinations with Testcontainers: new client with old cluster, old client with new cluster, and new client with new cluster.
Breaking changes the post hit. _type disappearing from search responses, is_write on an alias no longer accepting null, numeric bounds on date range queries needing to be strings, action.destructive_requires_name now defaulting to true (which broke tests that deleted indices by wildcard, so it was relaxed only in tests), and sorts on nested fields failing unless nested.path was set. Treat these as that codebase's experience, not a complete list.
The incident it reports. A routing error during one country's cutover created an extra index behind the same alias, so products appeared twice across that country until the stray index was dropped.
Alternate interview scenarios
"After restoring a snapshot onto the new cluster, how do you keep it in sync?"
A snapshot is a point in time, so the new cluster misses everything between the snapshot and the moment it starts receiving writes. Shadow live writes first, then replay the source streams from just before the snapshot, relying on idempotent, versioned writes so replayed events cannot overwrite newer ones. Watch document counts and index sizes on both clusters until they converge, then shadow reads.
"How do you test a period where some clusters run 7.x and others 8.x?"
Enable REST API compatibility on the 7.x High Level REST Client, which the docs say sends headers asking 8.x to honour 7.x requests and responses, and move the client later. Compatibility only bridges one major version, so it is a transition tool. Run the important query shapes against both versions in containers, including the features that justify the upgrade.
"How do you change a mapping on a live index without downtime?"
Most changes to an existing field's mapping cannot be made in place, so create a new index with the new mapping, reindex or replay into it, then move the alias in a single _aliases request with a remove and an add action, which the docs describe as atomic: the alias never points to both indices. If an alias does point to several indices, writes need an explicit is_write_index, and reads see both, which is how duplicates appear.
"Two indices are behind one alias and shoppers see duplicate products. What do you do?"
Confirm with the get-alias API which indices the alias resolves to, remove the stray index from the alias in one atomic _aliases call (or drop it, as the post's team did), and check document counts against the source. Afterwards, add a pre-cutover check that every alias resolves to exactly one index.
Sources (7)
- Zalando Engineering: Migrating From Elasticsearch 7.17 to Elasticsearch 8.x: Pitfalls and Learningsengineering.zalando.com
- Elasticsearch 8.19: Upgrade Elasticsearchelastic.co
- Elasticsearch: Upgrade an Elasticsearch cluster (node upgrade order)elastic.co
- Elasticsearch Java REST Client 7.17: Compatibility with Elasticsearch 8.xelastic.co
- Elasticsearch 8.19: REST API compatibilityelastic.co
- Elasticsearch: Aliaseselastic.co
- Elasticsearch API: Reindexelastic.co