Medium zalando companies

How does Elasticsearch avoid split brain, and why use dedicated master nodes?

The 30 second answer

Split brain means two halves of a partitioned cluster both elect a master and accept conflicting changes. Elasticsearch prevents it with quorums: a master election or cluster-state change needs responses from more than half of the voting configuration. From version seven the cluster manages that voting set itself. Dedicated masters keep elections and cluster-state updates away from heavy search and indexing load, so a busy data node does not destabilise the whole cluster.

The problem. A cluster has one elected master that owns the cluster state: which indices exist, their mappings and where every shard lives. If a network partition splits the nodes and each side elected its own master, both sides could accept writes and change the state independently. Reconciling them later would lose data. That is split brain.

The mechanism. The docs explain that decisions such as electing a master or committing a new cluster state are made "only after more than half of the nodes in the voting configuration have responded". In a partition at most one side can hold a majority, so at most one side can elect a master; the minority side cannot make progress. The quorums "are carefully chosen so the cluster does not have a split brain scenario".

What changed in version seven. Older clusters relied on an operator setting discovery.zen.minimum_master_nodes to a majority by hand, and a wrong value after adding nodes made split brain possible. From 7.0 that setting is permitted but ignored; Elasticsearch maintains the voting configuration itself as master-eligible nodes join and leave. Two operational rules remain: do not stop half or more of the voting nodes at once, and use the voting config exclusions API before removing many master-eligible nodes deliberately.

Why dedicated masters. The master does lightweight but critical work: elections, cluster-state publication and shard allocation decisions. If master-eligible nodes also run heavy searches and aggregations, a long GC pause or CPU saturation on a data node can delay cluster-state updates or trigger re-elections, which then triggers shard movement and makes the overload worse. Three small dedicated master-eligible nodes give a quorum that survives one failure and stays isolated from query load. The same isolation idea is why Zalando's outage post mentions dedicated coordinator nodes.

Upgrade order. Elastic's upgrade guidance upgrades data nodes first and master-eligible nodes last, because older nodes cannot always join a cluster with an upgraded master. Zalando's upgrade post mentions adding a human approval step so masters are not upgraded before data nodes.

Alternate interview scenarios

"Why not run two master-eligible nodes?"

With two voting nodes a majority is both of them, so losing either one stops the cluster from electing a master. Two gives no more fault tolerance than one. Three tolerates one failure, which is why three is the usual minimum for production.

"The cluster is red after a network blip and a node rejoined. What do you check first?"

Cluster health and the allocation explain API to see which shards are unassigned and why, the master node's logs for election and cluster-state publication timeouts, and whether long GC pauses on master-eligible nodes caused them to drop out. If masters share hardware with heavy data work, that is the structural fix.

Sources (4)
  1. Elasticsearch: Quorum-based decision makingelastic.co
  2. Elasticsearch 7.0 breaking changes: discovery changeselastic.co
  3. Elasticsearch: Upgrade an Elasticsearch cluster (node upgrade order)elastic.co
  4. Zalando Engineering: Migrating From Elasticsearch 7.17 to Elasticsearch 8.xengineering.zalando.com