Modular Cluster: Election of a New Primary and Failover#
In the case where nodes fail, the system follows a failover procedure. For details on the failover and DR process, refer to the topics in the Platform Guide.
If the primary database is lost, the failover process involves the election of a new primary database by the remaining database nodes. Each node in a cluster is allocated a number of votes that are used in the failover election of a new primary database - the election of a running node with the highest database weight.
The database weights for a node can be seen as the priority
value when
running the database config command.
Note
The database weight of a node does not necessarily match its number of votes.
For voting on a modular system, arbiters are added to the database nodes (N1 and N2) on the primary site, but the secondary site database node (N3) arbiter will be unused.
members:
192.168.100.4:27020:
priority: 30.0
stateStr: SECONDARY
storageEngine: WiredTiger
192.168.100.4:27030:
priority: 30.0
stateStr: ARBITER
storageEngine: WiredTiger
192.168.100.6:27020:
priority: 40.0
stateStr: PRIMARY
storageEngine: WiredTiger
192.168.100.6:27030:
priority: 40.0
stateStr: ARBITER
storageEngine: WiredTiger
192.168.100.8:27020:
priority: 10.0
stateStr: SECONDARY
storageEngine: WiredTiger
The maximum number of votes in a cluster should not exceed 5 and arbiter votes are added to nodes to provide a total of 5 votes.
The tables below show the system status and failover for a selection of scenarios for a 6 node cluster. Also refer to the topics on the specific DR scenarios. The abbreviations used are as follows:
Pri : Primary site
DR : DR site
N : node. Primary node is N1, secondary node is N2.
w : database weight
v : vote
a : arbiter vote
For example, for a 6 node cluster with 3 database nodes and 2 sites, initial votes per node are as follows:
Primary database nodes (N1, N2): 2 (each 1 + 1 arbiter) Secondary database node (N3): 1 (no arbiter vote)
Pri N1 w:40 v:1 a:1 |
Pri N2 w:30 v:1 a:1 |
DR N3 w:10 v:1 |
Votes |
System Status under scenario |
---|---|---|---|---|
Up |
Up |
Up |
5 |
System is functioning normally. |
Up |
Up |
Down |
4 |
Scenario: Loss of a Non-primary Server in the DR Site. System continues functioning normally. |
Up |
Up |
Up |
5 |
Scenario: Loss of an Application Server. System continues functioning normally. Some transactions may hang. |
Up |
Down |
Up |
3 |
Scenario: Loss of a Non-primary Node in the Primary Site. System continues functioning normally. |
Pri N1 w:40 v:1 a:1 |
Pri N2 w:30 v:1 a:1 |
DR N3 w:10 v:1 |
Votes |
System Status under scenario |
---|---|---|---|---|
Down |
Up |
Up |
3 |
Scenario: Loss of the Primary Database Server. Some downtime occurs. System automatically fails over to N2. |
Down |
Down |
Up |
1 |
Scenario: Loss of a Primary Site. Manual recovery required. |
Up |
Down |
Down |
2 |
Scenario: Loss of all secondary nodes. Manual recovery required. |
Down |
Up |
Down |
2 |
Scenario: Loss of all Primary and DR nodes. Manual recovery required. |