DR Failover and Recovery in a 2 Node Cluster

Important

A 2 node cluster will not fail over automatically.

With only two Unified nodes, with or without Web proxies, there is no High Availability. The database on the primary node is read/write, while the database on the secondary is read only.

Only redundancy is available.

  • If the primary node fails, a manual delete of the primary node on the secondary and a cluster provision will be needed.
  • If the secondary node fails, it needs to be replaced.

Scenario: Loss of Primary Node

  • The administrator deployed the 2-node cluster.

    $ cluster status
    
    Data Centre: jhb
                 application : AS01[172.29.42.100]
                               AS02[172.29.42.101]
    
                 webproxy :    AS01[172.29.42.100]
                               AS02[172.29.42.101]
    
                 database :    AS01[172.29.42.100]
                               AS02[172.29.42.101]
    

    Example database weights:

    $ database weight list
         172.29.42.100:
             weight: 20
         172.29.42.101:
             weight: 10
    
  • Node Failure: in the case where the primary node is lost on the Primary site:

    $ cluster status
    
    Data Centre: unknown
                 application : unknown_172.29.248.100[172.29.248.100] (not responding)
    
                 webproxy : unknown_172.29.248.100[172.29.248.100] (not responding)
    
                 database : unknown_172.29.248.100[172.29.248.100] (not responding)
    
    
    Data Centre: jhb
                 application : AS02[172.29.248.101]
    
                 webproxy : AS02[172.29.248.101]
    
                 database : AS02[172.29.248.101]
    

Recovery Steps

The primary node server is lost.

  1. It is decided to fail over to the secondary node:

    1. On the secondary node, remove the lost server from the cluster:

      cluster del 172.29.248.100

    2. On the secondary node, run cluster provision (it is recommended that this step is run in a terminal opened with the screen command).

      On the secondary node, check:

      $ cluster status
      
      Data Centre: jhb
                   application : AS02[172.29.248.101]
      
                   webproxy : AS02[172.29.248.101]
      
                   database : AS02[172.29.248.101]
      
  2. It is decided to recover the primary node:

    1. On the secondary node, remove the lost server from the cluster:

      cluster del 172.29.248.100

    2. On the secondary node, run cluster provision (it is recommended that this step is run in a terminal opened with the screen command).

      On the secondary node, check:

      $ cluster status
      
      Data Centre: jhb
                   application : AS02[172.29.248.101]
      
                   webproxy : AS02[172.29.248.101]
      
                   database : AS02[172.29.248.101]
      
    3. Switch on the newly installed server.

      On the secondary node, add the server. Run cluster add 172.29.42.100.

      On either node, check:

      $ cluster status
      
      Data Centre: jhb
                   application : AS01[172.29.42.100]
                                 AS02[172.29.42.101]
      
                   webproxy :    AS01[172.29.42.100]
                                 AS02[172.29.42.101]
      
                   database :    AS01[172.29.42.100]
                                 AS02[172.29.42.101]
      
    4. Configure the primary database. On the newly installed server, run cluster provision primary 172.29.42.100 (it is recommended that this step is run in a terminal opened with the screen command).

      Check database configuration on both nodes, for example:

      $ database config
          date:
              $date: 1549450382862
          heartbeatIntervalMillis: 2000
          members:
              172.29.42.100:27020:
                  priority: 20.0
                  stateStr: PRIMARY
                  storageEngine: WiredTiger
              172.29.42.100:27030:
                  priority: 1.0
                  stateStr: ARBITER
                  storageEngine: Unknown
              172.29.42.101:27020:
                  priority: 10.0
                  stateStr: SECONDARY
                  storageEngine: WiredTiger
          myState: 1
          ok: 1.0
          set: DEVICEAPI
          term: 8