Loss of the whole cluster and redeploying new servers#

Important

Backups should be created and restored in a screen session - see: Using the screen command.

The high level redeploy and backup restore steps are as follows:

Redeploy the cluster.
Store the backup you want to restore in a different location.
Recreate the remote backups on the primary node using backup create <loc-name> <URI>.
Copy the saved backup under the new UID folder on the remote backup server.
Do a backup list

For example:

pxetest:
   URI: sftp://sftpusr:********@172.29.42.249/AS03
   Backups:
      1 backups have been created - most recently 2014-08-21 10:24

A backup restore can now be run on the primary.

The example console output below shows the steps and process:

Identifying the database primary:

platform@AS01:~$ database primary
172.29.42.100

Listing the backups:

platform@AS01:~$ backup list
 localbackup:
    URI: file:///backups
    Backups:
       2 backups have been created - most recently 2014-08-21 17:59
 pxetest:
    URI: sftp://sftpusr:********@172.29.42.249/AS01
    Backups:
       2 backups have been created - most recently 2014-08-21 12:54

You have new mail in /var/mail/platform

Restoring the backup:

platform@AS01:~$ backup restore pxetest 2014-08-21 12:54
Services will be restarted during the restore. Do you wish to continue? y
Application <name>-deviceapi processes stopped.
Stopping Application while performing database restore


---------- AS02, ip=172.29.42.101, role=webproxy,application,database, loc=cpt


Stopping nginx:proxy

---------- AS01, ip=172.29.42.100, role=webproxy,application,database, loc=cpt


Application nginx processes stopped.


---------- AS02, ip=172.29.42.101, role=webproxy,application,database, loc=cpt


Application nginx processes stopped.


---------- AS04, ip=172.29.21.191, role=webproxy,application,database, loc=jhb


Application nginx processes stopped.


---------- AS03, ip=172.29.21.190, role=webproxy,application,database, loc=jhb


Application nginx processes stopped.
System restore starting from
  sftp://sftpusr:[email protected]/AS01/ba1e37deff1309edcc2595bf46c6bfc2a99ca164
   (1408625665)
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Thu Aug 21 12:54:25 2014
Successfully restored to
  /backups/appdata/restore_temp_1408699183, moving to /backups/appdata
Removing temporary files in /backups/appdata/restore_temp_1408699183
local
Dropping database <name>_FILES before restoring
MongoDB shell version: 2.6.1
connecting to: 127.0.0.1:27020/<name>_FILES
[object Object]
Repairing database <name>_FILES before restoring
MongoDB shell version: 2.6.1
connecting to: 127.0.0.1:27020/<name>_FILES
[object Object]
Dropping database PLATFORM before restoring
MongoDB shell version: 2.6.1
connecting to: 127.0.0.1:27020/PLATFORM
[object Object]
Repairing database PLATFORM before restoring
MongoDB shell version: 2.6.1
connecting to: 127.0.0.1:27020/PLATFORM
[object Object]
Dropping database <name> before restoring
MongoDB shell version: 2.6.1
connecting to: 127.0.0.1:27020/<name>
[object Object]
Repairing database <name> before restoring
MongoDB shell version: 2.6.1
connecting to: 127.0.0.1:27020/<name>
[object Object]
Dropping database <name>_LOCKING before restoring
MongoDB shell version: 2.6.1
connecting to: 127.0.0.1:27020/<name>_LOCKING
[object Object]
Repairing database <name>_LOCKING before restoring
MongoDB shell version: 2.6.1
connecting to: 127.0.0.1:27020/<name>_LOCKING
[object Object]
Dropping database admin before restoring
MongoDB shell version: 2.6.1
connecting to: 127.0.0.1:27020/admin
[object Object]
Repairing database admin before restoring
MongoDB shell version: 2.6.1
connecting to: 127.0.0.1:27020/admin
[object Object]
Trying with oplogReplay
Trying without oplogReplay
restore successfull
Restarting services

Application  processes stopped.

Application  processes started.

System settings have changed, please reboot using 'system reboot'