.. _upgrade_multinode: .. rst-class:: chapter-with-expand Unified Node Topology: Upgrade a Multinode Environment with the Delta Bundle ---------------------------------------------------------------------------- .. index:: cluster;cluster upgrade .. index:: cluster;cluster status .. index:: screen .. _19.1.1|EKB-1249: .. _19.1.2|EKB-2262: .. _19.2.1|EKB-2636: .. _19.2.1|VOSS-497|EKB-3126: .. _19.3.1|EKB-3806: .. _19.3.4|EKB-6669: .. _19.3.4|EKB-1969: .. _19.3.4|EKB-4132: .. _19.3.4|EKB-6084: .. note:: * While system upgrade takes approximately two hours at a single site, this may vary in accordance with your topology, number of devices and subscribers. Adjust your upgrade maintenance window to allow for your configuration. The standard **screen** command should be used where indicated, and the `reconnect` parameter is available if needed: * **screen** - start a new session * **screen -ls** - show sessions already available * **screen -r [screen PID]** - reconnect to a disconnected session We recommend using the **screen** command to avoid failures if the connection is interrupted whilst running the command. If the connection is interrupted whilst running the command in ``screen`` then the session can be retrieved by first listing the sessions PID currently running in screen: **screen -ls**, and then reconnecting to the session using **screen -r [screen PID]**. The version of **screen** used in VOSS Automate also supports the creation of a log file. If long-running commands will be run, the log file captures screen console output up to the session timeout. A message shows: :: timed out waiting for input: auto-logout To create a screen log file: 1. Run **screen** and wait for screen to open. 2. Press **-a** then **:** (colon). This will enter screen command mode at the bottom of the console. 3. Create your screen logfile in the ``media/`` directory: a. In screen command mode, type **logfile media/.log** #. Press **** #. Press **-a** and then **H** to start writing to the log file #. Run your commands. If the **screen** session times out, you can obtain console output from the log file, for example: **$ sftp platform@:media/.log** .. _Download-Files-and-Check-script: Download Files and Check .......................................... .. tabularcolumns:: |p{13.5cm}|p{4cm}| +-------------------------------------------------------------------------------------------------------+--------------------+ | Description and Steps | Notes and Status | +=======================================================================================================+====================+ | VOSS files: | | | | | | **https://voss.portalshape.com > Downloads > VOSS Automate > XXX > Upgrade** | | | | | | Download ``XXX-Delta-Bundle.script`` file, where XXX matches the release. | | | Transfer the ``XXX-Delta-Bundle.script`` file to the ``media/`` folder *of the primary Unified node*. | | | Two file transfer options: | .. raw:: html | | | | | Either using SFTP: | | | | | | |
** | type='checkbox' | | | id='done' | | * **cd media** | name='done' | | | unchecked> | | * **put ** | | | |
| | | | | | | | Or using SCP: | | | | | | | | | * **scp platform@:~/media** | | | | | | | | | | | | On the primary Unified node, verify that the ``.script`` file copied: | | | | | | | | | * **ls -l media/** | | | | | | | | | On the primary Unified node, verify that the original ``.sha256`` checksums on the | | | SFTP server match. | | | | | | | | | * **system checksum media/** | | | | | | ``Checksum: `` | | | | | | | | +-------------------------------------------------------------------------------------------------------+--------------------+ .. _Adaptations-Check: Adaptations Check .......................... .. tabularcolumns:: |p{13.5cm}|p{4cm}| +-------------------------------------------------------------------------------------------+--------------------+ | Description and Steps | Notes and Status | +===========================================================================================+====================+ | Identify installed adaptations and determine any effect on the upgrade plan. | | | | | | If the release is accompanied by Upgrade Notes, refer to the details. | .. raw:: html | | | | | |
| | | | | |
| | | | +-------------------------------------------------------------------------------------------+--------------------+ .. _Schedules-Transactions-Version-Check: Schedules, Transactions and Version Check .................................................. .. tabularcolumns:: |p{13.5cm}|p{4cm}| +----------------------------------------------------------------------------------------------+--------------------+ | Description and Steps | Notes and Status | +==============================================================================================+====================+ | Turn off any scheduled imports to prevent syncs triggering part way through the upgrade. | | | Two options are available: | | | | | | Individually for each job: | | | | | | 1. Log in on the GUI as a high level administrator above Provider level. | | | 2. Select the **Scheduling** menu to view scheduled jobs. | | | 3. Click each scheduled job. On the Base tab, uncheck the **Activate** check box. | | | | | | Mass modify: | | | | | | 1. On the GUI, export scheduled syncs into a bulk load sheet. | .. raw:: html | | 2. Modify the schedule settings to de-activate scheduled syncs. | | | 3. Import the sheet. |
| | | | | |
| | | | | Schedules enabled on the CLI: | | | | | | 1. Run **schedule list** to check if any schedules exist and overlap with the maintenance | | | window. | | | 2. For overlapping schedules, disable. Run **schedule disable **. | | | | | +----------------------------------------------------------------------------------------------+--------------------+ | Check for running imports. Either wait for them to complete or cancel them: | | | | | | 1. Log in on the GUI as a high level administrator above Provider level. | | | 2. Select the **Transaction** menu to view transactions. | | | 3. Filter the **Action** column: | | | | | | a. Choose **Status** as "Processing" and then choose each **Action** | | | that starts with "Import", for example, "Import Unity Connection". | | | b. Click **Search** and confirm there are no results. | | | c. If there are transactions to cancel, select them and click **Cancel**. | | | | | | | .. raw:: html | | | | | |
| | | | | |
| | | | | | | +----------------------------------------------------------------------------------------------+--------------------+ | | | | **Customized ``data/Settings``** | | | | | | If ``data/Settings`` instances have been modified, record these or export them as JSON. | | | | | | The modifications can be re-applied or exported JSON instances can be merged following | | | the upgrade. See: :ref:`Post-Template-Upgrade-Tasks`. | | | | | | **Version** | | | | | | Record the current version information. This is required for upgrade troubleshooting. | | | | | | * Log in on the GUI and record the information contained in the **About > Extended Version** | .. raw:: html | | | | | |
| | | | | |
| | | | | | | +----------------------------------------------------------------------------------------------+--------------------+ .. _Pre-Upgrade-Security-Health-Steps: Pre-Upgrade, Security and Health Steps ............................................... .. tabularcolumns:: |p{13.5cm}|p{4cm}| +-----------------------------------------------------------------------------------------+--------------------+ | Description and Steps | Notes and Status | +=========================================================================================+====================+ | Verify that the primary node is the active primary node at the time of upgrade. | | | | .. raw:: html | | **database config** | | | |
| | Example output :: | | | priority: |
| | stateStr: PRIMARY | | | storageEngine: WiredTiger | | | | | | Validate the system health. | | | | | | On the Primary Unified Node, verify cluster connectivity: | | | | | | * **cluster status** | | | | | | On each node verify network connectivity, disk status and NTP. | | | | | | | | | * **cluster check** | | | | | | If there is any sign of the paths below are over 80% full, a clean-up is needed | | | to avoid risk of for example full logs occurring during upgrade. | | | Clean-up steps are indicated next to the paths: | | | | | | :: | | | | | | / (call support if over 80%) | | | /var/log (run: log purge) | | | /opt/platform (remove any unnecessary files from /media directory) | | | /tmp (reboot) | | | | | | | | | On the Primary Unified Node, verify there are no pending Security Updates on any of the | | | nodes: | | | | | | * **cluster run all security check** | | | | | +-----------------------------------------------------------------------------------------+--------------------+ | | | | As part of the rollback procedure, ensure that | | | a suitable restore point is obtained prior to the start of the | | | activity, as per the guidelines for the infrastructure on which | | | the VOSS Automate platform is deployed. | | | | | | VOSS cannot guarantee that a restore point can be used to successfully | .. raw:: html | | restore VOSS Automate. If you cannot | | | restore the application from a restore point, your only recourse is to reinstall |
| | | | | |
| | | | | | | | | | | | | | | | | | | | | | | | | | Optional: If a backup is also required, use the | | | **backup add ** and | | | **backup create ** commands. For details, refer to the Platform Guide. | | | | | +-----------------------------------------------------------------------------------------+--------------------+ .. tabularcolumns:: |p{13.5cm}|p{4cm}| +-----------------------------------------------------------------------------------------+--------------------+ | Description and Steps | Notes and Status | +=========================================================================================+====================+ | Before upgrading, check all services, nodes and weights for the cluster: | | | | | | Make sure no services are stopped/broken. The message 'suspended waiting for mongo' | | | is normal on the fresh unified nodes. | | | | | | | | | * **cluster run all app status** | | | | | | | | | Make sure all application nodes show 3 or 5 nodes. | | | | | | | | | * **cluster run application cluster list** | | | | | | | | | Check that the database weights are set. It is *critical* to ensure the weights are set | | | before upgrading a cluster. | | | | | | | .. raw:: html | | * **cluster run application database weight list** | | | |
| | :: | | | 172.29.21.240: |
| | weight: 80 | | | 172.29.21.241: | | | weight: 70 | | | 172.29.21.243: | | | weight: 60 | | | 172.29.21.244: | | | weight: 50 | | | | | | | | | Verify the primary node in the primary site and ensure no nodes are in the | | | 'recovering' state (``stateStr`` is not ``RECOVERING``). On the primary node: | | | | | | | | | * **database config** | | | | | | | | +-----------------------------------------------------------------------------------------+--------------------+ .. _upgrade-Script: Upgrade ........................... .. tabularcolumns:: |p{13.5cm}|p{4cm}| +--------------------------------------------------------------------------------------+--------------------+ | Description and Steps | Notes and Status | +======================================================================================+====================+ | On the primary unified node: | | | | | | | | | * **screen** | .. raw:: html | | | | | Run (optionally with command parameters below): | | | | | | * **app install media/ delete-on-success yes --force** |
| | command. | | | |
| | * **database config** | | | | | | and verify the number of nodes are *uneven*, i.e. either 5 or 7. | | | If not, run: **cluster provision role database** and ensure an arbitrator shows | | | when you run **database config** again (``stateStr: ARBITER``). | | | | | | From release 19.1.2 and later, the ``delete-on-success`` parameter | | | and ``yes`` or ``no`` value have been added to remove or keep the the script file in | | | the ``media/`` directory after successful installation. | | | | | | Note that during the upgrade, phone registration data is cleared. A message will | | | show in the log: ``Remove phone registration data``. This is required so that old | | | values are not displayed, since after the upgrade this information is no longer | | | stored in the resource cache. | | +--------------------------------------------------------------------------------------+--------------------+ .. _Post-Upgrade-Security-Health-Steps: Post-Upgrade, Security and Health Steps ................................................ .. tabularcolumns:: |p{13.5cm}|p{4cm}| +--------------------------------------------------------------------------------------+--------------------+ | Description and Steps | Notes and Status | +======================================================================================+====================+ | On the primary unified node, verify the cluster status: | | | | | | * **cluster status** | .. raw:: html | | * **cluster check** | | | * If any of the above commands show errors, check for further details to assist | | | with troubleshooting: | | | |
| | | | | |
| | | | +--------------------------------------------------------------------------------------+--------------------+ | If upgrade is successful, the screen session can be closed by typing **exit** in the | | | screen terminal. If errors occurred, keep the screen terminal | | | open for troubleshooting purposes and contact VOSS support. | .. raw:: html | | | | | |
| | | | | |
| | | | +--------------------------------------------------------------------------------------+--------------------+ | Check for needed security updates. On the primary node, run: | | | | | | * **cluster run all security check** | | | | | | If one or more updates are required for any node, run on the primary Unified node: | | | | | | | | | | .. raw:: html | | * **cluster run all security update** | | | |
| | | | | |
| | * **cluster run notme system reboot** | | | | | | | | | If node messages: `` failed with timeout`` are displayed, | | | these can be ignored. | | | | | | | | | * **system reboot** | | | | | | | | | Since all services will be stopped, this takes some time. | | | | | | | | +--------------------------------------------------------------------------------------+--------------------+ .. _Post-Template-Upgrade-Tasks: Post Template Upgrade Tasks .................................... .. tabularcolumns:: |p{13.5cm}|p{4cm}| +-------------------------------------------------------------------------------------------+--------------------+ | Description and Steps | Notes and Status | +===========================================================================================+====================+ | | | | **Customized ``data/Settings``** | | | | | | Merge the previously backed up customized ``data/Settings`` with the latest settings | | | on the system by manually adding the differences or exporting the latest settings | | | to JSON, merging the customized changes and importing the JSON. | | | | | | | | | **Support for VG400 and VG450 Analogue Gateways** | | | | | | Before adding the VG400 or VG450 Gateway, the ``device/cucm/GatewayType`` | | | model needs to be imported for each Unified CM. | | | | | | 1. Create a Model Type List which includes the ``device/cucm/GatewayType`` model. | | | 2. Add the Model Type List to all the required Unified CM Data Syncs. | | | 3. Execute the Data Sync for all the required Unified CMs. | | | | | | **Verify the upgrade** | .. raw:: html | | | | | Log in on the GUI and check the information contained in the **About > Extended Version** | | | menu. Confirm that versions have upgraded: |
| | | | | If your web browser cannot open the user interface, clear your browser cache before |
| | trying to open the interface again. | | | | | +-------------------------------------------------------------------------------------------+--------------------+ | | .. raw:: html | | | | | |
| | | | | |
| | | | +-------------------------------------------------------------------------------------------+--------------------+ .. _Restore-Adaptations: Restore Adaptations ............................ .. tabularcolumns:: |p{13.5cm}|p{4cm}| +------------------------------------------------------------------------------------+--------------------+ | Description and Steps | Notes and Status | +====================================================================================+====================+ | Restore and adaptations prior to upgrade. | | | | | | If the release is accompanied by Upgrade Notes, refer to the details on adaptation | .. raw:: html | | impact. | | | |
| | | | | |
| | | | +------------------------------------------------------------------------------------+--------------------+ .. _Restore_Schedules: Restore Schedules .......................... .. tabularcolumns:: |p{13.5cm}|p{4cm}| +---------------------------------------------------------------------------------+--------------------+ | Description and Steps | Notes and Status | +=================================================================================+====================+ | Re-enable scheduled imports if any were disabled prior to the upgrade. | | | Two options are available: | | | | | | Individually for each job: | | | | | | 1. Log in on the GUI as a high level administrator above Provider level. | | | 2. Select the **Scheduling** menu to view scheduled jobs. | | | 3. Click each scheduled job. On the Base tab, check the **Activate** check box. | | | | | | Mass modify: | | | | | | 1. Modify the exported sheet of schedules to activate scheduled syncs. | | | 2. Import the bulk load sheet. | .. raw:: html | | | | | .. note:: | | | | | | Select the **Skip next execution** if you do not wish to execute schedules | | | overlapping the maintenance window, but only execute thereafter. | | | |
| | Run **schedule enable **. | | | |
| | | | | | | +---------------------------------------------------------------------------------+--------------------+ For overlapping schedules, disable. Run **schedule disable **. .. _Log-Files-Error-Checks: Log Files and Error Checks .................................... .. tabularcolumns:: |p{13.5cm}|p{4cm}| +----------------------------------------------------------------------------------+--------------------+ | Description and Steps | Notes and Status | +==================================================================================+====================+ | Inspect the output of the command line interface for upgrade errors. | .. raw:: html | | | | | Use the **log view** command to view any log files indicated in the error | | | messages, for example, run the command if the following message appears: |
| | 'log view platform/execute.log' | | | directory to an SFTP server: |
| | | | | * **log send sftp://x.x.x.x install** | | | | | +----------------------------------------------------------------------------------+--------------------+ | | .. raw:: html | | Log in on the GUI as system level administrator, go to | | | **Administration Tools > Transaction** and inspect the transactions |
| | | | | |
| | | | +----------------------------------------------------------------------------------+--------------------+