Upgrade a Multinode Environment with the ISO and Template

Note

  • When upgrading from VOSS-4-UC 18.1.3, refer to Upgrading from 18.1.3 to 20.1.1 - summary.

  • Upgrading to release 20.1.1 requires a system on 19.x, with security updates completed. The upgrade includes:

    • an upgrade to the underlying operating system to Ubuntu 18.04.4.
    • the installation of a new cluster check command available from the 20.1.1 ISO by running app install check_cluster.

  • While template installation and system upgrade takes approximately two hours at a single site, this may vary in accordance with your topology, number of devices and subscribers. Adjust your upgrade maintenance window to allow for your configuration.

    You can follow the progress on the Admin Portal transaction list.

  • When upgrading from CUCDM 11.5.3 Patch Bundle 2 or VOSS-4-UC 18.1 Patch Bundle 2 and earlier, re-import specified CUC models according to your current version. Refer to the final upgrade procedure step.

  • Tasks that are marked Prior to Maintenance Window can be completed a few days prior to the scheduled maintenance window so that VOSS support can be contacted if needed and in order to allow for reduce down time.

The standard screen command should be used where indicated. See: Using the screen command.

Download Files and Check (Prior to Maintenance Window)

Description and Steps Notes and Status

VOSS files: https://voss.portalshape.com > Downloads > VOSS-4-UC > 20.1.1_EFT > Upgrade

Download .iso and .template files.

  • Transfer the .iso file to the media/ folder of all nodes.
  • Transfer the .template file to the media/ folder of the primary node.

Two transfer options:

Either using SFTP:

  • sftp platform@<unified_node_hostname>
  • cd media
  • put <upgrade_iso_file>
  • put <upgrade_template_file>

Or using SCP:

  • scp <upgrade_iso_file> platform@<unified_node_ip_address>:~/media
  • scp <upgrade_template_file> platform@<unified_node_ip_address>:~/media

Verify that the .iso image and .template file copied:

  • ls -l media/

Verify that the original .sha256 checksums match.

  • system checksum media/<upgrade_iso_file>

    Checksum: <SHA256>

  • system checksum media/<upgrade_template_file>

    Checksum: <SHA256>

Security and Health Check Steps (Prior to Maintenance Window)

Description and Steps Notes and Status

Verify that the primary node is the active primary node at the time of upgrade.

database config

Ensure that the node on which the installation will be initiated has the stateStr parameter set to PRIMARY and has the highest priority number (highest priority number could vary depending on cluster layout).

Example output

<ip address>:27020:
  priority: <number>
  stateStr: PRIMARY
  storageEngine: WiredTiger

Validate the system health. For upgrade to release 20.1.1, carry out the following (19.x only):

  • system mount - mount the release 20.1.1 upgrade ISO.

  • app install check_cluster - install the new version of the cluster check command.

    For details, refer to Cluster Check.

  • cluster check - inspect the output of this command for warnings and errors. You can also use cluster check verbose to see more details. While warnings will not prevent an upgrade, it is advisable that these be resolved prior to upgrading where possible. Some warnings may be resolved by upgrading.

    For troubleshooting and resolutions, also refer to the Health Checks for Cluster Installations Guide and Platform Guide.

    If there is any sign of the paths below are over 80% full, a clean-up is needed, for example to avoid risk of full logs occurring during upgrade. Clean-up steps are indicated next to the paths:

    /              (call support if over 80%)
/var/log       (run: log purge)
/opt/platform  (remove any unnecessary files from /media directory)
/tmp           (reboot)

    On the Primary Unified Node, verify there are no pending Security Updates on any of the nodes.

Note

If you run cluster status after installing the new version of cluster check, any error message regarding a failed command can be ignored. This error message will not show after upgrade.

Schedules, Transactions and Version Check (Maintenance Window)

Description and Steps Notes and Status

Run cluster check and verify that no warnings and errors show.

Turn off any scheduled imports to prevent syncs triggering part way through the upgrade. Two options are available:

Individually for each job:

  1. Log in on the Admin Portal as a high level administrator above Provider level.
  2. Select the Scheduling menu to view scheduled jobs.
  3. Click each scheduled job. On the Base tab, uncheck the Activate check box.

Mass modify:

  1. On the Admin Portal, export scheduled syncs into a bulk load sheet.
  2. Modify the schedule settings to de-activate scheduled syncs.
  3. Import the sheet.

Check for running imports. Either wait for them to complete or cancel them:

  1. Log in on the Admin Portal as a high level administrator above Provider level.
  2. Select the Transaction menu to view transactions.
  3. Filter the Action column:
    1. Choose Status as “Processing” and then choose each Action that starts with “Import”, for example, “Import Unity Connection”.
    2. Click Search and confirm there are no results.
    3. If there are transactions to cancel, select them and click Cancel.

Customized ``data/Settings``

If data/Settings instances have been modified, record these or export them as JSON.

The modifications can be re-applied or exported JSON instances can be merged following the upgrade. See: Post Template Upgrade Tasks (Maintenance Window).

Version

Record the current version information. This is required for upgrade troubleshooting.

  • Log in on the Admin Portal and record the information contained in the About > Extended Version

Pre-Upgrade Steps (Maintenance Window)

Shutdown servers and take snapshots from VMware and then power on all servers, starting with the primary:

Use VMware snapshots. Consider the following:

  • VOSS cannot guarantee that a VMware snapshot can be used to successfully restore VOSS-4-UC or any Cisco HCS Management application. If you cannot restore the application from a snapshot, your only recourse is to reinstall the application.

  • When the backup is complete and you do not need the VMware snapshot for restore activities, delete the snapshot immediately to preserve LUN space.

  • cluster run notme system shutdown

    followed by:

    system shutdown

Log into VMware and take snapshots of all unified nodes and all web proxies.

After snapshots, restart the servers:

  • Power up the servers via VMware.

Optional: If a backup is required in addition to the snapshot, use the backup add <location-name> and backup create <location-name> commands. For details, refer to the Platform Guide.
Description and Steps Notes and Status

After snapshot and before upgrading: validate system health and check all services, nodes and weights for the cluster:

  • cluster run application cluster list

    Make sure all application nodes show 3 or 5 nodes.

  • cluster check - inspect the output of this command, for warnings and errors. You can also use cluster check verbose to see more details.

    • Make sure no services are stopped/broken. The message ‘suspended waiting for mongo’ is normal on the fresh unified nodes.

    • Check that the database weights are set. It is critical to ensure the weights are set before upgrading a cluster. Example output:

      172.29.21.240:
    weight: 80
172.29.21.241:
    weight: 70
172.29.21.243:
    weight: 60
172.29.21.244:
    weight: 50

    • Verify the primary node in the primary site and ensure no nodes are in the ‘recovering’ state (stateStr is not RECOVERING). On the primary node:

Upgrade (Maintenance Window)

Note

By default, the cluster upgrade is carried out in parallel on all nodes and without any backup in order to provide a fast upgrade.

Description and Steps Notes and Status

From VOSS-4-UC 18.1 or CUCDM 11.5.3 onwards, it is recommended that the upgrade steps are run in a terminal opened with the screen command.

Verify that the ISO has been uploaded to the media/ directory on each node. This will speed up the upgrade time.

On the primary unified node:

  • screen

  • cluster upgrade media/<upgrade_iso_file>

    Note

    The cluster upgrade command will also silently first run cluster check and the upgrade will fail if any error conditions exist.

All unused docker images except selfservice and voss_ubuntu images will be removed from the system at this stage.

Post-Upgrade, Security and Health Steps (Maintenance Window)

Description and Steps Notes and Status

On the primary unified node, verify the cluster status:

  • cluster check

  • If any of the above commands show errors, check for further details to assist with troubleshooting:

    cluster run all diag health
If upgrade is successful, the screen session can be closed by typing exit in the screen terminal. If errors occurred, keep the screen terminal open for troubleshooting purposes and contact VOSS support.

Complete all the security updates if needed:

If security updates needed from cluster check above, * cluster run all security update

The docker images selfservice and voss_ubuntu will be removed from the system at this stage.

Note: If the system reboots, do not carry out the next manual reboot step. When upgrading from pre-19.1.1, an automatic reboot should be expected.

Manual reboot only if needed:

  • cluster run notme system reboot

If node messages: <node name> failed with timeout are displayed, these can be ignored.

  • system reboot

Since all services will be stopped, this takes some time.

All unused docker images except selfservice and voss_ubuntu images will be removed from the system at this stage.

Database Filesystem Conversion (if required, Maintenance Window)

Important

This step is to be carried out only if you have not converted the file system before.

To check if the step is not required:

  • Run drives list and ensure that the LVM storage shows for all converted database nodes under Volume Groups. If the output of the drives list command contains dm-0 - mongodb:dbroot, the step is not required. Refer to the drives list command output example below.

The database convert_drive command provides parameters that allow for a flexible upgrade schedule in order to limit system downtime.

When the database convert_drive command is run, the voss-deviceapi service will be stopped first and started after completion. The command should therefore be run during a maintenance window while there are no running transactions.

The procedure and commands in this step depend on:

  • your topology
  • latency between data centers
  • upgrade maintenance windows - Window 1 to Window 3 represent chosen maintenance windows.

First inspect the table below for guidance on the commands to run according to your configuration and preferences.

  • Run all commands on the primary unified node:

    • Ensure states of database nodes are not DOWN - otherwise the command will fail

      database config (stateStr is not DOWN)

    • Ensure database weights are set and have 1 maximum weight - otherwise the command will fail

      database weight list (one weight value is maximum)

  • For 2 and 3 maintenance windows: after the upgrade (prior to Windows 2 and 3), only nodes with converted drives will generate valid backups.

    For example, if the primary drive is converted, backups from the primary node can be used to restore the database. If there is a database failover to the highest weight secondary node that was not converted, it will not be possible for backups to be generated on that secondary node until the drive is converted.

Topology Window 1 Window 2 Window 3 Commands (DC = valid data center name) Description
multinode Y    

database convert_drive secondary all

database convert_drive primary

 Recommended for a system with latency < 10ms.
multinode Y Y  

Window 1:

database convert_drive primary

Window 2:

database convert_drive secondary all

Can be used for a system with latency < or > 10ms.

Allows for smaller maintenance windows.

Cluster is not available during maintenance windows.
multinode Y Y Y

Window 1:

database convert_drive primary

Window 2:

database convert_drive secondary <first DC>

Window 3:

database convert_drive secondary <second DC>

Can be used for a system with latency > 10ms.

Allows for smaller maintenance windows.

Cluster is not available during maintenance windows.
Description and Steps Notes and Status

Database Filesystem Conversion step

Shut down all the nodes. Since all services will be stopped, this takes some time.

  • cluster run all system shutdown

  • Create a VMware snapshot for all the unified servers so that the system can easily be reverted in the case of a conversion error. Boot all the systems in VMware.

  • Run the convert_drive command with parameters according to the table above.

    Wait until it completes successfully.

  • database config

    Ensure that the storage engine for all converted database nodes shows as storageEngine: WiredTiger.

  • drives list

    Ensure that the LVM storage shows for all database nodes under Volume Groups.

In the example below, dbroot/dm-0 shows under Volume Groups, Logical volumes

$ drives list
Used disks and mountpoints:
sdc1 - services:backups
dm-0 - mongodb:dbroot

Unused disks:
none - if disks have been hot-mounted, it may be necessary to reboot the system

Unused mountpoints:
services:SWAPSPACE

Volume Groups
voss - 10.0 GB free, 60.0 GB total
Physical volumes:
sdd1
Logical volumes:
dbroot/dm-0 - 50.0 GB

Database Schema Upgrade (Maintenance Window)

Important

For VOSS-4-UC release 20.1.1, refer to the VOSS-4-UC 20.1.1 Release Changes and Impact document for details on model and workflow changes. Customizations related to these changes may be affected by this step.

Description and Steps Notes and Status

From VOSS-4-UC 18.1 or CUCDM 11.5.3 onwards, it is recommended that the upgrade steps are run in a terminal opened with the screen command.

On the primary unified node:

  • screen
  • voss upgrade_db

Check cluster status

  • cluster check

Template Upgrade (Maintenance Window)

Description and Steps Notes and Status

From VOSS-4-UC 18.1 or CUCDM 11.5.3 onwards, it is recommended that the upgrade steps are run in a terminal opened with the screen command.

On the primary unified node:

  • screen
  • app template media/<VOSS-4-UC.template>

The following message appears:

Running the DB-query to find the current environment's
existing solution deployment config...
  • Python functions are deployed
  • System artifacts are imported.

The template upgrade automatically detects the deployment mode: “Enterprise”, “Provider with HCM-F” or “Provider without HCM-F”. A message displays according to the selected deployment type. Check for one of the messages below:

Importing EnterpriseOverlay.json

Importing ProviderOverlay_Hcmf.json ...

Importing ProviderOverlay_Decoupled.json ...

The template install automatically restarts necessary applications. If a cluster is detected, the installation propagates changes throughout the cluster.

Description and Steps Notes and Status
Review the output from the app template command and confirm that the upgrade message appears: 
Deployment summary of PREVIOUS template solution
(i.e. BEFORE upgrade):
-------------------------------------------------


Product: [PRODUCT]
Version: [PREVIOUS PRODUCT RELEASE]
Iteration-version: [PREVIOUS ITERATION]
Platform-version: [PREVIOUS PLATFORM VERSION]

This is followed by updated product and version details:

Deployment summary of UPDATED template solution
(i.e. current values after installation):
-----------------------------------------------

Product: [PRODUCT]
Version: [UPDATED PRODUCT RELEASE]
Iteration-version: [UPDATED ITERATION]
Platform-version: [UPDATED PLATFORM VERSION]
Description and Steps Notes and Status
  • If no errors are indicated, make a backup or snapshot.

For an unsupported upgrade path, the install script stops with the message:

Upgrade failed due to unsupported upgrade path.
Please log in as sysadmin
and see Transaction logs for more detail.

You can restore to the backup or revert to the VM snapshot made before the upgrade.
If there are errors for another reason, the install script stops with a failure message listing the problem. Contact VOSS support.

Verify the extra_functions have the same checksum across the cluster.

  • cluster run application voss get_extra_functions_version -c

Post upgrade migrations:

On a single node of a cluster, run:

  • voss post-upgrade-migrations

Data migrations that are not critical to system operation can have significant execution time at scale. These need to be performed after the primary upgrade, allowing the migration to proceed whilst the system is in use - thereby limiting upgrade windows.

A transaction is queued on VOSS-4-UC and its progress is displayed as it executes.
Description and Steps Notes and Status

Check cluster status and health

  • cluster status

Post Template Upgrade Tasks (Maintenance Window)

Description and Steps Notes and Status

Customized ``data/Settings``

Merge the previously backed up customized data/Settings with the latest settings on the system by manually adding the differences or exporting the latest settings to JSON, merging the customized changes and importing the JSON.

Support for VG400 and VG450 Analogue Gateways

Before adding the VG400 or VG450 Gateway, the device/cucm/GatewayType model needs to be imported for each Unified CM.

  1. Create a Model Type List which includes the device/cucm/GatewayType model.
  2. Add the Model Type List to all the required Unified CM Data Syncs.
  3. Execute the Data Sync for all the required Unified CMs.

Verify the upgrade

Log in on the Admin Portal and check the information contained in the About > Extended Version menu. Confirm that versions have upgraded.

  • Release should show 20.1.1
  • Platform Version should show 20.1.1
  • Check themes on all roles are set correctly

Restore Schedules (Maintenance Window)

Description and Steps Notes and Status

Re-enable scheduled imports if any were disabled prior to the upgrade. Two options are available:

Individually for each job:

  1. Log in on the Admin Portal as a high level administrator above Provider level.
  2. Select the Scheduling menu to view scheduled jobs.
  3. Click each scheduled job. On the Base tab, check the Activate check box.

Mass modify:

  1. Modify the exported sheet of schedules to activate scheduled syncs.
  2. Import the bulk load sheet.

Release Specific Updates (Maintenance Window)

Description and Steps Notes and Status

When upgrading from CUCDM 11.5.3 Patch Bundle 2 or VOSS-4-UC 18.1 Patch Bundle 2 and earlier, re-import the following from all CUCM devices, since this upgrade deleted obsolete CUC timezone codes from the VOSS-4-UC database:

  • CUC models:

    device/cuc/TimeZone

Note:

This is a once off data migration step. If this was performed previously when upgrading to 19.1.x, then it does not have to be repeated.

After upgrading, obtain and install the following patch according to its accompanying MOP file:

  • Server Name: secure.voss-solutions.com
  • Path: /software/voss4uc/releases/Release-19.2.1
  • Patch Directory: Update_CUC_Localization_patch
  • Patch File: Update_CUC_Localization_patch.script
  • MOP File: MOP-Update_CUC_Localization.pdf

Note:

This is a once off data migration step. If this was performed previously when upgrading to 19.1.x, then it does not have to be repeated.

Re-import the following from all CUCM devices:

  • CUCM models:

device/cucm/PhoneType

For steps to create a custom data sync, refer to the chapter on Data Sync in the Core Feature Guide.

Note:

This is a once off data migration step. If this was performed previously when upgrading to 19.1.x, then it does not have to be repeated.

Log Files and Error Checks (Maintenance Window)

Description and Steps Notes and Status

Inspect the output of the command line interface for upgrade errors, for example File import failed! or Failed to execute command.

Use the log view command to view any log files indicated in the error messages, for example, run the command if the following message appears:

For more information refer to the execution log file with
'log view platform/execute.log'

For example, if it is required send all the install log files in the install directory to an SFTP server:

  • log send sftp://x.x.x.x install
Log in on the Admin Portal as system level administrator, go to Administration Tools > Transaction and inspect the transactions list for errors.

Upgrading from 18.1.3 to 20.1.1 - summary

Below are the summarized steps to upgrade from 18.1.3.

  • The steps require the necessary scripts, templates and ISOs to be in the media/ directory.
  • For details on the specific commands, refer to the corresponding steps above.
  • For general usage of commands to carry out tasks, refer to the Platform Guide.
Command and task sequence Comment
cluster status no service mismatch, all nodes ok
cluster run all diag disk check for disks over 90% full
database config ensure all unified nodes have a weight, and are in a good state: primary, secondary,arbiter
manual check stop / check for transactions running, stop where possible
external task create backup (snapshots)
cluster run all app install media/EKB-4124-18.1.3_patch.script refer to steps details above
cluster upgrade media/platform-install-19.2.1-1570776653.iso --force refer to steps details above
cluster run all security update --force refer to steps details above
cluster run all app install media/EKB-4124-18.1.3_patch.script refer to steps details above
external task create backup (snapshots)
cluster upgrade media/platform-install-20.1.1-<nnnnnnnnnn>.iso refer to preliminary and upgrade steps details above; <nnnnnnnnnn> matches the downloaded release ISO
cluster run all security update refer to steps details above
database config make sure all databases are in the correct state
database convert_drive secondary all refer to steps details above
voss upgrade_db refer to steps details above
app template media/<VOSS-4-UC.template> refer to steps details above