Modular Cluster Topology: Upgrade a Multinode Environment with the Delta Bundle#

Note

  • While system upgrade takes approximately two hours at a single site, this may vary in accordance with your topology, number of devices and subscribers. Adjust your upgrade maintenance window to allow for your configuration.

The standard screen command should be used where indicated. See: Using the screen command.

Determine the Primary database and application node in a Modular Cluster Topology#

Important

All upgrade steps are to be run from the primary application node. Where it is necessary to run database node commands from this application node, the primary database node IP will be passed in as a parameter with the command.

In order to run the commands for the upgrade on the nodes with the appropriate roles, determine:

  • The primary application node

  • The primary database node

  1. Log in on a node in your modular cluster.

  2. To find the primary application node in the cluster:

    $ cluster run all cluster primary role application

    Record the node entry where is_primary: true, for example:

    ---------- VOSS-UN-1, ip=192.168.100.3, role=webproxy,application, loc=cpt

    is_primary: true

  3. To find the primary database node in the cluster:

    $ cluster run all cluster primary role database

    Record the node entry IP where is_primary: true, for example:

    ---------- VOSS-UN-2, ip=192.168.100.4, role=database, loc=cpt


    is_primary: true

    This IP address will be used in command parameters during upgrade.

Note

When the cluster run all primary role <role> command is run, web proxy nodes will return a failure - this can be ignored. For example:

---------- VOSS-WP-1, ip=192.168.100.9, role=webproxy, loc=cpt
Invalid command syntax - refer to help descriptions

Download Files and Check#

Description and Steps

Notes and Status

VOSS files:

https://voss.portalshape.com > Downloads > VOSS Automate > XXX > Upgrade

Download XXX-Delta-Bundle.script file, where XXX matches the release. Transfer the XXX-Delta-Bundle.script file to the media/ folder of the primary application node. Two file transfer options:

Either using SFTP:

  • sftp platform@<primary_application_hostname>

  • cd media

  • put <XXX-Delta-Bundle.script>

Or using SCP:

  • scp <XXX-Delta-Bundle.script> platform@<primary_application_hostname>:~/media

On the primary application node, verify that the .script file copied:

  • ls -l media/

On the primary application node, verify that the original .sha256 checksums on the Download site match.

  • system checksum media/<XXX-Delta-Bundle.script>

    Checksum: <SHA256>

Adaptations Check#

Description and Steps

Notes and Status

Identify installed adaptations and determine any effect on the upgrade plan.

If the release is accompanied by Upgrade Notes, refer to the details.

Schedules, Transactions and Version Check#

Description and Steps

Notes and Status

Turn off any scheduled imports to prevent syncs triggering part way through the upgrade. Two options are available:

Individually for each job:

  1. Log in on the GUI as a high level administrator above Provider level.

  2. Select the Scheduling menu to view scheduled jobs.

  3. Click each scheduled job. On the Base tab, uncheck the Activate check box.

Mass modify:

  1. On the GUI, export scheduled syncs into a bulk load sheet.

  2. Modify the schedule settings to de-activate scheduled syncs.

  3. Import the sheet.

Schedules enabled on the CLI: on the primary application node:

  1. Check if any schedules exist. Run:

    cluster run all schedule list

    Record the IP addresses of nodes with schedules overlapping with the maintenance window.

  2. For overlapping schedules on a node, disable. Run:

    cluster run <node IP> schedule disable <job-name>

Check for running imports. Either wait for them to complete or cancel them:

  1. Log in on the GUI as a high level administrator above Provider level.

  2. Select the Transaction menu to view transactions.

  3. Filter the Action column:

    1. Choose Status as “Processing” and then choose each Action that starts with “Import”, for example, “Import Unity Connection”.

    2. Click Search and confirm there are no results.

    3. If there are transactions to cancel, select them and click Cancel.

Customized ``data/Settings``

If data/Settings instances have been modified, record these or export them as JSON.

The modifications can be re-applied or exported JSON instances can be merged following the upgrade. See: Post Template Upgrade Tasks.

Version

Record the current version information. This is required for upgrade troubleshooting.

  • Log in on the GUI and record the information contained in the About > Extended Version

Pre-Upgrade, Security and Health Steps#

Description and Steps

Notes and Status

Have the IP address available of the node determined to be the primary database node. Verify that the primary database node is the active primary node at the time of upgrade.

On the primary application node, run:

cluster run <primary db IP> database config

From the output, ensure that the primary database node stateStr parameter is set to PRIMARY and it has the highest priority:<number> (highest priority number could vary depending on cluster layout).

Example output

<ip address>:27020:
  priority: <number>
  stateStr: PRIMARY
  storageEngine: WiredTiger

Validate the system health on the primary application node:

Verify cluster connectivity

  • cluster run all cluster status

Verify network connectivity, disk status and NTP.

  • cluster check verbose

If there is any sign of the paths below are over 80% full, a clean-up is needed to avoid risk of for example full logs occurring during upgrade. Clean-up steps are indicated next to the paths:

/              (call support if over 80%)
/var/log       (run: log purge)
/opt/platform  (remove any unnecessary files from /media directory)
/tmp           (reboot)

On the primary application node, verify there are no pending Security Updates on any of the nodes:

  • cluster run all security check

As part of the rollback procedure, ensure that a suitable restore point is obtained prior to the start of the activity, as per the guidelines for the infrastructure on which the VOSS Automate platform is deployed.

VOSS cannot guarantee that a restore point can be used to successfully restore VOSS Automate. If you cannot restore the application from a restore point, your only recourse is to reinstall the application.

Optional: If a backup is also required

  • Database node backup:

    • Find the database node with the second highest weight. On the primary application node, run cluster run database database weight list to find this node.

    • Log in on the database node with the second highest weight.

    • Run backup add <SFTP-location-name> and backup create <SFTP-location-name> commands.

Backups for an application and web proxy type node are not required. A backup created on the secondary database node will include all the relevant data, excluding the web certificates.

For details, refer to the Platform Guide.

Description and Steps

Notes and Status

Before upgrading, check all services, nodes and weights for the cluster on the primary application node.

Make sure no services are stopped/broken. The message ‘suspended waiting for mongo’ is normal on the fresh database nodes.

  • cluster run all app status

Make sure all application nodes show.

  • cluster run application cluster list

Check that the database weights are set. It is critical to ensure the weights are set before upgrading a cluster. The command is run from the primary application node and is carried out on all database nodes

  • cluster run database database weight list

Example output:

---------- VOSS-UN-2, ip=192.168.100.4, role=database, loc=cpt

192.168.100.4:
    weight: 70
192.168.100.6:
    weight: 50
192.168.100.8:
    weight: 30

---------- VOSS-UN-4, ip=192.168.100.6, role=database, loc=cpt

[...]

Verify the primary database node in the primary site and ensure no nodes are in the ‘recovering’ state (stateStr is not RECOVERING). Run on the primary application node and use the primary database IP as parameter:

  • cluster run <primary db IP> database config

Upgrade#

Description and Steps

Notes and Status

On the primary application node:

  • screen

Run (optionally with command parameters below):

app install media/<script_file> delete-on-success yes --force

The upgrade will also silently run an updated version of cluster check and the upgrade will fail to proceed if any error conditions exist. Fix before proceeding. Refer to the Platform Guide for details on the new version of the cluster check command.

Run on the primary application node and use the primary database IP as parameter:

  • cluster run <primary db IP> database config

and verify the number of nodes are uneven, i.e. either 5 or 7. If not, run:

cluster run <primary db IP> cluster provision role database

Ensure an arbitrator shows when you run the command again:

cluster run <primary db IP> database config

(output shows: stateStr: ARBITER)

Note that during the upgrade, phone registration data is cleared. A message will show in the log: Remove phone registration data. This is required so that old values are not displayed, since after the upgrade this information is no longer stored in the resource cache.

Post-Upgrade, Security and Health Steps#

Description and Steps

Notes and Status

On the primary application node, verify the cluster status:

  • cluster status

  • cluster check

  • If any of the above commands show errors, check for further details to assist with troubleshooting:

    cluster run all diag health

If upgrade is successful, the screen session can be closed by typing exit in the screen terminal. If errors occurred, keep the screen terminal open for troubleshooting purposes and contact VOSS support.

Check for needed security updates. On the primary application node, run:

  • cluster run all security check

If one or more updates are required for any node, run on the primary application node:

  • cluster run all security update

Note: if the system reboots, do not carry out the next manual reboot step.

Manual reboot only if needed:

  • cluster run notme system reboot

If node messages: <node name> failed with timeout are displayed, these can be ignored.

  • system reboot

Since all services will be stopped, this takes some time.

Post Template Upgrade Tasks#

Description and Steps

Notes and Status

Restart the system from the command line if no reboot took place during the Post-Upgrade, Security and Health Steps.

On the primary application node, run:

cluster run notme system reboot

and then

system reboot

Import ``device/cucm/PhoneType``

In order for a security profile to be available for a Call Manager Analog Phone, the device/cucm/PhoneType model needs to be imported for each Unified CM.

  1. Create a Model Type List which includes the device/cucm/PhoneType model.

  2. Add the Model Type List to all the required Unified CM Data Syncs.

  3. Execute the Data Sync for all the required Unified CMs.

SSO Login URL check if needed

Verify the SSO Login URL if needed. Go to Single Sign On > SSO Identity Provider and ensure your URL matches the SSO Login URL value.

Customized ``data/Settings``

For releases prior to 21.1, merge the previously backed up customized data/Settings with the latest settings on the system by manually adding the differences or exporting the latest settings to JSON, merging the customized changes and importing the JSON.

Support for VG400 and VG450 Analogue Gateways

Before adding the VG400 or VG450 Gateway, the device/cucm/GatewayType model needs to be imported for each Unified CM.

  1. Create a Model Type List which includes the device/cucm/GatewayType model.

  2. Add the Model Type List to all the required Unified CM Data Syncs.

  3. Execute the Data Sync for all the required Unified CMs.

Verify the upgrade

Log in on the GUI and check the information contained in the About > Version menu. Confirm that versions have upgraded:

  • Release should show XXX

where XXX corresponds with the release number of the upgrade.

If your web browser cannot open the user interface, clear your browser cache before trying to open the interface again.

Verify that the web portal is configured to the required setting, either admin or classic. For further details on this setting, see Web Portal Configuration in the Platform Guide.

  • For configurations that make use of the Northbound Billing Integration (NBI), please check the service status of NBI and restart if necessary.

Restore Adaptations#

Description and Steps

Notes and Status

Restore and adaptations prior to upgrade.

If the release is accompanied by Upgrade Notes, refer to the details on adaptation impact.

Restore Schedules#

Description and Steps

Notes and Status

Re-enable scheduled imports if any were disabled prior to the upgrade. Two options are available:

Individually for each job:

  1. Log in on the GUI as a high level administrator above Provider level.

  2. Select the Scheduling menu to view scheduled jobs.

  3. Click each scheduled job. On the Base tab, check the Activate check box.

Mass modify:

  1. Modify the exported sheet of schedules to activate scheduled syncs.

  2. Import the bulk load sheet.

Note

Select the Skip next execution if you do not wish to execute schedules overlapping the maintenance window, but only execute thereafter.

Schedules enabled on the CLI. On the primary application node:

  1. For disabled schedules that were overlapping the maintenance window, enable.

    Verify disabled schedules: cluster run all schedule list.

    Record the IP addresses of nodes and scheduled jobs disabled.

  2. Enable schedules:

    cluster run <node IP> schedule enable <job-name>

Release Specific Updates#

Description and Steps

Notes and Status

When upgrading to release 21.3, users of Microsoft apps should after upgrade, select each Microsoft Tenant (relation/MicrosoftTenant) in the Admin GUI and click Save on it without making any changes.

This step is required so that VOSS Automate can communicate with the Tenant post upgrade.

Only if the following step was not carried out when upgrading to Release 21.3-PB1:

On the primary node, run:

voss migrate_summary_attributes data/InternalNumberInventory

Log Files and Error Checks#

Description and Steps

Notes and Status

Inspect the output of the command line interface for upgrade errors. On the primary application node, use the log view command to view any log files indicated in the error messages, for example, run the command if the following message appears:

For more information refer to the execution log file with
'log view platform/execute.log'

For example, if it is required send all the install log files in the install directory to an SFTP server:

  • log send sftp://x.x.x.x install

Log in on the GUI as system level administrator, go to Administration Tools > Transaction and inspect the transactions list for errors.

Licensing after Delta Bundle Upgrade#

Description and Steps

Notes and Status

From release 21.4 onwards, the deployment needs to be licensed. After installation, a 7-day grace period is available to license the product. Since license processing is only scheduled every hour, if you wish to license immediately, first run voss check-license from the primary application node CLI.

  1. Obtain the required license token from VOSS.

  2. Steps for GUI and CLI:

    1. To license through the GUI, follow steps indicated in Product License Management in the Core Feature Guide.

    2. To license through the CLI, follow steps indicated in Product Licensing in the Platform Guide.