Upgrade Automate#

Overview#

This section provides the steps for upgrading Automate with ISO and template, for all topologies. At each step in this procedure we’ve added labels to indicate the relevant topologies:

Unified node topology:
Modular cluster topology:
Single node cluster topology:

You can find out more about the Automate deployment topologies in the Automate Architecture and Hardware Specification Guide.

Before you start#

DONE

Review section “Prepare for Upgrade” before proceeding.

Important

Before starting the upgrade, ensure that the hardware version of each of your virtual machines (VMs) is at least version 11, compatible with ESXi 6.0 and up, and that your host CPU supports AVX (Advanced Vector Extensions).

A cluster check command in the Automate pre-upgrade steps checks for AVX support. To ensure that AVX support is added to the VMs, you’ll need to upgrade the compatibility of the VM in vCenter.

For the target version, before starting this upgrade, verify VMWare, Cloud deployments, and application version compatibility, as indicated in the Compatibility Matrix.

Prior to maintenance window#

Prior to the maintenance window, you will need to complete the following tasks:

Verify the primary database node and application node
Download and check files
Check the version

Verify the primary database node and application node#

DONE

Note

This task is optional for a single node cluster topology.

Verify the primary application node, run the following command on the node:

cluster primary role application
Note
- In a modular cluster topology, the application and database are on separate nodes.
- In a unified node topology you will need to ensure that the database and application status is primary on the same node (the node configured as “primary”). In this case (unified node), you’ll need to run the command on each node until you find the “primary” node. For example, the database node with the highest “weight” is “primary”.
The output should be true, for example:
```
platform@UN2:~$ cluster primary role application
is_primary: true
```
Verify the primary database node, run the following command on the node:

cluster primary role database
Note
- In a modular cluster topology, the application and database are on separate nodes.
- In a unified node topology you will need to ensure that the database and application status is “primary” on the same node (the node configured as “primary”). In this case (unified node), you’ll need to run the command on each node until you find the “primary” node. For example, the database node with the highest “weight” is “primary”.
The output should be true, for example:
```
platform@UN1:~$ cluster primary role database
is_primary: true
```

Download and check files#

DONE

Note

Ensure that the .iso file is available on all nodes.

Go to the download location for VOSS files (where XXX is the major version in your upgrade path requirement, for example, 24.2, if you’re upgrading to 24.2-PB1):

https://voss.portalshape.com > Downloads > VOSS Automate > XXX > Upgrade
Download .iso and .template files.
Transfer the files to the media/ folder, using either SFTP or SCP:
- Transfer the .iso file to the media/ folder of all nodes.
- Transfer the .template file to the media/ folder of the primary application node.
Transfer using SFTP:

For all nodes

sftp platform@<node_hostname>

cd media

put <upgrade_iso_file>

For primary application node

sftp platform@<application_node_hostname>

cd media

put <upgrade_template_file>

Transfer using SCP:

For all nodes

scp <upgrade_iso_file> platform@<node_ip_address>:~/media

For primary application node

scp <upgrade_template_file> platform@<application_node_ip_address>:~/media
Verify that the .iso image and .template file copied: ls -l media/
Verify that the original .sha256 checksums on the Download site match:

On any node, run:

cluster run all system checksum media/<upgrade_iso_file>

Note

If you have multiple nodes, run this command on only one node.

The output should be:
```
Checksum: <SHA256>
```
On the primary application node, run: system checksum media/<upgrade_template_file>

The output should be:
```
Checksum: <SHA256>
```

Version check#

DONE

If you have customized data settings (data/Settings), record these or export as JSON. Customizations can be re-applied or the exported JSON instances can be merged following the upgrade. See Post-template upgrade.
Record current version information for upgrade troubleshooting:
1. Log in to the Admin Portal.
2. Go to About > Version.
3. Make a note of the system version information.

Maintenance window#

In the maintenance window, you will need to complete the following tasks:

Perform security and health checks
Validate system health
Perform pre-upgrade steps
Upgrade
Perform post-upgrade and health check steps
Perform database schema upgrade
Perform template upgrade
Perform post-template upgrade steps
Inspect the log files and check for errors

Security and health checks#

DONE

If upgrading from [21.4-PB4, 21.4-PB5]:

Place the system in maintenance mode and suspend any scheduled transactions:

On an application node of the system, place the system in maintenance mode:

cluster maintenance-mode start

Scheduled transactions that are in progress will be allowed to complete, else, cancel data sync transactions that are in progress on the GUI. Refer to the Core Feature Guide. For details, see System Maintenance Mode in the Platform Guide.
Verify maintenance mode status:

cluster maintenance-mode status

Turn off scheduled imports:

If upgrading from [21.4, 21.4-PB1, 21.4-PB2, 21.4-PB3]:

Turn off any scheduled imports to prevent syncs triggering part way through the upgrade, either individually for each job, or mass modify:

Note

Schedules can easily be activated and deactivated via the Bulk Schedule Activation / Deactivation menu (available on the MVS-DataSync-Dashboard).

Individually for each job	Log in on the Admin Portal as a high level admin (above Provider). Select the Scheduling menu to view scheduled jobs. For each scheduled job, on the Base tab, clear the Activate checkbox to disable this setting.
Mass modify	In the Admin Portal, export scheduled syncs into a bulk load sheet. Modify schedule settings to de-activate scheduled syncs. Import the sheet.

Turn off schedules enabled on the CLI:
- Check if any schedules exist and overlap with the maintenance window: schedule list
- Disable overlapping schedules: schedule disable <job-name>

Note

This step is not relevant when upgrading a Single Node Cluster topology.

Verify that the primary database node is the active primary node at the time of upgrade:

database config

Note

A unified node topology will have the primary and database on the same node.

Note

This step is not relevant when upgrading a Single Node Cluster topology.

Ensure that the primary database node on which installation will be initiated has the stateStr parameter set to “PRIMARY” and has the highest priority number.

The highest priority number could vary depending on cluster layout.

Example output:

<ip address>:27020:
    priority: <number>
    stateStr: PRIMARY
    storageEngine: WiredTiger

<ip address>:27020:
    priority: 70.0
    stateStr: PRIMARY
    storageEngine: WiredTiger
<ip address>:27030:
    priority: 0.0
    stateStr: ARBITER
    storageEngine: WiredTiger
<ip address>:27020:
    priority: 50.0
    stateStr: SECONDARY
    storageEngine: WiredTiger
<ip address>:27030:
    priority: 0.0
    stateStr: ARBITER
    storageEngine: WiredTiger
<ip address>:27020:
    priority: 30.0
    stateStr: SECONDARY
    storageEngine: WiredTiger

Validate system health#

DONE

Verify that there are no pending security updates: security check

If any security updates are required, run security update
Mount upgrade ISO: system mount
Install the new version of the cluster check command: app install check_cluster

For details, see Cluster Check.
Run cluster check.

Inspect the output for warnings and errors. You can also use cluster check verbose to see more details, for example, to check that avx is enabled.

Review and resolve any warnings or errors before proceeding with the upgrade. Contact VOSS Support for assistance, if required.

For troubleshooting and resolutions, also refer to the Health Checks for Cluster Installations Guide and the Platform Guide.

If there is any sign that the paths below are over 80% full, a clean-up is required, for example, to avoid the risk of full logs during upgrade. Recommended steps to resolve are indicated at each path:

Path

Resolution

/

Contact VOSS Support if over 80%

/var/log

Run log purge

/opt/platform

Remove any unnecessary files from /media directory

/tmp

Reboot

Path	Resolution
/	Contact VOSS Support if over 80%
/var/log	Run `log purge`
/opt/platform	Remove any unnecessary files from /media directory
/tmp	Reboot

Note

If you run cluster status after installing the new version of cluster check, any error message regarding a failed command can be ignored. This error message will not show after upgrade.
Adaptation checks - if the GS SME Adaptation is installed, check for duplicate instances of of GS_SMETemplateData_DAT and delete any duplicates before upgrading to 24.2.

Pre-upgrade#

DONE

Obtain a suitable restore point as part of the rollback procedure (as per the guidelines for the infrastructure on which the VOSS Automate platform is deployed).

Important

All nodes must be powered off prior to creating the restore point, and must be powered back on again when the restore point is complete.

Optionally, if a backup is also required, use the following commands on the primary database node:

backup add <location-name>

backup create <location-name>

For details, see the Platform Guide.
Validate system health and check all services, nodes, and weights for the cluster:
1. Run cluster run application cluster list, and ensure that all application nodes show.
2. Run cluster check, then inspect the output of this command for warnings and errors. You can use the cluster check verbose command to see more details.
3. Ensure that no services are stopped or broken: app status
  
  The following message is normal on fresh database nodes:
```
suspended waiting for mongo ...
```
4. Important! Check that the database weights are set before upgrading a cluster. Example output:
```
<ip address>:
     weight: 80
 <ip address>:
     weight: 70
 <ip address>:
     weight: 60
 <ip address>:
     weight: 50
```
5. Verify the primary node in the primary site and ensure no nodes are in the recovering state (stateStr is not “RECOVERING”).

On the primary application node, verify that there are no pending security updates on any of the nodes:

cluster run all security check

If any security updates are required, run: security update

Upgrade#

DONE

It is recommended that the upgrade steps are run in a terminal opened with the screen command.

By default, the cluster upgrade is carried out in parallel on all nodes and without any backup in order to provide a fast upgrade.

For systems upgrading to 24.2 from 21.4.0 - 21.4-PB5, the VOSS platform maintenance mode starts automatically when running cluster upgrade. This prevents any new occurrences of scheduled transactions, including the 24.2 database syncs associated with insights sync. For details, see Insights Analytics in the Platform Guide.

Verify that the ISO has been uploaded to the media/ directory on each node. This speeds up the upgrade time.

On the primary database node (modular cluster) or primary unified node (unified node and single node cluster), run the following commands:

screen

cluster upgrade media/<upgrade_iso_file>
To remove a mount directory media/<iso_file basename> on nodes that may have remained after, for example, an upgrade, run:

cluster run all app cleanup
The system should reboot automatically. If it does not, perform manual reboot:

Topology

Command

cluster run notme system reboot

When all other nodes have rebooted, run system reboot on the local node.

system reboot

If the following node messages display, these can be ignored:
```
<node name> failed with timeout
```
Since all services will be stopped, this takes some time.
Press Ctrl + a, then \ to close screen.

Topology	Command
	`cluster run notme system reboot` When all other nodes have rebooted, run `system reboot` on the local node.
	`system reboot`

Post-upgrade and health check#

DONE

Check for required security updates. On the primary application node, run:

cluster run all security check
If security updates are required on any nodes, run the following on the primary application node:

cluster run all security update

If upgrading a Cloud deployment (Microsoft Azure or AWS), run: cluster check
Note

Contact VOSS Support for assistance if the following message displays at each node:
```
grub-pc: package in an undesired state
```
To resolve, VOSS Support runs the following command on each node (displayed for informational purposes only):

dpkg --configure -a

Following this command, prompts display in the text window. VOSS Support then performs the following (displayed for informational purposes only):
- At GRUB install devices, do not select any device. Press <Tab> to highlight, then <Ok>, and then press <Enter>.
- At Continuing without installing GRUB?, press <Yes>.
- Run cluster check again, and verify the error no longer displays.
If the system does not automatically reboot and you need to reboot manually - this takes some time as all services are stopped:

The system should reboot automatically. If it does not, perform manual reboot:

Topology

Command

cluster run notme system reboot

When all other nodes have rebooted, run system reboot on the local node.

system reboot

You can ignore the following node messages:
```
<node name> failed with timeout
```
Log in on the primary database node, then run:

cluster run database app status

If the report shows insights-voss-sync:realtime stopped* on some database, contact VOSS Support for assistance to perform the following on the primary database node (displayed for information only):
- Run /opt/platform/mags/insights-voss-sync-mag-script install database
  
  The output should be:
```
Configured Postgres secrets
```
- Verify that the database nodes now all have the correct mongo info:
  
  cluster run database diag config app insights-voss-sync /mongo
  
  All nodes should have the password/port/user shown as below:
```
mongo:
  password: ********
  port: 27020
  user: insights-platform
```
- Restart the insights-voss-sync:real-time service on all database nodes:
  
  cluster run database app start insights-voss-sync:real-time
Note

All unused docker images except “selfservice” and “voss_ubuntu” images will be removed from the system at this stage.
Verify cluster status. On the primary node, run: cluster check
If there are any errors, for details that may help with troubleshooting, run the following:

cluster run all diag health
To remove a mount directory (media/<iso_file basename>) on nodes that may have remained, after an upgrade for example, run the following on the primary database node:

cluster run all app cleanup
If the upgrade succeeds, type exit in the terminal to close the screen session.

If there are errors, keep the screen terminal open for troubleshooting, and contact VOSS Support.

Topology	Command
	`cluster run notme system reboot` When all other nodes have rebooted, run `system reboot` on the local node.
	`system reboot`

Database schema upgrade#

DONE

It is recommended that the upgrade steps are run in a terminal opened with the screen command.

On the primary application node, run the following:

screen

voss upgrade_db

voss upgrade_db
Check cluster status: cluster check

Template upgrade#

DONE

It is recommended that the upgrade steps are run in a terminal opened with the screen command.

On the primary application node, run the following commands:

screen

app template media/<VOSS Automate.template>

View the message that displays:

Running the DB-query to find the current environment's existing solution deployment config …

View progress:
- Python functions are deployed
- System artifacts are imported
  Note
  
  To perform fewer upgrade steps, updates of instances of some models are skipped, where:
  - data/CallManager instance does not exist as instance in data/NetworkDeviceList
  - data/CallManager instance exists, but data/NetworkDeviceList is empty
  - Call Manager AXL Generic Driver and Call Manager Control Center Services match the data/CallManager IP
- The template upgrade automatically detects the deployment mode, Enterprise or Provider. A system message displays for the selected deployment mode, for example:
  
  On Enterprise deployment:
```
Importing EnterpriseOverlay.json
```
  On Provider deployment:
```
Importing ProviderOverlay.json
```
- The template install automatically restarts necessary applications. If a cluster is detected, the installation propagates changes throughout the cluster.

Run app template, then review the output to verify that the upgrade message displays:

Deployment summary of PREVIOUS template solution
(i.e. BEFORE upgrade):
-------------------------------------------------

Product: [PRODUCT]
Version: [PREVIOUS PRODUCT RELEASE]
Iteration-version: [PREVIOUS ITERATION]
Platform-version: [PREVIOUS PLATFORM VERSION]

This is followed by updated product and version details:

Deployment summary of UPDATED template solution
(i.e. current values after installation):
-----------------------------------------------

Product: [PRODUCT]
Version: [UPDATED PRODUCT RELEASE]
Iteration-version: [UPDATED ITERATION]
Platform-version: [UPDATED PLATFORM VERSION]

If no errors are indicated, create a restore point.

As part of the rollback procedure, ensure that a suitable restore point is obtained prior to the start of the activity, as per the guidelines for the infrastructure on which the VOSS Automate platform is deployed.

For unsupported upgrade paths, the install script stops with the message:
```
Upgrade failed due to unsupported upgrade path.
Please log in as sysadmin and see Transaction logs for more detail.
```
You can roll back as per the guidelines for the infrastructure on which the VOSS Automate platform is deployed.

If there are errors for another reason, the install script stops with a failure message listing the problem. Contact VOSS Support.
On the primary application node, run the following command to verify that the extra_functions have the same checksum across the cluster:

Topology

Command

cluster run application voss get_extra_functions_version -c

cluster run voss get_extra_functions
For post-upgrade migrations, run the following command on a single application node of a cluster:

voss post-upgrade-migrations

Data migrations that are not critical to system operation can have significant execution time at scale. These need to be performed after the primary upgrade, allowing the migration to proceed while the system is in use - thereby limiting upgrade windows.
View transaction progress. A transaction is queued on VOSS Automate and its progress displays as it executes.
On the primary database node, check cluster status and health: cluster status

Topology	Command
	`cluster run application voss get_extra_functions_version -c`
	`cluster run voss get_extra_functions`

Post-template upgrade#

DONE

Verify the upgrade:
1. Log in on the Admin Portal, and check version details in About > Version.
  
  If your web browser can’t open the user interface, clear your browser cache before trying to open the interface again.
2. Confirm that versions are upgraded (where XXX is the release version).
  - Release should display XXX
  - Platform version should display XXX
Check that themes on all roles are set correctly.
For configurations using Northbound Billing Integration (NBI), check the service status of NBI, and restart if necessary.

Log files and error checks#

DONE

Inspect the output of the command line interface for upgrade errors, for example, “File import failed!” or “Failed to execute command”.
If there are any errors referring to log files, for example:
```
For more information refer to the execution log file with ``log view platform/execute.log``
```
Then run the log view command on the primary application node command to view any log files indicated in the error messages.

If required, send all the install log files in the install directory to an SFTP server:

log send sftp://x.x.x.x install
Log in on the Admin Portal as system level admin, then go to Administration Tools > Transaction, and inspect the transaction list for errors.

Post-maintenance window#

In the post-maintenance part of the upgrade you will need to perform the following tasks:

End the maintenance window and restore schedules
Apply the license
Mount the Insights disk

End maintenance window and restore schedules#

DONE

If you’re upgrading from 21.4 or 21.4.-PBx to 24.2, then, on the CLI, run the following command to end the VOSS maintenance window:

cluster maintenance-mode stop

Scheduled data sync transactions can now resume, including insights sync operations added in 24.1. For details, see Maintenance Mode in the Platform Guide.

Restore schedules.

Schedules can easily be activated and deactivated from the Bulk Schedule Activation / Deactivation menu available on the MVS-DataSync-Dashboard.

If upgrading from [21.4, 21.4-PB1, 21.4-PB2, 21.4-PB3]:

Re-enable scheduled imports if any were disabled prior to the upgrade - either individually for each job, or mass modify:

Individually for each job	Log in on the Admin Portal as a high level admin (above Provider). Select the Scheduling menu to view scheduled jobs. Click each scheduled job, and on the Base tab, select the Activate checkbox.
Mass modify	Modify the exported sheet of schedules to activate scheduled syncs. Import the sheet. If you don’t want to execute schedules overlapping the maintenance window but only execute afterwards, select Skip next execution.

For schedules enabled on the CLI, enable any disabled schedules that were overlapping the maintenance window:

schedule enable <job-name>

Licensing#

DONE

The Automate deployment requires a license. After installation, a 7-day grace period is available to license the product.

Since license processing is only scheduled every hour, if you wish to license immediately, first run voss check-license from the primary application node CLI.

Obtain the required license token from VOSS.
Apply the license:
- If applying a license via the GUI, follow the steps indicated in the Product License Management section of the Core Feature Guide.
- If applying a license through the CLI, follow the steps indicated in Product Licensing in the Platform Guide.

Mount the Insights disk#

DONE

On each database/unified node, assign the insights-voss-sync:database mount point to the drive added for the Insights database prior to upgrade.

For example, if drives list shows the added disk as …

Unused disks:
sde

Then run the following command on each database/unified node where the drive has been added:

drives add sde insights-voss-sync:database

Sample output:

$ drives add sde insights-voss-sync:database
Configuration setting "devices/scan_lvs" unknown.
Configuration setting "devices/allow_mixed_block_sizes" unknown.
WARNING: Failed to connect to lvmetad. Falling back to device scanning.
71ad98e0-7622-49ad-9fg9-db04055e82bc
Application insights-voss-sync processes stopped.
Migrating data to new drive - this can take several minutes
Data migration complete - reassigning drive
Checking that /dev/sde1 is mounted
Checking that /dev/dm-0 is mounted
/opt/platform/apps/mongodb/dbroot
Checking that /dev/sdc1 is mounted
/backups

Application services:firewall processes stopped.
Reconfiguring applications...
Application insights-voss-sync processes started.

The following message can be ignored on release 24.1:

Warning: Failed to connect to lvmetad. Falling back to device scanning.

Note

On Automate 24.2, the initial management of dashboards on the GUI and use of VOSS Wingman is available after the first scheduled delta-sync of data (scheduled to run every 30 minutes).

No manual sync is therefore required after upgrade. For details, see the Insights Analytics section of the Platform Guide.

For all nodes	`scp <upgrade_iso_file> platform@<node_ip_address>:~/media`
For primary application node	`scp <upgrade_template_file> platform@<application_node_ip_address>:~/media`

Upgrade Automate

On this page

Upgrade Automate#

Overview#

Before you start#

Prior to maintenance window#

Verify the primary database node and application node#

Download and check files#

Version check#

Maintenance window#

Security and health checks#

Validate system health#

Pre-upgrade#

Upgrade#

Post-upgrade and health check#

Database schema upgrade#

Template upgrade#

Post-template upgrade#

Log files and error checks#

Post-maintenance window#

End maintenance window and restore schedules#

Licensing#

Mount the Insights disk#