.. _upgrade_multinode_ISO:

.. rst-class:: chapter-with-expand

Modular Cluster Topology: Upgrade a Multinode Environment with the ISO and Template
-----------------------------------------------------------------------------------

.. index:: cluster;cluster check
.. index:: cluster;cluster upgrade
.. index:: screen
.. index:: database;database convert_drive
.. index:: app; app cleanup

    
    
.. note::    

  * When upgrading from an existing Modular Cluster Topology that 
    was available since VOSS Automate 21.1, use the steps listed here.
  * Tasks that are marked **Prior to Maintenance Window** can be completed a few days prior to
    the scheduled maintenance window so that VOSS support can be contacted if needed and
    in order to allow for reduce down time.

   
The standard **screen** command should be used where indicated.
See: :ref:`screen-command`.

.. rubric:: Primary database and application node in a Modular Cluster Topology

* Verify the *primary application node* (UN2) with the **cluster primary role application**
  command run on the node. The output should be `true`, for example:

   ::

      platform@UN2:~$ cluster primary role application
      is_primary: true


* Verify the *primary database node* (UN1) with the **cluster primary role database**
  command run on the node. The output should be `true`, for example:

   ::

      platform@UN1:~$ cluster primary role database
      is_primary: true


.. _modular-Prior-to-Maintenance-Window-Download-Files-and-Check-ISO:

Download Files and Check (Prior to Maintenance Window) 
.......................................................


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+------------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                    | Notes and Status   |
+==========================================================================================+====================+
|                                                                                          |                    |
| VOSS files: **https://voss.portalshape.com > Downloads > VOSS Automate > XXX > Upgrade** |                    |
|                                                                                          |                    |
| Download ``.iso``/``.ova`` and ``.template`` files, where XXX is the release number.     |                    |
|                                                                                          |                    |
| * Transfer the ``.iso`` file to the ``media/`` folder of the primary database node.      |                    |
| * Transfer the ``.template`` file to the ``media/`` folder of the primary application    |                    |
|   node.                                                                                  |                    |
|                                                                                          |                    |
| Two transfer options:                                                                    |                    |
|                                                                                          | .. raw:: html      |
| Either using SFTP:                                                                       |                    |
|                                                                                          |                    |
|                                                                                          |    <div><input     |
| * **sftp platform@<unified_node_hostname>**                                              |    type='checkbox' |
|                                                                                          |    id='done'       |
| * **cd media**                                                                           |    name='done'     |
|                                                                                          |    unchecked>      |
| * **put <upgrade_iso_or_ova_file>**                                                      |    <label          |
|                                                                                          |    for='isdone'>   |
| * **put <upgrade_template_file>**                                                        |    Done</label>    |
|                                                                                          |    </div>          |
|                                                                                          |                    |
|                                                                                          |                    |
| Or using SCP:                                                                            |                    |
|                                                                                          |                    |
|                                                                                          |                    |
| * **scp <upgrade_iso_or_ova_file> platform@<unified_node_ip_address>:~/media**           |                    |
|                                                                                          |                    |
| * **scp <upgrade_template_file> platform@<unified_node_ip_address>:~/media**             |                    |
|                                                                                          |                    |
|                                                                                          |                    |
| Verify that the ``.iso`` image and ``.template`` file copied:                            |                    |
|                                                                                          |                    |
|                                                                                          |                    |
| * **ls -l media/**                                                                       |                    |
|                                                                                          |                    |
|                                                                                          |                    |
| Verify that the original ``.sha256`` checksums on the                                    |                    |
| Download site match.                                                                     |                    |
|                                                                                          |                    |
|                                                                                          |                    |
| * primary database node: **system checksum media/<upgrade_iso_or_ova_file>**             |                    |
|                                                                                          |                    |
|   ``Checksum: <SHA256>``                                                                 |                    |
|                                                                                          |                    |
| * primary application node: **system checksum media/<upgrade_template_file>**            |                    |
|                                                                                          |                    |
|   ``Checksum: <SHA256>``                                                                 |                    |
|                                                                                          |                    |
|                                                                                          |                    |
+------------------------------------------------------------------------------------------+--------------------+


.. _modular-Prior-to-Maintenance-Window-Security-Health-Steps:

Security and Health Check Steps (Prior to Maintenance Window)
.............................................................


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+----------------------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                              | Notes and Status   |
+====================================================================================================+====================+
| Verify that the primary database node is the active primary node at the time of upgrade.           |                    |
|                                                                                                    |                    |
| **database config**                                                                                | .. raw:: html      |
|                                                                                                    |                    |
| Ensure that the node on which the installation will be initiated has the ``stateStr``              |    <div><input     |
| parameter set to **PRIMARY** and has the **highest** ``priority`` **number**                       |    type='checkbox' |
| (highest priority number could vary depending on cluster layout).                                  |    id='done'       |
|                                                                                                    |    name='done'     |
| Example output ::                                                                                  |    unchecked>      |
|                                                                                                    |    <label          |
|   <ip address>:27020:                                                                              |    for='isdone'>   |
|     priority: <number>                                                                             |    Done</label>    |
|     stateStr: PRIMARY                                                                              |    </div>          |
|     storageEngine: WiredTiger                                                                      |                    |
|                                                                                                    |                    |
|                                                                                                    |                    |
| * **cluster check** - inspect the output of this command for warnings and errors. You              |                    |
|   can also use **cluster check verbose** to see more details. While warnings will not              |                    |
|   prevent an upgrade, it is advisable that these be resolved prior to upgrading where              |                    |
|   possible. Some warnings may be resolved by upgrading.                                            |                    |
|                                                                                                    |                    |
|   For troubleshooting and resolutions,                                                             |                    |
|   also refer to the *Health Checks for Cluster Installations Guide* and *Platform Guide*.          |                    |
|                                                                                                    |                    |
|   If there is any sign of the paths below are over 80% full, a clean-up is needed,                 |                    |
|   for example to avoid risk of full logs occurring during upgrade.                                 |                    |
|   Clean-up steps are indicated next to the paths:                                                  |                    |
|                                                                                                    |                    |
|   ::                                                                                               |                    |
|                                                                                                    |                    |
|       /              (call support if over 80%)                                                    |                    |
|       /var/log       (run: log purge)                                                              |                    |
|       /opt/platform  (remove any unnecessary files from /media directory)                          |                    |
|       /tmp           (reboot)                                                                      |                    |
|                                                                                                    |                    |
|                                                                                                    |                    |
|   On the Primary Unified Node, verify there are no pending Security Updates on any of the          |                    |
|   nodes.                                                                                           |                    |
|                                                                                                    |                    |
| .. note::                                                                                          |                    |
|                                                                                                    |                    |
|    If you run **cluster status** after installing the new version of **cluster check**, any        |                    |
|    error message regarding a failed command can be ignored. This error message will not show       |                    |
|    after upgrade.                                                                                  |                    |
|                                                                                                    |                    |
| * **Adaptation check** - if the *GS SME Adaptation* is installed, check for duplicate instances of |                    |
|   of ``GS_SMETemplateData_DAT`` and deleted any duplicates before upgrading to 21.2.               |                    |
+----------------------------------------------------------------------------------------------------+--------------------+


.. _modular-Maintenance-Window-Schedules-Transactions-Version-Check:

Schedules, Transactions and Version Check (Maintenance Window) 
..................................................................


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+-------------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                     | Notes and Status   |
+===========================================================================================+====================+
| Run **cluster check** and verify that no warnings and errors show.                        |                    |
|                                                                                           |                    |
| Turn off any scheduled imports to prevent syncs triggering part way through the upgrade.  |                    |
| Two options are available:                                                                |                    |
|                                                                                           |                    |
| Individually for each job:                                                                |                    |
|                                                                                           |                    |
| 1. Log in on the Admin Portal as a high level administrator above Provider level.         |                    |
| 2. Select the **Scheduling** menu to view scheduled jobs.                                 |                    |
| 3. Click each scheduled job. On the Base tab, uncheck the **Activate** check box.         |                    |
|                                                                                           |                    |
| Mass modify:                                                                              |                    |
|                                                                                           |                    |
| 1. On the Admin Portal, export scheduled syncs into a bulk load sheet.                    | .. raw:: html      |
| 2. Modify the schedule settings to de-activate scheduled syncs.                           |                    |
| 3. Import the sheet.                                                                      |    <div><input     |
|                                                                                           |    type='checkbox' |
| Schedules enabled on the primary application node CLI:                                    |    id='done'       |
|                                                                                           |    name='done'     |
| 1. Run **schedule list** to check if any schedules exist and overlap with the maintenance |     unchecked>     |
|    window.                                                                                |    <label          |
| 2. For overlapping schedules, disable. Run **schedule disable <job-name>**.               |    for='isdone'>   |
|                                                                                           |    Done</label>    |
|                                                                                           |    </div>          |
|                                                                                           |                    |
|                                                                                           |                    |
+-------------------------------------------------------------------------------------------+--------------------+
| Check for running imports. Either wait for them to complete or cancel them:               |                    |
|                                                                                           |                    |
| 1. Log in on the Admin Portal as a high level administrator above Provider level.         |                    |
| 2. Select the **Transaction** menu to view transactions.                                  |                    |
| 3. Filter the **Action** column:                                                          |                    |
|                                                                                           |                    |
|    a. Choose **Status** as "Processing" and then choose each **Action**                   |                    |
|       that starts with "Import", for example, "Import Unity Connection".                  |                    |
|    b. Click **Search** and confirm there are no results.                                  |                    |
|    c. If there are transactions to cancel, select them and click **Cancel**.              |                    |
|                                                                                           |                    |
|                                                                                           | .. raw:: html      |
|                                                                                           |                    |
|                                                                                           |    <div><input     |
|                                                                                           |    type='checkbox' |
|                                                                                           |    id='done'       |
|                                                                                           |    name='done'     |
|                                                                                           |    unchecked>      |
|                                                                                           |    <label          |
|                                                                                           |    for='isdone'>   |
|                                                                                           |    Done</label>    |
|                                                                                           |    </div>          |
|                                                                                           |                    |
+-------------------------------------------------------------------------------------------+--------------------+
|                                                                                           |                    |
| **Version**                                                                               |                    |
|                                                                                           |                    |
| Record the current version information. This is required for upgrade troubleshooting.     |                    |
|                                                                                           |                    |
| * Log in on the Admin Portal and record the information contained in the menu:            | .. raw:: html      |
|   **About > Version**                                                                     |                    |
|                                                                                           |    <div><input     |
|                                                                                           |    type='checkbox' |
|                                                                                           |    id='done'       |
|                                                                                           |    name='done'     |
|                                                                                           |    unchecked>      |
|                                                                                           |    <label          |
|                                                                                           |    for='isdone'>   |
|                                                                                           |    Done</label>    |
|                                                                                           |    </div>          |
|                                                                                           |                    |
|                                                                                           |                    |
+-------------------------------------------------------------------------------------------+--------------------+

.. _modular-Maintenance-Window-Pre-Upgrade-Steps:

Pre-Upgrade Steps (Maintenance Window)  
...............................................


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+-----------------------------------------------------------------------------------------+--------------------+
| As part of the rollback procedure, ensure that                                          |                    |
| a suitable restore point is obtained prior to the start of the                          |                    |
| activity, as per the guidelines for the infrastructure on which                         |                    |
| the VOSS Automate platform is deployed.                                                 |                    |
|                                                                                         |                    |
|                                                                                         | .. raw:: html      |
|                                                                                         |                    |
|                                                                                         |    <div><input     |
|                                                                                         |    type='checkbox' |
|                                                                                         |    id='done'       |
|                                                                                         |    name='done'     |
|                                                                                         |    unchecked>      |
|                                                                                         |    <label          |
|                                                                                         |    for='isdone'>   |
|                                                                                         |    Done</label>    |
|                                                                                         |    </div>          |
|                                                                                         |                    |
| Optional: If a backup is also required - on the primary database node, use the          |                    |
| **backup add <location-name>** and                                                      |                    |
| **backup create <location-name>** commands. For details, refer to the *Platform Guide*. |                    |
|                                                                                         |                    |
+-----------------------------------------------------------------------------------------+--------------------+

   
.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+---------------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                       | Notes and Status   |
+=============================================================================================+====================+
| After restore point creation and before upgrading: validate system health and check all     |                    |
| services, nodes and weights for the cluster:                                                |                    |
|                                                                                             |                    |
| * **cluster run application cluster list**                                                  |                    |
|                                                                                             |                    |
|   Make sure all application nodes show.                                                     |                    |
|                                                                                             |                    |
|                                                                                             |                    |
|                                                                                             |                    |
|                                                                                             |                    |
|                                                                                             | .. raw:: html      |
|                                                                                             |                    |
|                                                                                             |    <div><input     |
| * **cluster check** - inspect the output of this command, for warnings and errors. You      |    type='checkbox' |
|   can also use **cluster check verbose** to see more details.                               |    id='done'       |
|                                                                                             |    name='done'     |
|   * Make sure no services are stopped/broken. The message 'suspended waiting for mongo'     |    unchecked>      |
|     is normal on the fresh database nodes.                                                  |    <label          |
|                                                                                             |    for='isdone'>   |
|   * Check that the database weights are set. It is *critical* to ensure the weights are set |    Done</label>    |
|     before upgrading a cluster.                                                             |    </div>          |
|     Example output:                                                                         |                    |
|                                                                                             |                    |
|     ::                                                                                      |                    |
|                                                                                             |                    |
|        172.29.21.240:                                                                       |                    |
|            weight: 80                                                                       |                    |
|        172.29.21.241:                                                                       |                    |
|            weight: 70                                                                       |                    |
|        172.29.21.243:                                                                       |                    |
|            weight: 60                                                                       |                    |
|        172.29.21.244:                                                                       |                    |
|            weight: 50                                                                       |                    |
|                                                                                             |                    |
|   * Verify the primary node in the primary site and ensure no nodes are in the              |                    |
|     'recovering' state (``stateStr`` is not ``RECOVERING``). On the primary node:           |                    |
|                                                                                             |                    |
| On the primary application node, verify there are no pending Security Updates on any of the |                    |
| nodes:                                                                                      |                    |
|                                                                                             |                    |
|  * **cluster run all security check**                                                       |                    |
|                                                                                             |                    |
|                                                                                             |                    |
+---------------------------------------------------------------------------------------------+--------------------+

.. _modular-Maintenance-Window-upgrade-ISO:

Upgrade (Maintenance Window)  
............................


.. note::


   By default, the cluster upgrade is carried out in parallel on all nodes and without any
   backup in order to provide a fast upgrade.



.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+--------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                | Notes and Status   |
+======================================================================================+====================+
| It is recommended that the upgrade steps                                             |                    |
| are run in a terminal opened with the **screen** command.                            |                    |
|                                                                                      |                    |
| Verify that the ISO has been uploaded to the ``media/`` directory on each node. This |                    |
| will speed up the upgrade time.                                                      |                    |
|                                                                                      |                    |
| On the primary database node:                                                        |                    |
|                                                                                      |                    |
|                                                                                      |                    |
| * **screen**                                                                         | .. raw:: html      |
|                                                                                      |                    |
|                                                                                      |                    |
| * **cluster upgrade media/<upgrade_iso_file>**                                       |    <div><input     |
|                                                                                      |    type='checkbox' |
|                                                                                      |    id='done'       |
|                                                                                      |    name='done'     |
|                                                                                      |    unchecked>      |
|                                                                                      |    <label          |
|                                                                                      |    for='isdone'>   |
|                                                                                      |    Done</label>    |
|                                                                                      |    </div>          |
|                                                                                      |                    |
|                                                                                      |                    |
| Close **screen**: ``Ctrl-a \``                                                       |                    |
|                                                                                      |                    |
+--------------------------------------------------------------------------------------+--------------------+

All unused docker images except ``selfservice`` and ``voss_ubuntu`` images
will be removed from the system at this stage.


.. _modular-Maintenance-Window-Post-Upgrade-Security-Health-Steps:

Post-Upgrade and Health Steps  (Maintenance Window)
.............................................................


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+--------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                | Notes and Status   |
+======================================================================================+====================+
| On the primary database node, verify the cluster status:                             |                    |
|                                                                                      |                    |
| * **cluster check**                                                                  | .. raw:: html      |
|                                                                                      |                    |
| * If any of the above commands show errors, check for further details to assist      |    <div><input     |
|   with troubleshooting:                                                              |    type='checkbox' |
|                                                                                      |    id='done'       |
|   **cluster run all diag health**                                                    |    name='done'     |
|                                                                                      |    unchecked>      |
|                                                                                      |    <label          |
|                                                                                      |    for='isdone'>   |
|                                                                                      |    Done</label>    |
|                                                                                      |    </div>          |
|                                                                                      |                    |
+--------------------------------------------------------------------------------------+--------------------+
| To remove a mount directory ``media/<iso_file basename>`` on nodes that may          |                    |
| have remained after for example an upgrade, run:                                     |                    |
|                                                                                      | .. raw:: html      |
| **cluster run all app cleanup**                                                      |                    |
|                                                                                      |    <div><input     |
| on the primary database node.                                                        |    type='checkbox' |
|                                                                                      |    id='done'       |
|                                                                                      |    name='done'     |
|                                                                                      |    unchecked>      |
|                                                                                      |    <label          |
|                                                                                      |    for='isdone'>   |
|                                                                                      |    Done</label>    |
|                                                                                      |    </div>          |
|                                                                                      |                    |
+--------------------------------------------------------------------------------------+--------------------+
| Check for needed security updates. On the primary application node, run:             |                    |
|                                                                                      |                    |
| * **cluster run all security check**                                                 |                    |
|                                                                                      |                    |
| If one  or more updates are required for any node, run on the primary application    |                    |
| node:                                                                                |                    |
|                                                                                      |                    |
|                                                                                      | .. raw:: html      |
| * **cluster run all security update**                                                |                    |
|                                                                                      |    <div><input     |
|                                                                                      |    type='checkbox' |
|                                                                                      |    id='done'       |
|                                                                                      |    name='done'     |
| Note: *if the system reboots, do not carry out the next manual reboot step*.         |    unchecked>      |
|                                                                                      |    <label          |
| Manual reboot *only if needed*:                                                      |    for='isdone'>   |
|                                                                                      |    Done</label>    |
|                                                                                      |    </div>          |
| * **cluster run notme system reboot**                                                |                    |
|                                                                                      |                    |
|                                                                                      |                    |
| If node messages: ``<node name> failed with timeout`` are displayed,                 |                    |
| these can be ignored.                                                                |                    |
|                                                                                      |                    |
|                                                                                      |                    |
| * **system reboot**                                                                  |                    |
|                                                                                      |                    |
|                                                                                      |                    |
| Since all services will be stopped, this takes some time.                            |                    |
|                                                                                      |                    |
|                                                                                      |                    |
+--------------------------------------------------------------------------------------+--------------------+
| If upgrade is successful, the screen session can be closed by typing **exit** in the |                    |
| screen terminal. If errors occurred, keep the screen terminal                        |                    |
| open for troubleshooting purposes and contact VOSS support.                          | .. raw:: html      |
|                                                                                      |                    |
|                                                                                      |    <div><input     |
|                                                                                      |    type='checkbox' |
|                                                                                      |    id='done'       |
|                                                                                      |    name='done'     |
|                                                                                      |    unchecked>      |
|                                                                                      |    <label          |
|                                                                                      |    for='isdone'>   |
|                                                                                      |    Done</label>    |
|                                                                                      |    </div>          |
|                                                                                      |                    |
+--------------------------------------------------------------------------------------+--------------------+



.. _modular-Maintenance-Window-Database-Schema-Upgrade:


Database Schema Upgrade (Maintenance Window) 
.............................................


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+---------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                 | Notes and Status   |
+=======================================================================================+====================+
| It is recommended that the upgrade steps                                              |                    |
| are run in a terminal opened with the **screen** command.                             |                    |
|                                                                                       |                    |
| On the primary database node:                                                         |                    |
|                                                                                       | .. raw:: html      |
| * **screen**                                                                          |                    |
|                                                                                       |    <div><input     |
| * **voss upgrade_db**                                                                 |    type='checkbox' |
|                                                                                       |    id='done'       |
|                                                                                       |    name='done'     |
| Check cluster status                                                                  |    unchecked>      |
|                                                                                       |    <label          |
| * **cluster check**                                                                   |    for='isdone'>   |
|                                                                                       |    Done</label>    |
|                                                                                       |    </div>          |
|                                                                                       |                    |
+---------------------------------------------------------------------------------------+--------------------+


.. _modular-Maintenance-Window-Template-Upgrade-ISO:


Template Upgrade (Maintenance Window) 
.....................................



.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+---------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                 | Notes and Status   |
+=======================================================================================+====================+
| It is recommended that the upgrade steps                                              |                    |
| are run in a terminal opened with the **screen** command.                             |                    |
|                                                                                       |                    |
| On the primary application node:                                                      |                    |
|                                                                                       | .. raw:: html      |
| * **screen**                                                                          |                    |
|                                                                                       |    <div><input     |
| * **app template media/<VOSS Automate.template>**                                     |    type='checkbox' |
|                                                                                       |    id='done'       |
|                                                                                       |    name='done'     |
|                                                                                       |    unchecked>      |
|                                                                                       |    <label          |
|                                                                                       |    for='isdone'>   |
|                                                                                       |    Done</label>    |
|                                                                                       |    </div>          |
|                                                                                       |                    |
+---------------------------------------------------------------------------------------+--------------------+



The following message appears:                                                                   
                                                                                                 
::                                                                                               
                                                                                                 
   Running the DB-query to find the current environment's 
   existing solution deployment config... 
                                                                                                 
* Python functions are deployed                                                                  
* System artifacts are imported.                                                                 

  .. note::

     In order to carry out fewer upgrade steps, the updates of instances
     of the some models are skipped in the cases where:
     
     * ``data/CallManager`` instance does not exist as instance in ``data/NetworkDeviceList``
     * ``data/CallManager`` instance exists, but ``data/NetworkDeviceList`` is empty
     * Call Manager AXL Generic Driver and Call Manager Control Center Services
       match the ``data/CallManager`` IP

The template upgrade automatically detects the deployment mode: "Enterprise",                    
"Provider with HCM-F" or "Provider without HCM-F".                                               
A message displays according to the selected deployment type.
Check for one of the messages below:                                    
                                                                                                 
::                                                                                               
                                                                                                 
   Importing EnterpriseOverlay.json
                                                                                                
   Importing ProviderOverlay_Hcmf.json ...
                                                                                                
   Importing ProviderOverlay_Decoupled.json ...
                                                                                                 
The template install automatically restarts necessary applications. If a cluster is detected,    
the installation propagates changes throughout the cluster.                                      


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+------------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                    | Notes and Status   |
+==========================================================================================+====================+
|                                                                                          |                    |
| Review the output from the **app template** command and confirm that the upgrade message | .. raw:: html      |
| appears:                                                                                 |                    |
|                                                                                          |    <div><input     |
|                                                                                          |    type='checkbox' |
|                                                                                          |    id='done'       |
|                                                                                          |    name='done'     |
|                                                                                          |    unchecked>      |
|                                                                                          |    <label          |
|                                                                                          |    for='isdone'>   |
|                                                                                          |    Done</label>    |
|                                                                                          |    </div>          |
|                                                                                          |                    |
|                                                                                          |                    |
+------------------------------------------------------------------------------------------+--------------------+

::

 Deployment summary of PREVIOUS template solution
 (i.e. BEFORE upgrade):
 -------------------------------------------------                      
                                                                                              
                                                                                              
 Product: [PRODUCT]                                                                           
 Version: [PREVIOUS PRODUCT RELEASE]                                                          
 Iteration-version: [PREVIOUS ITERATION]                                                      
 Platform-version: [PREVIOUS PLATFORM VERSION]                                                
                                                                                              
                                                                                               
This is followed by updated product and version details:
                                                                                               
::

  Deployment summary of UPDATED template solution 
  (i.e. current values after installation):    
  -----------------------------------------------    
                                                                                               
  Product: [PRODUCT]                                                                           
  Version: [UPDATED PRODUCT RELEASE]                                                           
  Iteration-version: [UPDATED ITERATION]                                                       
  Platform-version: [UPDATED PLATFORM VERSION]                                                 
                                                                                               
                                                                                               
                                                                                               


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+-----------------------------------------------------------------------------+--------------------+
| Description and Steps                                                       | Notes and Status   |
+=============================================================================+====================+
|                                                                             |                    |
|                                                                             | .. raw:: html      |
|                                                                             |                    |
| * If no errors are indicated, create a restore point.                       |    <div><input     |
|                                                                             |    type='checkbox' |
|                                                                             |    id='done'       |
|                                                                             |    name='done'     |
|                                                                             |    unchecked>      |
|                                                                             |    <label          |
|                                                                             |    for='isdone'>   |
|                                                                             |    Done</label>    |
|                                                                             |    </div>          |
|                                                                             |                    |
|   As part of the rollback procedure, ensure that                            |                    |
|   a suitable restore point is obtained prior to the start of the            |                    |
|   activity, as per the guidelines for the infrastructure on which           |                    |
|   the VOSS Automate platform is deployed.                                   |                    |
+-----------------------------------------------------------------------------+--------------------+
|                                                                             |                    |
|                                                                             | .. raw:: html      |
|                                                                             |                    |
| For an unsupported upgrade path, the install script stops with the message: |    <div><input     |
|                                                                             |    type='checkbox' |
|                                                                             |    id='done'       |
| ::                                                                          |    name='done'     |
|                                                                             |    unchecked>      |
|                                                                             |    <label          |
|     Upgrade failed due to unsupported upgrade path.                         |    for='isdone'>   |
|     Please log in as sysadmin                                               |    Done</label>    |
|     and see Transaction logs for more detail.                               |    </div>          |
|                                                                             |                    |
| You can roll back as per the guidelines for the infrastructure on which     |                    |
| the VOSS Automate platform is deployed.                                     |                    |
|                                                                             |                    |
+-----------------------------------------------------------------------------+--------------------+
|                                                                             |                    |
|                                                                             | .. raw:: html      |
| If there are errors for another reason, the install script stops with a     |                    |
| failure message listing the problem. Contact VOSS support.                  |    <div><input     |
|                                                                             |    type='checkbox' |
|                                                                             |    id='done'       |
|                                                                             |    name='done'     |
|                                                                             |    unchecked>      |
|                                                                             |    <label          |
|                                                                             |    for='isdone'>   |
|                                                                             |    Done</label>    |
|                                                                             |    </div>          |
|                                                                             |                    |
+-----------------------------------------------------------------------------+--------------------+
|                                                                             |                    |
| On the primary application node,                                            | .. raw:: html      |
| verify the ``extra_functions`` have the *same checksum* across the cluster. |                    |
|                                                                             |    <div><input     |
| * **cluster run application voss get_extra_functions_version -c**           |    type='checkbox' |
|                                                                             |    id='done'       |
|                                                                             |    name='done'     |
|                                                                             |    unchecked>      |
|                                                                             |    <label          |
|                                                                             |    for='isdone'>   |
|                                                                             |    Done</label>    |
|                                                                             |    </div>          |
|                                                                             |                    |
+-----------------------------------------------------------------------------+--------------------+
| Post upgrade migrations:                                                    |                    |
|                                                                             |                    |
| On a single application node of a cluster, run:                             | .. raw:: html      |
|                                                                             |                    |
| * **voss post-upgrade-migrations**                                          |    <div><input     |
|                                                                             |    type='checkbox' |
|                                                                             |    id='done'       |
|                                                                             |    name='done'     |
|                                                                             |    unchecked>      |
|                                                                             |    <label          |
|                                                                             |    for='isdone'>   |
|                                                                             |    Done</label>    |
|                                                                             |    </div>          |
|                                                                             |                    |
+-----------------------------------------------------------------------------+--------------------+


Data migrations that are not critical to system operation can have significant
execution time at scale. These need to be performed after the primary upgrade,
allowing the migration to proceed whilst the system is in use - thereby limiting
upgrade windows.

A transaction is queued on VOSS Automate and its progress is displayed as it executes.




.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+----------------------------------+--------------------+
| Description and Steps            | Notes and Status   |
+==================================+====================+
|                                  |                    |
| Check cluster status and health  | .. raw:: html      |
|  - on the primary database node: |                    |
|                                  |                    |
| * **cluster status**             |    <div><input     |
|                                  |    type='checkbox' |
|                                  |    id='done'       |
|                                  |    name='done'     |
|                                  |    unchecked>      |
|                                  |    <label          |
|                                  |    for='isdone'>   |
|                                  |    Done</label>    |
|                                  |    </div>          |
|                                  |                    |
+----------------------------------+--------------------+


.. _modular-Maintenance-Window-Post-Template-Upgrade-Tasks:


Post Template Upgrade Tasks (Maintenance Window) 
................................................


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+--------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                | Notes and Status   |
+======================================================================================+====================+
|                                                                                      |                    |
| **SSO Login URL check if needed**                                                    |                    |
|                                                                                      |                    |
| Verify the SSO Login URL if needed. Go to **Single Sign On > SSO Identity Provider** |                    |
| and ensure your URL matches the **SSO Login URL** value.                             |                    |
|                                                                                      |                    |
|                                                                                      |                    |
| **Customized ``data/Settings``**                                                     |                    |
|                                                                                      |                    |
| Merge the previously backed up customized ``data/Settings`` with the latest settings |                    |
| on the system by manually adding the differences or exporting the latest settings    |                    |
| to JSON, merging the customized changes and importing the JSON.                      |                    |
|                                                                                      |                    |
|                                                                                      |                    |
| **Support for VG400 and VG450 Analogue Gateways**                                    |                    |
|                                                                                      |                    |
| Before adding the VG400 or VG450 Gateway, the ``device/cucm/GatewayType``            |                    |
| model needs to be imported for each Unified CM.                                      |                    |
|                                                                                      |                    |
| 1. Create a Model Type List which includes the ``device/cucm/GatewayType`` model.    |                    |
| 2. Add the Model Type List to all the required Unified CM Data Syncs.                |                    |
| 3. Execute the Data Sync for all the required Unified CMs.                           |                    |
|                                                                                      |                    |
|                                                                                      |                    |
| **Verify the upgrade**                                                               | .. raw:: html      |
|                                                                                      |                    |
| Log in on the Admin Portal and check the information contained in the                |                    |
| **About > Version** menu. Confirm that versions have upgraded.                       |    <div><input     |
|                                                                                      |    type='checkbox' |
| * **Release** should show 21.3                                                       |    id='done'       |
| * **Platform Version** should show 21.3                                              |    name='done'     |
|                                                                                      |    unchecked>      |
|                                                                                      |    <label          |
|                                                                                      |    for='isdone'>   |
|                                                                                      |    Done</label>    |
|                                                                                      |    </div>          |
|                                                                                      |                    |
+--------------------------------------------------------------------------------------+--------------------+
|                                                                                      | .. raw:: html      |
|                                                                                      |                    |
|                                                                                      |    <div><input     |
| * Check themes on all roles are set correctly                                        |    type='checkbox' |
|                                                                                      |    id='done'       |
|                                                                                      |    name='done'     |
|                                                                                      |    unchecked>      |
|                                                                                      |    <label          |
|                                                                                      |    for='isdone'>   |
|                                                                                      |    Done</label>    |
|                                                                                      |    </div>          |
|                                                                                      |                    |
+--------------------------------------------------------------------------------------+--------------------+
|                                                                                      | .. raw:: html      |
|                                                                                      |                    |
|                                                                                      |    <div><input     |
| * For configurations that make use of the Northbound Billing Integration (NBI),      |    type='checkbox' |
|   please check the service status of NBI and restart if necessary.                   |    id='done'       |
|                                                                                      |    name='done'     |
|                                                                                      |    unchecked>      |
|                                                                                      |    <label          |
|                                                                                      |    for='isdone'>   |
|                                                                                      |    Done</label>    |
|                                                                                      |    </div>          |
|                                                                                      |                    |
+--------------------------------------------------------------------------------------+--------------------+


.. _modular-Maintenance-Window-Restore_Schedules:

Restore Schedules (Maintenance Window) 
......................................


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+--------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                | Notes and Status   |
+======================================================================================+====================+
| Re-enable scheduled imports if any were disabled prior to the upgrade.               |                    |
| Two options are available:                                                           |                    |
|                                                                                      |                    |
| Individually for each job:                                                           |                    |
|                                                                                      |                    |
| 1. Log in on the Admin Portal as a high level administrator above Provider           |                    |
|    level.                                                                            |                    |
| 2. Select the **Scheduling** menu to view scheduled jobs.                            |                    |
| 3. Click each scheduled job. On the Base tab, check the **Activate** check box.      |                    |
|                                                                                      |                    |
| Mass modify:                                                                         |                    |
|                                                                                      |                    |
| 1. Modify the exported sheet of schedules to activate scheduled syncs.               |                    |
| 2. Import the bulk load sheet.                                                       | .. raw:: html      |
|                                                                                      |                    |
| .. note::                                                                            |                    |
|                                                                                      |                    |
|    Select the **Skip next execution** option if you do not wish to execute schedules |                    |
|    overlapping the maintenance window, but only execute thereafter.                  |                    |
|                                                                                      |                    |
|                                                                                      |                    |
| Schedules enabled on the CLI of the primary application node:                        |    <div><input     |
|                                                                                      |    type='checkbox' |
| 1. For disabled schedules that were overlapping the maintenance window, enable.      |    id='done'       |
|                                                                                      |    name='done'     |
|    Run **schedule enable <job-name>**.                                               |    unchecked>      |
|                                                                                      |    <label          |
|                                                                                      |    for='isdone'>   |
|                                                                                      |    Done</label>    |
|                                                                                      |    </div>          |
|                                                                                      |                    |
|                                                                                      |                    |
+--------------------------------------------------------------------------------------+--------------------+

.. _modular-Maintenance-Window-Release-specific-updates:


Release Specific Updates (Maintenance Window) 
................................................

.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+-------------------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                           | Notes and Status   |
+=================================================================================================+====================+
| When upgrading from CUCDM 11.5.3 Patch Bundle 2 or VOSS-4-UC 18.1 Patch Bundle 2 and earlier,   |                    |
| re-import the following from all CUCM devices, since this upgrade deleted obsolete CUC timezone |                    |
| codes from the VOSS Automate database:                                                          |                    |
|                                                                                                 |                    |
| * CUC models:                                                                                   | .. raw:: html      |
|                                                                                                 |                    |
|   ``device/cuc/TimeZone``                                                                       |    <div><input     |
|                                                                                                 |    type='checkbox' |
|                                                                                                 |    id='done'       |
|                                                                                                 |    name='done'     |
| Note:                                                                                           |    unchecked>      |
|                                                                                                 |    <label          |
| This is a once off data migration step. If this was performed previously when upgrading         |    for='isdone'>   |
| to 19.1.x, then it does not have to be repeated.                                                |    Done</label>    |
|                                                                                                 |    </div>          |
+-------------------------------------------------------------------------------------------------+--------------------+
| After upgrading, obtain and install the following patch according to its accompanying MOP file: |                    |
|                                                                                                 |                    |
| * **Server Name**: ``secure.voss-solutions.com``                                                | .. raw:: html      |
| * **Path**: ``/software/voss4uc/releases/Release-19.2.1``                                       |                    |
| * **Patch Directory**: ``Update_CUC_Localization_patch``                                        |    <div><input     |
| * **Patch File**: ``Update_CUC_Localization_patch.script``                                      |    type='checkbox' |
| * **MOP File**: ``MOP-Update_CUC_Localization.pdf``                                             |    id='done'       |
|                                                                                                 |    name='done'     |
|                                                                                                 |    unchecked>      |
| Note:                                                                                           |    <label          |
|                                                                                                 |    for='isdone'>   |
| This is a once off data migration step. If this was performed previously when upgrading         |    Done</label>    |
| to 19.1.x, then it does not have to be repeated.                                                |    </div>          |
|                                                                                                 |                    |
+-------------------------------------------------------------------------------------------------+--------------------+
| Re-import the following from all CUCM devices:                                                  |                    |
|                                                                                                 |                    |
| * CUCM models:                                                                                  | .. raw:: html      |
|                                                                                                 |                    |
| ``device/cucm/PhoneType``                                                                       |    <div><input     |
|                                                                                                 |    type='checkbox' |
| For steps to create a custom data sync, refer to the chapter on Data Sync                       |    id='done'       |
| in the Core Feature Guide.                                                                      |    name='done'     |
|                                                                                                 |    unchecked>      |
|                                                                                                 |    <label          |
| Note:                                                                                           |    for='isdone'>   |
|                                                                                                 |    Done</label>    |
| This is a once off data migration step. If this was performed previously when upgrading         |    </div>          |
| to 19.1.x, then it does not have to be repeated.                                                |                    |
+-------------------------------------------------------------------------------------------------+--------------------+
|                                                                                                 |                    |
|  User Management migration updates default authentication types on SSO Identity Providers.      | .. raw:: html      |
|  If an SSO Identity Provider exists at the provider hierarchy level, the                        |                    |
|  default authentication settings:                                                               |    <div><input     |
|                                                                                                 |    type='checkbox' |
|  * Authentication Scope: Current hierarchy level and below                                      |    id='done'       |
|  * User Sync Type: All users                                                                    |    name='done'     |
|                                                                                                 |    unchecked>      |
|  will not allow any non-SSO user logins (typically local administrators).                       |    <label          |
|  The solution is to log in as higher level administrator account (full access)                  |    for='isdone'>   |
|  and set the SSO Identity Provider:                                                             |    Done</label>    |
|                                                                                                 |    </div>          |
|  * Authentication Scope: Current hierarchy level only                                           |                    |
|  * User Sync Type: LDAP synced users only                                                       |                    |
|                                                                                                 |                    |
|  Please refer to the *SSO Identity Provider: Field Reference* topic in the Core Feature Guide.  |                    |
+-------------------------------------------------------------------------------------------------+--------------------+
|                                                                                                 |                    |
|  When upgrading to release 21.3, users of Microsoft apps should after upgrade, select each      | .. raw:: html      |
|  Microsoft Tenant (``relation/MicrosoftTenant``) in the Admin GUI and click **Save** on it      |                    |
|  without making any changes.                                                                    |    <div><input     |
|                                                                                                 |    type='checkbox' |
|  This step is required so that VOSS Automate can communicate with the Tenant post upgrade.      |    id='done'       |
|                                                                                                 |    name='done'     |
|                                                                                                 |    unchecked>      |
|                                                                                                 |    <label          |
|                                                                                                 |    for='isdone'>   |
|                                                                                                 |    Done</label>    |
|                                                                                                 |    </div>          |
+-------------------------------------------------------------------------------------------------+--------------------+


.. _modular-Maintenance-Window-Log-Files-Error-Checks:


Log Files and Error Checks (Maintenance Window) 
................................................

.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+----------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                            | Notes and Status   |
+==================================================================================+====================+
| Inspect the output of the command line interface for upgrade errors,             | .. raw:: html      |
| for example ``File import failed!`` or ``Failed to execute command``.            |                    |
|                                                                                  |                    |
| On the primary application node,                                                 |                    |
| use the **log view** command to view any log files indicated in the error        |                    |
| messages, for example, run the command if the following message appears:         |    <div><input     |
|                                                                                  |    type='checkbox' |
| ::                                                                               |    id='done'       |
|                                                                                  |    name='done'     |
|    For more information refer to the execution log file with                     |    unchecked>      |
|    'log view platform/execute.log'                                               |    <label          |
|                                                                                  |    for='isdone'>   |
| For example, if it is required send all the install log files in the ``install`` |    Done</label>    |
| directory to an SFTP server:                                                     |    </div>          |
|                                                                                  |                    |
| * **log send sftp://x.x.x.x install**                                            |                    |
|                                                                                  |                    |
+----------------------------------------------------------------------------------+--------------------+
|                                                                                  | .. raw:: html      |
| Log in on the Admin Portal as system level administrator, go to                  |                    |
| **Administration Tools > Transaction** and inspect the transactions              |    <div><input     |
| list for errors.                                                                 |    type='checkbox' |
|                                                                                  |    id='done'       |
|                                                                                  |    name='done'     |
|                                                                                  |    unchecked>      |
|                                                                                  |    <label          |
|                                                                                  |    for='isdone'>   |
|                                                                                  |    Done</label>    |
|                                                                                  |    </div>          |
|                                                                                  |                    |
+----------------------------------------------------------------------------------+--------------------+