.. _modular-upgrade_multinode-delta:

.. rst-class:: chapter-with-expand

Modular Cluster Topology: Upgrade a Multinode Environment with the Delta Bundle
-------------------------------------------------------------------------------

.. index:: cluster;cluster upgrade
.. index:: cluster;cluster status
.. index:: cluster;cluster run
.. index:: cluster;cluster run <IP>
.. index:: cluster;cluster run application
.. index:: cluster;cluster run database
.. index:: cluster;cluster run all
.. index:: cluster;cluster primary role application
.. index:: cluster;cluster primary role database
.. index:: screen

    
.. note::    

  * While system upgrade takes approximately two hours
    at a single site, this may vary in accordance with your topology, number of
    devices and subscribers. Adjust your upgrade maintenance window
    to allow for your configuration.
   
The standard **screen** command should be used where indicated.
See: :ref:`screen-command`.


Determine the Primary database and application node in a Modular Cluster Topology
...................................................................................

.. important::

   All upgrade steps are to be run from the *primary application node*. Where it is
   necessary to run database node commands from this application node, the primary
   database node IP will be passed in as a parameter with the command.

In order to run the commands for the upgrade on the nodes with the appropriate roles,
determine:

* The *primary application node*
* The *primary database node*

1. Log in on a node in your modular cluster.
2. To find the *primary application node* in the cluster:

   ::

      $ cluster run all cluster primary role application

   Record the node entry where ``is_primary: true``, for example:

   ::

      ---------- VOSS-UN-1, ip=192.168.100.3, role=webproxy,application, loc=cpt

          is_primary: true

3. To find the *primary database node* in the cluster:

   ::

      $ cluster run all cluster primary role database

   Record the node entry IP where ``is_primary: true``, for example:

   ::

       ---------- VOSS-UN-2, ip=192.168.100.4, role=database, loc=cpt
       
       
           is_primary: true
       
   This IP address will be used in command parameters during upgrade.

.. note::

   When the **cluster run all primary role <role>** command is run,
   web proxy nodes will return a failure - this can be ignored.
   For example:

   ::
   
      ---------- VOSS-WP-1, ip=192.168.100.9, role=webproxy, loc=cpt
      Invalid command syntax - refer to help descriptions



.. _modular-Download-Files-and-Check-script:

Download Files and Check 
..........................................


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+-----------------------------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                                     | Notes and Status   |
+===========================================================================================================+====================+
| VOSS files:                                                                                               |                    |
|                                                                                                           |                    |
| **https://voss.portalshape.com > Downloads > VOSS Automate > XXX > Upgrade**                              |                    |
|                                                                                                           |                    |
| Download ``XXX-Delta-Bundle.script`` file, where ``XXX`` matches the release.                             |                    |
| Transfer the ``XXX-Delta-Bundle.script`` file to the ``media/`` folder *of the primary application node*. |                    |
| Two file transfer options:                                                                                | .. raw:: html      |
|                                                                                                           |                    |
| Either using SFTP:                                                                                        |                    |
|                                                                                                           |                    |
|                                                                                                           |    <div><input     |
| * **sftp platform@<primary_application_hostname>**                                                        |    type='checkbox' |
|                                                                                                           |    id='done'       |
| * **cd media**                                                                                            |    name='done'     |
|                                                                                                           |    unchecked>      |
| * **put <XXX-Delta-Bundle.script>**                                                                       |    <label          |
|                                                                                                           |    for='isdone'>   |
|                                                                                                           |    Done</label>    |
|                                                                                                           |    </div>          |
|                                                                                                           |                    |
|                                                                                                           |                    |
| Or using SCP:                                                                                             |                    |
|                                                                                                           |                    |
|                                                                                                           |                    |
| * **scp <XXX-Delta-Bundle.script> platform@<primary_application_hostname>:~/media**                       |                    |
|                                                                                                           |                    |
|                                                                                                           |                    |
|                                                                                                           |                    |
| On the primary application node, verify that the ``.script`` file copied:                                 |                    |
|                                                                                                           |                    |
|                                                                                                           |                    |
| * **ls -l media/**                                                                                        |                    |
|                                                                                                           |                    |
|                                                                                                           |                    |
| On the primary application node, verify that the original ``.sha256`` checksums on the                    |                    |
| Download site match.                                                                                      |                    |
|                                                                                                           |                    |
|                                                                                                           |                    |
| * **system checksum media/<XXX-Delta-Bundle.script>**                                                     |                    |
|                                                                                                           |                    |
|   ``Checksum: <SHA256>``                                                                                  |                    |
|                                                                                                           |                    |
|                                                                                                           |                    |
+-----------------------------------------------------------------------------------------------------------+--------------------+

.. _modular-Adaptations-Check:

Adaptations Check 
..........................

.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                        | Notes and Status   |
+==============================================================================+====================+
| Identify installed adaptations and determine any effect on the upgrade plan. |                    |
|                                                                              |                    |
| If the release is accompanied by Upgrade Notes, refer to the details.        | .. raw:: html      |
|                                                                              |                    |
|                                                                              |    <div><input     |
|                                                                              |    type='checkbox' |
|                                                                              |    id='done'       |
|                                                                              |    name='done'     |
|                                                                              |    unchecked>      |
|                                                                              |    <label          |
|                                                                              |    for='isdone'>   |
|                                                                              |    Done</label>    |
|                                                                              |    </div>          |
|                                                                              |                    |
+------------------------------------------------------------------------------+--------------------+


.. _modular-Schedules-Transactions-Version-Check:

Schedules, Transactions and Version Check 
..................................................


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+----------------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                        | Notes and Status   |
+==============================================================================================+====================+
| Turn off any scheduled imports to prevent syncs triggering part way through the upgrade.     |                    |
| Two options are available:                                                                   |                    |
|                                                                                              |                    |
| Individually for each job:                                                                   |                    |
|                                                                                              |                    |
| 1. Log in on the GUI as a high level administrator above Provider level.                     |                    |
| 2. Select the **Scheduling** menu to view scheduled jobs.                                    |                    |
| 3. Click each scheduled job. On the Base tab, uncheck the **Activate** check box.            |                    |
|                                                                                              |                    |
| Mass modify:                                                                                 |                    |
|                                                                                              |                    |
| 1. On the GUI, export scheduled syncs into a bulk load sheet.                                | .. raw:: html      |
| 2. Modify the schedule settings to de-activate scheduled syncs.                              |                    |
| 3. Import the sheet.                                                                         |    <div><input     |
|                                                                                              |    type='checkbox' |
|                                                                                              |    id='done'       |
|                                                                                              |    name='done'     |
|                                                                                              |    unchecked>      |
|                                                                                              |    <label          |
|                                                                                              |    for='isdone'>   |
|                                                                                              |    Done</label>    |
|                                                                                              |    </div>          |
|                                                                                              |                    |
| Schedules enabled on the CLI:  on the primary application node:                              |                    |
|                                                                                              |                    |
| 1. Check if any schedules exist. Run:                                                        |                    |
|                                                                                              |                    |
|    **cluster run all schedule list**                                                         |                    |
|                                                                                              |                    |
|    Record the IP addresses of nodes with schedules overlapping with the maintenance window.  |                    |
|                                                                                              |                    |
| 2. For overlapping schedules on a node, disable. Run:                                        |                    |
|                                                                                              |                    |
|    **cluster run <node IP> schedule disable <job-name>**                                     |                    |
|                                                                                              |                    |
+----------------------------------------------------------------------------------------------+--------------------+
| Check for running imports. Either wait for them to complete or cancel them:                  |                    |
|                                                                                              |                    |
| 1. Log in on the GUI as a high level administrator above Provider level.                     |                    |
| 2. Select the **Transaction** menu to view transactions.                                     |                    |
| 3. Filter the **Action** column:                                                             |                    |
|                                                                                              |                    |
|    a. Choose **Status** as "Processing" and then choose each **Action**                      |                    |
|       that starts with "Import", for example, "Import Unity Connection".                     |                    |
|    b. Click **Search** and confirm there are no results.                                     |                    |
|    c. If there are transactions to cancel, select them and click **Cancel**.                 |                    |
|                                                                                              |                    |
|                                                                                              | .. raw:: html      |
|                                                                                              |                    |
|                                                                                              |    <div><input     |
|                                                                                              |    type='checkbox' |
|                                                                                              |    id='done'       |
|                                                                                              |    name='done'     |
|                                                                                              |    unchecked>      |
|                                                                                              |    <label          |
|                                                                                              |    for='isdone'>   |
|                                                                                              |    Done</label>    |
|                                                                                              |    </div>          |
|                                                                                              |                    |
|                                                                                              |                    |
+----------------------------------------------------------------------------------------------+--------------------+
|                                                                                              |                    |
| **Customized ``data/Settings``**                                                             |                    |
|                                                                                              |                    |
| If ``data/Settings`` instances have been modified, record these or export them as JSON.      |                    |
|                                                                                              |                    |
| The modifications can be re-applied or exported JSON instances can be merged following       |                    |
| the upgrade. See: :ref:`Post-Template-Upgrade-Tasks`.                                        |                    |
|                                                                                              |                    |
| **Version**                                                                                  |                    |
|                                                                                              |                    |
| Record the current version information. This is required for upgrade troubleshooting.        |                    |
|                                                                                              |                    |
| * Log in on the GUI and record the information contained in the **About > Extended Version** | .. raw:: html      |
|                                                                                              |                    |
|                                                                                              |    <div><input     |
|                                                                                              |    type='checkbox' |
|                                                                                              |    id='done'       |
|                                                                                              |    name='done'     |
|                                                                                              |    unchecked>      |
|                                                                                              |    <label          |
|                                                                                              |    for='isdone'>   |
|                                                                                              |    Done</label>    |
|                                                                                              |    </div>          |
|                                                                                              |                    |
|                                                                                              |                    |
+----------------------------------------------------------------------------------------------+--------------------+


.. _modular-Pre-Upgrade-Security-Health-Steps:

Pre-Upgrade, Security and Health Steps 
...............................................



.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+------------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                    | Notes and Status   |
+==========================================================================================+====================+
| Have the IP address available of the node determined to be the *primary database node*.  |                    |
| Verify that the primary database node is the active primary node at the time of upgrade. |                    |
|                                                                                          |                    |
| On the primary application node, run:                                                    |                    |
|                                                                                          | .. raw:: html      |
| **cluster run <primary db IP> database config**                                          |                    |
|                                                                                          |    <div><input     |
| From the output, ensure that the primary database node ``stateStr``                      |    type='checkbox' |
| parameter is set to **PRIMARY** and it has the *highest* ``priority:<number>``           |    id='done'       |
| (highest priority number could vary depending on cluster layout).                        |    name='done'     |
|                                                                                          |    unchecked>      |
| Example output ::                                                                        |    <label          |
|                                                                                          |    for='isdone'>   |
|   <ip address>:27020:                                                                    |    Done</label>    |
|     priority: <number>                                                                   |    </div>          |
|     stateStr: PRIMARY                                                                    |                    |
|     storageEngine: WiredTiger                                                            |                    |
|                                                                                          |                    |
| Validate the system health on the primary application node:                              |                    |
|                                                                                          |                    |
| Verify cluster connectivity                                                              |                    |
|                                                                                          |                    |
| * **cluster run all cluster status**                                                     |                    |
|                                                                                          |                    |
| Verify network connectivity, disk status and NTP.                                        |                    |
|                                                                                          |                    |
|                                                                                          |                    |
| * **cluster check verbose**                                                              |                    |
|                                                                                          |                    |
| If there is any sign of the paths below are over 80% full, a clean-up is needed          |                    |
| to avoid risk of for example full logs occurring during upgrade.                         |                    |
| Clean-up steps are indicated next to the paths:                                          |                    |
|                                                                                          |                    |
| ::                                                                                       |                    |
|                                                                                          |                    |
|     /              (call support if over 80%)                                            |                    |
|     /var/log       (run: log purge)                                                      |                    |
|     /opt/platform  (remove any unnecessary files from /media directory)                  |                    |
|     /tmp           (reboot)                                                              |                    |
|                                                                                          |                    |
|                                                                                          |                    |
| On the primary application node, verify there are no pending Security Updates on         |                    |
| any of the nodes:                                                                        |                    |
|                                                                                          |                    |
|  * **cluster run all security check**                                                    |                    |
|                                                                                          |                    |
+------------------------------------------------------------------------------------------+--------------------+
|                                                                                          |                    |
| As part of the rollback procedure, ensure that                                           |                    |
| a suitable restore point is obtained prior to the start of the                           |                    |
| activity, as per the guidelines for the infrastructure on which                          |                    |
| the VOSS Automate platform is deployed.                                                  |                    |
|                                                                                          |                    |
| VOSS cannot guarantee that a restore point can be used to successfully                   | .. raw:: html      |
| restore VOSS Automate. If you cannot                                                     |                    |
| restore the application from a restore point, your only recourse is to reinstall         |    <div><input     |
| the application.                                                                         |    type='checkbox' |
|                                                                                          |    id='done'       |
|                                                                                          |    name='done'     |
|                                                                                          |    unchecked>      |
|                                                                                          |    <label          |
| Optional: If a backup is also required                                                   |    for='isdone'>   |
|                                                                                          |    Done</label>    |
| * Application node backup:                                                               |    </div>          |
|                                                                                          |                    |
|   On the primary application node, use the **backup add <location-name>** and            |                    |
|   **backup create <location-name>** commands. For details, refer to the Platform Guide.  |                    |
| * Database node backup:                                                                  |                    |
|                                                                                          |                    |
|   * Find the database node with the *second* highest weight.                             |                    |
|     On the primary application node, run                                                 |                    |
|     **cluster run database database weight list** to find this node.                     |                    |
|   * Log in on the database node with the *second* highest weight.                        |                    |
|   * Run **backup add <location-name>** and **backup create <location-name>** commands.   |                    |
|     For details, refer to the Platform Guide.                                            |                    |
+------------------------------------------------------------------------------------------+--------------------+

.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+-----------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                   | Notes and Status   |
+=========================================================================================+====================+
| Before upgrading, check all services, nodes and weights for the cluster on the          |                    |
| primary application node.                                                               |                    |
|                                                                                         |                    |
| Make sure no services are stopped/broken. The message 'suspended waiting for mongo'     |                    |
| is normal on the fresh database nodes.                                                  |                    |
|                                                                                         |                    |
|                                                                                         |                    |
| * **cluster run all app status**                                                        |                    |
|                                                                                         |                    |
|                                                                                         |                    |
| Make sure all application nodes show.                                                   |                    |
|                                                                                         |                    |
|                                                                                         |                    |
| * **cluster run application cluster list**                                              |                    |
|                                                                                         |                    |
|                                                                                         |                    |
| Check that the database weights are set. It is *critical* to ensure the weights are set |                    |
| before upgrading a cluster. The command is run from the primary application node        |                    |
| and is carried out on all database nodes                                                |                    |
|                                                                                         | .. raw:: html      |
| * **cluster run database database weight list**                                         |                    |
|                                                                                         |    <div><input     |
|                                                                                         |    type='checkbox' |
|                                                                                         |    id='done'       |
| Example output:                                                                         |    name='done'     |
|                                                                                         |    unchecked>      |
| ::                                                                                      |    <label          |
|                                                                                         |    for='isdone'>   |
|    ---------- VOSS-UN-2, ip=192.168.100.4, role=database, loc=cpt                       |    Done</label>    |
|                                                                                         |    </div>          |
|    192.168.100.4:                                                                       |                    |
|        weight: 70                                                                       |                    |
|    192.168.100.6:                                                                       |                    |
|        weight: 50                                                                       |                    |
|    192.168.100.8:                                                                       |                    |
|        weight: 30                                                                       |                    |
|                                                                                         |                    |
|    ---------- VOSS-UN-4, ip=192.168.100.6, role=database, loc=cpt                       |                    |
|                                                                                         |                    |
|    [...]                                                                                |                    |
|                                                                                         |                    |
| Verify the primary database node in the primary site and ensure no nodes are in the     |                    |
| 'recovering' state (``stateStr`` is not ``RECOVERING``). Run on the primary application |                    |
| node and use the primary database IP as parameter:                                      |                    |
|                                                                                         |                    |
| * **cluster run <primary db IP> database config**                                       |                    |
|                                                                                         |                    |
|                                                                                         |                    |
+-----------------------------------------------------------------------------------------+--------------------+


.. _modular-upgrade-Script:

Upgrade 
...........................


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+-------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                               | Notes and Status   |
+=====================================================================================+====================+
| On the primary application node:                                                    |                    |
|                                                                                     |                    |
|                                                                                     |                    |
| * **screen**                                                                        | .. raw:: html      |
|                                                                                     |                    |
| Run (optionally with command parameters below):                                     |                    |
|                                                                                     |                    |
| ``app install media/<script_file> delete-on-success yes --force``                   |    <div><input     |
|                                                                                     |    type='checkbox' |
| The upgrade will also silently run an updated version of **cluster check** and the  |    id='done'       |
| upgrade will fail to proceed if any error conditions exist. Fix before proceeding.  |    name='done'     |
| Refer to the Platform Guide for details on the new version of the **cluster check** |    unchecked>      |
| command.                                                                            |    <label          |
|                                                                                     |    for='isdone'>   |
| Run on the primary application node and use the primary database IP as parameter:   |    Done</label>    |
|                                                                                     |    </div>          |
| * **cluster run <primary db IP> database config**                                   |                    |
|                                                                                     |                    |
| and verify the number of nodes are *uneven*, i.e. either 5 or 7. If not, run:       |                    |
|                                                                                     |                    |
| **cluster run <primary db IP> cluster provision role database**                     |                    |
|                                                                                     |                    |
| Ensure an arbitrator shows when you run the command again:                          |                    |
|                                                                                     |                    |
| **cluster run <primary db IP> database config**                                     |                    |
|                                                                                     |                    |
| (output shows: ``stateStr: ARBITER``)                                               |                    |
|                                                                                     |                    |
| Note that during the upgrade, phone registration data is cleared. A message will    |                    |
| show in the log: ``Remove phone registration data``. This is required so that old   |                    |
| values are not displayed, since after the upgrade this information is no longer     |                    |
| stored in the resource cache.                                                       |                    |
+-------------------------------------------------------------------------------------+--------------------+





.. _modular-Post-Upgrade-Security-Health-Steps:

Post-Upgrade, Security and Health Steps 
................................................


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+--------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                | Notes and Status   |
+======================================================================================+====================+
| On the primary application node, verify the cluster status:                          |                    |
|                                                                                      |                    |
| * **cluster status**                                                                 | .. raw:: html      |
| * **cluster check**                                                                  |                    |
| * If any of the above commands show errors, check for further details to assist      |                    |
|   with troubleshooting:                                                              |                    |
|                                                                                      |    <div><input     |
|   **cluster run all diag health**                                                    |    type='checkbox' |
|                                                                                      |    id='done'       |
|                                                                                      |    name='done'     |
|                                                                                      |    unchecked>      |
|                                                                                      |    <label          |
|                                                                                      |    for='isdone'>   |
|                                                                                      |    Done</label>    |
|                                                                                      |    </div>          |
|                                                                                      |                    |
+--------------------------------------------------------------------------------------+--------------------+
| If upgrade is successful, the screen session can be closed by typing **exit** in the |                    |
| screen terminal. If errors occurred, keep the screen terminal                        |                    |
| open for troubleshooting purposes and contact VOSS support.                          | .. raw:: html      |
|                                                                                      |                    |
|                                                                                      |    <div><input     |
|                                                                                      |    type='checkbox' |
|                                                                                      |    id='done'       |
|                                                                                      |    name='done'     |
|                                                                                      |    unchecked>      |
|                                                                                      |    <label          |
|                                                                                      |    for='isdone'>   |
|                                                                                      |    Done</label>    |
|                                                                                      |    </div>          |
|                                                                                      |                    |
+--------------------------------------------------------------------------------------+--------------------+
| Check for needed security updates. On the primary application node, run:             |                    |
|                                                                                      |                    |
| * **cluster run all security check**                                                 |                    |
|                                                                                      |                    |
| If one  or more updates are required for any node, run on the primary application    |                    |
| node:                                                                                |                    |
|                                                                                      |                    |
|                                                                                      | .. raw:: html      |
| * **cluster run all security update**                                                |                    |
|                                                                                      |    <div><input     |
|                                                                                      |    type='checkbox' |
|                                                                                      |    id='done'       |
|                                                                                      |    name='done'     |
| Note: *if the system reboots, do not carry out the next manual reboot step*.         |    unchecked>      |
|                                                                                      |    <label          |
| Manual reboot *only if needed*:                                                      |    for='isdone'>   |
|                                                                                      |    Done</label>    |
|                                                                                      |    </div>          |
| * **cluster run notme system reboot**                                                |                    |
|                                                                                      |                    |
|                                                                                      |                    |
| If node messages: ``<node name> failed with timeout`` are displayed,                 |                    |
| these can be ignored.                                                                |                    |
|                                                                                      |                    |
|                                                                                      |                    |
| * **system reboot**                                                                  |                    |
|                                                                                      |                    |
|                                                                                      |                    |
| Since all services will be stopped, this takes some time.                            |                    |
|                                                                                      |                    |
|                                                                                      |                    |
+--------------------------------------------------------------------------------------+--------------------+

.. _modular-Post-Template-Upgrade-Tasks:


Post Template Upgrade Tasks 
....................................

.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+-----------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                   | Notes and Status   |
+=========================================================================================+====================+
|                                                                                         |                    |
| **SSO Login URL check if needed**                                                       |                    |
|                                                                                         |                    |
| Verify the SSO Login URL if needed. Go to **Single Sign On > SSO Identity Provider**    |                    |
| and ensure your URL matches the **SSO Login URL** value.                                |                    |
|                                                                                         |                    |
|                                                                                         |                    |
| **Customized ``data/Settings``**                                                        |                    |
|                                                                                         |                    |
| For releases prior to 21.1, merge the previously backed up customized ``data/Settings`` |                    |
| with the latest settings on the system by manually adding the differences or exporting  |                    |
| the latest settings to JSON, merging the customized changes and importing the JSON.     |                    |
|                                                                                         |                    |
|                                                                                         |                    |
| **Support for VG400 and VG450 Analogue Gateways**                                       |                    |
|                                                                                         |                    |
| Before adding the VG400 or VG450 Gateway, the ``device/cucm/GatewayType``               |                    |
| model needs to be imported for each Unified CM.                                         |                    |
|                                                                                         |                    |
| 1. Create a Model Type List which includes the ``device/cucm/GatewayType`` model.       |                    |
| 2. Add the Model Type List to all the required Unified CM Data Syncs.                   |                    |
| 3. Execute the Data Sync for all the required Unified CMs.                              |                    |
|                                                                                         |                    |
| **Verify the upgrade**                                                                  | .. raw:: html      |
|                                                                                         |                    |
| Log in on the GUI and check the information contained in the **About > Version**        |                    |
| menu. Confirm that versions have upgraded:                                              |    <div><input     |
|                                                                                         |    type='checkbox' |
| * **Release** should show ``XXX``                                                       |    id='done'       |
|                                                                                         |    name='done'     |
|                                                                                         |    unchecked>      |
| where ``XXX`` corresponds with the release number of the upgrade.                       |    <label          |
|                                                                                         |    for='isdone'>   |
|                                                                                         |    Done</label>    |
| If your web browser cannot open the user interface, clear your browser cache before     |    </div>          |
| trying to open the interface again.                                                     |                    |
|                                                                                         |                    |
+-----------------------------------------------------------------------------------------+--------------------+
|                                                                                         | .. raw:: html      |
|                                                                                         |                    |
|                                                                                         |    <div><input     |
| * For configurations that make use of the Northbound Billing Integration (NBI),         |    type='checkbox' |
|   please check the service status of NBI and restart if necessary.                      |    id='done'       |
|                                                                                         |    name='done'     |
|                                                                                         |    unchecked>      |
|                                                                                         |    <label          |
|                                                                                         |    for='isdone'>   |
|                                                                                         |    Done</label>    |
|                                                                                         |    </div>          |
|                                                                                         |                    |
+-----------------------------------------------------------------------------------------+--------------------+


.. _modular-Restore-Adaptations:

Restore Adaptations 
............................

.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                              | Notes and Status   |
+====================================================================================+====================+
| Restore and adaptations prior to upgrade.                                          |                    |
|                                                                                    |                    |
| If the release is accompanied by Upgrade Notes, refer to the details on adaptation | .. raw:: html      |
| impact.                                                                            |                    |
|                                                                                    |    <div><input     |
|                                                                                    |    type='checkbox' |
|                                                                                    |    id='done'       |
|                                                                                    |    name='done'     |
|                                                                                    |    unchecked>      |
|                                                                                    |    <label          |
|                                                                                    |    for='isdone'>   |
|                                                                                    |    Done</label>    |
|                                                                                    |    </div>          |
|                                                                                    |                    |
+------------------------------------------------------------------------------------+--------------------+

.. _modular-Restore_Schedules:

Restore Schedules 
..........................


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+---------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                           | Notes and Status   |
+=================================================================================+====================+
| Re-enable scheduled imports if any were disabled prior to the upgrade.          |                    |
| Two options are available:                                                      |                    |
|                                                                                 |                    |
| Individually for each job:                                                      |                    |
|                                                                                 |                    |
| 1. Log in on the GUI as a high level administrator above Provider level.        |                    |
| 2. Select the **Scheduling** menu to view scheduled jobs.                       |                    |
| 3. Click each scheduled job. On the Base tab, check the **Activate** check box. |                    |
|                                                                                 |                    |
| Mass modify:                                                                    |                    |
|                                                                                 |                    |
| 1. Modify the exported sheet of schedules to activate scheduled syncs.          |                    |
| 2. Import the bulk load sheet.                                                  | .. raw:: html      |
|                                                                                 |                    |
| .. note::                                                                       |                    |
|                                                                                 |                    |
|    Select the **Skip next execution**  if you do not wish to execute schedules  |                    |
|    overlapping the maintenance window, but only execute thereafter.             |                    |
|                                                                                 |    <div><input     |
| Schedules enabled on the CLI. On the primary application node:                  |    type='checkbox' |
|                                                                                 |    id='done'       |
| 1. For disabled schedules that were overlapping the maintenance window, enable. |    name='done'     |
|                                                                                 |    unchecked>      |
|    Verify disabled schedules: **cluster run all schedule list**.                |    <label          |
|                                                                                 |    for='isdone'>   |
|    Record the IP addresses of nodes and scheduled jobs disabled.                |    Done</label>    |
| 2. Enable schedules:                                                            |    </div>          |
|                                                                                 |                    |
|    **cluster run <node IP> schedule enable <job-name>**                         |                    |
|                                                                                 |                    |
+---------------------------------------------------------------------------------+--------------------+



.. _delta-modular-upgrade-release-specific-updates:


Release Specific Updates 
................................................

.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+-----------------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                         | Notes and Status   |
+===============================================================================================+====================+
|                                                                                               |                    |
|  When upgrading to release 21.3, users of Microsoft apps should after upgrade, select each    | .. raw:: html      |
|  Microsoft Tenant (``relation/MicrosoftTenant``) in the Admin GUI and click **Save** on it    |                    |
|  without making any changes.                                                                  |    <div><input     |
|                                                                                               |    type='checkbox' |
|  This step is required so that VOSS Automate can communicate with the Tenant post upgrade.    |    id='done'       |
|                                                                                               |    name='done'     |
|                                                                                               |    unchecked>      |
|                                                                                               |    <label          |
|                                                                                               |    for='isdone'>   |
|                                                                                               |    Done</label>    |
|                                                                                               |    </div>          |
+-----------------------------------------------------------------------------------------------+--------------------+







.. _modular-Log-Files-Error-Checks:


Log Files and Error Checks 
....................................

.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+----------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                            | Notes and Status   |
+==================================================================================+====================+
| Inspect the output of the command line interface for upgrade errors.             | .. raw:: html      |
| On the primary application node,                                                 |                    |
| use the **log view** command to view any log files indicated in the error        |                    |
| messages, for example, run the command if the following message appears:         |    <div><input     |
|                                                                                  |    type='checkbox' |
| ::                                                                               |    id='done'       |
|                                                                                  |    name='done'     |
|    For more information refer to the execution log file with                     |    unchecked>      |
|    'log view platform/execute.log'                                               |    <label          |
|                                                                                  |    for='isdone'>   |
| For example, if it is required send all the install log files in the ``install`` |    Done</label>    |
| directory to an SFTP server:                                                     |    </div>          |
|                                                                                  |                    |
| * **log send sftp://x.x.x.x install**                                            |                    |
|                                                                                  |                    |
+----------------------------------------------------------------------------------+--------------------+
|                                                                                  | .. raw:: html      |
| Log in on the GUI as system level administrator, go to                           |                    |
| **Administration Tools > Transaction** and inspect the transactions              |    <div><input     |
| list for errors.                                                                 |    type='checkbox' |
|                                                                                  |    id='done'       |
|                                                                                  |    name='done'     |
|                                                                                  |    unchecked>      |
|                                                                                  |    <label          |
|                                                                                  |    for='isdone'>   |
|                                                                                  |    Done</label>    |
|                                                                                  |    </div>          |
|                                                                                  |                    |
+----------------------------------------------------------------------------------+--------------------+