.. _upgrade_multinode_ISO:

.. rst-class:: chapter-with-expand

Unified Node Topology: Upgrade a Multinode Environment with the ISO and Template
-----------------------------------------------------------------------------------

.. index:: cluster;cluster check
.. index:: cluster;cluster upgrade
.. index:: screen
.. index:: database;database convert_drive
.. index:: app; app cleanup

.. _19.1.1|EKB-1249:
.. _19.1.2|EKB-2262:
.. _19.2.1|EKB-2636:
.. _19.2.1|VOSS-497|EKB-3126:
.. _20.1.1|VOSS-661:    
.. _20.1.1|VOSS-661|EKB-4836:    
.. _20.1.1|EKB-5552:    
.. _20.1.1|VOSS-724|EKB-4923:
.. _20.1.1|EKB-6669:    
.. _21.4|VOSS-872:   
.. _24.1|VOSS-1187:
    
    
.. important::    

  * Before upgrading to release 24.1, ensure that an additional disk is available
    for the Insights database.

    .. raw:: html 

       See: <a href="../platform/tasks-backup-restore-new-add-more-space-large-restore.html">Adding Hard Disk Space</a>
       and <a href="automate-system-hardware-specification.html">VOSS Automate Hardware Specifications</a>.

    .. raw:: latex

       See the Adding Hard Disk Space topic in the Platform Guide and VOSS Automate Hardware Specifications in the Architecture and Hardware Specification Guide.

    This disk is needed to assign to the ``insights-voss-sync:database`` mount point.
    See: :ref:`Post-maintenance-mount-insights-disk`.

  * Before upgrading to release 24.1, ensure that sufficient time is allocated to the maintenance
    window. This may vary in accordance with your topology, number of devices and subscribers.

    The information below serves as a guideline VOSS support can be contacted if further
    guidance is required:

    * Cluster upgrade: 4h
    * Template install: 2.5h
    * For a 500K Data User system (13Mil RESOURCE documents), the expected ``upgrade_db`` step is about 12h.
    * For a 160K Data User system (2.5Mil RESOURCE documents), the expected ``upgrade_db`` step is about 2.5h.
    
    You can follow the progress on the Admin Portal transaction list.

  * Tasks that are marked **Prior to Maintenance Window** can be completed a few days prior to
    the scheduled maintenance window so that VOSS support can be contacted if needed and
    in order to allow for reduce down time.
 

The standard **screen** command should be used where indicated.
See: :ref:`screen-command`.



.. _Prior-to-maintenance-window-Download-Files-and-Check-ISO:

Download Files and Check (Prior to Maintenance Window) 
.......................................................


.. note::

   Ensure that the ``.iso`` file is available on *all* nodes.


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+--------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                | Notes and Status   |
+======================================================================================+====================+
| VOSS files:                                                                          |                    |
|                                                                                      |                    |
| **https://voss.portalshape.com > Downloads > VOSS Automate > XXX > Upgrade**         |                    |
|                                                                                      |                    |
| Download ``.iso`` and ``.template`` files, where XXX matches the release.            |                    |
|                                                                                      |                    |
| * Transfer the ``.iso`` file to the ``media/`` folder of all nodes.                  |                    |
| * Transfer the ``.template`` file to the ``media/`` folder of the primary node.      |                    |
|                                                                                      |                    |
| Two transfer options:                                                                |                    |
|                                                                                      | .. raw:: html      |
| Either using SFTP:                                                                   |                    |
|                                                                                      |                    |
|                                                                                      |    <div><input     |
| * **sftp platform@<unified_node_hostname>**                                          |    type='checkbox' |
|                                                                                      |    id='done'       |
| * **cd media**                                                                       |    name='done'     |
|                                                                                      |    unchecked>      |
| * **put <upgrade_iso_file>**                                                         |    <label          |
|                                                                                      |    for='isdone'>   |
| * **put <upgrade_template_file>**                                                    |    Done</label>    |
|                                                                                      |    </div>          |
|                                                                                      |                    |
|                                                                                      |                    |
| Or using SCP:                                                                        |                    |
|                                                                                      |                    |
|                                                                                      |                    |
| * **scp <upgrade_iso_file> platform@<unified_node_ip_address>:~/media**              |                    |
|                                                                                      |                    |
| * **scp <upgrade_template_file> platform@<unified_node_ip_address>:~/media**         |                    |
|                                                                                      |                    |
|                                                                                      |                    |
| Verify that the ``.iso`` image and ``.template`` file copied:                        |                    |
|                                                                                      |                    |
|                                                                                      |                    |
| * **ls -l media/**                                                                   |                    |
|                                                                                      |                    |
|                                                                                      |                    |
| Verify that the original ``.sha256`` checksums on the                                |                    |
| Download site server match.                                                          |                    |
|                                                                                      |                    |
|                                                                                      |                    |
| * **system checksum media/<upgrade_iso_file>**                                       |                    |
|                                                                                      |                    |
|   ``Checksum: <SHA256>``                                                             |                    |
|                                                                                      |                    |
| * **system checksum media/<upgrade_template_file>**                                  |                    |
|                                                                                      |                    |
|   ``Checksum: <SHA256>``                                                             |                    |
|                                                                                      |                    |
|                                                                                      |                    |
+--------------------------------------------------------------------------------------+--------------------+


.. _Prior-to-maintenance-window-Security-Health-Steps:

Security and Health Check Steps (Prior to Maintenance Window)
.............................................................


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+--------------------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                            | Notes and Status   |
+==================================================================================================+====================+
| Verify that the primary node is the active primary node at the time of upgrade.                  |                    |
|                                                                                                  |                    |
| **database config**                                                                              | .. raw:: html      |
|                                                                                                  |                    |
| Ensure that the node on which the installation will be initiated has the ``stateStr``            |    <div><input     |
| parameter set to **PRIMARY** and has the **highest** ``priority`` **number**                     |    type='checkbox' |
| (highest priority number could vary depending on cluster layout).                                |    id='done'       |
|                                                                                                  |    name='done'     |
| Example output ::                                                                                |    unchecked>      |
|                                                                                                  |    <label          |
|   <ip address>:27020:                                                                            |    for='isdone'>   |
|     priority: <number>                                                                           |    Done</label>    |
|     stateStr: PRIMARY                                                                            |    </div>          |
|     storageEngine: WiredTiger                                                                    |                    |
|                                                                                                  |                    |
| Validate the system health. Carry out the following:                                             |                    |
|                                                                                                  |                    |
| * **system mount** - mount upgrade ISO.                                                          |                    |
| * **app install check_cluster** - install the new version of the **cluster check**               |                    |
|   command.                                                                                       |                    |
|                                                                                                  |                    |
|   .. raw:: latex                                                                                 |                    |
|                                                                                                  |                    |
|      For details, refer to the "Cluster Check" topic in the Platform Guide.                      |                    |
|                                                                                                  |                    |
|   .. raw:: html                                                                                  |                    |
|                                                                                                  |                    |
|      For details, refer to <a href="../platform/reference-cluster-check.html">Cluster Check</a>. |                    |
|                                                                                                  |                    |
| * **cluster check** - inspect the output of this command for warnings and errors. You            |                    |
|   can also use **cluster check verbose** to see more details, for example, ``avx`` enabled.      |                    |
|   While warnings will not prevent an upgrade, it is advisable that these be resolved prior to    |                    |
|   upgrading where possible. Some warnings may be resolved by upgrading.                          |                    |
|                                                                                                  |                    |
|   For troubleshooting and resolutions,                                                           |                    |
|   also refer to the *Health Checks for Cluster Installations Guide* and *Platform Guide*.        |                    |
|                                                                                                  |                    |
|   If there is any sign of the paths below are over 80% full, a clean-up is needed,               |                    |
|   for example to avoid risk of full logs occurring during upgrade.                               |                    |
|   Clean-up steps are indicated next to the paths:                                                |                    |
|                                                                                                  |                    |
|   ::                                                                                             |                    |
|                                                                                                  |                    |
|       /              (call support if over 80%)                                                  |                    |
|       /var/log       (run: log purge)                                                            |                    |
|       /opt/platform  (remove any unnecessary files from /media directory)                        |                    |
|       /tmp           (reboot)                                                                    |                    |
|                                                                                                  |                    |
|                                                                                                  |                    |
|   On the Primary Unified Node, verify there are no pending Security Updates on any of the        |                    |
|   nodes.                                                                                         |                    |
|                                                                                                  |                    |
| .. note::                                                                                        |                    |
|                                                                                                  |                    |
|    If you run **cluster status** after installing the new version of **cluster check**, any      |                    |
|    error message regarding a failed command can be ignored. This error message will not show     |                    |
|    after upgrade.                                                                                |                    |
|                                                                                                  |                    |
+--------------------------------------------------------------------------------------------------+--------------------+

.. _maintenance-window-Schedules-Transactions-Version-Check:

Version Check (Maintenance Window) 
..................................................................


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+-------------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                     | Notes and Status   |
+===========================================================================================+====================+
|                                                                                           |                    |
| **Customized ``data/Settings``**                                                          |                    |
|                                                                                           |                    |
| If ``data/Settings`` instances have been modified, record these or export them as JSON.   |                    |
|                                                                                           |                    |
| The modifications can be re-applied or exported JSON instances can be merged following    |                    |
| the upgrade. See: :ref:`maintenance-window-post-template-upgrade-tasks`.                  |                    |
|                                                                                           |                    |
|                                                                                           |                    |
| **Version**                                                                               |                    |
|                                                                                           |                    |
| Record the current version information. This is required for upgrade troubleshooting.     |                    |
|                                                                                           |                    |
| * Log in on the Admin Portal and record the information contained in the menu:            | .. raw:: html      |
|   **About > Version**                                                                     |                    |
|                                                                                           |    <div><input     |
|                                                                                           |    type='checkbox' |
|                                                                                           |    id='done'       |
|                                                                                           |    name='done'     |
|                                                                                           |    unchecked>      |
|                                                                                           |    <label          |
|                                                                                           |    for='isdone'>   |
|                                                                                           |    Done</label>    |
|                                                                                           |    </div>          |
|                                                                                           |                    |
|                                                                                           |                    |
+-------------------------------------------------------------------------------------------+--------------------+

.. _maintenance-window-Pre-Upgrade-Steps:

Pre-Upgrade Steps (Maintenance Window)  
...............................................


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+-----------------------------------------------------------------------------------------+--------------------+
| As part of the rollback procedure, ensure that                                          |                    |
| a suitable restore point is obtained prior to the start of the                          |                    |
| activity, as per the guidelines for the infrastructure on which                         |                    |
| the VOSS Automate platform is deployed.                                                 |                    |
|                                                                                         |                    |
|                                                                                         | .. raw:: html      |
|                                                                                         |                    |
|                                                                                         |    <div><input     |
|                                                                                         |    type='checkbox' |
|                                                                                         |    id='done'       |
|                                                                                         |    name='done'     |
|                                                                                         |    unchecked>      |
|                                                                                         |    <label          |
|                                                                                         |    for='isdone'>   |
|                                                                                         |    Done</label>    |
|                                                                                         |    </div>          |
|                                                                                         |                    |
| Optional: If a backup is also required, use the                                         |                    |
| **backup add <location-name>** and                                                      |                    |
| **backup create <location-name>** commands. For details, refer to the *Platform Guide*. |                    |
|                                                                                         |                    |
+-----------------------------------------------------------------------------------------+--------------------+

   
.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+---------------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                       | Notes and Status   |
+=============================================================================================+====================+
| After restore point creation and before upgrading: validate system health and check all     |                    |
| services, nodes and weights for the cluster:                                                |                    |
|                                                                                             |                    |
| * **cluster run application cluster list**                                                  |                    |
|                                                                                             |                    |
|   Make sure all application nodes show 4 or 6 nodes.                                        |                    |
|                                                                                             |                    |
|                                                                                             |                    |
|                                                                                             |                    |
|                                                                                             |                    |
|                                                                                             | .. raw:: html      |
|                                                                                             |                    |
|                                                                                             |    <div><input     |
| * **cluster check** - inspect the output of this command, for warnings and errors. You      |    type='checkbox' |
|   can also use **cluster check verbose** to see more details.                               |    id='done'       |
|                                                                                             |    name='done'     |
|   * Make sure no services are stopped/broken. The message 'suspended waiting for mongo'     |    unchecked>      |
|     is normal on the fresh unified nodes.                                                   |    <label          |
|                                                                                             |    for='isdone'>   |
|   * Check that the database weights are set. It is *critical* to ensure the weights are set |    Done</label>    |
|     before upgrading a cluster.                                                             |    </div>          |
|     Example output:                                                                         |                    |
|                                                                                             |                    |
|     ::                                                                                      |                    |
|                                                                                             |                    |
|        172.29.21.240:                                                                       |                    |
|            weight: 80                                                                       |                    |
|        172.29.21.241:                                                                       |                    |
|            weight: 70                                                                       |                    |
|        172.29.21.243:                                                                       |                    |
|            weight: 60                                                                       |                    |
|        172.29.21.244:                                                                       |                    |
|            weight: 50                                                                       |                    |
|                                                                                             |                    |
|   * Verify the primary node in the primary site and ensure no nodes are in the              |                    |
|     'recovering' state (``stateStr`` is not ``RECOVERING``). On the primary node:           |                    |
|                                                                                             |                    |
+---------------------------------------------------------------------------------------------+--------------------+


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+----------------------------------------------------------------------------------------+--------------------+
| The following step is needed if own private certificate and generated SAN certificates |                    |
| are required and the ``web cert gen_csr`` command was run.                             |                    |
| For details, refer to the Web Certificate Setup Options topic in the Platform Guide.   |                    |
|                                                                                        |                    |
| The steps below are needed to check if a CSR private key exists but no associated      |                    |
| signed certificate is available.                                                       | .. raw:: html      |
|                                                                                        |                    |
| Request VOSS support to run on the CLI as ``root`` user, the following command:        |    <div><input     |
|                                                                                        |    type='checkbox' |
| ::                                                                                     |    id='done'       |
|                                                                                        |    name='done'     |
|    for LST in /opt/platform/apps/nginx/config/csr/*;                                   |    unchecked>      |
|    do openssl x509 -in $LST -text -noout >/dev/null                                    |    <label          |
|    2>&1 && SIGNED="$LST"; done                                                         |    for='isdone'>   |
|                                                                                        |    Done</label>    |
|    echo $SIGNED                                                                        |    </div>          |
|                                                                                        |                    |
|                                                                                        |                    |
| If the ``echo $SIGNED`` command output is blank, back up the ``csr/`` directory with   |                    |
| for example the following command:                                                     |                    |
|                                                                                        |                    |
| ::                                                                                     |                    |
|                                                                                        |                    |
|    mv /opt/platform/apps/nginx/config/csr/ /opt/platform/apps/nginx/config/csrbackup   |                    |
|                                                                                        |                    |
+----------------------------------------------------------------------------------------+--------------------+

.. _maintenance-window-upgrade-ISO:

Upgrade (Maintenance Window)  
............................


.. note::


   * By default, the cluster upgrade is carried out in parallel on all nodes and without any
     backup in order to provide a fast upgrade.
   * For systems *upgrading to 24.1 from 21.4.0 - 21.4-PB5*:
     
     * The VOSS platform maintenance mode will be started automatically
       when the **cluster upgrade** command is run. This prevents any new occurrences of scheduled transactions, including the
       24.1 database syncs associated with **insights sync**. For details on **insights sync**, see the *Insights Analytics* topic
       in the Platform Guide.
     * The **cluster maintenance-mode stop** command must however be run manually
       after the maintenance window of the upgrade: :ref:`maintenance-window-manual-stop`.

   For details on the VOSS platform maintenance mode, see the *Maintenance Mode* topic in the Platform Guide.
  


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+--------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                | Notes and Status   |
+======================================================================================+====================+
| It is recommended that the upgrade steps                                             |                    |
| are run in a terminal opened with the **screen** command.                            |                    |
|                                                                                      |                    |
| Verify that the ISO has been uploaded to the ``media/`` directory on each node. This |                    |
| will speed up the upgrade time.                                                      |                    |
|                                                                                      |                    |
| On the primary unified node:                                                         |                    |
|                                                                                      |                    |
|                                                                                      |                    |
| * **screen**                                                                         | .. raw:: html      |
|                                                                                      |                    |
|                                                                                      |                    |
| * **cluster upgrade media/<upgrade_iso_file>**                                       |    <div><input     |
|                                                                                      |    type='checkbox' |
| Note: If the system reboots, do not carry out the next manual reboot step.           |    id='done'       |
|                                                                                      |    name='done'     |
|                                                                                      |    unchecked>      |
| Manual reboot *only if needed*:                                                      |    <label          |
|                                                                                      |    for='isdone'>   |
| * **cluster run notme system reboot**                                                |    Done</label>    |
|                                                                                      |    </div>          |
| If node messages: ``<node name> failed with timeout`` are displayed,                 |                    |
| these can be ignored.                                                                |                    |
|                                                                                      |                    |
| * **system reboot**                                                                  |                    |
|                                                                                      |                    |
| Since all services will be stopped, this takes some time.                            |                    |
|                                                                                      |                    |
| Close **screen**: ``Ctrl-a \``                                                       |                    |
|                                                                                      |                    |
+--------------------------------------------------------------------------------------+--------------------+

All unused docker images except ``selfservice`` and ``voss_ubuntu`` images
will be removed from the system at this stage.


.. _maintenance-window-Post-Upgrade-Security-Health-Steps:

Post-Upgrade, Security and Health Steps  (Maintenance Window)
.............................................................


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+--------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                | Notes and Status   |
+======================================================================================+====================+
| On each unified node, assign the ``insights-voss-sync:database`` mount point to the  |                    |
| drive added for the Insights database prior to upgrade.                              |                    |
|                                                                                      | .. raw:: html      |
| For example, if ``drives list`` shows the added disk as:                             |                    |
|                                                                                      |    <div><input     |
| ::                                                                                   |    type='checkbox' |
|                                                                                      |    id='done'       |
|   Unused disks:                                                                      |    name='done'     |
|   sde                                                                                |    unchecked>      |
|                                                                                      |    <label          |
| then run the command                                                                 |    for='isdone'>   |
|                                                                                      |    Done</label>    |
| ``drives add sde insights-voss-sync:database``                                       |    </div>          |
|                                                                                      |                    |
| on each unified node where the drive has been added.                                 |                    |
|                                                                                      |                    |
| Output is typically (the message below can be ignored on release 24.1:               |                    |
|                                                                                      |                    |
| ``WARNING: Failed to connect to lvmetad. Falling back to device scanning.``          |                    |
|                                                                                      |                    |
| )                                                                                    |                    |
|                                                                                      |                    |
|                                                                                      |                    |
|                                                                                      |                    |
|                                                                                      |                    |
|                                                                                      |                    |
+--------------------------------------------------------------------------------------+--------------------+
| On the primary unified node, verify the cluster status:                              |                    |
|                                                                                      |                    |
| * **cluster check**                                                                  | .. raw:: html      |
|                                                                                      |                    |
| * If any of the above commands show errors, check for further details to assist      |    <div><input     |
|   with troubleshooting:                                                              |    type='checkbox' |
|                                                                                      |    id='done'       |
|   **cluster run all diag health**                                                    |    name='done'     |
|                                                                                      |    unchecked>      |
|                                                                                      |    <label          |
|                                                                                      |    for='isdone'>   |
|                                                                                      |    Done</label>    |
|                                                                                      |    </div>          |
|                                                                                      |                    |
+--------------------------------------------------------------------------------------+--------------------+
| Check for needed security updates. On the primary node, run:                         |                    |
|                                                                                      |                    |
| * **cluster run all security check**                                                 |                    |
|                                                                                      |                    |
| If one  or more updates are required for any node, run on the primary Unified node:  |                    |
|                                                                                      |                    |
|                                                                                      |                    |
|                                                                                      | .. raw:: html      |
| * **cluster run all security update**                                                |                    |
|                                                                                      |    <div><input     |
|                                                                                      |    type='checkbox' |
|                                                                                      |    id='done'       |
|                                                                                      |    name='done'     |
| Note: *if the system reboots, do not carry out the next manual reboot step*.         |    unchecked>      |
|                                                                                      |    <label          |
| Manual reboot *only if needed*:                                                      |    for='isdone'>   |
|                                                                                      |    Done</label>    |
|                                                                                      |    </div>          |
| * **cluster run notme system reboot**                                                |                    |
|                                                                                      |                    |
|                                                                                      |                    |
| If node messages: ``<node name> failed with timeout`` are displayed,                 |                    |
| these can be ignored.                                                                |                    |
|                                                                                      |                    |
|                                                                                      |                    |
| * **system reboot**                                                                  |                    |
|                                                                                      |                    |
|                                                                                      |                    |
| Since all services will be stopped, this takes some time.                            |                    |
|                                                                                      |                    |
+--------------------------------------------------------------------------------------+--------------------+
| To remove a mount directory ``media/<iso_file basename>`` on nodes that may          |                    |
| have remained after for example an upgrade, run:                                     |                    |
|                                                                                      | .. raw:: html      |
| **cluster run all app cleanup**                                                      |                    |
|                                                                                      |    <div><input     |
|                                                                                      |    type='checkbox' |
|                                                                                      |    id='done'       |
|                                                                                      |    name='done'     |
|                                                                                      |    unchecked>      |
|                                                                                      |    <label          |
|                                                                                      |    for='isdone'>   |
|                                                                                      |    Done</label>    |
|                                                                                      |    </div>          |
|                                                                                      |                    |
+--------------------------------------------------------------------------------------+--------------------+
| If upgrade is successful, the screen session can be closed by typing **exit** in the |                    |
| screen terminal. If errors occurred, keep the screen terminal                        |                    |
| open for troubleshooting purposes and contact VOSS support.                          | .. raw:: html      |
|                                                                                      |                    |
|                                                                                      |    <div><input     |
|                                                                                      |    type='checkbox' |
|                                                                                      |    id='done'       |
|                                                                                      |    name='done'     |
|                                                                                      |    unchecked>      |
|                                                                                      |    <label          |
|                                                                                      |    for='isdone'>   |
|                                                                                      |    Done</label>    |
|                                                                                      |    </div>          |
|                                                                                      |                    |
+--------------------------------------------------------------------------------------+--------------------+

.. _maintenance-window-Database-Schema-Upgrade:


Database Schema Upgrade (Maintenance Window) 
.............................................



.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+---------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                 | Notes and Status   |
+=======================================================================================+====================+
| It is recommended that the upgrade steps                                              |                    |
| are run in a terminal opened with the **screen** command.                             |                    |
|                                                                                       |                    |
| On the primary unified node:                                                          |                    |
|                                                                                       | .. raw:: html      |
| * **screen**                                                                          |                    |
|                                                                                       |    <div><input     |
| * **voss upgrade_db**                                                                 |    type='checkbox' |
|                                                                                       |    id='done'       |
|                                                                                       |    name='done'     |
| Check cluster status                                                                  |    unchecked>      |
|                                                                                       |    <label          |
| * **cluster check**                                                                   |    for='isdone'>   |
|                                                                                       |    Done</label>    |
|                                                                                       |    </div>          |
|                                                                                       |                    |
+---------------------------------------------------------------------------------------+--------------------+


.. _maintenance-window-Template-Upgrade-ISO:


Template Upgrade (Maintenance Window) 
.....................................



.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+---------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                 | Notes and Status   |
+=======================================================================================+====================+
| It is recommended that the upgrade steps                                              |                    |
| are run in a terminal opened with the **screen** command.                             |                    |
|                                                                                       |                    |
| On the primary unified node:                                                          |                    |
|                                                                                       | .. raw:: html      |
| * **screen**                                                                          |                    |
|                                                                                       |    <div><input     |
| * **app template media/<VOSS Automate.template>**                                     |    type='checkbox' |
|                                                                                       |    id='done'       |
|                                                                                       |    name='done'     |
|                                                                                       |    unchecked>      |
|                                                                                       |    <label          |
|                                                                                       |    for='isdone'>   |
|                                                                                       |    Done</label>    |
|                                                                                       |    </div>          |
|                                                                                       |                    |
+---------------------------------------------------------------------------------------+--------------------+



The following message appears:                                                                   
                                                                                                 
::                                                                                               
                                                                                                 
   Running the DB-query to find the current environment's 
   existing solution deployment config... 
                                                                                                 
* Python functions are deployed                                                                  
* System artifacts are imported.                                                                 

  .. note::

     In order to carry out fewer upgrade steps, the updates of instances
     of some models are skipped in the cases where:
     
     * ``data/CallManager`` instance does not exist as instance in ``data/NetworkDeviceList``
     * ``data/CallManager`` instance exists, but ``data/NetworkDeviceList`` is empty
     * Call Manager AXL Generic Driver and Call Manager Control Center Services
       match the ``data/CallManager`` IP

The template upgrade automatically detects the deployment mode: "Enterprise" or "Provider"

A message displays according to the selected deployment type.
Check for one of the messages below:                                    
                                                                                                 
::                                                                                               
                                                                                                 
   Importing EnterpriseOverlay.json
                                                                                                                                                                                         
   Importing ProviderOverlay.json ...
                                                                                                 
The template install automatically restarts necessary applications. If a cluster is detected,    
the installation propagates changes throughout the cluster.                                      


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+------------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                    | Notes and Status   |
+==========================================================================================+====================+
|                                                                                          |                    |
| Review the output from the **app template** command and confirm that the upgrade message | .. raw:: html      |
| appears:                                                                                 |                    |
|                                                                                          |    <div><input     |
|                                                                                          |    type='checkbox' |
|                                                                                          |    id='done'       |
|                                                                                          |    name='done'     |
|                                                                                          |    unchecked>      |
|                                                                                          |    <label          |
|                                                                                          |    for='isdone'>   |
|                                                                                          |    Done</label>    |
|                                                                                          |    </div>          |
|                                                                                          |                    |
|                                                                                          |                    |
| ::                                                                                       |                    |
|                                                                                          |                    |
|    Deployment summary of PREVIOUS template solution                                      |                    |
|    (i.e. BEFORE upgrade):                                                                |                    |
|    -------------------------------------------------                                     |                    |
|                                                                                          |                    |
|                                                                                          |                    |
|    Product: [PRODUCT]                                                                    |                    |
|    Version: [PREVIOUS PRODUCT RELEASE]                                                   |                    |
|    Iteration-version: [PREVIOUS ITERATION]                                               |                    |
|    Platform-version: [PREVIOUS PLATFORM VERSION]                                         |                    |
|                                                                                          |                    |
|                                                                                          |                    |
| This is followed by updated product and version details:                                 |                    |
|                                                                                          |                    |
| ::                                                                                       |                    |
|                                                                                          |                    |
|     Deployment summary of UPDATED template solution                                      |                    |
|     (i.e. current values after installation):                                            |                    |
|     -----------------------------------------------                                      |                    |
|                                                                                          |                    |
|     Product: [PRODUCT]                                                                   |                    |
|     Version: [UPDATED PRODUCT RELEASE]                                                   |                    |
|     Iteration-version: [UPDATED ITERATION]                                               |                    |
|     Platform-version: [UPDATED PLATFORM VERSION]                                         |                    |
|                                                                                          |                    |
+------------------------------------------------------------------------------------------+--------------------+
                                                                                                
                                                                                               


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+-----------------------------------------------------------------------------+--------------------+
| Description and Steps                                                       | Notes and Status   |
+=============================================================================+====================+
|                                                                             |                    |
|                                                                             | .. raw:: html      |
|                                                                             |                    |
| * If no errors are indicated, create a restore point.                       |    <div><input     |
|                                                                             |    type='checkbox' |
|   This restore point can be used if post-upgrade patches                    |    id='done'       |
|   that may be required, fail.                                               |    name='done'     |
|                                                                             |    unchecked>      |
|                                                                             |    <label          |
|                                                                             |    for='isdone'>   |
|                                                                             |    Done</label>    |
|                                                                             |    </div>          |
|                                                                             |                    |
|   As part of the rollback procedure, ensure that                            |                    |
|   a suitable restore point is obtained prior to the start of the            |                    |
|   activity, as per the guidelines for the infrastructure on which           |                    |
|   the VOSS Automate platform is deployed.                                   |                    |
+-----------------------------------------------------------------------------+--------------------+
|                                                                             |                    |
|                                                                             | .. raw:: html      |
|                                                                             |                    |
| For an unsupported upgrade path, the install script stops with the message: |    <div><input     |
|                                                                             |    type='checkbox' |
|                                                                             |    id='done'       |
| ::                                                                          |    name='done'     |
|                                                                             |    unchecked>      |
|                                                                             |    <label          |
|     Upgrade failed due to unsupported upgrade path.                         |    for='isdone'>   |
|     Please log in as sysadmin                                               |    Done</label>    |
|     and see Transaction logs for more detail.                               |    </div>          |
|                                                                             |                    |
| You can roll back as per the guidelines for the infrastructure on which     |                    |
| the VOSS Automate platform is deployed.                                     |                    |
|                                                                             |                    |
+-----------------------------------------------------------------------------+--------------------+
|                                                                             |                    |
|                                                                             | .. raw:: html      |
| If there are errors for another reason, the install script stops with a     |                    |
| failure message listing the problem. Contact VOSS support.                  |    <div><input     |
|                                                                             |    type='checkbox' |
|                                                                             |    id='done'       |
|                                                                             |    name='done'     |
|                                                                             |    unchecked>      |
|                                                                             |    <label          |
|                                                                             |    for='isdone'>   |
|                                                                             |    Done</label>    |
|                                                                             |    </div>          |
|                                                                             |                    |
+-----------------------------------------------------------------------------+--------------------+
|                                                                             |                    |
|                                                                             | .. raw:: html      |
| Verify the ``extra_functions`` have the *same checksum* across the cluster. |                    |
|                                                                             |    <div><input     |
| * **cluster run application voss get_extra_functions_version -c**           |    type='checkbox' |
|                                                                             |    id='done'       |
|                                                                             |    name='done'     |
|                                                                             |    unchecked>      |
|                                                                             |    <label          |
|                                                                             |    for='isdone'>   |
|                                                                             |    Done</label>    |
|                                                                             |    </div>          |
|                                                                             |                    |
+-----------------------------------------------------------------------------+--------------------+
| Post upgrade migrations:                                                    |                    |
|                                                                             |                    |
| On a single node of a cluster, run:                                         | .. raw:: html      |
|                                                                             |                    |
| * **voss post-upgrade-migrations**                                          |    <div><input     |
|                                                                             |    type='checkbox' |
|                                                                             |    id='done'       |
|                                                                             |    name='done'     |
|                                                                             |    unchecked>      |
|                                                                             |    <label          |
|                                                                             |    for='isdone'>   |
|                                                                             |    Done</label>    |
|                                                                             |    </div>          |
|                                                                             |                    |
+-----------------------------------------------------------------------------+--------------------+


Data migrations that are not critical to system operation can have significant
execution time at scale. These need to be performed after the primary upgrade,
allowing the migration to proceed whilst the system is in use - thereby limiting
upgrade windows.

A transaction is queued on VOSS Automate and its progress is displayed as it executes.




.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+---------------------------------+--------------------+
| Description and Steps           | Notes and Status   |
+=================================+====================+
|                                 |                    |
| Check cluster status and health | .. raw:: html      |
|                                 |                    |
| * **cluster status**            |    <div><input     |
|                                 |    type='checkbox' |
|                                 |    id='done'       |
|                                 |    name='done'     |
|                                 |    unchecked>      |
|                                 |    <label          |
|                                 |    for='isdone'>   |
|                                 |    Done</label>    |
|                                 |    </div>          |
|                                 |                    |
+---------------------------------+--------------------+


.. _maintenance-window-post-template-upgrade-tasks:


Post Template Upgrade Tasks (Maintenance Window) 
................................................


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+--------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                | Notes and Status   |
+======================================================================================+====================+
|                                                                                      |                    |
| **Verify the upgrade**                                                               | .. raw:: html      |
|                                                                                      |                    |
| Log in on the Admin Portal and check the information contained in the                |                    |
| **About > Version** menu. Confirm that versions have upgraded.                       |    <div><input     |
|                                                                                      |    type='checkbox' |
| * **Release** should show ``XXX``                                                    |    id='done'       |
| * **Platform Version** should show ``XXX``                                           |    name='done'     |
|                                                                                      |    unchecked>      |
| where ``XXX`` corresponds with the release number of the upgrade.                    |    <label          |
|                                                                                      |    for='isdone'>   |
|                                                                                      |    Done</label>    |
|                                                                                      |    </div>          |
|                                                                                      |                    |
+--------------------------------------------------------------------------------------+--------------------+
|                                                                                      | .. raw:: html      |
|                                                                                      |                    |
|                                                                                      |    <div><input     |
| * Check themes on all roles are set correctly                                        |    type='checkbox' |
|                                                                                      |    id='done'       |
|                                                                                      |    name='done'     |
|                                                                                      |    unchecked>      |
|                                                                                      |    <label          |
|                                                                                      |    for='isdone'>   |
|                                                                                      |    Done</label>    |
|                                                                                      |    </div>          |
|                                                                                      |                    |
+--------------------------------------------------------------------------------------+--------------------+
|                                                                                      | .. raw:: html      |
|                                                                                      |                    |
|                                                                                      |    <div><input     |
| * For configurations that make use of the Northbound Billing Integration (NBI),      |    type='checkbox' |
|   please check the service status of NBI and restart if necessary.                   |    id='done'       |
|                                                                                      |    name='done'     |
|                                                                                      |    unchecked>      |
|                                                                                      |    <label          |
|                                                                                      |    for='isdone'>   |
|                                                                                      |    Done</label>    |
|                                                                                      |    </div>          |
|                                                                                      |                    |
+--------------------------------------------------------------------------------------+--------------------+



.. _maintenance-window-Log-Files-Error-Checks:


Log Files and Error Checks (Maintenance Window) 
................................................

.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+----------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                            | Notes and Status   |
+==================================================================================+====================+
| Inspect the output of the command line interface for upgrade errors,             | .. raw:: html      |
| for example ``File import failed!`` or ``Failed to execute command``.            |                    |
|                                                                                  |                    |
| Use the **log view** command to view any log files indicated in the error        |                    |
| messages, for example, run the command if the following message appears:         |    <div><input     |
|                                                                                  |    type='checkbox' |
| ::                                                                               |    id='done'       |
|                                                                                  |    name='done'     |
|    For more information refer to the execution log file with                     |    unchecked>      |
|    'log view platform/execute.log'                                               |    <label          |
|                                                                                  |    for='isdone'>   |
| For example, if it is required send all the install log files in the ``install`` |    Done</label>    |
| directory to an SFTP server:                                                     |    </div>          |
|                                                                                  |                    |
| * **log send sftp://x.x.x.x install**                                            |                    |
|                                                                                  |                    |
+----------------------------------------------------------------------------------+--------------------+
|                                                                                  | .. raw:: html      |
| Log in on the Admin Portal as system level administrator, go to                  |                    |
| **Administration Tools > Transaction** and inspect the transactions              |    <div><input     |
| list for errors.                                                                 |    type='checkbox' |
|                                                                                  |    id='done'       |
|                                                                                  |    name='done'     |
|                                                                                  |    unchecked>      |
|                                                                                  |    <label          |
|                                                                                  |    for='isdone'>   |
|                                                                                  |    Done</label>    |
|                                                                                  |    </div>          |
|                                                                                  |                    |
+----------------------------------------------------------------------------------+--------------------+



.. _maintenance-window-manual-stop:

Manually Stop the Maintenance Window 
.................................................................


.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+----------------------------------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                                          | Notes and Status   |
+================================================================================================================+====================+
| On the CLI:                                                                                                    |                    |
|                                                                                                                |                    |
| Run the **cluster maintenance-mode stop** command to end the automatic start of the VOSS maintenance mode when | .. raw:: html      |
| the **cluster upgrade** command was run when upgrading to 24.1 from 21.4.0 - 21.4-PB5.                         |                    |
|                                                                                                                |    <div><input     |
|                                                                                                                |    type='checkbox' |
| This will allow synced transactions to resume, including **insights sync** operations added in 24.1.           |    id='done'       |
|                                                                                                                |    name='done'     |
| For details on the VOSS platform maintenance mode, see the *Maintenance Mode* topic in the Platform Guide.     |    unchecked>      |
|                                                                                                                |    <label          |
|                                                                                                                |    for='isdone'>   |
|                                                                                                                |    Done</label>    |
|                                                                                                                |    </div>          |
|                                                                                                                |                    |
+----------------------------------------------------------------------------------------------------------------+--------------------+




.. _Post-maintenance-window-Licensing:

Licensing (outside, after Maintenance Window) 
.................................................................

.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+-------------------------------------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                                             | Notes and Status   |
+===================================================================================================================+====================+
| From release 21.4 onwards, the deployment needs to be licensed.                                                   | .. raw:: html      |
| After installation, a 7-day grace period is available to license the product.                                     |                    |
| Since license processing is only scheduled every hour, if you wish to license immediately,                        |    <div><input     |
| first run **voss check-license** from the primary unified node CLI.                                               |    type='checkbox' |
|                                                                                                                   |    id='done'       |
| 1. Obtain the required license token from VOSS.                                                                   |    name='done'     |
| 2. Steps for GUI and CLI:                                                                                         |    unchecked>      |
|                                                                                                                   |    <label          |
|    a. To license through the GUI, follow steps indicated in Product License Management in the Core Feature Guide. |    for='isdone'>   |
|    b. To license through the CLI, follow steps indicated in Product Licensing in the Platform Guide.              |    Done</label>    |
|                                                                                                                   |    </div>          |
|                                                                                                                   |                    |
+-------------------------------------------------------------------------------------------------------------------+--------------------+


.. _Post-maintenance-mount-insights-disk:

Mount the Insights disk (outside, after Maintenance Window)
.............................................................

.. tabularcolumns:: |p{13.5cm}|p{4cm}|

+---------------------------------------------------------------------------------------+--------------------+
| Description and Steps                                                                 | Notes and Status   |
+=======================================================================================+====================+
| *On each unified node*, assign the ``insights-voss-sync:database`` mount point to the |                    |
| drive added for the Insights database prior to upgrade.                               |                    |
|                                                                                       | .. raw:: html      |
| For example, if ``drives list`` shows the added disk as:                              |                    |
|                                                                                       |    <div><input     |
| ::                                                                                    |    type='checkbox' |
|                                                                                       |    id='done'       |
|    Unused disks:                                                                      |    name='done'     |
|    sde                                                                                |    unchecked>      |
|                                                                                       |    <label          |
| then run the command                                                                  |    for='isdone'>   |
|                                                                                       |    Done</label>    |
| ``drives add sde insights-voss-sync:database``                                        |    </div>          |
|                                                                                       |                    |
| on each unified node where the drive has been added.                                  |                    |
|                                                                                       |                    |
| Sample output (the message below can be ignored on release 24.1:                      |                    |
|                                                                                       |                    |
|  ``WARNING: Failed to connect to lvmetad. Falling back to device scanning.``)         |                    |
|                                                                                       |                    |
| ::                                                                                    |                    |
|                                                                                       |                    |
|    $ drives add sde insights-voss-sync:database                                       |                    |
|    Configuration setting "devices/scan_lvs" unknown.                                  |                    |
|    Configuration setting "devices/allow_mixed_block_sizes" unknown.                   |                    |
|    WARNING: Failed to connect to lvmetad. Falling back to device scanning.            |                    |
|    71ad98e0-7622-49ad-9fg9-db04055e82bc                                               |                    |
|    Application insights-voss-sync processes stopped.                                  |                    |
|    Migrating data to new drive - this can take several minutes                        |                    |
|    Data migration complete - reassigning drive                                        |                    |
|    Checking that /dev/sde1 is mounted                                                 |                    |
|    Checking that /dev/dm-0 is mounted                                                 |                    |
|    /opt/platform/apps/mongodb/dbroot                                                  |                    |
|    Checking that /dev/sdc1 is mounted                                                 |                    |
|    /backups                                                                           |                    |
|                                                                                       |                    |
|    Application services:firewall processes stopped.                                   |                    |
|    Reconfiguring applications...                                                      |                    |
|    Application insights-voss-sync processes started.                                  |                    |
|                                                                                       |                    |
+---------------------------------------------------------------------------------------+--------------------+