.. _insights-analytics:

Insights Analytics
-------------------

.. _24.1|VOSS-1187:
.. _24.2|EKB-21112:
.. _24.2|EKB-22574:
.. _24.2|EKB-22058:
.. _25.1|EKB-24311:
.. _25.3|VOSS-1559:
.. _25.4|VOSS-1490:


.. index:: tmux
.. index:: insights sync;insights sync schedule time;insights sync schedule list;insights sync run;insights sync enable;insights sync disable


Applications (insights-api, insights-voss-sync) 
..................................................

Two applications are available for managing data between Automate and Insights, *insights-api* and 
*insights-voss-sync*: 

.. note:: 

   To display these applications, run ``app status``.

* **insights-api**

  * The API service allowing communication between Automate and Insights
  * Manages the Insights Analytics used for VOSS Wingman and dashboard widgets in the Automate GUI
  * The communication is associated with a specific Automate session for authentication so that 
    session timeout and logout will also terminate this specific communication


* **insights-voss-sync**

  * The sync application
  * Resides on the database node
  * Manages data syncs between the Automate and Insights databases
  * Comprises the following sync components and processes:

    * ``real-time``, ``database`` 

      Components that ensure initial creation of database tables and indexes required for the sync 
      between Automate and Insights databases and changes to the *Resource* collection.

      The ``real-time`` app runs and completes after the creation of database tables and indexes to 
      ensure the accurate representation of data in widgets on Automate GUI dashboards. 
      See :ref:`synced-automate-models`.


    * ``full-sync``, ``full-transaction``, ``multivendor``, ``delta-sync``, ``sync-check``, ``sync-check-arbitrator``

      Sync processes that enable additional control and include default schedules 
      for data syncs between the Automate and Insights databases.



.. note:: 

   The Insights database is installed on each database node in the VOSS system topology. The 
   ``insights-voss-sync`` resides on the database. The Automate API resides on the application node.



.. rubric:: Related topics

* 
  .. raw:: latex

     Wingman in the Core Feature Guide

  .. raw:: html
  
     <a href="../voss-wingman.html">Wingman</a> 

* 
  .. raw:: latex

     Introduction to Automate dashboards in the Core Feature Guide

  .. raw:: html
  
     <a href="../automate-dashboards-overview.html">Introduction to Automate dashboards</a> 


Sync types
...........

full-sync
''''''''''

A ``full-sync`` is a ``delta-sync`` with full refresh: 

* Disabled by default. 
* Can be run manually. See :ref:`manual-sync`.
* Executes the equivalent of the following command and parameter to fetch all data from the 
  beginning of time (Jan 1, 1970): ``delta-sync --full-refresh``
* Concludes with a ``multivendor`` sync, refreshing multi vendor database tables 
* No output displays in the ``tmux`` session while this sync is in progress.
* Pushes data to Arbitrator, if you have an Arbitrator resource configured

  :: 

      $ insights sync run full-sync
      Syncing can be a long process. Do you wish to continue?yes



full-transaction
''''''''''''''''''

The ``full-transaction`` sync is a *transaction data* sync: 

* Runs hourly, every day (default)
* Syncs transaction data since previous run of this sync. 
* The first sync defaults to the last 90 days or 10 million transactions. 
* Only completed transactions are synced
* Excludes queued and processing transactions 
* A cleanup script runs daily to ensure the number of stored transactions remain under 10 million.
* Excludes syncing of system and data sync transactions

  ::

      $ insights sync run full-transaction
      Syncing can be a long process. Do you wish to continue?yes



Since the ``delta-sync`` excludes transaction data you'll need to run a ``full-transaction`` sync 
manually or use the Platform system scheduling feature. 

To sync transaction data, perform *either* of the following: 
   
* Manually run the following command: ``insights sync run full-transaction``
* Use the Platform system scheduling feature. See: :ref:`cli-scheduling`.





.. note:: 

   Upon upgrade to Automate 25.1 from an earlier version, the default ``full-transaction`` sync schedule 
   is *hourly*.


multivendor sync
''''''''''''''''''''

The ``multivendor`` sync: 

* Syncs multi vendor subscriber details
* Runs hourly, every day (default)
* Syncs multi vendor subscriber details for the multi vendor overview dashboard



delta-sync 
''''''''''''

The ``delta-sync``: 

* Runs every 30 minutes (default)
* Only available as a manual sync using the following command: ``insights sync run``

  See :ref:`manual-sync`.

* Initially it carries out a full sync and thereafter, an incremental resource sync. 
* Allows for syncing of specific Automate objects defined by the ``data/ReporterModels`` instance, 
  which maps Automate and Insights databases. 
* Fetches changed data incrementally (data changed since previous fetch)
* Deletes stale data 
* Excludes ``multivendor`` sync
* Can also be run using the following parameter to execute a full sync of Insights data: ``--full-refresh``
* Pushes data to Arbitrator, if you have an Arbitrator resource configured



sync-check
''''''''''''''

The ``sync-check``: 

* Runs once a day (configurable)
* Performs a check to ensure that the Automate and Insights databases have the same data (it runs a count). 
* Updates any models that differ (after performing the comparison)

.. note:: 

   For reporter resources (``data/ReporterResource``) configured as *standalone* - Microsoft Teams Room 
   Calendar (``device/msgraph/CalendarView``) and Webex Workspace Metrics (``device/spark/WorkspaceMetrics``) 
   models - ``sync-check`` is an *append-only* time-series ingestion that reads from dedicated Mongo 
   collections. If recent events are not visible on dashboards, run a manual ``sync-check`` or adjust the 
   schedule. For more information about standalone reporter resources, see 
   *Introduction to Automate dashboards*.

.. rubric:: Related topics

*
  .. raw:: latex

     Introduction to Meeting Rooms Management

  .. raw:: html

     <a href="../intro-meeting-rooms-management.html">Introduction to Meeting Rooms Management</a> 




sync-check-arbitrator
''''''''''''''''''''''''

The ``sync-check-arbitrator``: 

* Extends the checks performed with ``sync-check``
* Runs once a day
* Inspects available data sources and populates the Arbitrator database with equivalent data 
  (inserts) for the following Automate models (defaults):

  * ``device/msgraph/MsolUser`` (pushes MS 0365 user details (including hierarchies) to call records on Arbitrator)
  * ``device/msgraph/Room``
  * ``device/spark/Place``
  * ``device/spark/User``

  If any of these models change, data is pushed to Arbitrator.

  .. note::

     The data source has an encrypted password, so ``insights-voss-sync`` decrypts the password via 
     the Automate API as part of the workflow.


.. rubric:: Pre-requisites

The following needs to be in place before running ``sync-check-arbitrator``: 
    
* Release 25.3 and up, Arbitrator and Automate
* The Automate password set up via Arbitrator's CLI menu (*Change Automate Public Database Password*). 
  This password must match the Dashboard data source on Automate.
* Run a ``full-sync`` before running ``sync-check-arbitrator``
* A Microsoft tenant on Automate that has MSOL users, and an Arbitrator (username = *automate*, and 
  password matching the password set up for Automate on Arbitrator)

Provided all requirements are met, you can now run ``sync-check-arbitrator``: 

:: 

   insights sync run sync-check-arbitrator

Or sync only specific models, any of the default models:

:: 

   insights sync run sync-check-arbitrator --models "device/msgraph/MsolUser,device/spark/Place"


.. rubric:: Related topics

* 
  .. raw:: html
     
     See <a href="../concepts-voss-insights-arbitrators.html">Push hierarchy details to Arbitrator</a>
     
  .. raw:: latex
     
     See *Push hierarchy details to Arbitrator* in the Core Feature Guide.



arbquery
............

The ``arbquery`` process is a standalone, scheduled ingestion process responsible for retrieving 
aggregated, multi-customer (partitioned) analytics data from configured Arbitrator databases 
and imports it into the local Insights database. It is designed to query a recent time window 
(for example, the last 24 hours). This gives the local platform access to 
customer-specific, aggregated usage (call/meeting) metrics that are only available on 
Arbitrators (e.g., Cisco/Webex/Teams call analytics).

.. note:: 

   ``sync-check-arbitrator`` pushes data to Arbitrators; ``arbquery`` pulls aggregated data from 
   Arbitrators.


The ``arbquery`` process: 

* Connects to each configured Arbitrator 
* Executes a predefined SQL aggregation query on each Arbitrator to query a time window 
  (default: last 24 hours, configurable) and retrieves the result
* Before import, deletes overlapping existing rows for the same time window and the same Arbitrator to 
  avoid duplication 
* Imports fresh aggregated data
* Feeds the data to dashboards that rely on Arbitrator-side metrics (Webex Room/Workspace analytics, 
  Teams Room meeting analytics, cross-vendor room metrics) 

.. note:: 

   For best results, ensure regular ``delta-sync`` / ``sync-check`` jobs are healthy and that 
   ``sync-check-arbitrator`` has aligned schemas on the Arbitrators.







Log files for insights-voss-sync
...................................

Log files for ``insights-voss-sync`` components can be inspected and monitored. For example:

::

   insights-voss-sync/real-time.log
   insights-voss-sync/full-transaction.log
   insights-voss-sync/full-sync.log
   insights-voss-sync/delta-sync.log
   insights-voss-sync/multivendor.log
   insights-voss-sync/database.log


Since these log files can grow to be large files, they are by default rotated on an hourly basis 
to conserve disk space.



Monitoring sync progress
''''''''''''''''''''''''''

Syncs initially take some time. Inspect related log files to monitor sync progress: 
   
:: 

   $ log follow process/insights-voss-sync.full-sync.log
   $ log follow process/insights-voss-sync.delta-sync.log
   $ log follow process/insights-voss-sync.full-transaction.log
   $ log follow process/insights-voss-sync.multivendor.log







.. rubric:: Related topics 

* :ref:`platform-logs` (how to view logs)



Sync and service management commands
....................................


Scheduled syncs
''''''''''''''''

Default schedules are predefined (see output below), but you can also schedule Insights data syncs manually. 

.. note:: 

   If the initial management of dashboards on the GUI is required prior to the first scheduled sync 
   you can run the following syncs manually: ``full-sync``, ``full-transaction``, ``multivendor``, 
   ``delta-sync``

   See :ref:`manual-sync`. 


* Schedule Insights data syncs manually and specify daily and weekly schedules: ``insights sync schedule time``
* View default syncs and their *disabled* statuses: 

  ::

     platform@VOSS:~$ insights sync schedule list
         schedule: 
             delta-sync: 
                 hour: every
                 minute: every/30
                 weekday: every
             full-sync: 
                 disabled: true
                 hour: 1
                 minute: 0
                 weekday: 6
             full-transaction: 
                 hour: every/1
                 minute: 0
                 weekday: 0
             multivendor: 
                 hour: every
                 minute: 0
                 weekday: every

* Inspect the current state of schedules: ``schedule list`` 

.. note:: 

   Since standalone reporter resources ingest during ``sync-check``, consider increasing the ``sync-check`` 
   frequency if your dashboards require fresher time-series data. You can also run ``insights sync run sync check`` 
   manually as needed. For more information on standalone reporter resources, see *Introduction to Automate dashboards*. 



.. rubric:: Schedule Commands

The schedule command takes three options: sync type name, time, weekday

Sync type (``{sync-type}``) can be:

* ``full-sync``
* ``full-transaction``
* ``multivendor``


Time and weekday are set up separately. 

Time can be an interval, using ``every`` or ``every/{n}``:

* ``every``

  :: 
     
     insights sync schedule time {sync-type} every <{n} hours|{n} minutes>

* ``every/{n}``

  :: 

     insights sync schedule time {sync-type} <every/<{n} hours|every/{n} minutes>

  .. note:: 
    
     With ``{n}``:

     * hours: 0 - 23
     * minutes: 0 - 59  



.. rubric:: Set the schedule for a specific time: 

::
  
   insights sync schedule time {sync-type} <{hour} {minute}>

Example (for a 2:15 AM sync on the weekday of the setup):
 
:: 
  
   insights sync schedule time {sync-type} 02 15



.. rubric:: Set a weekly schedule:

:: 

   insights sync schedule time {sync-type} weekly {weekday}


Examples:

* Change the time setup to be a weekly Friday sync:  ``insights sync schedule time {sync-type} weekly 5``
* Change the time setup to be a weekly every day:    ``insights sync schedule time {sync-type} weekly every``


Value convention for ``{weekday}``:

* ``0`` is Sunday
* ``1`` is Monday
* ``2`` is Tuesday
* ``3`` is Wednesday
* ``4`` is Thursday
* ``5`` is Friday
* ``6`` is Saturday
* ``every`` is every day



.. rubric:: Disable/enable scheduled Insights syncs

To enable or disable a scheduled Insights sync, run the following command: 

:: 
    
    insights sync schedule <enable|disable> {sync-type}



.. _manual-sync:

Manual syncs
''''''''''''

Scheduled syncs and the ``real-time`` sync keeps dashboard widget data in sync.

You can run a sync manually at any time to ensure that the Automate database and 
Insights reporter database are in sync and that dashboards reflect current data.

.. note::

   * A ``full sync`` can be a long process, so if manually executing, ensure you're doing this 
     at a suitable time. 
     
     Sync durations depend on the amount of data in the database and, for 
     ``full-transaction`` syncs, the amount of activity against that data since the last sync, 
     or 90 days / 10 million transactions. A daily cleanup script ensures that number of stored 
     transactions remain under 10 million.

   * If a sync is in progress, another sync of the same type won't run. 
   * Run manual sync commands in a terminal opened with the ``tmux`` command.
          
     .. raw:: html
     
        For details, see <a href="../platform/reference-tmux-command.html">Using the tmux command</a>
     
     .. raw:: latex
     
        For details, see *Using the tmux command* in the Platform Guide.




.. rubric:: Manually run different types of insights syncs
 
Use the following command with options for sync type: ``insights sync run``

For example: ``insights sync run {sync-type}``

Sync type ``{sync-type}`` can be:

* ``delta-sync``
* ``full-sync``
* ``full-transaction``
* ``multivendor``
* ``sync check``
* ``sync-check-arbitrator``

.. note:: 

   If upgrading to release 24.2, an initial manual sync is not required for dashboard management since 
   the ``delta-sync`` runs every 30 minutes and will then execute this first sync.




.. _synced-automate-models:

Synced Automate models
........................

Several Automate database models are synced from the Insights database (the table lists examples). These models 
also serve as query sources for VOSS Wingman in the Automate GUI and by some Automate dashboard 
widgets.

See the topic on *Wingman* and *Dashboards* in the Core Feature Guide.

When any of these models are updated, the ``delta-sync`` sync process creates the matching INSERT, 
UPDATE or DELETE database operation in the Insights database so that any widgets on the 
Automate GUI dashboards that reference corresponding data, remains synced.

.. note:: 

   Reporter resources configured as *standalone* are time-series models (``device/msgraph/CalendarView``, 
   ``device/spark/WorkspaceMetrics``). Their data is collected into dedicated Mongo collections and 
   appended to *collect* database tables during ``sync-check``. These models are not reconciled via 
   ``delta-sync`` and follow a 13-month retention policy with monthly partitions. For more 
   information on standalone reporter resources, see *Introduction to Automate dashboards*. 


.. tabularcolumns:: |p{6cm}|p{4cm}|p{5cm}|

.. csv-table:: Insights Sync Table
   :file: insights-synced-data.csv
   :class: longtable
   :header-rows: 1
   :widths: 6 4 5



.. rubric:: Related topics

* 
  .. raw:: latex

     Wingman in the Core Feature Guide

  .. raw:: html
  
     <a href="../voss-wingman.html">Wingman</a> 

* 
  .. raw:: latex

     Introduction to Automate dashboards in the Core Feature Guide

  .. raw:: html
  
     <a href="../automate-dashboards-overview.html">Introduction to Automate dashboards</a>