Insights Analytics

Applications (insights-api, insights-voss-sync)

Two applications are available for managing data between Automate and Insights, insights-api and insights-voss-sync:

Note

To display these applications, run app status.

• insights-api

  • The API service allowing communication between Automate and Insights

  • Manages the Insights Analytics used for VOSS Wingman and dashboard widgets in the Automate GUI

  • The communication is associated with a specific Automate session for authentication so that session timeout and logout will also terminate this specific communication

• insights-voss-sync

  • The sync application

  • Resides on the database node

  • Manages data syncs between the Automate and Insights databases

  • Comprises the following sync components and processes:

    • real-time, database

      Components that ensure initial creation of database tables and indexes required for the sync between Automate and Insights databases and changes to the Resource collection.

      The real-time app runs and completes after the creation of database tables and indexes to ensure the accurate representation of data in widgets on Automate GUI dashboards. See Synced Automate models.

    • full-sync, full-transaction, multivendor, delta-sync, sync-check, sync-check-arbitrator

      Sync processes that enable additional control and include default schedules for data syncs between the Automate and Insights databases.

Note

The Insights database is installed on each database node in the VOSS system topology. The insights-voss-sync resides on the database. The Automate API resides on the application node.

Related topics

• Wingman
• Introduction to Automate dashboards

Sync types

full-sync

A full-sync is a delta-sync with full refresh:

• Disabled by default.

• Can be run manually. See Manual syncs.

• Executes the equivalent of the following command and parameter to fetch all data from the beginning of time (Jan 1, 1970): delta-sync --full-refresh

• Concludes with a multivendor sync, refreshing multi vendor database tables

• No output displays in the tmux session while this sync is in progress.

• Pushes data to Arbitrator, if you have an Arbitrator resource configured

  $ insights sync run full-sync
  Syncing can be a long process. Do you wish to continue?yes
  

full-transaction

The full-transaction sync is a transaction data sync:

• Runs hourly, every day (default)

• Syncs transaction data since previous run of this sync.

• The first sync defaults to the last 90 days or 10 million transactions.

• Only completed transactions are synced

• Excludes queued and processing transactions

• A cleanup script runs daily to ensure the number of stored transactions remain under 10 million.

• Excludes syncing of system and data sync transactions

  $ insights sync run full-transaction
  Syncing can be a long process. Do you wish to continue?yes
  

Since the delta-sync excludes transaction data you’ll need to run a full-transaction sync manually or use the Platform system scheduling feature.

To sync transaction data, perform either of the following:

• Manually run the following command: insights sync run full-transaction

• Use the Platform system scheduling feature. See: Scheduling.

Note

Upon upgrade to Automate 25.1 from an earlier version, the default full-transaction sync schedule is hourly.

multivendor sync

The multivendor sync:

• Syncs multi vendor subscriber details

• Runs hourly, every day (default)

• Syncs multi vendor subscriber details for the multi vendor overview dashboard

delta-sync

The delta-sync:

• Runs every 30 minutes (default)

• Only available as a manual sync using the following command: insights sync run

  See Manual syncs.

• Initially it carries out a full sync and thereafter, an incremental resource sync.

• Allows for syncing of specific Automate objects defined by the data/ReporterModels instance, which maps Automate and Insights databases.

• Fetches changed data incrementally (data changed since previous fetch)

• Deletes stale data

• Excludes multivendor sync

• Can also be run using the following parameter to execute a full sync of Insights data: --full-refresh

• Pushes data to Arbitrator, if you have an Arbitrator resource configured

sync-check

The sync-check:

• Runs once a day

• Performs a check to ensure that the Automate and Insights databases have the same data (it runs a count).

• Updates any models that differ (after performing the comparison)

sync-check-arbitrator

The sync-check-arbitrator:

• Extends the checks performed with sync-check

• Runs once a day

• Inspects available data sources and populates the Arbitrator database with equivalent data (inserts) for the following Automate models (defaults):

  • device/msgraph/MsolUser (pushes MS 0365 user details (including hierarchies) to call records on Arbitrator)

  • device/msgraph/Room

  • device/spark/Place

  • device/spark/User

  If any of these models change, data is pushed to Arbitrator.

  Note

  The data source has an encrypted password, so insights-voss-sync decrypts the password via the Automate API as part of the workflow.

Pre-requisites

The following needs to be in place before running sync-check-arbitrator:

• Release 25.3 and up, Arbitrator and Automate

• The Automate password set up via Arbitrator’s CLI menu (Change Automate Public Database Password). This password must match the Dashboard data source on Automate.

• Run a full-sync before running sync-check-arbitrator

• A Microsoft tenant on Automate that has MSOL users, and an Arbitrator (username = automate, and password matching the password set up for Automate on Arbitrator)

Provided all requirements are met, you can now run sync-check-arbitrator:

insights sync run sync-check-arbitrator

Or sync only specific models, any of the default models:

insights sync run sync-check-arbitrator --models "device/msgraph/MsolUser,device/spark/Place"

Related topics

• See Push hierarchy details to Arbitrator

Log files for insights-voss-sync

Log files for insights-voss-sync components can be inspected and monitored. For example:

insights-voss-sync/real-time.log
insights-voss-sync/full-transaction.log
insights-voss-sync/full-sync.log
insights-voss-sync/delta-sync.log
insights-voss-sync/multivendor.log
insights-voss-sync/database.log

Since these log files can grow to be large files, they are by default rotated on an hourly basis to conserve disk space.

Monitoring sync progress

Syncs initially take some time. Inspect related log files to monitor sync progress:

$ log follow process/insights-voss-sync.full-sync.log
$ log follow process/insights-voss-sync.delta-sync.log
$ log follow process/insights-voss-sync.full-transaction.log
$ log follow process/insights-voss-sync.multivendor.log

Related topics

• Logs (how to view logs)

Sync and service management commands

Scheduled syncs

Default schedules are predefined (see output below), but you can also schedule Insights data syncs manually.

Note

If the initial management of dashboards on the GUI is required prior to the first scheduled sync you can run the following syncs manually: full-sync, full-transaction, multivendor, delta-sync

See Manual syncs.

• Schedule Insights data syncs manually and specify daily and weekly schedules: insights sync schedule time

• View default syncs and their disabled statuses:

  platform@VOSS:~$ insights sync schedule list
      schedule:
          delta-sync:
              hour: every
              minute: every/30
              weekday: every
          full-sync:
              disabled: true
              hour: 1
              minute: 0
              weekday: 6
          full-transaction:
              hour: every/1
              minute: 0
              weekday: 0
          multivendor:
              hour: every
              minute: 0
              weekday: every
  

• Inspect the current state of schedules: schedule list

Schedule Commands

The schedule command takes three options: sync type name, time, weekday

Sync type ({sync-type}) can be:

• full-sync

• full-transaction

• multivendor

Time and weekday are set up separately.

Time can be an interval, using every or every/{n}:

• every

  insights sync schedule time {sync-type} every <{n} hours|{n} minutes>
  

• every/{n}

  insights sync schedule time {sync-type} <every/<{n} hours|every/{n} minutes>
  

  Note

  With {n}:

  • hours: 0 - 23

  • minutes: 0 - 59

Set the schedule for a specific time:

insights sync schedule time {sync-type} <{hour} {minute}>

Example (for a 2:15 AM sync on the weekday of the setup):

insights sync schedule time {sync-type} 02 15

Set a weekly schedule:

insights sync schedule time {sync-type} weekly {weekday}

Examples:

• Change the time setup to be a weekly Friday sync: insights sync schedule time {sync-type} weekly 5

• Change the time setup to be a weekly every day: insights sync schedule time {sync-type} weekly every

Value convention for {weekday}:

• 0 is Sunday

• 1 is Monday

• 2 is Tuesday

• 3 is Wednesday

• 4 is Thursday

• 5 is Friday

• 6 is Saturday

• every is every day

Disable/enable scheduled Insights syncs

To enable or disable a scheduled Insights sync, run the following command:

insights sync schedule <enable|disable> {sync-type}

Manual syncs

Scheduled syncs and the real-time sync keeps dashboard widget data in sync.

You can run a sync manually at any time to ensure that the Automate database and Insights reporter database are in sync and that dashboards reflect current data.

Note

• A full sync can be a long process, so if manually executing, ensure you’re doing this at a suitable time.

  Sync durations depend on the amount of data in the database and, for full-transaction syncs, the amount of activity against that data since the last sync, or 90 days / 10 million transactions. A daily cleanup script ensures that number of stored transactions remain under 10 million.

• If a sync is in progress, another sync of the same type won’t run.

• Run manual sync commands in a terminal opened with the tmux command.

  For details, see Using the tmux command

Manually run different types of insights syncs

Use the following command with options for sync type: insights sync run

For example: insights sync run {sync-type}

Sync type {sync-type} can be:

• delta-sync

• full-sync

• full-transaction

• multivendor

• sync check

• sync-check-arbitrator

Note

If upgrading to release 24.2, an initial manual sync is not required for dashboard management since the delta-sync runs every 30 minutes and will then execute this first sync.

Synced Automate models

Several Automate database models are synced the Insights database (the table lists examples). These models also serve as query sources for VOSS Wingman in the Automate GUI and by some Automate dashboard widgets.

See the topic on Wingman and Dashboards in the Core Feature Guide.

When any of these models are updated, the delta-sync sync process creates the matching INSERT, UPDATE or DELETE database operation in the Insights database so that any widgets on the Automate GUI dashboards that reference corresponding data, remains synced.

Insights Sync Table

model_type	friendly_name	friendly_model_type
data/MonitoringCluster	Monitoring Platform Cluster
data/MonitoringQueue	Monitoring Transaction Queue
data/MonitoringSessions	Monitoring Sessions
data/MetricDatabaseCollectionStats	Monitoring Database Stats
data/LicenseAuditCounts	Automate License Counts
data/BaseSiteDAT	Site	relation/HcsSiteREL
data/CountLimit	Subscriber Count Limit
data/HcsDpDNE164AssociateDAT	E164 Number Association
data/HcsDpE164InventoryDAT	E164 Number Inventory	data/HcsDpE164InventoryDAT
data/HcsEntitlementProfileDAT	Entitlement Profile	data/HcsEntitlementProfileDAT
data/HierarchyNode	Hierarchy
data/InternalNumberInventory	Number Inventory	relation/NumberInventoryREL
data/MicrosoftSubscriberQasStaging	Microsoft Subscriber Staging	data/MicrosoftSubscriberQasStaging
data/User	User	relation/User
device/cuc/User	Cisco CUC User	relation/Voicemail
device/cuc/UserLicense	Cisco CUC User License
device/cuc/Callhandler	Cisco CUC Call Handler	device/cuc/Callhandler
device/cucm/CallPickupGroup	Cisco UCM Call Pickup Group	device/cucm/CallPickupGroup
device/cucm/DeviceProfile	Cisco UCM Device Profile	relation/SubscriberDeviceProfile
device/cucm/Gateway	Cisco UCM Gateway
device/cucm/GatewaySccpEndpoints	Cisco UCM Gateway SCCP Endpoints
device/cucm/HcsLicense	Cisco UCM License
device/cucm/HuntList	Cisco UCM Hunt List
device/cucm/HuntPilot	Cisco UCM Hunt Pilot	relation/HuntGroupRelation
device/cucm/LicensingResourceUsage	Cisco UCM Licensing Resource Usage
device/cucm/Line	Cisco UCM Line	relation/LineRelation
device/cucm/LineGroup	Cisco UCM Line Group
device/cucm/Phone	Cisco UCM Phone	relation/SubscriberPhone
device/cucm/PhoneType	Cisco UCM Phone Type
device/cucm/RemoteDestination	Cisco UCM Remote Destination
device/cucm/RemoteDestinationProfile	Cisco UCM Remote Destination Profile	relation/SingleNumberReachREL
device/cucm/User	Cisco UCM User	relation/Subscriber
device/pexip/Conference	Pexip Conference
device/pexip/ConferenceAlias	Pexip Conference Alias
device/msexchangeonline/UserMailbox	Microsoft Exchange Online User Mailbox	device/msexchangeonline/UserMailbox
device/msgraph/MsolAccountSku	Microsoft O365 User SKU
device/msgraph/MsolUser	Microsoft O365 User	relation/MicrosoftSubscriber
device/msgraph/Room	Microsoft Teams Rooms
device/msteamsonline/CsAutoAttendant	Microsoft Teams Auto Attendant	device/msteamsonline/CsAutoAttendant
device/msteamsonline/CsCallQueue	Microsoft Teams Call Queue	device/msteamsonline/CsCallQueue
device/msteamsonline/CsOnlineUser	Microsoft Teams User	device/msteamsonline/CsOnlineUser
device/msteamsonline/CsTeamsClientConfiguration	Microsoft Teams Client Configuration
device/spark/Announcements	Webex Calling Announcements	device/spark/Announcements
device/spark/AutoAttendants	Webex Calling Auto Attendants	relation/WebexAutoAttendants
device/spark/CallParkExtensions	Webex Calling Call Park Extensions	relation/WebexCallParkExtensions
device/spark/CallParkGroup	Webex Calling Call Park Group	relation/WebexCallParkGroup
device/spark/CallPickup	Webex Calling Call Pickup	relation/WebexCallPickup
device/spark/Device	Webex Calling Device	relation/WebexDevice
device/spark/HuntGroup	Webex Calling Hunt Group	relation/WebexHuntGroup
device/spark/Group	Webex Calling Group	device/spark/Group
device/spark/License	Webex Calling License	device/spark/License
device/spark/Place	Webex Calling Place	relation/WebexTeamsPlace
device/spark/Number	Webex Calling Number	device/spark/Number
device/spark/Schedules	Webex Calling Schedule	relation/WebexSchedules
device/spark/Team	Webex Calling Team	device/spark/Team
device/spark/User	Webex Calling User	relation/SparkUser
device/uccx/Agent	Cisco UCCX Agent	relation/UccxAgent
device/webex/User	Cisco Webex User	device/webex/User

Related topics

• Wingman
• Introduction to Automate dashboards