[Index]

Model: data/DataSync

Data Sync

To access the latest documentation, go to Documentation and Resources at: https://voss.portalshape.com

Data on devices may be updated within VOSS Automate, or directly on the device. For this reason, cached VOSS Automate data should be periodically synchronized with the data on devices. For example:

VOSS Automate data syncs allow you to dynamically synchronize cached VOSS Automate data with data on devices. The data sync instance is associated with the connection parameters of a device type in VOSS Automate.

Supported devices include:

Individual add, update, and delete operations carried out by a data sync instance can be disabled on the user interface. If no operation is selected, the default behavior is maintained.

Related Topics

Sync Overview in the Best Practices Guide

Data Sync Types in the Core Feature Guide

Data Sync Settings

The table describes a number of key settings that are available for data sync:

Model Type lists

Define the entities to pull in a given sync. For example:

  • Only pull in device/cucm/User records from Cisco Unified CM.
Model Instance filters

Limit a sync to a subset of entities in a sync. For example:

  • Pull in users with a primary extension starting with 1.

A system-level administrator will need to expose this setting on the Admin Portal.

Actions

Select which actions are active for a sync (Add/ Update/Delete).

  • Update requires more effort to run because this typically involves a GET API call for each record, which must then be compared to VOSS Automate data.

  • Add/Delete can be determined from the initial list API calls.

    To save time on the sync, you may wish to disable Update if you only require Add/Delete.

Quick Import

Uses the list API responses to update the VOSS Automate cache, and won't perform individual GET calls for each entity for the update.

Recommended when the list response contains all values for the entity, or where only the key settings must be updated.

Removing individual GETs speeds up the sync, since VOSS Automate is not waiting for the API responses when there are a many entities to update.

This is useful if the list and GET responses are required, or if you only need the summary data from the list view.

Note

Quick Import is generally not recommended, and should be used only for syncing device/cuc/ImportUser.

However, initially there is an exception to the performance improvement of a Quick Import sync with device/cuc/User:

Synchronous and Asynchronous Data Sync

By default, a data sync is asynchronous; that is, other tasks can be carried out while the sync is in progress.

However, a data sync can be set to be synchronous so that a workflow step can, for example, wait for the sync process to complete.

Asynchronous imports initiated by a data sync are standalone transactions; that is, they aren't child transactions of the data sync execute transaction. Synchronous imports initiated by a data sync are children of the data sync execute transaction.

Data Sync Types

VOSS Automate provides the following data sync types:

Data sync type Description
Pull from Device

Available to all device types.

  • Pull all data from the device
  • Pull only the schema from the device (used for LDAP)
  • Pull data from the Change Notification Feature local data collection
Purge Local Resources

Available to all device types.

  • Purge data from the cache
Push to Device

Available only to Cisco Unified CM devices

  • Push data in the cache to the device
Change Notification Sync Available only to Cisco Unified CM devices

Note

A quick import option is available to fetch only summary data that is contained in a list operation response and not the data for all instances/fields. See Data Sync Overview in the Core Guide for details.

Generally, for all sync types, VOSS Automate builds up the lists of entities from both VOSS Automate and the device, and compares them, using the key for the device entity. The key is typically the unique identifier (ID) for the record in the device we're syncing with. For example, for Unified CM, the ID is the pkid, which is the internal Unified CM database ID.

For subscribers, a sync builds up the list of device/cucm/Users in VOSS Automate and then requests from the Unified CM the lists of users it currently has for the comparison. Differences in the lists are handled according to each sync type.

Related Topics

Data Sync Overview in the Core Feature Guide

Change Notification Feature Overview in the Core Feature Guide

Pull from Device

For sync type Pull from Device, the VOSS Automate resource is updated where the same key is present in both lists. In this case, the device data is the master and the VOSS Automate system model data is updated with the device data.

For example, let's say new data is added to the Unified CM, so that the VOSS Automate system data state for a Unified CM device/cucm/User does not show instances that are shown on the Unified CM.

In this case, a pull data sync synchronizes the system data with the Unified CM data. For example, a user's Department may be updated on the Unified CM, but the update only shows on the system after a Pull from Device sync. If a user resource is created in Unified CM but not in VOSS Automate, this adds the device/cucm/User instance into VOSS Automate at the level the pull sync was run from, for example, at the customer level.

When deleting a VOSS Automate resource from the device, so that the key is in the VOSS Automate list but not in the device list, a pull sync removes the resource in VOSS Automate. For example, if the resource is a user in VOSS Automate but not in Unified CM, the pull sync removes the device/cucm/User record in VOSS Automate.

To restrict the number of records removed in VOSS Automate, ensure you have the following named macro at the hierarchy where the sync takes place:

PULL_SYNC_DELETE_THRESHOLD_<device_type>

For details, see Pull Sync Delete Threshold topic in the Advanced Configuration Guide.

When pulling device data, for example LDAP users from an LDAP device, the results returned to VOSS Automate depend on the LDAP server configuration. For example, if the returned results exceed the LDAP server configured maximum, and if the server does not support paging, an appropriate error message is returned.

Push to Device

Sync type Push to Device is available only to Cisco Unified CM device types.

In a Push to Device sync type, devices are synchronized with the VOSS Automate system data state, which is the primary data state.

Keys found in both lists are ignored. Existing records are not updated in either direction.

In the device/cucm/User example, if the same user exists on both VOSS Automate and on Unified CM, no update occurs in either direction. Detailed settings may still not match after a Push to Device sync.

Important

When performing a push sync, it is important to consider data dependencies between different models.

For example, data dependencies may exist between users and phones in the Cisco Unified CM. In this case, if a user is associated to a phone (via the associated devices on the user), you can't add the user if the phone does not yet exist in in Cisco Unified CM.

On the other hand, for ownerID on the phone, pushing the phone first will fail since the user isn't in place.

This might mean running the push sync multiple times so it loads in the required order, or you may need to modify data (such as removing device association) to allow the push sync to succeed.

Note

The keys list sync logic described in this topic implies that in case of a reversion of the Unified CM to restores/inactive partitions, the end-state of the relevant pkids may differ to their state the last time VOSS Automate was in sync with Unified CM (before a restore), particularly if testing occurred in between. This means you may, for example, have a user with the same username in both VOSS Automate and Unified CM, but if that user's pkid in Unified CM now differs to the one in VOSS Automate from previous syncs or interactions, they will be seen as different users even though they have the same usernames.

Change Notification Sync

Sync type Change Notification Sync is available only to Cisco Unified CM device types.

A Change Notification Sync is a pull sync of changes stored in the local collection that is updated by the Change Notification Collector service.

For more details on Change Notification Sync, see the related topics in Data Sync section of the Core Feature Guide.

Purge Local Resources

In a Purge Local Resources sync type, all resources or instances of device information that exists in the system are deleted. Entities in the device are not deleted.

Note

The default purge syncs created when adding a CUCM, CUC, LDAP or CCX server type are disabled by default. To use the purge sync, the "Remove" check box must first be cleared on the "Disabled Operations" tab of the relevant sync.

This sync type is typically used when cleaning up the system. The system displays a warning before executing an enabled purge sync.

See the following sample device type syncs:

Full Sync

A full pull sync, when it runs, empties the changes from the data collection as they don't need to be processed by the Change Notification Sync. Use the disabled operations and the model type list of the full sync to filter the changes to remove. If a model instance filter is included, no changes are removed.

Enable a Scheduled Data Sync

This procedure enables the scheduled data sync so that it executes regularly.

Note

Setting up a CUCM or CUC device in VOSS Automate ...

Enable the scheuled data sync

  1. Log in as provider administrator.

  2. Go to (default menus) Administration Tools > Scheduling.

  3. On the Scheduling page, choose the schedule instance that matches this naming convention:

    HcsSync-<ip_address>-<device_name>-SCHED. For example:

    HcsSync-192.0.2.24-CUCM01-SCHED

  4. Select the Active check box.

  5. Select the Multiple Executions tab, and update the interval, as required.

  6. Click Save.

    The full data sync executes immediately, and executes again according to the schedule.

Manually Run the Default Data Sync

You can always manually run the default data sync when there have been updates to Cisco Unified Communications Manager (CUCM) or Cisco Unity Connection (CUC) devices that need to be synced into VOSS Automate.

Note

Manually running the change notification sync is not supported.

Perform these steps:

  1. Log in as provider or reseller administrator.
  2. Go to (default menu Device Management > Advanced > Perform Publisher Actions).
  3. From the Action drop-down, choose Import.
  4. From the App Type drop-down, choose CUCM Device or CUC Device.
  5. From the Clusters Available box, choose the device, move it to the Selected box, and click Save.

Controlling a Data Sync with a Model Type List

Using a Model Type List (MTL), you can control the types of data that are synced into VOSS Automate from devices from vendors, such as:

and so on.

Controlling the types of data that are synced can greatly improve sync performance. The MTL is a list of device models associated with the device type, for example, Phone and Line device models that are associated with the Unified CM device.

These are the possible types of Model Type Lists:

A data sync created with an empty Model Type List attribute results in the subsequent import(s) synchronizing all device models for the corresponding device.

Here is an example of an include MTL:

A data sync using this MTL will sync all Media Resource Group, Media Resource Lists, Music on Hold servers and audio sources, and Media Termination Points. No other data will be synced from Unified CM.

It is recommended to define MTLs for sets of data that are being modified on the device directly. An example is Unified CM because this is where the bulk of the configuration data for each customer resides.

By defining MTLs that target specific data sets rather than doing a full sync, the performance of VOSS Automate can be maintained with better response times and quicker transaction execution.

Note

Some Unified CM device models to avoid unless needed are Users, Phones, and Lines, as there may be large numbers of these in the Unified CM and result in a lengthy data sync operation.

Data sync overhead can be further reduced if you want to sync only new and deleted instances of the device model and not updates to existing instances. This can be done by unchecking the Refresh Existing (Changed) Data check box on the Data Sync configuration page. This check box controls whether existing device model instances are updated in VOSS Automate in addition to importing new instances and removing deleted instances. If checked, all device model instances must be synced and examined. If unchecked, only new and deleted instances need to be imported and the data sync will run considerably faster.

Create a Custom Data Sync

Create a custom data sync to use a targeted Model Type List.

Perform these steps:

  1. Log in as provider admin or higher.

  2. Choose Administration Tools > Data Sync.

  3. Click Add.

  4. Enter the name of the Data Sync in the Name field.

    It is recommend to use a naming convention that makes it easy to identify the data syncs in the list view, such as C1Pull-CUCM01-DS where C1 is the customer name, Pull is the data sync type, CUCM01 is the name of the Cisco Unified Communications Manager, and DS stands for Data Sync. You could also include the type of data included in the sync, such as C1Pull-CUCM01-MediaResources-DS.

  5. From the Device Type drop-down, choose the Device Type you are syncing from.

  6. From the Sync Type drop-down, choose Pull from Device.

  7. From the Dependency Resolution drop-down, choose Default.

  8. Select the Execute Asynchronously and Refresh Existing (Changed) Data check boxes.

    Execute Asynchronously means that the sync request will return a reply before its complete when executed from the API. Refresh Existing (Changed) Data means that all instances of the device models specified in the Model Type List will be updated.

  9. Select the Force Refresh of Data check box if a data update is required regardless of whether data has changed on the device. This option would for example be used if it is required that update workflows be run upon a data sync.

  10. From the Model Type List drop-down, choose the targeted Model Type List you defined earlier.

  11. Leave Synchronization Order and Model Instance Filter blank.

  12. Click + next to Device Filters to add an entry to the list.

    1. From the Attribute Name drop-down, choose host.
    2. From the Condition drop-down, choose Equals.
    3. From the Value drop-down, choose the hostname/IP address of the device.

    Note

    Workflows can be added to, and executed by a custom data sync to perform specific data sync operations.

  13. In the Workflows section, include workflows in the custom data sync if you want to perform specific data sync operations, otherwise leave the Workflows section empty. For example, if you want to move remote destinations from the Customer hierarchy level to the Site level, choose the RD_Overbuild_PWF_wrapper workflow from the Workflow drop-down.

  14. From the Transaction Log Level drop-down, choose the log level for the data sync. For a description of the list of log levels, see Transaction Log Levels. The default log level is Warning.

    You can for example reduce the log level for PULL device syncs in order to reduce the size of transaction logs. This is useful where large numbers of transactions are archived regularly.

  15. Click Save.

Next Steps

To run the custom data sync, click the data sync from the Data Sync list and click Execute.

Unified CM Change Notification Feature Alerts

The Unified CM Change Notification Feature (CNF) is enabled to display alerts. You don't have to configure the change notification feature alerts manually in the VOSS Automate. The administrator gets the alerts when something goes wrong with the collector process.

The administrators can view the alerts at the hierarchy level they log in and all the levels below that hierarchy. For example, if an alert is raised at the customer level (sys.hcs.provider.reseller.customer), then the provider, reseller, and customer administrators can see that alert but not site administrator. A Site administrator doesn't have access to view the alert. All the administrators have read and delete permissions to the alerts.

When a change notification feature alert is raised, the Messages indicator on the VOSS Automate Admin Portal shows the alert. Clicking the Messages or Notifications button on the toolbar shows a pop-up and a message that alerts have been raised. Clicking on the message, the user is navigated to the list of alert messages on the Alerts list view (default menu Administration Tools > Alerts).

CNF alerts have the following distinct properties:

Note

Administrators can also filter alerts by any of the alert fields.

VOSS Automate displays change notification feature alerts for the following error scenarios:

The administrator reads, inspects, acts on (for example, run a full sync on the device), and then manages alerts of the Change Notification collection service. The administrator can delete the alert from the list only when the issue that raised the alert has been resolved.

Note

If the Administrators forget to remove the change notification feature alert after resolving it, the alert will still be shown when they log in to VOSS Automate. We strongly recommend removing the alert after resolving it.

Data Sync Blacklist

Administrators with permissions to access the Global instance of the settings in the data/Settings model, can create a list of device attributes that will not trigger any update workflows that may have been defined to execute during the data sync. These attributes are here called Data Sync Blacklist attributes and are excluded from data sync considerations.

The reason for this list of attributes is that while data sync operations can have a performance impact, some data sync attribute changes do not require data sync workflows to be carried out.

Note however that the local device cache will still be updated with the updated attribute data. No update workflows will be run, though. The transaction logs will indicate the updated device cache, but the transactions for these attributes instances will show as:

"Device changes on blacklisted attributes only. Updating cache, skipping workflows."

Note

After release 20.1.1 or applying patch EKB-4362-19.2.1_patch, the previously blacklisted LDAP attributes are no longer imported during LDAP syncronization:

For device/ldap/user:

A number of blacklisted attributes have been added by default:

Blacklisted ``device/ldap/user``

Blacklisted ``device/cucm/User``

Blacklisted ``device/cucm/Phone``

Blacklisted ``device/ldap/userProxy``

Alert Field Descriptions

Field Description
ID * The unique ID of the alert.
Code * The code of the alert.
Category * The category of the alert.
Severity The severity of the alert.
Message The message describing the alert.
Count The number of times this alert has occurred.
Latest Alert The last time this alert occurred.

Provides data synchronization capabilities with devices (eg. CUCM, CUC, LDAP)

Model Details: data/DataSync

Title Description Details
Name * The name that is given to the Data Sync instance.
  • Field Name: name
  • Type: String
Description A description for the Data Sync instance.
  • Field Name: description
  • Type: String
Device Type * The type of devices to be synchronized. Choices are based on connection parameter data models of the devices supported by the current system.
  • Field Name: device_type
  • Type: String
  • Choices: ["Active Directory", "Active Directory Hybrid", "Assurance Arbitrator", "AzureADOnline", "Call Manager AXL Generic Driver", "Call Manager Control Center Services", "Cisco HCM-F", "Cisco Unified CM", "Cisco Unified Contact Center Express", "Cisco Unity Connection", "Cisco WebEx", "Exchange", "Exchange Hybrid", "Exchange Online", "IMP GUI", "LDAP", "MSExchangeOnline", "MSGraph", "MSTeamsOnline", "Microsoft Online", "OracleECB", "OracleSBC", "PRSCallControl", "Pexip Infinity Conferencing Platform", "PowerShell", "Service Now", "SkypeForBusiness", "SkypeForBusiness Hybrid", "SkypeForBusiness Online", "Spark", "Unity Connection GUI", "Zoom", "sfb2015"]
Sync Type The selected Sync Type. The Type can be Pull from Device, Push to Device, Merge with Device, or Purge Local. For Push, the system data state is the master state and the model types of the connection parameter are synchronized with it. For Pull, the model type states of the connection parameter is the master and the system model data is synchronized with these. For Merge, data synchronization takes place in both directions without overwrite of either data. For Purge Local, resources all resources that exist in the system will be deleted, although the entities in the device are not deleted - this is useful when cleaning up the system.
  • Field Name: sync_type
  • Type: String
  • Choices: ["Pull from Device", "Push to Device", "Merge with Device", "Purge Local Resources", "Change Notification Sync"]
Dependency Resolution The strategy for handling failures due to missing dependencies. Only 'Best Effort' resolution is supported. 'Best Effort' resolution attempts to brute force the synchronization as much as possible by re-iterating over failed imports until there are no errors.
  • Field Name: dependency_resolution
  • Type: String
  • Choices: ["Best Effort", "Default"]
Quick Import (Warning: Summary Data Only) Enable with caution: Indicates whether import operations will fetch full data for all instances or just use the summary data. Only use with Model types where all of the required data is in the summary data eg: device/cuc/ImportUser model.
  • Field Name: quick_import
  • Type: Boolean
Import Schema Only Indicates whether import operations will fetch schema only and no data.
  • Field Name: schema_only
  • Type: Boolean
Import Data Only Indicates whether import operations will fetch data only and no schema.
  • Field Name: no_schema
  • Type: Boolean
Execute Asynchronously Indicates whether the Data Sync must be executed Asynchronously or Synchronously. Default: True
  • Field Name: asynchronous
  • Type: Boolean
  • Default: True
Refresh Existing (Changed) Data Indicates whether the Data Sync refreshes existing data that has changed on the device.
  • Field Name: refresh_existing_data
  • Type: Boolean
Force Refresh Of Data Indicates whether a update action is executed regardless of whether the device has changed or not
  • Field Name: force_update_flag
  • Type: Boolean
Number of Changes To Process Specifies how many changes to process during a device notification.
  • Field Name: num_changes_to_process
  • Type: Integer
Disabled Operations Indicates which operations should be disabled for this Data Sync.
  • Field Name: disabled_operations
  • Type: Object
Add Disable the "add" operation.
  • Field Name: disabled_operations.add
  • Type: Boolean
Update Disable the "update" operation.
  • Field Name: disabled_operations.update
  • Type: Boolean
Remove Disable the "remove" operation.
  • Field Name: disabled_operations.remove
  • Type: Boolean
Model Type List The selected 'exclusion/inclusion' model type list that was created as a model instance of the Model Type List. See: Model Type List.
  • Field Name: model_type_list
  • Type: String
  • Target: data/ModelTypeList
  • Target attr: name
  • Format: uri
Synchronization Order The selected 'ordered' model type list that was created as a model instance of the Model Type List. This list dictates the order in which models will be synchronized. See: Model Type List.
  • Field Name: sync_order
  • Type: String
  • Target: data/ModelTypeList
  • Target attr: name
  • Format: uri
Model Instance Filter An instance of Model Instance Filter data model. This instance defines the rules for filtering out the models to be synchronized. See: Model Instance Filter.
  • Field Name: model_instance_filter
  • Type: String
  • Target: data/ModelInstanceFilter
  • Target attr: name
  • Format: uri
Device Filters A refinement on devices that the data sync applies to. These filters are optional. If they are not specified, data sync will be applied to all instances of the device type supported by this Data Sync instance. If specified, only the devices that match the given filters will be synchronized.
  • Field Name: device_filters.[n]
  • Type: Array
Attribute Name * The model attribute name to which the condition applies.
  • Field Name: device_filters.[n].attr_name
  • Type: String
  • Choices: [""]
Condition * If the Condition is Equals or Not Equals, then the Value to test the condition against. If the Condition is In or Not In, then Value is a list of values to test against.
  • Field Name: device_filters.[n].condition
  • Type: String
  • Choices: ["Equals", "Not Equals", "In", "Not In"]
Value The selected Attribute Name is tested against the Value that can also be a list of values, such as a list of host names. The Condition tests against the Value or list of values.
  • Field Name: device_filters.[n].value
  • Type: String
  • Choices: [""]
Values If the Condition is In or Not In, then the Values list the items to test the condition against.
  • Field Name: value_list.[n]
  • Type: Array
Workflows A list of workflows that is to be executed as part of data sync actions eg. when a resource is added, modified, or deleted
  • Field Name: workflows.[n]
  • Type: Array
Model Type The model type for which the workflow is to be executed.
  • Field Name: workflows.[n].model_type
  • Type: String
  • Format: uri
Operation * The operation for which the workflow is to be executed.
  • Field Name: workflows.[n].operation
  • Type: String
  • Choices: ["Add", "Update", "Delete"]
Phase * The operation phase at which the workflow is to be executed eg. Pre-Execution (before the operation is performed) or Post-Execution (after the operation is performed)
  • Field Name: workflows.[n].phase
  • Type: String
  • Choices: ["Pre Execution", "Post Execution"]
Workflow * The workflow instance that will be executed.
  • Field Name: workflows.[n].workflow
  • Type: String
  • Target: data/ProvisioningWorkflow
  • Target attr: name
  • Format: uri
Synchronous If selected the workflow will be executed synchronously, otherwise asynchronously (default).
  • Field Name: workflows.[n].synchronous
  • Type: Boolean
Transaction Log Level The transaction log level to be used for this Data Sync and its immediate sub-transactions. Default is Warning when this field is not set. Default: 30
  • Field Name: transaction_log_level
  • Type: Integer
  • Default: 30
  • Choices: ["Use System Settings", "Debug", "Verbose", "Info", "Warning", "Error", "Disabled"]