[Index]

Model: data/DataSync

Data Sync

Full HTML Help

Tip

Use the Action search to navigate Automate

Overview

Data syncs can be performed from within Automate or directly on a device. For this reason, cached Automate data should be periodically synced with data on devices.

Examples:

Automate data syncs allow you to dynamically synchronize cached Automate data with data on devices. The data sync instance is associated with the connection parameters of a device type in Automate.

Supported devices include:

Individual add, update, and delete operations carried out by a data sync instance can be disabled on the user interface. If no operation is selected, the default behavior is maintained.

For Microsoft deployments, it is recommended that you enable Prevent Duplicate Numbers in the Global Settings to force the system to fail transactions involving workflow steps that create a copy of a number that already exists in Automate, for example, when creating a number range or when syncing in users. For details, see:

Prevent duplicate numbers in the Core Feature Guide

Global Settings (Number Inventory tab/panel) in the Core Feature Guide

Related topics

Sync overview in the Best Practices Guide

Data sync types in the Core Feature Guide

Model instance filters in the Core Feature Guide

Prevent duplicate numbers in the Core Feature Guide

Data sync settings

To view the summary list of configured data syncs in Automate, go to the Data Sync page. The list view displays basic details for each available data sync, including a number of summary attributes that provide additional details about the data sync.

To view data sync settings, click on a data sync in the list to open the configuration page.


The table describes a number of key settings that are available for data syncs:

Settings Description
Model Type lists

Define the entities to pull in a given sync. For example:

  • Only pull in device/cucm/User records from CUCM.
Model Instance filters

Limit a sync to a subset of entities in a sync. For example:

  • Pull in users with a primary extension starting with 1.

A system-level administrator will need to expose this setting in the Admin Portal.

  • The Purge Unmatched Records option can be enabled to purge all Automate data that does not match the active filters during sync. This allows for a maintenance free scheduled sync that will keep the cache records in line with the filter.
Disabled Operations

Choose which operations are enabled for a sync (Add/Update/Remove).

  • Update requires more effort to run because this typically involves a GET API call for each record, which must then be compared to Automate data.
  • Add/Remove can be determined from the initial list API calls.

To save time on the sync, you may wish to disable Update if you only require Add/Delete.

From the list view of Data Sync entries, these fields are also available as columns (summary attributes) and can for example be used to filter the list of data syncs by the operations.

Important

Remove is disabled by default if you've selected a model instance filter for the sync. You'll need to enable Remove if you intend for the sync to purge (remove) cache records that are excluded in the model instance filter.

If you're upgrading to 21.4-PB4 from an earlier version of Automate, a migration script disables Remove for any syncs that have a model instance filter applied. This is to prevent the sync from unexpectedly removing a large number of records after the upgrade.
Quick Import

Uses the list API responses to update the Automate cache, and won't perform individual GET calls for each entity for the update.

Recommended when the list response contains all values for the entity, or where only the key settings must be updated.

Removing individual GETs speeds up the sync, since Automate is not waiting for the API responses when there are a many entities to update. This is useful if the list and GET responses are required, or if you only need the summary data from the list view.

This setting is disabled by default for syncs related to device/spark/user syncs (SyncSparkUsers and SyncSpark).

This setting is enabled by default for syncs related to data/MSGraph syncs (Syncs Microsoft License data from Microsoft Graph).

Note

Quick Imports can improve sync performance but must be used with extreme caution to avoid unintended changes. Typically, Quick Import is recommended only for Microsoft-related data syncs (MSOperatorConnect, MSGraph, MSTeamsOnline, MSExchangeOnline) and for Webex App/Teams-related syncs that do not include the Webex App/Teams user (device/Spark/user).

The default Full Sync instance provided for CUC also does not have Quick Import enabled.

Synchronous and asynchronous data sync

By default, a data sync is asynchronous; that is, other tasks can be carried out while the sync is in progress.

However, a data sync can be set to be synchronous so that a workflow step can, for example, wait for the sync process to complete.

Asynchronous imports initiated by a data sync are standalone transactions; that is, they aren't child transactions of the data sync execute transaction. Synchronous imports initiated by a data sync are children of the data sync execute transaction.

Note

By default, syncs related to data/MSGraph syncs (Syncs Microsoft License data from Microsoft Graph) are synchronous.

Data sync types

Full HTML Help

Overview

Automate provides the following data sync types:

Data sync type Description
Pull from device

Available to all device types.

  • Pull all data from the device
  • Pull only the schema from the device (used for LDAP)
  • Pull data from the Change Notification Feature local data collection
Purge local resources

Available to all device types.

  • Purge data from the cache
Push to device

Available only to Cisco UCM devices

  • Push data in the cache to the device
Change notification sync Available only to Cisco UCM devices

Note

A quick import option is available to fetch only summary data that is contained in a list operation response and not the data for all instances/fields. See Data Sync Overview in the Core Guide for details.

Generally, for all sync types, Automate builds up the lists of entities from both Automate and the device, and compares them, using the key for the device entity. The key is typically the unique identifier (ID) for the record in the device we're syncing with. For example, for Cisco UCM, the ID is the pkid, which is the internal Cisco UCM database ID.

For users, a sync builds up the list of device/cucm/Users in Automate and then requests from the Cisco UCM the lists of users it currently has for the comparison. Differences in the lists are handled according to each sync type.

Related topics

Data Sync Overview in the Core Feature Guide

Change Notification Feature Overview in the Core Feature Guide

Pull from device

For sync type Pull from Device, the Automate resource is updated where the same key is present in both lists. In this case, the device data is the master and the Automate system model data is updated with the device data.

For example, let's say new data is added to the Cisco UCM, so that the Automate system data state for a Cisco UCM device/cucm/User does not show instances that are shown on the Cisco UCM.

In this case, a pull data sync synchronizes the system data with the Cisco UCM data. For example, a user's Department may be updated on the Cisco UCM, but the update only shows on the system after a Pull from Device sync. If a user resource is created in Cisco UCM but not in Automate, this adds the device/cucm/User instance into Automate at the level the pull sync was run from, for example, at the customer level.

When deleting a Automate resource from the device, so that the key is in the Automate list but not in the device list, a pull sync removes the resource in Automate. For example, if the resource is a user in Automate but not in Cisco UCM, the pull sync removes the device/cucm/User record in Automate.

To restrict the number of records removed in Automate, ensure you have the following named macro at the hierarchy where the sync takes place:

PULL_SYNC_DELETE_THRESHOLD_<device_type>

For details, see Pull Sync Delete Threshold topic in the Advanced Configuration Guide.

When pulling device data, for example LDAP users from an LDAP device, the results returned to Automate depend on the LDAP server configuration. For example, if the returned results exceed the LDAP server configured maximum, and if the server does not support paging, an appropriate error message is returned.

For Microsoft 365 syncs, a Max page size (default 1000) setting can be adjusted if the error "Template output exceeded memory limit" is shown.

For details, see the Configure Microsoft Tenant Connection Parameters topic in the Core Guide.

Push to Device

Sync type Push to Device is available only to Cisco UCM device types.

In a Push to Device sync type, devices are synchronized with the Automate system data state, which is the primary data state.

Keys found in both lists are ignored. Existing records are not updated in either direction.

In the device/cucm/User example, if the same user exists on both Automate and on Cisco UCM, no update occurs in either direction. Detailed settings may still not match after a Push to Device sync.

Important

When performing a push sync, it is important to consider data dependencies between different models.

For example, data dependencies may exist between users and phones in the Cisco UCM. In this case, if a user is associated to a phone (via the associated devices on the user), you can't add the user if the phone does not yet exist in in Cisco UCM.

On the other hand, for ownerID on the phone, pushing the phone first will fail since the user isn't in place.

This might mean running the push sync multiple times so it loads in the required order, or you may need to modify data (such as removing device association) to allow the push sync to succeed.

Note

The keys list sync logic described in this topic implies that in case of a reversion of the Cisco UCM to restores/inactive partitions, the end-state of the relevant pkids may differ to their state the last time Automate was in sync with Cisco UCM (before a restore), particularly if testing occurred in between. This means you may, for example, have a user with the same username in both Automate and Cisco UCM, but if that user's pkid in Cisco UCM now differs to the one in Automate from previous syncs or interactions, they will be seen as different users even though they have the same usernames.

Change Notification Sync

Sync type Change Notification Sync is available only to Cisco UCM device types.

A Change Notification Sync is a pull sync of changes stored in the local collection that is updated by the Change Notification Collector service.

For more details on Change Notification Sync, see the related topics in Data Sync section of the Core Feature Guide.

Purge Local Resources

In a Purge Local Resources sync type, all resources or instances of device information that exists in the system are deleted. Entities in the device are not deleted.

Note

The default purge syncs created when adding a Cisco UCM, CUC, LDAP or CCX server type are disabled by default. To use the purge sync, the "Remove" check box must first be cleared on the "Disabled Operations" tab of the relevant sync.

This sync type is typically used when cleaning up the system. The system displays a warning before executing an enabled purge sync.

See the following sample device type syncs:

Full sync

Full HTML Help

A full pull sync, when it runs, empties the changes from the data collection as they don't need to be processed by the Change Notification sync. Use the disabled operations and the model type list (MTL) of the full sync to filter the changes to remove. If a model instance filter is included, no changes are removed.

It is recommended to use the CUCXN Exclude ImportUser MTL on Cisco Unity Connection data syncs in order to avoid unneeded data and slowing the sync time.

Enable a scheduled data sync

Full HTML Help

Tip

Use the Action search to navigate Automate

This procedure enables the scheduled data sync so that it executes regularly.

Note

Setting up a Cisco UCM or CUC device in Automate:

Enable the scheduled data sync

  1. Log in as provider administrator.

  2. Go to the Scheduling page.

  3. Choose the schedule instance that matches this naming convention: HcsSync-<ip_address>-<device_name>-SCHED

    For example: HcsSync-192.0.2.24-CUCM01-SCHED

  4. Select the Active check box.

  5. Select the Multiple Executions tab, and update the interval, as required.

  6. Click Save.

    The full data sync executes immediately, and executes again according to the schedule.

Manually run the default data sync

Full HTML Help

Tip

Use the Action search to navigate Automate

You can always manually run the default data sync when there have been updates to Cisco Unified Communications Manager (UCM) or Cisco Unity Connection (CUC) devices that need to be synced into Automate.

Note

Manually running the change notification sync is not supported.

  1. Log in as provider or reseller administrator.
  2. Go to Perform Publisher Actions.
  3. From the Action drop-down, choose Import.
  4. From the App Type drop-down, choose CUCM Device or CUC Device.
  5. From the Clusters Available box, choose the device, move it to the Selected box, and click Save.

Controlling a data sync with a model type list

Full HTML Help

Using a Model Type List (MTL), you can control the types of data that are synced into Automate from devices from vendors, such as:

Controlling the types of data that are synced can greatly improve sync performance. The MTL is a list of device models associated with the device type, for example, Phone and line device models that are associated with the Cisco UCM device.

These are the possible types of model type lists:

Model type list Description
Include Selected Model Types This list represents the device models to explicitly include in the data sync.
Exclude Selected Model Types This list represents the device models to explicitly exclude from the data sync.
Ordered List This list represents the device models to explicitly include in the data sync in the order they must be synced.

A data sync created with an empty Model Type List attribute results in the subsequent import(s) synchronizing all device models for the corresponding device.

Here is an example of an include MTL:

A data sync using this MTL will sync all Media Resource Group, Media Resource Lists, Music on Hold servers and audio sources, and Media Termination Points. No other data will be synced from Unified CM.

It is recommended to define MTLs for sets of data that are being modified on the device directly. An example is Unified CM because this is where the bulk of the configuration data for each customer resides.

By defining MTLs that target specific data sets rather than doing a full sync, the performance of VOSS Automate can be maintained with better response times and quicker transaction execution.

Note

Some Cisco UCM device models to avoid unless needed are users, phones, and lines, as there may be large numbers of these in the Cisco UCM and result in a lengthy data sync operation.

Data sync overhead can be further reduced if you want to sync only new and deleted instances of the device model and not updates to existing instances. This can be done by unchecking the Refresh Existing (Changed) Data checkbox on the Data Sync configuration page. This check box controls whether existing device model instances are updated in Automate in addition to importing new instances and removing deleted instances. If checked, all device model instances must be synced and examined. If unchecked, only new and deleted instances need to be imported and the data sync will run considerably faster.

Create a custom data sync

Full HTML Help

Tip

Use the Action search to navigate Automate

Create a custom data sync to use a targeted model type list.

  1. Log in as provider admin or higher.

  2. Go to the Data Sync page.

  3. Click the Plus icon (+) to add a new record.

  4. Fill out a name for the data sync in the Name field.

    It is recommend to use a naming convention that makes it easy to identify the data syncs in the list view, such as C1Pull-CUCM01-DS, where:

    You could also include the type of data included in the sync, such as C1Pull-CUCM01-MediaResources-DS.

  5. From the Device Type drop-down, choose the device type you're syncing from.

  6. From the Sync Type drop-down, choose Pull from Device.

  7. From the Dependency Resolution drop-down, choose Default.

  8. Select the Execute Asynchronously and Refresh Existing (Changed) Data checkboxes.

    Execute Asynchronously means that the sync request will return a reply before it's complete when executed from the API. Refresh Existing (Changed) Data means that all instances of the device models specified in the model type list will be updated.

  9. Select the Force Refresh of Data checkbox if a data update is required regardless of whether data has changed on the device. This option would for example be used if it is required that update workflows be run upon a data sync.

  10. From the Model Type List drop-down, choose the targeted model type list you defined earlier.

  11. Leave Synchronization Order and Model Instance Filter blank.

  12. Click the Plus icon (+) adjacent to Device Filters to add an entry to the list.

    1. From the Attribute Name drop-down, choose host.
    2. From the Condition drop-down, choose Equals.
    3. From the Value drop-down, choose the hostname/IP address of the device.

    Note

    Workflows can be added to, and executed by a custom data sync to perform specific data sync operations.

  13. In the Workflows section, include workflows in the custom data sync if you want to perform specific data sync operations, otherwise leave the Workflows section empty. For example, if you want to move remote destinations from the customer hierarchy level to the site level, choose the RD_Overbuild_PWF_wrapper workflow from the Workflow drop-down.

    If the Synchronous checkbox is enabled, the workflows will execute in sync with data sync transaction, so that the parent pre- and post workflow steps are honoured. If disabled, the workflows will not be child transactions of the data sync execute transaction.

  14. From the Transaction Log Level drop-down, choose the log level for the data sync. For a description of the list of log levels, see Transaction Log Levels. The default log level is Warning.

    You can for example reduce the log level for PULL device syncs in order to reduce the size of transaction logs. This is useful where large numbers of transactions are archived regularly.

  15. Click Save.

Next steps

To run the custom data sync, click the data sync from the Data Sync list and click Execute.

Cisco UCM change notification alerts

Full HTML Help

Overview

The Cisco Unified Communications Manager (Cisco UCM) change notification (CNF) feature is enabled to display alerts. There is no need to configure CNF alerts manually in Automate - admin users receive alerts when the collector process is in an error state.

Administrators can view and delete alerts at the hierarchy they log in at, and at lower hierarchies. For example, Provider, Reseller, and Customer administrators can view alerts raised at the Customer level (sys.hcs.provider.reseller.customer), but a Site admin cannot view such alerts.

When a CNF alert is raised, you're notified of the alert via the toolbar Messages icon in the Admin Portal. Click the Messages or Notifications toolbar button to view the alerts, or go to Alerts.

Properties of CNF alerts

Change notification (CNF) alerts have the following properties:

Property Description
ID A generated unique identifier of the target device of the collector. For CUCM, the ID displays the host name, port, and hierarchy.
Code An error or warning code associated with the alert.
Category The alert category - Device Change Notification Collector
Severity

Automate displays severity codes and messages as follows (“{}” indicate device or number placeholders in the messages).

Each alert has some properties, for example, severity (Error, Warning or Info), the number of times that the same alert has been raised, and the time stamp of the last alert instance.
Message Displays error message description and the statement to fix the error.
Count Displays the number of times the alert has occurred for a specific device.
Latest Alert The last time this alert occurred.

Note

Administrators can filter alerts by any of the alert fields.

CNF error scenarios

Automate displays change notification (CNF) alerts for the following error scenarios:

The administrator reads, inspects, acts on (for example, run a full sync on the device), and then manages alerts of the CNF collection service. The administrator can delete the alert from the list only when the issue that raised the alert has been resolved.

Note

If the Administrators forget to remove the CNF alert after resolving it, the alert will still be shown when they log in to Automate. We strongly recommend removing the alert after resolving it.

Alert Field Descriptions

Full HTML Help

Field Description
ID * The unique ID of the alert.
Code * The code of the alert.
Category * The category of the alert.
Severity The severity of the alert.
Message The message describing the alert.
Count The number of times this alert has occurred.
Latest Alert The last time this alert occurred.

Provides data synchronization capabilities with devices (eg. CUCM, CUC, LDAP)

Model Details: data/DataSync

Title Description Details
Name * The name that is given to the Data Sync instance.
  • Field Name: name
  • Type: String
Description A description for the Data Sync instance.
  • Field Name: description
  • Type: String
Device Type * The type of devices to be synchronized. Choices are based on connection parameter data models of the devices supported by the current system.
  • Field Name: device_type
  • Type: String
  • Choices: ["Active Directory", "Active Directory Hybrid", "Assurance Arbitrator", "AudioCodes", "AzureADOnline", "BroadsoftOCIP", "Call Manager AXL Generic Driver", "Call Manager Control Center Services", "Cisco HCM-F", "Cisco Unified CM", "Cisco Unified Contact Center Express", "Cisco Unity Connection", "Cisco WebEx", "Exchange", "Exchange Hybrid", "Exchange Online", "IMP GUI", "LDAP", "MSExchangeOnline", "MSGraph", "MSOperatorConnect", "MSTeamsOnline", "Microsoft Online", "Netflow", "OracleECB", "OracleSBC", "PRSCallControl", "Pexip Infinity Conferencing Platform", "PowerShell", "Roomos", "Service Now", "ServiceNowOnline", "SkypeForBusiness", "SkypeForBusiness Hybrid", "SkypeForBusiness Online", "Spark", "Unity Connection GUI", "Zoom", "sfb2015"]
Sync Type The selected Sync Type. The Type can be Pull from Device, Push to Device, Merge with Device, or Purge Local. For Push, the system data state is the master state and the model types of the connection parameter are synchronized with it. For Pull, the model type states of the connection parameter is the master and the system model data is synchronized with these. For Merge, data synchronization takes place in both directions without overwrite of either data. For Purge Local, resources all resources that exist in the system will be deleted, although the entities in the device are not deleted - this is useful when cleaning up the system.
  • Field Name: sync_type
  • Type: String
  • Choices: ["Pull from Device", "Push to Device", "Merge with Device", "Purge Local Resources", "Change Notification Sync"]
Dependency Resolution The strategy for handling failures due to missing dependencies. Only 'Best Effort' resolution is supported. 'Best Effort' resolution attempts to brute force the synchronization as much as possible by re-iterating over failed imports until there are no errors.
  • Field Name: dependency_resolution
  • Type: String
  • Choices: ["Best Effort", "Default"]
Quick Import (Warning: Summary Data Only) Enable with caution: Indicates whether import operations will fetch full data for all instances or just use the summary data. Only use with Model types where all of the required data is in the summary data eg: device/cuc/ImportUser model.
  • Field Name: quick_import
  • Type: Boolean
Import Schema Only Indicates whether import operations will fetch schema only and no data.
  • Field Name: schema_only
  • Type: Boolean
Import Data Only Indicates whether import operations will fetch data only and no schema.
  • Field Name: no_schema
  • Type: Boolean
Execute Asynchronously Indicates whether the Data Sync must be executed Asynchronously or Synchronously. Default: True
  • Field Name: asynchronous
  • Type: Boolean
  • Default: True
Refresh Existing (Changed) Data Indicates whether the Data Sync refreshes existing data that has changed on the device.
  • Field Name: refresh_existing_data
  • Type: Boolean
Force Refresh Of Data Indicates whether a update action is executed regardless of whether the device has changed or not
  • Field Name: force_update_flag
  • Type: Boolean
Number of Changes To Process Specifies how many changes to process during a device notification.
  • Field Name: num_changes_to_process
  • Type: Integer
Disabled Operations Indicates which operations should be disabled for this Data Sync.
  • Field Name: disabled_operations
  • Type: Object
Disable Add Operation Disable the "add" operation.
  • Field Name: disabled_operations.add
  • Type: Boolean
Disable Update Operation Disable the "update" operation.
  • Field Name: disabled_operations.update
  • Type: Boolean
Disable Remove Operation Disable the "remove" operation.
  • Field Name: disabled_operations.remove
  • Type: Boolean
Model Type List The selected 'exclusion/inclusion' model type list that was created as a model instance of the Model Type List. See: Model Type List.
  • Field Name: model_type_list
  • Type: String
  • Target: data/ModelTypeList
  • Target attr: name
  • Format: uri
Synchronization Order The selected 'ordered' model type list that was created as a model instance of the Model Type List. This list dictates the order in which models will be synchronized. See: Model Type List.
  • Field Name: sync_order
  • Type: String
  • Target: data/ModelTypeList
  • Target attr: name
  • Format: uri
Model Instance Filter Provides the ability to filter on a subset of records for the chosen model type. Note: Model Instance filters do not work with CUCM Change Notification sync types. If a model instance filter is needed for a CUCM element, this model type should be excluded from the change notification sync and a separate sync should be se up for this.
  • Field Name: model_instance_filter
  • Type: String
  • Target: data/ModelInstanceFilter
  • Target attr: name
  • Format: uri
Purge Unmatched Records(Warning: Existing data not matching Filter will be removed) Purge all cache records which do not match the filters. Warning: Existing data not matching Filter will be removed
  • Field Name: purge_unmatched
  • Type: Boolean
Device Filters A refinement on devices that the data sync applies to. These filters are optional. If they are not specified, data sync will be applied to all instances of the device type supported by this Data Sync instance. If specified, only the devices that match the given filters will be synchronized.
  • Field Name: device_filters.[n]
  • Type: Array
Attribute Name * The model attribute name to which the condition applies.
  • Field Name: device_filters.[n].attr_name
  • Type: String
  • Choices: [""]
Condition * If the Condition is Equals or Not Equals, then the Value to test the condition against. If the Condition is In or Not In, then Value is a list of values to test against.
  • Field Name: device_filters.[n].condition
  • Type: String
  • Choices: ["Equals", "Not Equals", "In", "Not In"]
Value The selected Attribute Name is tested against the Value that can also be a list of values, such as a list of host names. The Condition tests against the Value or list of values.
  • Field Name: device_filters.[n].value
  • Type: String
  • Choices: [""]
Values If the Condition is In or Not In, then the Values list the items to test the condition against.
  • Field Name: value_list.[n]
  • Type: Array
Workflows A list of workflows that is to be executed as part of data sync actions eg. when a resource is added, modified, or deleted
  • Field Name: workflows.[n]
  • Type: Array
Model Type The model type for which the workflow is to be executed.
  • Field Name: workflows.[n].model_type
  • Type: String
  • Format: uri
Operation * The operation for which the workflow is to be executed.
  • Field Name: workflows.[n].operation
  • Type: String
  • Choices: ["Add", "Update", "Delete"]
Phase * The operation phase at which the workflow is to be executed eg. Pre-Execution (before the operation is performed) or Post-Execution (after the operation is performed)
  • Field Name: workflows.[n].phase
  • Type: String
  • Choices: ["Pre Execution", "Post Execution"]
Workflow * The workflow instance that will be executed.
  • Field Name: workflows.[n].workflow
  • Type: String
  • Target: data/ProvisioningWorkflow
  • Target attr: name
  • Format: uri
Synchronous If selected the workflow will be executed synchronously, otherwise asynchronously (default).
  • Field Name: workflows.[n].synchronous
  • Type: Boolean
Transaction Log Level The transaction log level to be used for this Data Sync and its immediate sub-transactions. Default is Warning when this field is not set. Default: 30
  • Field Name: transaction_log_level
  • Type: Integer
  • Default: 30
  • Choices: ["Use System Settings", "Debug", "Verbose", "Info", "Warning", "Error", "Disabled"]