[Index]

Model: data/DataSync

Data Sync

Full HTML Help

Overview

Data syncs can be performed from within Automate or directly on a device. For this reason, cached Automate data should be periodically synced with data on devices.

Examples:

• When an instance of a Cisco UCM is added to the system, its data is imported and cached.

• When instances are added, updated, or deleted from the Cisco UCM, the cached data in Automate becomes out of sync with data on the device.

• When deleting data from Cisco UCM before deleting it from Automate, the system displays the following error: "The specified resource could not be found"

  This means the resource is out of sync, and Automate may need to re-sync with Cisco UCM in order to delete it or update it.

Automate data syncs allow you to dynamically synchronize cached Automate data with data on devices. The data sync instance is associated with the connection parameters of a device type in Automate.

Supported devices include:

• Cisco Unified CM (Cisco UCM)
• Cisco Unity Connection (CUC)
• LDAP
• Webex
• MSTeamsOnline (Microsoft Teams)
• MSGraph

Individual add, update, and delete operations carried out by a data sync instance can be disabled on the user interface. If no operation is selected, the default behavior is maintained.

For Microsoft deployments, it is recommended that you enable Prevent Duplicate Numbers in the Global Settings to force the system to fail transactions involving workflow steps that create a copy of a number that already exists in Automate, for example, when creating a number range or when syncing in users. For details, see:

Prevent duplicate numbers in the Core Feature Guide

Global Settings (Number Inventory tab/panel) in the Core Feature Guide

Related topics

Sync overview in the Best Practices Guide

Data sync types in the Core Feature Guide

Model instance filters in the Core Feature Guide

Prevent duplicate numbers in the Core Feature Guide

Data sync settings

To view the summary list of configured data syncs in Automate, go to the Data Sync page. The list view displays basic details for each available data sync, including a number of summary attributes that provide additional details about the data sync.

To view data sync settings, click on a data sync in the list to open the configuration page.

The table describes a number of key settings that are available for data syncs:

Synchronous and asynchronous data sync

By default, a data sync is asynchronous; that is, other tasks can be carried out while the sync is in progress.

However, a data sync can be set to be synchronous so that a workflow step can, for example, wait for the sync process to complete.

Asynchronous imports initiated by a data sync are standalone transactions; that is, they aren't child transactions of the data sync execute transaction. Synchronous imports initiated by a data sync are children of the data sync execute transaction.

Data sync types

Full HTML Help

Overview

Automate provides the following data sync types:

• Pull from Device

  This data sync type is available to all device types:

  • Pull all data from the device
  • Pull only the schema from the device (used for LDAP)
  • Pull data from the Change Notification Feature (CNF) local data collection

• Push to Device: Available only to Cisco UCM devices. Push data in the cache to the device

• Change Notification (CNF): Available only to Cisco UCM devices

• Purge Local Resources: Available only to Cisco UCM devices

For all sync types, Automate typically builds up the lists of entities from both Automate and the device and compares them, using the key for the device entity.

The key is typically the unique identifier (ID) for the record in the device we're syncing with. For example, for Cisco UCM, the ID is the pkid, which is the internal Cisco UCM database ID.

For users, a sync builds up the list of device/cucm/Users in Automate and then requests from the Cisco UCM the lists of users it currently has for the comparison. Differences in the lists are handled based on the sync type.

Related topics

Data Sync Overview in the Core Feature Guide

Cisco UCM Change Notification Sync (CNF) in the Core Feature Guide

Pull from Device

In a Pull from Device sync type, the Automate resource is updated where the same key is present in both lists. In this case, the device data is the master and the Automate system model data is updated with the device data.

For example, let's say new data is added to the Cisco UCM, so that the Automate system data state for a Cisco UCM device/cucm/User does not show instances that are shown on the Cisco UCM.

In this case, a pull data sync synchronizes the system data with the Cisco UCM data. For example, a user's Department may be updated on the Cisco UCM, but the update only shows on the system after a Pull from Device sync. If a user resource is created in Cisco UCM but not in Automate, this adds the device/cucm/User instance into Automate at the level the pull sync was run from, for example, at the customer level.

When deleting a Automate resource from the device, so that the key is in the Automate list but not in the device list, a pull sync removes the resource in Automate. For example, if the resource is a user in Automate but not in Cisco UCM, the pull sync removes the device/cucm/User record in Automate.

To restrict the number of records removed in Automate, ensure you have the following named macro at the hierarchy where the sync takes place:

PULL_SYNC_DELETE_THRESHOLD_<device_type>

For details, see Pull Sync Delete Threshold topic in the Advanced Configuration Guide.

When pulling device data, for example LDAP users from an LDAP device, the results returned to Automate depend on the LDAP server configuration. For example, if the returned results exceed the LDAP server configured maximum, and if the server does not support paging, an appropriate error message is returned.

For Microsoft 365 syncs, a Max page size (default 1000) setting can be adjusted if the error "Template output exceeded memory limit" is shown.

For details, see Microsoft tenant setup in the Core Guide.

Push to Device

The Push to Device sync type (available only to Cisco UCM device types), synchronizes devices with the Automate system data state, which is the primary data state.

• When deleting device data from Automate so that the key is in the device list but not in the Automate list (for example, delete user in Automate), the user is removed from Cisco UCM. The user will not exist on the device or on Automate.
• When adding new device data to Automate so that the resource shows instances that are not shown on the device, a push data sync synchronizes the device data with the Automate data. For example, adding a device/cucm/User instance to Automate and running a Push to Device sync adds the user record to Cisco UCM.

Keys found in both lists are ignored. Existing records are not updated in either direction.

In the device/cucm/User example, if the same user exists on both Automate and on Cisco UCM, no update occurs in either direction. Detailed settings may still not match after a Push to Device sync.

Change Notification (CNF)

The CNF sync type is available only to Cisco UCM device types.

A CNF sync type is a pull sync of changes stored in the local collection that is updated by the Change Notification Collector service.

For more details on Change Notification Sync, see the related topics in Data Sync section of the Core Feature Guide.

Purge Local Resources

A Purge Local Resources sync type deletes all resources or instances of device information that exists in the system are deleted. Entities in the device are not deleted.

This sync type is typically used when cleaning up the system. The system displays a warning before executing an enabled purge sync.

See the following sample device type syncs:

• HcsPurge-{{CUCMHostname}}--{{CUCMClusterName}}-DS
• HcsUserPurgeDS-{{CUCMHostname}}--{{CUCMClusterName}}
• HcsPhonePurgeDS-{{CUCMHostname}}--{{CUCMClusterName}}
• HcsPurge-{{CUCXHostname}}--{{CUCXClusterName}}-DS
• PurgeUccx-{{UCCXHostName}}
• HcsLdapUserPurge--{{UniqueID}}
• PurgeSpark{{CustomerName}}

Full sync

Full HTML Help

A full pull sync, when it runs, empties the changes from the data collection as they don't need to be processed by the Change Notification sync. Use the disabled operations and the model type list (MTL) of the full sync to filter the changes to remove. If a model instance filter is included, no changes are removed.

It is recommended to use the CUCXN Exclude ImportUser MTL on Cisco Unity Connection data syncs in order to avoid unneeded data and slowing the sync time.

Manually run the default data sync

Full HTML Help

You can always manually run the default data sync when there have been updates to Cisco Unified Communications Manager (UCM) or Cisco Unity Connection (CUC) devices that need to be synced into Automate.

1. Log in as provider or reseller administrator.
2. Go to Perform Publisher Actions.
3. From the Action drop-down, choose Import.
4. From the App Type drop-down, choose CUCM Device or CUC Device.
5. From the Clusters Available box, choose the device, move it to the Selected box, and click Save.

Create a custom data sync

Full HTML Help

Create a custom data sync to use a targeted model type list.

1. Log in as provider admin or higher.

2. Go to the Data Sync page.

3. Click the Plus icon (+) to add a new record.

4. Fill out a name for the data sync in the Name field.

   It is recommend to use a naming convention that makes it easy to identify the data syncs in the list view, such as C1Pull-CUCM01-DS, where:

   • C1 is the customer name
   • Pull is the data sync type
   • CUCM01 is the name of the Cisco UCM
   • DS is the acronym for Data Sync

   You could also include the type of data included in the sync, such as C1Pull-CUCM01-MediaResources-DS.

5. From the Device Type drop-down, choose the device type you're syncing from.

6. From the Sync Type drop-down, choose Pull from Device.

7. From the Dependency Resolution drop-down, choose Default.

8. Select the Execute Asynchronously and Refresh Existing (Changed) Data checkboxes.

   Execute Asynchronously means that the sync request will return a reply before it's complete when executed from the API. Refresh Existing (Changed) Data means that all instances of the device models specified in the model type list will be updated.

9. Select the Force Refresh of Data checkbox if a data update is required regardless of whether data has changed on the device. This option would for example be used if it is required that update workflows be run upon a data sync.

10. From the Model Type List drop-down, choose the targeted model type list you defined earlier.

11. Leave Synchronization Order and Model Instance Filter blank.

12. Click the Plus icon (+) adjacent to Device Filters to add an entry to the list.

    1. From the Attribute Name drop-down, choose host.
    2. From the Condition drop-down, choose Equals.
    3. From the Value drop-down, choose the hostname/IP address of the device.

    Note

    Workflows can be added to, and executed by a custom data sync to perform specific data sync operations.

13. In the Workflows section, include workflows in the custom data sync if you want to perform specific data sync operations, otherwise leave the Workflows section empty. For example, if you want to move remote destinations from the customer hierarchy level to the site level, choose the RD_Overbuild_PWF_wrapper workflow from the Workflow drop-down.

    If the Synchronous checkbox is enabled, the workflows will execute in sync with data sync transaction, so that the parent pre- and post workflow steps are honoured. If disabled, the workflows will not be child transactions of the data sync execute transaction.

14. From the Transaction Log Level drop-down, choose the log level for the data sync. For a description of the list of log levels, see Transaction Log Levels. The default log level is Warning.

    You can for example reduce the log level for PULL device syncs in order to reduce the size of transaction logs. This is useful where large numbers of transactions are archived regularly.

15. Click Save.

Next steps

To run the custom data sync, click the data sync from the Data Sync list and click Execute.

Cisco UCM Change Notification (CNF) Sync

Full HTML Help

Overview

Partners can use Cisco UCM's Change Notification (CNF) functionality to process update sync operations faster. This feature is disabled by default. You can enable it if required.

This topic provides guidelines for setting up a sync schedule, and lists associated performance implications.

These guidelines are derived from concepts related to total processing capacity. The total number of updates processed in a time period is the sum of all updates across the customers selected for update in that time period. In this case, the time period is one hour. In this example, it is assumed that each customer has 1000 UCM-related changes in that hour.

Recommendations provided in the following table indicates that 5 customers can run in parallel (concurrently), and therefore a total of 5,000 changes processed in total.

Partners exceeding the recommendation of 5 concurrent customers may notice a performance degradation, and the full set of required changes may not complete within that hour. Alternatively, if the number of changes for any customer is significantly higher than 1,000, or if the total number of changes is significantly greater than 5,000, the supported concurrency number may be less than 5.

If some of the planned changes do not complete within the hour noted in the table, those changes are completed the next time that particular customer is scheduled for a sync.

If the number of changes for any customer is so large that the changes continually exceed those that can be processed in one hour, it will eventually result in a full sync. For such customers, it is recommended to schedule within an hour where less than 5 customers execute concurrently.

Staggering CNF syncs across customers

This section provides an example and considerations for change notification (CNF) syncs across customers.

If a partner has twenty customers who want to use CNF sync, only schedule a maximum of five CNF syncs to run concurrently. This means that syncs would run as follows:

• 1st hour, for example 12:00

  Customer 1 to Customer 5

• 2nd hour, for example 13:00

  Customer 6 to Customer 10

• 3rd hour, for example 14:00

  Customer 11 to Customer 15

• 4th hour, for example 15:00

  Customer 16 to Customer 20

• 5th hour, for example 16:00 (Begin repeating customers)

  Customer 1 - Customer 5

• and so on.

The preceding example means that the CNF sync schedule per customer must run at 4 hour intervals. Therefore, there are 6 CNF syncs per customer within a 24 hour window. With each CNF sync processing up to 1k changes, there are:

• A total of 6k changes processed per customer in a 24 hour window
• A total of 120k changes processed across all 20 customers in a 24 hour window

CNF components

Automate's interaction with the Cisco Unified Communication Manager (Cisco UCM) Change Notification Feature (CNF) sync has two primary components:

The Automate data collector retrieves change records from Cisco UCM based according to the predefined interval. By default, this is every 300 seconds (3 minutes).

When running a CNF sync, Automate processes the change records that are collected as follows:

Update syncs perform GET API calls only for changed records, which reduces the time it takes to run the sync. This is particularly beneficial for large UC installations.

With a data collector polling period of 300 seconds and a CNF sync scheduled for every 24 hours, the process is as follows:

• Every 300 seconds (5 minutes) the polling collector retrieves all current changes from Cisco UCM.

• The polling repeats every 5 minutes, and updates the Automate cache.

• After 24 hours, the CNF sync runs, and processes all the changes that Automate stored over that 24hr period.

  The sync duration depends on the number of changes to process, since each changed object requires an AXL GET API request. A GET AXL request is only required for objects changed in the time between syncs.

For example, on a system with 10 000 users, if 100 of these users are changed, then only 100 GET AXL request are required in a CNF sync. By contrast, a non-CNF sync would require 10 000 GET AXL requests to update these 100 users.

The Automate data collection can store up to 200,000 changes from a single Cisco UCM cluster.

• A system warning is triggered when the data collection storage reaches 75% of capacity

• A sync error is triggered when the 200 000 changes capacity is reached - see Errors and Troubleshooting Change Notification Processes

  To prevent the sync error, it is recommended that you have a regular, scheduled CNF sync running.

Provides data synchronization capabilities with devices (eg. CUCM, CUC, LDAP)

Model Details: data/DataSync