.. _cisco-ucm-cnf-syncs: Cisco UCM Change Notification (CNF) Sync -------------------------------------------- .. _25.4|EKB-27467: Overview ......... Partners can use Cisco UCM's *Change Notification* (CNF) functionality to process *update sync* operations faster. This feature is disabled by default. You can enable it if required. This topic provides guidelines for setting up a sync schedule, and lists associated performance implications. .. note:: Further details for using *CNF* is provided elsewhere in the Automate documentation. The changes described here are not transactions so details don't display in the transaction log, but in special logs created for CNF. These guidelines are derived from concepts related to total processing capacity. The total number of updates processed in a time period is the sum of all updates across the customers selected for update in that time period. In this case, the time period is one hour. In this example, it is assumed that each customer has 1000 UCM-related changes in that hour. Recommendations provided in the following table indicates that 5 customers can run in parallel (concurrently), and therefore a total of 5,000 changes processed in total. Partners exceeding the recommendation of 5 concurrent customers may notice a performance degradation, and the full set of required changes may not complete within that hour. Alternatively, if the number of changes for any customer is significantly higher than 1,000, or if the total number of changes is significantly greater than 5,000, the supported concurrency number may be less than 5. If some of the planned changes do not complete within the hour noted in the table, those changes are completed the next time that particular customer is scheduled for a sync. If the number of changes for any customer is so large that the changes continually exceed those that can be processed in one hour, it will eventually result in a full sync. For such customers, it is recommended to schedule within an hour where less than 5 customers execute concurrently. +------------------------------------------+----------------------------------------------------------------------------------------------+ | Configuration | Recommendation | +==========================================+==============================================================================================+ | Maximum number of concurrent CNF sync | 5 | +------------------------------------------+----------------------------------------------------------------------------------------------+ | Maximum number of changes processed per | 1,000 | | CNF sync | | +------------------------------------------+----------------------------------------------------------------------------------------------+ | CNF sync schedule frequency | Once per hour per customer - This is subject to the staggering of CNF sync across customers. | +------------------------------------------+----------------------------------------------------------------------------------------------+ | Staggering of CNF syncs across customers | Factor of maximum changes processed and | | | maximum number of concurrent CNF syncs. | +------------------------------------------+----------------------------------------------------------------------------------------------+ | CNF collector frequency | Initial recommendation is 15 minutes. | +------------------------------------------+----------------------------------------------------------------------------------------------+ | When is full sync required? | Weekends only or when there are CNF | | | alerts prompting for full sync. | +------------------------------------------+----------------------------------------------------------------------------------------------+ .. note:: It is recommended that you disable CNF if you experience significant performance degradation. Contact your support representative if you have any performance concerns. Staggering CNF syncs across customers ........................................ This section provides an example and considerations for change notification (CNF) syncs across customers. If a partner has twenty customers who want to use CNF sync, only schedule a maximum of five CNF syncs to run concurrently. This means that syncs would run as follows: * 1st hour, for example 12:00 Customer 1 to Customer 5 * 2nd hour, for example 13:00 Customer 6 to Customer 10 * 3rd hour, for example 14:00 Customer 11 to Customer 15 * 4th hour, for example 15:00 Customer 16 to Customer 20 * 5th hour, for example 16:00 (Begin repeating customers) Customer 1 - Customer 5 * and so on. The preceding example means that the CNF sync schedule per customer must run at 4 hour intervals. Therefore, there are 6 CNF syncs per customer within a 24 hour window. With each CNF sync processing up to 1k changes, there are: * A total of 6k changes processed per customer in a 24 hour window * A total of 120k changes processed across all 20 customers in a 24 hour window .. _voss-cnf-overview: CNF components ................. Automate's interaction with the Cisco Unified Communication Manager (Cisco UCM) Change Notification Feature (CNF) sync has two primary components: .. tabularcolumns:: |p{6cm}|p{9cm}| +--------------------------+-------------------------------------------------------------------+ | Component | Description | +==========================+===================================================================+ | Data Collector | Collects changes from Cisco UCM and updates the VOSS cache on the | | | predefined frequency (defaults to every 300 seconds). | | | | | | This collector must be enabled to collect the changes, otherwise | | | the sync won't process any changes. | +--------------------------+-------------------------------------------------------------------+ | Change Notification Sync | A change notification (CNF) sync type that processes the changes | | | changes the data collector places in the VOSS cache. | | | | | | A scheduled sync must be set up and enabled so that the changes | | | are processed within a reasonable period. | | | | | | The CNF sync can also be run adhoc, if required, around the | | | the schedule. | +--------------------------+-------------------------------------------------------------------+ The Automate data collector retrieves change records from Cisco UCM based according to the predefined interval. By default, this is every 300 seconds (3 minutes). When running a CNF sync, Automate processes the change records that are collected as follows: .. tabularcolumns:: |p{6cm}|p{9cm}| +---------------+-------------------------------------------------------------------+ | Operation | Description | +===============+===================================================================+ | Add | Performs a GET API call to retrieve the full record, and adds it | | | to Automate. | +---------------+-------------------------------------------------------------------+ | Update | Performs a GET API call to retrieve the full record, and updates | | | updates the record in Automate. | +---------------+-------------------------------------------------------------------+ | Del | Removes the record from Automate. | +---------------+-------------------------------------------------------------------+ Update syncs perform GET API calls only for changed records, which reduces the time it takes to run the sync. This is particularly beneficial for large UC installations. With a data collector polling period of 300 seconds and a CNF sync scheduled for every 24 hours, the process is as follows: * Every 300 seconds (5 minutes) the polling collector retrieves all current changes from Cisco UCM. * The polling repeats every 5 minutes, and updates the Automate cache. * After 24 hours, the CNF sync runs, and processes all the changes that Automate stored over that 24hr period. The sync duration depends on the number of changes to process, since each changed object requires an AXL GET API request. A GET AXL request is only required for objects changed in the time between syncs. For example, on a system with 10 000 users, if 100 of these users are changed, then only 100 GET AXL request are required in a CNF sync. By contrast, a non-CNF sync would require 10 000 GET AXL requests to update these 100 users. The Automate data collection can store up to 200,000 changes from a single Cisco UCM cluster. * A system warning is triggered when the data collection storage reaches 75% of capacity * A sync error is triggered when the 200 000 changes capacity is reached - see :ref:`errors-troubleshooting-CNF-processes` To prevent the sync error, it is recommended that you have a regular, scheduled CNF sync running. .. _detailed-CUCM-CNF-functionality: Cisco UCM and CNF '''''''''''''''''' The Cisco UCM CNF capability supports all the objects that are available via AXL. Typically, this means that everything Automate can manage in Cisco UCM is available via change notification. Data that Automate pulls from Cisco UCM that is *not* via AXL, includes: * ``device/cucm/PhoneType`` - this is a combination of thinAXL so would not auto-update. This includes when you add or update phone types in Cisco Unified CM with COP files or via Cisco Unified CM upgrades. So a non-CNF sync is still required for this model. * Phone Status and IP Address - this is pulled into the system when the phones are viewed in Automate (list view or individual phone).  This is not via AXL so would not be updated via change notification or even a normal sync at this point. In Cisco UCM, the change queue cache is stored in memory and is limited to 100,000 changes. The cache can fill quickly depending on the types of changes performed. For example, if an XSI (IP Phone) Service has been configured for 10,000 phones and the service is deleted, the cache will include one entry representing the deletion of the service plus 10,000 phone updates indicating the service was removed from each device. The polling period from the Cisco Unified CM is configurable and the timing should be considered based on how frequent configuration changes are being made in Cisco Unified CM. The default in Automate when polling is enabled is 300 seconds but it can be modified to be longer (up to 7200 seconds) as desired. .. _CUCM-setup-for-CNF: Cisco UCM setup to use CNF ........................... There are two settings in the Cisco Unified Communications Manager (Cisco UCM) to check and update to ensure change notification (CNF) is enabled and set up for the right queue size (accessed via service parameters: - **System > Service Parameters > Cisco Database Layer Monitor** then click the **Advanced** button): .. tabularcolumns:: |p{6cm}|p{9cm}| +-------------------------------+-----------------------------------+ | Service Parameter Name | Setting | +===============================+===================================+ | AXL Change Notification | This should be set to "On" | +-------------------------------+-----------------------------------+ | AXL Change Notification Queue | This has a default of 20000. For | | Size | a typical system, it is suggested | | | this is changed to the maximum of | | | 100000 to reduce the chance of | | | changes being missed under heavy | | | provisioning tasks. | +-------------------------------+-----------------------------------+ .. _setup-enable-disable-CNF-CUCM: Enable / disable CNF syncs for a Cisco UCM cluster ................................................... This section describes the high level steps for enabling change notification (CNF) syncs for a Cisco Unified Communications Manager (Cisco UCM) cluster in Automate: .. note:: Perform these steps in reverse order to disable change notification for the cluster. 1. Enable and configure the service in Cisco UCM - see :ref:`CUCM-setup-for-CNF` #. Enable change notification on the Cisco UCM cluster in Automate - see :ref:`detailed-VOSS-Automate-CNF-functionality` #. Review the change notification settings for the cluster - see :ref:`detailed-VOSS-Automate-CNF-functionality` #. Review or create the required data sync instances for change notification, for the cluster. .. raw:: latex See Introduction to data sync in the Core Feature Guide .. raw:: html See Introduction to data sync .. note:: The actual number of syncs and their setup will depend on the needs for your system and the design. See the Best Practices Guide for guidance on sync logic and recommended setups. If further recommendations or guidance is needed, contact your VOSS account team or VOSS support. #. Review or create required schedules for the data sync(s) created in the previous step, and activate the schedule(s). See :ref:`enable-a-scheduled-data-sync`. When activating the schedule, select the **Multiple Executions** mode(s) you require (**Specific**, **Calendar**, or **Timed**). Any modes not selected are considered disabled and do not need to be toggled. .. note:: Follow the guidance for scheduling around syncs to ensure the load on the system is optimized. At least one sync schedule should be activated for the CNF sync setup to be complete. .. _detailed-VOSS-Automate-CNF-functionality: Automate change notification ............................... .. _21.3-PB3|EKB-12307: This section provides more details on the functionality of the Change Notification Feature (CNF) components in Automate. Change notification collector '''''''''''''''''''''''''''''''' The ``data/DeviceChanges`` model has an instance per UCM cluster and will provide the data collector status and pending changes in the cache for that cluster. An instance of the model will appear for all UCM clusters whether change notification is enabled or not. It gives you access to: * **Base** tab * **Last Collection Time** - time the changes were last collected from the device * **Pending Change Notifications** - a view of pending changes collected for different types of models and by type of change (Add/Update/Delete). By default, this is ``device/cucm/User``, ``device/cucm/Line`` and ``device/cucm/Phone``. However, these models are adjustable on the **Settings** tab. If additional model types to the defaults above are added to the list, these are also shown. The remaining model types are all grouped in a single row called ``Other``. * **Settings** tab * **Polling Interval (seconds)** (300-7200 seconds with default 300) - the duration for collection of changes from the device * **Enable Change Collection** - enable or disable the change collector for that device. * **Ignored Operations** - you can select certain operations (Add/Mod/Del) to not be collected. Typically you want to collect all changes; however this option can be used to ignore some changes for specific scenarios if needed (for example, you will only handle updates via the CNF sync). * **Displayed Model Types** - here you can configure which models you want to see summary stats for on the **Base** tab. You can add, remove or change models to meet specific needs (for example, ``deviceProfile`` for extension mobility profiles, ``remoteDestination`` for SNR remote destinations, and so on). The ``data/DeviceChanges`` model should be included in menu layouts for roles that need access to this level of detail for the CNF syncs. Change notification sync type '''''''''''''''''''''''''''''''' When a Change Notification sync type is used in a data sync, there are a number of differences in the sync behavior in comparison with a normal pull sync: * A GUI portal rule on the Data Sync interface will change some of the settings visible on the Data Sync GUI interface when the **Sync Type** is set to Change Notification Sync. This selection hides settings that are not relevant and exposes new settings for this type only. * **Number of Changes to Process** - This input field becomes available from the Data Sync interface. Leaving the input box blank or typing in 0 will mean the sync will process all the pending changes collected - subject to the selected model type list and Disabled Operations set up on the sync. If you enter a number, the sync will process that number of changes only and leave any additional changes in the change collection for the next sync. Typically this value should be 0 or blank, unless there is a specific reason to limit the number of changes to process, for example when managing how long the sync may run. * A **Model Type List** (MTL) can be set up and selected to be associated with UCM change notification collection. This list allows a user to whitelist/blacklist certain model types from the Call Manager change notification collector service so that change notifications which are not in the MTLs do not accumulate and possibly trigger the maximum changes counter which prevents any new changes from being collected. All other visible settings are the same as with a normal pull sync, for example, device filters, workflows, and so on. When a sync runs (either a normal pull sync or a change notification sync), it will clear out the change notification collection of any model types and changes processed for that cluster. The model type lists and disabled operations define which models and types of actions are processed in either a pull or a change notification sync: * The model type list (if one is assigned) assigned to the sync will determine which model type changes will be processed from the collected changes (for example, ``device/cucm/User`` for user entities only). * The **Disabled Operations** tab defines if any of the types of changes are ignored. For example, selecting **Remove** will ignore delete changes. Pull Sync and Change Notification Sync details: * A pull sync does not utilize the change notification collection as a source of data. However, it will clear the collection for the models types it processes. * A *full* pull sync (a pull sync without a model type list) will clear the change collection as part of the sync process since it is pulling *all* the latest information from the UCM. * A pull sync with a model type list defined (for example one that contains ``device/cucm/User``) will clear the change collection of any ``device/cucm/User`` changes, since it is syncing all the user information anyway. All other model types and changes will be left in the collection. * If a pull sync is run with **Disabled Operations** selected (for example, **Add** is selected) this will process the pending changes for Update and Delete actions for any matching models. However, *all* actions for the matching model will be cleared from the cache, *including Add actions*. * A Change Notification Sync utilizes the change collection as its source of information and will clear that changes from the change collection for any model types it is processing. * A *full* Change notification sync (CNF sync without a model type list) will process *all* the pending changes and clear the change collection (unless limited by a value in the **Number of Changes to Process** setting on the sync. Then only that number of changes will be processed and cleared). * A change notification sync with a model type list defined (for example contains ``device/cucm/User``) will process all the pending changes for the ``device/cucm/User`` model type and clear those from the change collection. * If a change notification sync is run with **Disabled Operations** (for example, **Add** is selected), it will process the Update and Delete changes for the matching models. However, *all* actions for the matching model(s) will be cleared from the cache, *including Add actions*. This sync behavior means that you may wish to set up multiple syncs for a cluster to handle different types of sync and sync schedules to meet your needs. Ensure that you generally have all the model types covered in your scheduled syncs if CNF is enabled, otherwise some changes may never be cleared from the change collection, thereby taking up space. For additional considerations and information around sync setup best practices, see the Best Practices Guide. Setup to enable change notification ................................................... Enabling the Cisco UCM Change Notification Feature (CNF) capability is completed on a per Cisco UCM cluster basis. This can be done on the Cisco UCM Server configuration page for a publisher via the **Publisher** tab and selecting the **Enable Change Notification Sync** check box. When selected and saved, the system will: * Enable the data collector for that cluster * Create a CNF sync type for the cluster * Create a schedule for the CNF sync (disabled by default). * These settings should all be reviewed, adjusted, or additional instances created to meet your needs. See further information in: * The Best Practices Guide * The System Monitoring Configuration section in the Advanced Configuration Guide on sync best practices for different scenarios and other considerations. .. note:: The auto-created CNF schedule runs based on the **Multiple Executions** options you select when you activate it. Options that are not selected are treated as off. You do not need to toggle **Use Specific Executions** or **Use Calendar Executions** on/off to make the schedule run. Select only the options you intend to use. For more detail, see :ref:`scheduling`. * A full sync with the Cisco UCM cluster should be executed just before or after enabling CNF for the cluster. This can be part of changing the setting for an existing cluster or adding a new publisher. Currently, both actions will invoke a full sync of the cluster. However, if the sync is not completed during the add/modify of the publisher, then one should be initiated. When CNF is disabled on the Publisher configuration page (or if the cluster is removed from the system), the following will occur: * The auto-generated schedule that was added during enabling will be removed. Any additional custom scheduled added will not be removed automatically and should be removed before disabling the change notification for the cluster to avoid unnecessary syncs running. * The auto-generated CNF sync type that was added during enabling will be removed. Any additional custom CNF sync types for the cluster added will not be removed automatically and should be removed before disabling the change notification for the cluster to avoid unnecessary syncs being set up. * The data collector for the cluster will be disabled. .. note:: If the collector is only disabled via the ``data/DeviceChanges`` model, then the schedules and sync will remain. This is the best approach if you need to temporarily disable the CNF sync (for example, for a maintenance window). .. rubric:: Related topics * For guidance on configuring execution modes and understanding **Last Executed (UTC Time)**, see :ref:`scheduling`. .. _errors-troubleshooting-CNF-processes: Troubleshooting Change Notification (CNF) ............................................... .. tip:: **Quick check for CNF schedules**: If a CUCM CNF schedule appears enabled but changes are not being processed as expected, verify: * The CNF schedule is **Active** and has at least one **Multiple Executions** option selected (**Specific**, **Calendar**, or **Timed**). * The next execution time is in the future (or activate with **Skip execution on activation** if you want to avoid an immediate catch-up run). * The CNF **collector** is enabled and is accumulating changes (see *Change notification collector*). You do not need to toggle **Use Specific Executions** or **Use Calendar Executions** to make the schedule run. A number of scenarios may result in error conditions in the change notification (CNF) process. In this case, the system is able to display alerts automatically, which means it won't be necessary to configure change notification (CNF) alerts manually. Administrators can view the alerts at the hierarchy level they log in at and all the levels below that hierarchy. For example, if an alert is raised at the customer level (sys.hcs.provider.reseller.c1), then the provider, reseller, and customer administrators can see that alert, but not the site administrators. All the administrators have read and delete permissions to the alerts. When a CNF alert is raised, the Notifications indicator on the Admin Portal GUI displays the alert. Clicking the notifications launches a dialog and a message that alerts have been raised. Click on the message to be able to go to the list of alert messages, which are also accessible via the **Alerts** page. Properties of CNF alerts '''''''''''''''''''''''''''''' CNF alerts have these properties: * ID: A generated identifier of the target device of the collector For Unified CM, the ID shows the host name, port, and hierarchy. * Code: An error or warning code associated with the alert. * Alert category: The category of the alert - Device Change Notification Collector * Severity: Automate displays severity codes and messages as follows ("{}" indicate device or number placeholders in the messages). Each alert has some properties, for example, severity (Error, Warning or Info), the number of times that the same alert has been raised, and the time stamp of the last alert instance. * Message: Displays error message description and the statement to fix the error. * Count: Displays the number of times the alert has occurred for a specific device. * Latest Alert: Displays the last time this alert occurred. .. note:: Administrators can also filter alerts by any of the alert fields. Error scenarios for CNF alerts '''''''''''''''''''''''''''''''' Automate displays change notification alerts for the following error scenarios: .. important:: Errors codes not discussed in this section may be relevant for alerts that are more internal, and you may need to raise a support ticket for further investigation. * Warning: * 45000: Unprocessed changes at 75% of limit for device {}. Please configure and run the necessary data syncs. * Error: * 40000: Device change notifications are not supported for device {}. * 40001: Device change notification data for device {} has been lost. Tracking data has been repaired and collector process will continue. Some changes may have been lost, please run a full sync on the device. * 40002: Device change notification tracking data for device {} has become corrupted. Tracking data has been repaired and collector process will continue. Some changes may have been lost, please run a full sync on the device. * 40003: Device change notification tracking DB write for device {} failed. The collector process will continue to attempt DB writes. Please investigate the database write failure. * 40004: Device change notification data DB write for device {} failed. The collector process will continue to attempt DB writes. Please investigate the database write failure. * 40005: Unable to repair device change notification tracking data for device {}. * 40006: Too many unprocessed changes recorded for device {}. No new changes will be recorded until at least {} changes are processed. Please configure and run the necessary data syncs. * 40008: Could not update pending changes data for device {}. {}. The administrator reads, inspects, acts on (for example, run a full sync on the device), and then manages alerts of the Change Notification collection service. The administrator can delete the alert from the list only when the issue that raised the alert has been resolved. .. note:: If the Administrators forget to remove the change notification feature alert after resolving it, the alert will still be shown when they log in to Automate. We strongly recommend removing the alert after resolving it. Change Cache Full on Cisco UCM '''''''''''''''''''''''''''''''' If the Cisco UCM maximum number of stored change records is exceeded, the Cisco UCM drops the oldest changes that have not been collected. This can happen if the polling time in Automate is set up to be too long or the Cisco UCM is experiencing a very high level of changes. In this case, the system automatically attempts to recover once it receives the polling error. This activity is logged as an alert in the system and provides the outcome, either of the following: * recovery was successful (alert code 40001 or 40002) * recovery was not successful (alert code 40005) If recovery is successful, you may want to review and consider a full sync as some changes would have been lost (the oldest changes in the Cisco UCM cache). If recovery was not successful, a full sync is required to update and to get CNF functioning again. The full sync is needed as changes would have been missed from the Cisco UCM, and a clean sync is required in order to start processing changes again. In this situation, application info log messages are logged as well - "Repaired change notification tracking data for device {}" or "Unable to repair change notification tracking data for device {}" Change Collection Full for a CUCM Cluster '''''''''''''''''''''''''''''''''''''''''''' If the Automate change collection for a given Cisco UCM cluster exceeds the maximum changes - 200,000 - then an alert with code 40006 is raised. This alert means that no further changes are collected from the Cisco UCM until some of the pending changes are processed. This can be carried out by an administrator executing a sync for that Cisco UCM cluster to clear some of the changes. If the next scheduled sync is not too far ahead in time, then waiting for the next scheduled sync to run may be acceptable.