Troubleshooting Change Notification (CNF)#

Overview#

A number of scenarios may result in error conditions in the change notification (CNF) process. In this case, VOSS Automate is able to display alerts automatically, which means it won’t be necessary to configure change notification (CNF) alerts manually.

Administrators can view the alerts at the hierarchy level they log in at and all the levels below that hierarchy. For example, if an alert is raised at the customer level (sys.hcs.provider.reseller.c1), then the provider, reseller, and customer administrators can see that alert, but not the site administrators. All the administrators have read and delete permissions to the alerts.

When a CNF alert is raised, the Notifications indicator on the Admin Portal GUI displays the alert. Clicking the notifications launches a dialog and a message that alerts have been raised. Click on the message to be able to go to the list of alert messages, which are also accessible via (default menus) Administration Tools > Alerts.

Properties for CNF Alerts#

CNF alerts have these properties:

  • ID: A generated identifier of the target device of the collector For Unified CM, the ID shows the host name, port, and hierarchy.

  • Code: An error or warning code associated with the alert.

  • Alert category: The category of the alert - Device Change Notification Collector

  • Severity: VOSS Automate displays severity codes and messages as follows (“{}” indicate device or number placeholders in the messages). Each alert has some properties, for example, severity (Error, Warning or Info), the number of times that the same alert has been raised, and the time stamp of the last alert instance.

  • Message: Displays error message description and the statement to fix the error.

  • Count: Displays the number of times the alert has occurred for a specific device.

  • Latest Alert: Displays the last time this alert occurred.

Note

Administrators can also filter alerts by any of the alert fields.

Error Scenarios for CNF Alerts#

VOSS Automate displays change notification alerts for the following error scenarios:

Important

Errors codes not discussed in this section may be relevant for alerts that are more internal, and you may need to raise a support ticket for further investigation.

  • Warning:

    • 45000: Unprocessed changes at 75% of limit for device {}. Please configure and run the necessary data syncs.

  • Error:

    • 40000: Device change notifications are not supported for device {}.

    • 40001: Device change notification data for device {} has been lost. Tracking data has been repaired and collector process will continue. Some changes may have been lost, please run a full sync on the device.

    • 40002: Device change notification tracking data for device {} has become corrupted. Tracking data has been repaired and collector process will continue. Some changes may have been lost, please run a full sync on the device.

    • 40003: Device change notification tracking DB write for device {} failed. The collector process will continue to attempt DB writes. Please investigate the database write failure.

    • 40004: Device change notification data DB write for device {} failed. The collector process will continue to attempt DB writes. Please investigate the database write failure.

    • 40005: Unable to repair device change notification tracking data for device {}.

    • 40006: Too many unprocessed changes recorded for device {}. No new changes will be recorded until at least {} changes are processed. Please configure and run the necessary data syncs.

    • 40008: Could not update pending changes data for device {}. {}.

The administrator reads, inspects, acts on (for example, run a full sync on the device), and then manages alerts of the Change Notification collection service. The administrator can delete the alert from the list only when the issue that raised the alert has been resolved.

Note

If the Administrators forget to remove the change notification feature alert after resolving it, the alert will still be shown when they log in to VOSS Automate. We strongly recommend removing the alert after resolving it.

Change Cache Full on CUCM#

If the CUCM maximum number of stored change records is exceeded, the CUCM drops the oldest changes that have not been collected. This can happen if the polling time in VOSS Automate is set up to be too long or the CUCM is experiencing a very high level of changes. In this case, the system automatically attempts to recover once it receives the polling error. This activity is logged as an Alert in the system and provides the outcome, either of the following:

  • recovery was successful (alert code 40001 or 40002)

  • recovery was not successful (alert code 40005)

If recovery is successful, you may want to review and consider a full sync as some changes would have been lost (the oldest changes in the CUCM cache).

If recovery was not successful, a full sync is required to update and to get CNF functioning again. The full sync is needed as changes would have been missed from the CUCM, and a clean sync is required in order to start processing changes again. In this situation, application info log messages are logged as well - “Repaired change notification tracking data for device {}” or “Unable to repair change notification tracking data for device {}”

Change Collection Full for a CUCM Cluster#

If the VOSS Automate change collection for a given CUCM cluster exceeds the maximum changes - 200,000 - then an alert with code 40006 is raised. This alert means that no further changes are collected from the CUCM until some of the pending changes are processed. This can be carried out by an administrator executing a sync for that CUCM cluster to clear some of the changes. If the next scheduled sync is not too far ahead in time, then waiting for the next scheduled sync to run may be acceptable.