Errors and Troubleshooting Change Notification Processes¶
A number of scenarios may result in error conditions in the change notification process. The VOSS Automate system is enabled to display alerts automatically in this case, so that it is not necessary to configure the change notification feature (CNF) alerts manually.
Administrators can view the alerts at the hierarchy level they log in at and all the levels below that hierarchy. For example, if an alert is raised at the customer level (sys.hcs.provider.reseller.c1), then the provider, reseller, and customer administrators can see that alert, but not the site administrators. All the administrators have read and delete permissions to the alerts.
When a change notification feature alert is raised, the Notifications indicator on the VOSS Automate Admin Portal shows the alert. Clicking the Notifications button shows a pop-up and a message that alerts have been raised. By clicking on the message, users are directed to the list of alert messages which can also be accessed via the menu under Administration Tools > Alerts.
CNF alerts have the following distinct properties:
ID: A generated identifier of the target device of the collector For Unified CM, the ID shows the host name, port, and hierarchy.
Code: An error or warning code associated with the alert.
Alert category: The category of the alert - Device Change Notification Collector
Severity: VOSS Automate displays severity codes and messages as follows (“{}” indicate device or number placeholders in the messages). Each alert has some properties, for example, severity (Error, Warning or Info), the number of times that the same alert has been raised, and the time stamp of the last alert instance.
Message: Displays error message description and the statement to fix the error.
Count: Displays the number of times the alert has occurred for a specific device.
Latest Alert: Displays the last time this alert occurred.
Note
Administrators can also filter alerts by any of the alert fields.
VOSS Automate displays change notification feature alerts for the following error scenarios:
Warning:
45000: Unprocessed changes at 75% of limit for device {}. Please configure and run the necessary data syncs.
Error:
40000: Device change notifications are not supported for device {}.
40001: Device change notification data for device {} has been lost. Tracking data has been repaired and collector process will continue. Some changes may have been lost, please run a full sync on the device.
40002: Device change notification tracking data for device {} has become corrupted. Tracking data has been repaired and collector process will continue. Some changes may have been lost, please run a full sync on the device.
40003: Device change notification tracking DB write for device {} failed. The collector process will continue to attempt DB writes. Please investigate the database write failure.
40004: Device change notification data DB write for device {} failed. The collector process will continue to attempt DB writes. Please investigate the database write failure.
40005: Unable to repair device change notification tracking data for device {}.
40006: Too many unprocessed changes recorded for device {}. No new changes will be recorded until at least {} changes are processed. Please configure and run the necessary data syncs.
40008: Could not update pending changes data for device {}. {}.
The administrator reads, inspects, acts on (for example, run a full sync on the device), and then manages alerts of the Change Notification collection service. The administrator can delete the alert from the list only when the issue that raised the alert has been resolved.
Note
If the Administrators forget to remove the change notification feature alert after resolving it, the alert will still be shown when they log in to VOSS Automate. We strongly recommend removing the alert after resolving it.
Change Cache Full on Cisco Unified CM¶
If the Cisco Unified CM maximum number of stored change records is exceeded (see detailed Cisco Unified CM functionality section for more details on the limit and configuration) then the Cisco Unified CM will drop the oldest changes that have not been collected. This can happen if the polling time in VOSS Automate is set up to be too long or the Cisco Unified CM is experiencing a very high level of changes (see the detailed VOSS Automate functionality section for more details on polling configuration). When this situation occurs, VOSS Automate will get an error on polling and will try to recover. This activity is logged as an Alert in the system and provides the outcome - recovery was successful (alert code 40001 or 40002) or recovery was not successful (alert code 40005).
In the event the recovery was successful, you may want to review and consider a full sync as some changes would have been lost (the oldest changes in the Cisco Unified CM cache).
In the event the recovery was not successful, then a full sync is required to update and to get change notification functioning again. The full sync is needed as changes would have been missed from the Cisco Unified CM and we need to be at a clean sync in order to start processes changes again.
In this situation, application info log messages are logged as well - “Repaired change notification tracking data for device {}” or “Unable to repair change notification tracking data for device {}”
VOSS Automate Change Collection full for a Cisco Unified CM cluster¶
If the VOSS Automate change collection for a given Cisco Unified CM cluster exceeds the maximum changes - 200,000 - then an alert with code 40006 is raised. This alert means that no further changes are collected from the Cisco Unified CM until some of the pending changes are processed. This can be carried out by an administrator executing a sync for that Cisco Unified CM cluster to clear some of the changes. If the next scheduled sync is not too far ahead in time, then waiting for the next scheduled sync to run may be acceptable.
Other errors¶
The other error codes listed for the alerts are more internal in nature and should result in a VOSS support ticket being raised for further investigation.