Notification Messages#

There are 3 levels of notifications:

info - these are informational messages intended to communicate a positive event.
warn - these are non-critical notifications indicating that there is a temporary error
error - these are critical messages indicating a system fault that is affecting system functionality

Backups#

Category	Level	Message
Backup monitoring	error	ERROR: The last backup was more than 2 days ago …
Backup monitoring	info	INFO: Backups now runs regularly …
Backups	error	DISK_USAGE_LIMITS /backups
Backups	info	DISK_USAGE_LIMITS /backups
Backups	info	Backup completed…
Backups	info	Restore successful…
Backups	error	Restore failed ..
Backups	error	Backup failed .

Examples#

Oct 10 21:39:17 robot-slave snmptrapd[480]: 2022-10-10 21:39:17 <UNKNOWN> [UDP: [192.168.100.3]:59765->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (306489) 0:51:04.89
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "Backup completed"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Backup was successfully created at localbackup"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 0
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 21:39:27 robot-slave snmptrapd[480]: 2022-10-10 21:39:27 <UNKNOWN> [UDP: [192.168.100.3]:25962->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (307540) 0:51:15.40
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "INFO: Backups now runs regularly"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Backup list:
#012
#012    localbackup:
#012        backups:
#012            1 backups have been created - most recently 2022-10-10 19:38
#012        uri: file:///backups
#012
#012
#012
#012"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 0
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 21:39:30 robot-slave snmptrapd[480]: 2022-10-10 21:39:30 <UNKNOWN> [UDP: [192.168.100.3]:14347->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (307808) 0:51:18.08
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "ERROR: The last backup was more than 2 days ago"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Backup list:
#012
#012    localbackup:
#012        backups:
#012            1 backups have been created - most recently 2022-10-10 19:38
#012        uri: file:///backups
#012
#012
#012
#012"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Resolution:

Investigate if the scheduler is set correctly and that the backup is set up correctly. Run a manual backup to test if it is working as it should. Otherwise, call Support to investigate.

Oct 10 21:40:06 robot-slave snmptrapd[480]: 2022-10-10 21:40:06 <UNKNOWN> [UDP: [192.168.100.3]:53579->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (311359) 0:51:53.59
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "backup.py : num_sockets exeeded warning threshold of 6 with 7"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Command backup.py attribute num_sockets
#012Current average value 7
#012Error threshold 10
#012Warning threshold 6
#012"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 21:42:01 robot-slave snmptrapd[480]: 2022-10-10 21:42:01 <UNKNOWN> [UDP: [192.168.100.3]:7560->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (322901) 0:53:49.01
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "Backup completed"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Backup was successfully created at sftpbackup"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 0
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 22:17:09 robot-slave snmptrapd[480]: 2022-10-10 22:17:09 <UNKNOWN> [UDP: [192.168.100.3]:36106->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (533696) 1:28:56.96
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "Backup failed"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Backup failed - error running pre-backup method for mongodb"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 22:22:14 robot-slave snmptrapd[480]: 2022-10-10 22:22:14 <UNKNOWN> [UDP: [192.168.100.3]:21378->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (564173) 1:34:01.73
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "Backup failed"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Backup to localbackup failed to verify"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Resolution:

There may have been some form of corruption during the time that the backup was run. Run backup manually and see whether the same failure is detected. If it is the case, investigate. Otherwise, contact L2 Support to investigate.

Database in Cluster Failover#

Category	Level	Message
Cluster Failover	error	ERROR: The db is failing over constantly within 5 min …
Cluster Failover	info	INFO: The db failover status returned to normal …

Database Maintenance and Usage#

Category	Level	Message
Database Maintenance	error	ERROR: Database maintenance not scheduled…
Database Maintenance	info	INFO: Database maintenance is scheduled …
Database Usage	warn	WARN: Database %s exceeded threshold …
Database Usage	info	INFO: Database %s returned to normal …

Resolution if no schedule

Create schedule and time:

With for example a schedule name dbtxn:

schedule add dbtxn voss transaction archive or schedule add dbtxn voss transaction delete

schedule time dbtxn weekly 0

See: Enable Database Scheduling.

Examples#

Oct 10 22:43:20 robot-slave snmptrapd[480]: 2022-10-10 22:43:19 <UNKNOWN> [UDP: [192.168.100.3]:41903->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (690787) 1:55:07.87
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "WARN: Database transaction count exceeded threshold"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "count: 13332"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 22:43:24 robot-slave snmptrapd[480]: 2022-10-10 22:43:24 <UNKNOWN> [UDP: [192.168.100.3]:25484->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (691198) 1:55:11.98
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "INFO: Database transaction count returned to normal"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "The transaction count returned to normal."
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 0
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 22:43:28 robot-slave snmptrapd[480]: 2022-10-10 22:43:28 <UNKNOWN> [UDP: [192.168.100.3]:11973->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (691609) 1:55:16.09
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "WARN: Database transaction size exceeded threshold"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "size: 32264767"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Resolution for transaction size

The following commands can be used to resolve this:

voss transaction delete <days> [limit <number>]

voss transaction archive <days>

Disk Latency and Status#

Category	Level	Message
Disk Latency	error	ERROR: Disk slow ‘, ‘Disk latency info:…
Disk Latency	info	INFO: The disk latency returned to normal
Disk Status	error	DISK ALMOST FULL: Disk {} is more than {} percent full
Disk Status	warn	DISK STATUS: Disk {} is now running below {} percent. Disk %s cleared Disk status after it was cleared
Disk Status	info	DISK STATUS: Disk {} is now running below {} percent. Disk %s cleared Disk status after it was cleared:
Disk Status	error	DISK_USAGE_LIMITS /
Disk Status	error	DISK_USAGE_LIMITS /backups
Disk Status	error	DISK ALMOST FULL: Disk /var/log is more than {} percent full
Disk Status	info	DISK_USAGE_LIMITS /
Disk Status	info	DISK_USAGE_LIMITS /backups
Disk Status	info	DISK_USAGE_LIMITS /var/log
Disk Status	audit	DISK ALMOST FULL: Disk /var/log is more than {} percent full
Platform	error	DISK_USAGE_LIMITS /opt/platform
Platform	info	DISK_USAGE_LIMITS /opt/platform

Examples#

Oct 11 00:02:46 robot-slave snmptrapd[480]: 2022-10-11 00:02:46 <UNKNOWN> [UDP: [192.168.100.3]:63595->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (1167361) 3:14:33.61
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "DISK ALMOST FULL: Disk /var/log is more than 80 percent full"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Use log purge to purge all rotated logs
#012
#012Current disk status:
#012Filesystem: /dev/sdb1
#012      Size: 9.9G
#012      Used: 9.5G
#012     Avail: 0
#012      Use%: 100%
#012Mounted on: /var/log"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 11 00:03:46 robot-slave snmptrapd[480]: 2022-10-11 00:03:46 <UNKNOWN> [UDP: [192.168.100.3]:4569->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (1173418) 3:15:34.18
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "DISK STATUS:  Disk /var/log is now running below 80 percent"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Disk /var/log cleared
#012Disk status after it was cleared:
#012Filesystem: /dev/sdb1
#012      Size: 9.9G
#012      Used: 31M
#012     Avail: 9.4G
#012      Use%: 1%
#012Mounted on: /var/log"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 0
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 11 00:05:19 robot-slave snmptrapd[480]: 2022-10-11 00:05:19 <UNKNOWN> [UDP: [192.168.100.3]:36811->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (1182726) 3:17:07.26
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "ERROR: Disk slow "
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Disk latency info:
#012
#012Disk latency:34.44ms
#012"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 26 10:59:57 robot-slave snmptrapd[546]: 2022-10-26 10:59:57 <UNKNOWN> [UDP: [192.168.100.3]:33587->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (135921) 0:22:39.21
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "DISK ALMOST FULL: Disk /opt/platform/apps/mongodb/dbroot is more than 80 percent full"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Contact support to free space
#012
#012Current disk status:
#012      Use%: 92%
#012      Used: 207G
#012     Avail: 19G
#012 use_limit: 92
#012Filesystem: /dev/mapper/voss-dbroot
#012Mounted on: /opt/platform/apps/mongodb/dbroot
#012    Message is truncated"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 26 10:59:58 robot-slave snmptrapd[546]: 2022-10-26 10:59:58 <UNKNOWN> [UDP: [192.168.100.3]:48675->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (136014) 0:22:40.14
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "DISK STATUS:  Disk /opt/platform/apps/mongodb/dbroot is now running below 80 percent"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Disk /opt/platform/apps/mongodb/dbroot cleared
#012Disk status after it was cleared:
#012      Use%: 1%
#012      Used: 1.3G
#012     Avail: 224G
#012 use_limit: 1
#012Filesystem: /dev/mapper/voss-dbroot
#012Mounted on: /opt/pla  Message is truncated"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 0
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Resolution:

This trap indicates that there is heavy disk activity. This will be normal in many situations and should not always require immediate action. The one reason for this is that there are many transactions spawned at the same time. The disk utilization can be relieved by cancelling a few current transactions or by rescheduling some others.

However, this should be monitored closely and should it persist over several hours then Support should be contacted to investigate.

Oct 11 00:01:27 robot-slave snmptrapd[480]: 2022-10-11 00:01:27 <UNKNOWN> [UDP: [192.168.100.3]:22430->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (1159464) 3:13:14.64
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "ERROR: Disk full"
#011iso.3.6.1.2.1.88.2.1.2.0 = ""
#011iso.3.6.1.2.1.88.2.1.3.0 = ""
#011iso.3.6.1.2.1.88.2.1.4.0 = OID: iso.3.6.1.4.1.2021.9.1.100.1
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"
#011iso.3.6.1.4.1.2021.9.1.2.1 = STRING: "/"
#011iso.3.6.1.4.1.2021.9.1.101.1 = STRING: "/: less than 30% free (= 0%)"

Oct 11 00:02:28 robot-slave snmptrapd[480]: 2022-10-11 00:02:28 <UNKNOWN> [UDP: [192.168.100.3]:25418->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (1165577) 3:14:15.77
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "ERROR: Disk full"
#011iso.3.6.1.2.1.88.2.1.2.0 = ""
#011iso.3.6.1.2.1.88.2.1.3.0 = ""
#011iso.3.6.1.2.1.88.2.1.4.0 = OID: iso.3.6.1.4.1.2021.9.1.100.3
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"
#011iso.3.6.1.4.1.2021.9.1.2.3 = STRING: "/var/log"
#011iso.3.6.1.4.1.2021.9.1.101.3 = STRING: "/var/log: less than 10% free (= 0%)"

Oct 11 00:02:46 robot-slave snmptrapd[480]: 2022-10-11 00:02:46 <UNKNOWN> [UDP: [192.168.100.3]:63595->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (1167361) 3:14:33.61
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "DISK ALMOST FULL: Disk /var/log is more than 80 percent full"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Use log purge to purge all rotated logs
#012
#012Current disk status:
#012Filesystem: /dev/sdb1
#012      Size: 9.9G
#012      Used: 9.5G
#012     Avail: 0
#012      Use%: 100%
#012Mounted on: /var/log"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Disk Full Resolution

It depends on which disk is full:

If it is the media or backup disks, then clean up the disk space.
If it is any other disk then contact Support immediately.

DNS and Domain#

Category	Level	Message
DNS	warn	WARNING: No dns configured for %s…
DNS	info	INFO: dns is now configured for %s …
Domain	warn	WARNING: No domain configured for %s…
Domain	info	INFO: domain is now configured for %s…

Examples#

Oct 10 21:31:32 robot-slave snmptrapd[480]: 2022-10-10 21:31:32 <UNKNOWN> [UDP: [192.168.100.3]:21656->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (260016) 0:43:20.16
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "WARNING: No dns configured for VOSS-192.168.100.3"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "It is recommended that the dns is configured.
#012
#012To configure dns use the following command:
#012network dns <server1> <server2>"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Resolution:

Configure a DNS for the system with network dns add <server>.

Oct 10 21:32:51 robot-slave snmptrapd[480]: 2022-10-10 21:32:51 <UNKNOWN> [UDP: [192.168.100.3]:48247->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (267891) 0:44:38.91
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "INFO: dns is now configured for VOSS-192.168.100.3"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "DNS cleared"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 0
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 21:33:56 robot-slave snmptrapd[480]: 2022-10-10 21:33:56 <UNKNOWN> [UDP: [192.168.100.3]:36367->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (274443) 0:45:44.43
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "INFO: domain is now configured for VOSS-192.168.100.3"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Domain cleared"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 0
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 21:34:00 robot-slave snmptrapd[480]: 2022-10-10 21:34:00 <UNKNOWN> [UDP: [192.168.100.3]:50982->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (274843) 0:45:48.43
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "WARNING: No domain configured for VOSS-192.168.100.3"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "It is recommended that the domain is configured.
#012
#012To configure the domain use the following command:
#012network domain <domain-name>"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Resolution:

Configure domain with network domain <name>.

Mail#

Category	Level	Message
External Mail Monitoring	warn	WARNING: Not all notify levels is configured with an external email address …
External Mail Monitoring	info	INFO: All notify levels is now configured with an external email address ..
Health email	error	ERROR: Trouble sending health email
Health email	info	INFO: Health emails is now being sent
Local Mail Monitoring	warn	WARNING: The total messages in the local mailbox for %s has reached in excess of 200..
Local Mail Monitoring	info	INFO: Messages for %s auto archived as it reached more than 500…
Local Mail Monitoring	info	INFO: The total local messages for %s is now under 200…

Examples#

Oct 10 21:05:39 robot-slave snmptrapd[480]: 2022-10-10 21:05:39 <UNKNOWN> [UDP: [192.168.100.3]:8550->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (104723) 0:17:27.23
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "INFO: All notify levels is now configured with an external email address"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "    notifications:
#012        emailrelay: 127.0.0.1
#012        level:
#012            error:
#012                mailto:platform@localhost
#012                snmp://[email protected]
#012            info:
#012          Message is truncated"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 0
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 21:27:29 robot-slave snmptrapd[480]: 2022-10-10 21:27:29 <UNKNOWN> [UDP: [192.168.100.3]:5434->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (235697) 0:39:16.97
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "WARNING: The total messages in the local mailbox for platform has reached in excess of 200"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Use the following mail commands to manage the local mailbox:
#012
#012mail del <number>      - delete the selected mail
#012mail del <from
#> <to
#> - deletes the selected range of mail message
#012mail del all          Message is truncated"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 21:27:46 robot-slave snmptrapd[480]: 2022-10-10 21:27:46 <UNKNOWN> [UDP: [192.168.100.3]:32395->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (237369) 0:39:33.69
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "INFO: Messages for platform auto archived as it reached more than 500"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Use the following command to view archived messages:
#012
#012log view /var/log/platform/archived_mail.log"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 0
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Notification message from (1, 3, 6, 1, 6, 1, 1):('192.29.22.122', 25067):
Var-binds:
1.3.6.1.2.1.1.3.0 = 7420086
1.3.6.1.6.3.1.1.4.1.0 = 1.3.6.1.2.1.88.2.0.1
1.3.6.1.2.1.88.2.1.1.0 = WARNING:  Some notify levels are configured with a local email address
1.3.6.1.2.1.88.2.1.3.0 = Notify list:
    notifications:
    emailrelay: 172.29.42.30
    level:
    audit:
    snmp://[email protected]
    error:
    mailto:xlatform@loc  Message is truncated
1.3.6.1.2.1.88.2.1.5.0 = 1
1.3.6.1.2.1.1.5.0 = UN1-192.29.22.122

Resolution:

Ensure that all notifications are non-local.

License#

Category	Level	Message
License	error	ERROR: License file generation failed The license audit report scheduled for %s was not successful …
License	info	INFO: License file generated The license audit report scheduled for %s was successful…

Examples#

Oct 10 22:46:06 robot-slave snmptrapd[480]: 2022-10-10 22:46:06 <UNKNOWN> [UDP: [192.168.100.3]:33539->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (707420) 1:57:54.20
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "ERROR: License file generation failed"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "The license audit report scheduled for October 2022 was not successful.
#012Please contact your VOSS account manager. "
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 23:00:41 robot-slave snmptrapd[480]: 2022-10-10 23:00:41 <UNKNOWN> [UDP: [192.168.100.3]:52074->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (794896) 2:12:28.96
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "INFO: License file generated"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "The license audit report scheduled for October 2022 was successful."
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Nov  9 10:31:46 robot-slave snmptrapd[391]: 2022-11-09 10:31:46 <UNKNOWN> [UDP: [192.168.100.3]:56185->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (63460) 0:10:34.60
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "Automate License Service Error"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "ID: AUTOMATE_LICENSE, Code: 12003, Occurences: 1, Latest Occurence: 2022-11-09T08:31:43.994Z"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Nov  9 10:32:09 robot-slave snmptrapd[391]: 2022-11-09 10:32:09 <UNKNOWN> [UDP: [192.168.100.3]:4445->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (65783) 0:10:57.83
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "ProcessStart"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Application start command completed license-service:license-service"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Nov  9 10:32:51 robot-slave snmptrapd[391]: 2022-11-09 10:32:51 <UNKNOWN> [UDP: [192.168.100.3]:24239->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (69973) 0:11:39.73
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "Automate License Expiry"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "ID: AUTOMATE_LICENSE, Code: 12002, Occurences: 1, Latest Occurence: 2022-11-09T08:32:48.826Z"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Resolution

See: Product Licensing.

Logs#

Category	Level	Message
Log Processing	error	System unable to send {} messages to {}…
Logging	error	ERROR: Log files larger than 1Gig found in /var/log …
Logging	info	NFO: /var/log rotated …
Logs Rotate	warn	Forcing log rotation as /var/log is at $USAGE usage: $message
Logs Rotate	warn	Autopurging /var/log due to extremely high disk usage. Usage $USAGE: $message

Examples#

Oct 11 00:01:42 robot-slave snmptrapd[480]: 2022-10-11 00:01:42 <UNKNOWN> [UDP: [192.168.100.3]:47625->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (1160957) 3:13:29.57
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "ERROR: Log files larger than 1Gig found in /var/log "
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Logrotation was executed to rotate the following logs:
#012/var/log/big-log.log: 1.5G"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 11 00:01:43 robot-slave snmptrapd[480]: 2022-10-11 00:01:43 <UNKNOWN> [UDP: [192.168.100.3]:6397->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (1161096) 3:13:30.96
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "INFO: /var/log rotated"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "/var/log rotated"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 0
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 11 00:02:45 robot-slave snmptrapd[480]: 2022-10-11 00:02:45 <UNKNOWN> [UDP: [192.168.100.3]:45283->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (1167292) 3:14:32.92
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "Logs"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Forcing log rotation as /var/log is at 100 usage. [platform_mon.py] reading config file /etc/logrotate.conf
#012including /etc/logrotate.d
#012reading config file alternatives
#012reading config file apt
#012reading   Message is truncated"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 0
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 11 00:02:45 robot-slave snmptrapd[480]: 2022-10-11 00:02:45 <UNKNOWN> [UDP: [192.168.100.3]:45283->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (1167292) 3:14:32.92
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "Logs"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Forcing log rotation as /var/log is at 100 usage. [platform_mon.py] reading config file /etc/logrotate.conf
#012including /etc/logrotate.d
#012reading config file alternatives
#012reading config file apt
#012reading   Message is truncated"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 0
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 11 00:02:45 robot-slave snmptrapd[480]: 2022-10-11 00:02:45 <UNKNOWN> [UDP: [192.168.100.3]:43304->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (1167346) 3:14:33.46
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "Logs"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Autopurging /var/log due to extremely high disk usage. [platform_mon.py] Usage 100: Deleting:
#0124.0K
#011/var/log/alternatives.log.2.gz
#012104K
#011/var/log/apt/term.log.3.gz
#0128.0K
#011/var/log/apt/history.log.1.gz
#01212K  Message is truncated"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 0
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Network Monitoring#

Category	Level	Message
Network Monitoring	error	ERROR: Network Failures. The following network failures occurred: …
Network Monitoring	info	INFO: Network failures resolved …

Examples#

Mar 22 14:07:58 robot-sl snmptrapd[1214]:
2019-03-22 14:07:58 <UNKNOWN>
[UDP: [192.168.100.3]:18751->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (1155411) 3:12:34.11
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "ERROR: Network Failures"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "The following network failures occured: netntp: 172.29.1.15"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Mar 29 13:57:11 robot-sl snmptrapd
[1234]: 2019-03-29 13:57:11 <UNKNOWN>
[UDP: [192.168.100.3]:32794->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (165816) 0:27:38.16
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "INFO: Network failures resolved"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Network failures resolved"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 0
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Memory Monitoring#

Category	Level	Message
Memory Monitoring	error	Critiical: ERROR: High memory usage Memory activity: …
Memory Monitoring	info	INFO: Memory usage returned to normal Memory more than 1024MB

Examples#

Oct 11 00:05:02 robot-slave snmptrapd[480]: 2022-10-11 00:05:02 <UNKNOWN> [UDP: [192.168.100.3]:40305->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (1180986) 3:16:49.86
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "ERROR: High memory usage"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Memory activity:
#012
#012223Mb"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 11 00:05:16 robot-slave snmptrapd[480]: 2022-10-11 00:05:16 <UNKNOWN> [UDP: [192.168.100.3]:35079->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (1182434) 3:17:04.34
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "INFO: Memory usage returned to normal"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Memory more than 1024MB"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 0
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 11 00:18:13 robot-slave snmptrapd[480]: 2022-10-11 00:18:13 <UNKNOWN> [UDP: [192.168.100.3]:34120->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (1260111) 3:30:01.11
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "ERROR: Memory swap error"
#011iso.3.6.1.2.1.88.2.1.2.0 = ""
#011iso.3.6.1.2.1.88.2.1.3.0 = ""
#011iso.3.6.1.2.1.88.2.1.4.0 = OID: iso.3.6.1.4.1.2021.4.100.0
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"
#011iso.3.6.1.4.1.2021.4.2.0 = STRING: "swap"
#011iso.3.6.1.4.1.2021.4.101.0 = STRING: "Running out of swap space (0)"

NTP#

Category	Level	Message
NTP	error	ERROR: No ntp configured for %s …
NTP	info	INFO: ntp is now configured for %s …
NTP Daemon	warn	WARNING: The ntp daemon has stopped on %s …
NTP Daemon	info	INFO: The ntp daemon is now running on %s …
NTP Offset	warn	WARNING: The ntp offset exceeds 1 second on %s ..
NTP Offset	info	INFO: The ntp offset restored to normal on %s …

Examples#

Oct 10 21:28:15 robot-slave snmptrapd[480]: 2022-10-10 21:28:15 <UNKNOWN> [UDP: [192.168.100.3]:62399->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (240260) 0:40:02.60
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "ERROR: No ntp configured for VOSS-192.168.100.3"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "It is mandatory that the ntp is configured.
#012
#012To configure ntp use the following command:
#012network ntp <server1> <server2>"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 21:28:23 robot-slave snmptrapd[480]: 2022-10-10 21:28:23 <UNKNOWN> [UDP: [192.168.100.3]:40886->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (241068) 0:40:10.68
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "INFO: ntp is now configured for VOSS-192.168.100.3"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "NTP cleared"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 0
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 21:29:12 robot-slave snmptrapd[480]: 2022-10-10 21:29:12 <UNKNOWN> [UDP: [192.168.100.3]:41584->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (246013) 0:41:00.13
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "WARNING: The ntp daemon has stopped on VOSS-192.168.100.3"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Run 'app start services:time' to restart ntpd"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 21:29:44 robot-slave snmptrapd[480]: 2022-10-10 21:29:44 <UNKNOWN> [UDP: [192.168.100.3]:64806->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (249163) 0:41:31.63
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "INFO: The ntp daemon is now running on VOSS-192.168.100.3"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "ntp daemon running"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 0
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Security Update#

Category	Level	Message
Security Update	warn	WARNING: Security Updates available …
Security Update	info	INFO: Security Updates applied …

Service Monitoring#

Category	Level	Message
Service Monitoring	error	ERROR: Service Failures …
Service Monitoring	info	INFO: Services started successfully …

Examples#

Oct 10 23:00:35 robot-slave snmptrapd[480]: 2022-10-10 23:00:35 <UNKNOWN> [UDP: [192.168.100.3]:15622->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (794329) 2:12:23.29
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "INFO: Services started successfully"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "selfservice v22.1.0 (2022-10-10 13:57)
#012   |-node                  running
#012voss-deviceapi v22.1.0 (2022-10-10 13:57)
#012   |-voss-cnf_collector    running
#012   |-voss-queue            running
#012   |-voss-wsgi  Message is truncated"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 0
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 11 00:03:03 robot-slave snmptrapd[480]: 2022-10-11 00:03:03 <UNKNOWN> [UDP: [192.168.100.3]:41278->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (1169102) 3:14:51.02
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "INFO: Services started successfully"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "0"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 0
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Service Startup Changes Made#

Severity levels, messages and resolution:

Info : ProcessRestart, ProcessStart

Resolution: If this is an unexpected event, call Support should be called for further investigation. This trap can also be triggered as expected, when app start or system reboot is run.
Warning: ProcessWarning

Resolution: This trap should be seen when a process or service is being restarted or stopped. If this is an unexpected event, call Support should be called for further investigation.
Error : ProcessStop, ProcessError

Resolution: If this is an unexpected event, call Support should be called for further investigation. This trap can also be triggered as expected, when app stop or system reboot is run.

Examples#

Oct 10 22:23:43 robot-slave snmptrapd[480]: 2022-10-10 22:23:43 <UNKNOWN> [UDP: [192.168.100.3]:26646->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (573107) 1:35:31.07
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "ProcessRestart"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Applications are restarting services:firewall"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 22:23:54 robot-slave snmptrapd[480]: 2022-10-10 22:23:54 <UNKNOWN> [UDP: [192.168.100.3]:54975->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (574250) 1:35:42.50
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "ProcessStart"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Application start command completed services:firewall"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 22:23:45 robot-slave snmptrapd[480]: 2022-10-10 22:23:45 <UNKNOWN> [UDP: [192.168.100.3]:61828->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (573306) 1:35:33.06
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "ProcessWarning"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Application was killed services:firewall"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 22:28:06 robot-slave snmptrapd[480]: 2022-10-10 22:28:06 <UNKNOWN> [UDP: [192.168.100.3]:34679->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (599418) 1:39:54.18
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "ProcessStop"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Applications are stopping mongodb"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 11 00:00:34 robot-slave snmptrapd[480]: 2022-10-11 00:00:34 <UNKNOWN> [UDP: [192.168.100.3]:48544->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (1154165) 3:12:21.65
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "ProcessError"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Application failed with error 1 nginx:proxy"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Excessive Load#

Load average interval	<LoadIdx>	<LoadError>	<LoadMessage>
1 minute	1	Load-1	1 min Load Average too high (= 2.52)
5 minute	2	Load-5	5 min Load Average too high (= 1.27)
15 minute	3	Load-15	15 min Load Average too high (= 1.27)

Examples#

Oct 11 00:13:48 robot-slave snmptrapd[480]: 2022-10-11 00:13:48 <UNKNOWN> [UDP: [192.168.100.3]:10511->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (1233585) 3:25:35.85
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "ERROR: Excessive load"
#011iso.3.6.1.2.1.88.2.1.2.0 = ""
#011iso.3.6.1.2.1.88.2.1.3.0 = ""
#011iso.3.6.1.2.1.88.2.1.4.0 = OID: iso.3.6.1.4.1.2021.10.1.100.1
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"
#011iso.3.6.1.4.1.2021.10.1.2.1 = STRING: "Load-1"
#011iso.3.6.1.4.1.2021.10.1.101.1 = STRING: "1 min Load Average too high (= 5.59)"

Oct 11 00:13:48 robot-slave snmptrapd[480]: 2022-10-11 00:13:48 <UNKNOWN> [UDP: [192.168.100.3]:16438->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (1233630) 3:25:36.30
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "ERROR: Excessive load"
#011iso.3.6.1.2.1.88.2.1.2.0 = ""
#011iso.3.6.1.2.1.88.2.1.3.0 = ""
#011iso.3.6.1.2.1.88.2.1.4.0 = OID: iso.3.6.1.4.1.2021.10.1.100.2
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"
#011iso.3.6.1.4.1.2021.10.1.2.2 = STRING: "Load-5"
#011iso.3.6.1.4.1.2021.10.1.101.2 = STRING: "5 min Load Average too high (= 2.43)"

Oct 11 00:15:48 robot-slave snmptrapd[480]: 2022-10-11 00:15:48 <UNKNOWN> [UDP: [192.168.100.3]:43788->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (1245577) 3:27:35.77
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "ERROR: Excessive load"
#011iso.3.6.1.2.1.88.2.1.2.0 = ""
#011iso.3.6.1.2.1.88.2.1.3.0 = ""
#011iso.3.6.1.2.1.88.2.1.4.0 = OID: iso.3.6.1.4.1.2021.10.1.100.3
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"
#011iso.3.6.1.4.1.2021.10.1.2.3 = STRING: "Load-15"
#011iso.3.6.1.4.1.2021.10.1.101.3 = STRING: "15 min Load Average too high (= 3.14)"

Transaction Archive#

Category	Level	Message
Transaction Archive	error	ERROR: Transaction archive files older than 30 days…
Transaction Archive	info	INFO: Transaction archive files OK…

Upgrade#

Category	Level	Message
Upgrade	error	upgrade failed upgrade failed as other activity is in progress …

Examples#

Oct 10 22:30:17 robot-slave snmptrapd[480]: 2022-10-10 22:30:17 <UNKNOWN> [UDP: [192.168.100.3]:32373->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (612524) 1:42:05.24
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "upgrade failed"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "upgrade failed as other activity is in progress"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 0
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

NGINX Status#

Level:

Error: nginx upstream failure, upstream <node> server <server> failed

Info: nginx upstreams OK

Examples#

Mar 25 13:30:12 robot-sl snmptrapd[1214]:
2019-03-25 13:30:12 <UNKNOWN>
[UDP: [172.29.21.129]:63573->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (293333) 0:48:53.33
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "nginx upstream failure"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "upstream selfservice server 192.29.22.122:443 failed: <urlopen error timed out>"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "WP2-172.29.21.129"

Mar 25 13:34:02 robot-sl snmptrapd[1214]:
2019-03-25 13:34:02 <UNKNOWN>
[UDP: [172.29.21.129]:31265->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (316311) 0:52:43.11
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "nginx upstreams OK"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "nginx upstream servers returned to normal"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 0
#011iso.3.6.1.2.1.1.5.0 = STRING: "WP2-172.29.21.129"

Web Certificate#

A trap is generated when the nginx web certificate is about to expire.

Three intervals are checked which will return a result:

30 days prior to expiry
14 days prior to expiry
1 day prior to expiry

After expiry, an alert is sent every day.

Category	Level	Message
Web Certificate Maintenance	error	ERROR: Web certificate about to expire..
Web Certificate Maintenance	info	INFO: Web certificate has been renewed …

Examples#

Oct 10 22:58:45 robot-slave snmptrapd[480]: 2022-10-10 22:58:42 <UNKNOWN> [UDP: [192.168.100.3]:39688->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (782988) 2:10:29.88
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "Certificate_Maintenance"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "ID: WEB_CERTIFICATE_VOSS, Code: -1, Occurences: 1, Latest Occurence: 2022-10-10T20:58:41.312Z"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 22:58:45 robot-slave snmptrapd[480]: 2022-10-10 22:58:45 <UNKNOWN> [UDP: [192.168.100.3]:39098->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (783366) 2:10:33.66
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "ERROR: Web certificate about to expire"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Cert will expire in less than 30 day(s)"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 22:58:48 robot-slave snmptrapd[480]: 2022-10-10 22:58:48 <UNKNOWN> [UDP: [192.168.100.3]:15074->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (783622) 2:10:36.22
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "INFO: Web certificate has been renewed"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Web certificate has been renewed"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Example GUI alert shown in the data/Alert instance in the Database

AlertCode: “25005”
Category: “Certificate_Maintenance”
AlertId: “WEB_CERTIFICATE_<certificate-name>”
Message: “The certificate will expire in less than {} day(s).” (number of days - alert will result)
Severity: WARNING
alert_timestamp: <timestamp>

Example error raised through platform monitoring

Notification message from (1, 3, 6, 1, 6, 1, 1):('10.120.1.203', 5194):
Var-binds:
3.6.1.2.1.1.3.0 = 16128774
3.6.1.6.3.1.1.4.1.0 = 1.3.6.1.2.1.88.2.0.1
3.6.1.2.1.88.2.1.1.0 = ERROR: Web certificate about to expire
3.6.1.2.1.88.2.1.3.0 = Cert will expire in less than {} day(s)
3.6.1.2.1.88.2.1.5.0 = 1
3.6.1.2.1.1.5.0 = VOSS-UN-1

Resolution

Renew the web certificate - see: Web Certificate Setup Options.

Thresholds#

Examples#

Oct 10 21:30:04 robot-slave snmptrapd[480]: 2022-10-10 21:30:04 <UNKNOWN> [UDP: [192.168.100.3]:63587->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (251207) 0:41:52.07
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "rsyslogd : num_files exceeded maximum value of 30 with 78"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Command rsyslogd attribute num_files
#012Current maximum value 78
#012Error threshold 50
#012Warning threshold 33
#012"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 21:30:04 robot-slave snmptrapd[480]: 2022-10-10 21:30:04 <UNKNOWN> [UDP: [192.168.100.3]:63524->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (251245) 0:41:52.45
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "sysmon : num_sockets exceeded maximum value of 10 with 11"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Command sysmon attribute num_sockets
#012Current maximum value 11
#012Error threshold 10
#012Warning threshold 6
#012"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"


Oct 10 22:40:03 robot-slave snmptrapd[480]: 2022-10-10 22:40:03 <UNKNOWN> [UDP: [192.168.100.3]:62002->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (671171) 1:51:51.71
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "containerd : num_sockets exceeded maximum value of 16 with 17"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Command containerd attribute num_sockets
#012Current maximum value 17
#012Error threshold 10
#012Warning threshold 6
#012"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"


Oct 10 22:50:03 robot-slave snmptrapd[480]: 2022-10-10 22:50:03 <UNKNOWN> [UDP: [192.168.100.3]:34433->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (731163) 2:01:51.63
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "execute : num_sockets exeeded warning threshold of 6 with 6"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Command execute attribute num_sockets
#012Current average value 6
#012Error threshold 10
#012Warning threshold 6
#012"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 23:50:04 robot-slave snmptrapd[480]: 2022-10-10 23:50:04 <UNKNOWN> [UDP: [192.168.100.3]:15565->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (1091211) 3:01:52.11
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "containerd-shim : num_files exceeded maximum value of 19 with 29"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Command containerd-shim attribute num_files
#012Current maximum value 29
#012Error threshold 20
#012Warning threshold 13
#012"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 23:50:05 robot-slave snmptrapd[480]: 2022-10-10 23:50:05 <UNKNOWN> [UDP: [192.168.100.3]:2699->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (1091258) 3:01:52.58
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "checkmongoalive : num_files exeeded warning threshold of 13 with 20"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Command checkmongoalive attribute num_files
#012Current average value 20
#012Error threshold 20
#012Warning threshold 13
#012"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Resolution:

Check if mongodb is up and running: Use the command app status to see if mongodb is running. If not, run app start mongodb. Else contact VOSS support.

Oct 10 21:40:06 robot-slave snmptrapd[480]: 2022-10-10 21:40:06 <UNKNOWN> [UDP: [192.168.100.3]:53579->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (311359) 0:51:53.59
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "backup.py : num_sockets exeeded warning threshold of 6 with 7"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Command backup.py attribute num_sockets
#012Current average value 7
#012Error threshold 10
#012Warning threshold 6
#012"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 22:00:16 robot-slave snmptrapd[480]: 2022-10-10 22:00:16 <UNKNOWN> [UDP: [192.168.100.3]:40320->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (432428) 1:12:04.28
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "docker-proxy : num_sockets exceeded maximum value of 7 with 27"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Command docker-proxy attribute num_sockets
#012Current maximum value 27
#012Error threshold 20
#012Warning threshold 13
#012"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 10 22:00:17 robot-slave snmptrapd[480]: 2022-10-10 22:00:17 <UNKNOWN> [UDP: [192.168.100.3]:39273->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (432500) 1:12:05.00
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "docker-proxy : num_files exceeded maximum value of 17 with 57"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Command docker-proxy attribute num_files
#012Current maximum value 57
#012Error threshold 20
#012Warning threshold 13
#012"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Oct 11 00:00:15 robot-slave snmptrapd[480]: 2022-10-11 00:00:15 <UNKNOWN> [UDP: [192.168.100.3]:55586->[192.168.100.25]:162]:
#012iso.3.6.1.2.1.1.3.0 = Timeticks: (1152330) 3:12:03.30
#011iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.2.1.88.2.0.1
#011iso.3.6.1.2.1.88.2.1.1.0 = STRING: "dockerd : num_files exceeded maximum value of 78 with 84"
#011iso.3.6.1.2.1.88.2.1.3.0 = STRING: "Command dockerd attribute num_files
#012Current maximum value 84
#012Error threshold 80
#012Warning threshold 53
#012"
#011iso.3.6.1.2.1.88.2.1.5.0 = INTEGER: 1
#011iso.3.6.1.2.1.1.5.0 = STRING: "VOSS"

Notification Messages

On this page

Notification Messages#

Backups#

Examples#

Database in Cluster Failover#

Database Maintenance and Usage#

Examples#

Disk Latency and Status#

Examples#

DNS and Domain#

Examples#

Mail#

Examples#

License#

Examples#

Logs#

Examples#

Network Monitoring#

Examples#

Memory Monitoring#

Examples#

NTP#

Examples#

Security Update#

Service Monitoring#

Examples#

Service Startup Changes Made#

Examples#

Excessive Load#

Examples#

Transaction Archive#

Upgrade#

Examples#

NGINX Status#

Examples#

Web Certificate#

Examples#

Thresholds#

Examples#