Statistics Troubleshooting
Revised: August 10, 2011, OL-25016-01
Introduction
This chapter provides the information needed for monitoring and troubleshooting statistics events and alarms. This chapter is divided into the following sections:
•Statistics Events and Alarms—Provides a brief overview of each statistics event and alarm
•Monitoring Statistics Events—Provides the information needed for monitoring and correcting the statistics events
•Troubleshooting Statistics Alarms—Provides the information needed for troubleshooting and correcting the statistics alarms
Statistics Events and Alarms
This section provides a brief overview of all of the statistics events and alarms for the Cisco BTS 10200 Softswitch; the event and alarms are arranged in numerical order. Table 11-1 lists all of the statistics events and alarms by severity.
|
Note Click the statistics message number in Table 11-1 to display information about the event or alarm.
|
Statistics (1)
Table 11-2 lists the details of the Statistics (1) informational event. For additional information, refer to the "Test Report—Statistics (1)" section.
Table 11-2 Statistics (1) Details
Description |
Test Report |
Severity |
Information |
Threshold |
10000 |
Throttle |
0 |
Statistics (2)
Table 11-3 lists the details of the Statistics (2) informational event. For additional information, refer to the "Call Agent Measurement Collection Started—Statistics (2)" section.
Table 11-3 Statistics (2) Details
Description |
Call Agent Measurement Collection Started |
Severity |
Information |
Threshold |
100 |
Throttle |
0 |
Datawords |
Start Time—STRING [40] |
Primary Cause |
Indicates that the 15-minute traffic measurement collection process has started on the Call Agent. |
Primary Action |
No action is necessary. |
Statistics (3)
Table 11-3 lists the details of the Statistics (3) informational event. For additional information, refer to the "Call Agent Measurement Collection Finished—Statistics (3)" section.
Table 11-4 Statistics (3) Details
Description |
Call Agent Measurement Collection Finished |
Severity |
Information |
Threshold |
100 |
Throttle |
0 |
Datawords |
End Time—STRING [40] |
Primary Cause |
Indicates that the 15-minute traffic measurement collection process has completed on the Call Agent. |
Primary Action |
No action is necessary. |
Statistics (4)
Table 11-5 lists the details of the Statistics (4) informational event. For additional information, refer to the "Plain Old Telephone Service/Centrex/Telecommunications Data Link Monitor Feature Server Measurement Collection Started—Statistics (4)" section.
Table 11-5 Statistics (4) Details
Description |
Plain Old Telephone Service/Centrex/Telecommunications Data Link Monitor Feature Server Measurement Collection Started (POTS/CTX/TDM Feature Server Measurement Collection Started) |
Severity |
Information |
Threshold |
100 |
Throttle |
0 |
Datawords |
Start Time—STRING [40] |
Primary Cause |
Indicates that the 15-minute traffic measurement collection process has started on the plain old telephone service (POTS)/Centrex/Tandem Feature Server. |
Primary Action |
No action is necessary. |
Statistics (5)
Table 11-6 lists the details of the Statistics (5) informational event. For additional information, refer to the "Plain Old Telephone Service/Centrex/Telecommunications Data Link Monitor Feature Server Measurement Collection Finished—Statistics (5)" section.
Table 11-6 Statistics (5) Details
Description |
Plain Old Telephone Service/Centrex/Telecommunications Data Link Monitor Feature Server Measurement Collection Finished (POTS/CTX/TDM Feature Server Measurement Collection Finished) |
Severity |
Information |
Threshold |
100 |
Throttle |
0 |
Datawords |
End Time—STRING [40] |
Primary Cause |
Indicates that the 15-minute traffic measurement collection process has finished on the POTS/Centrex/Tandem Feature Server. |
Primary Action |
No action is necessary. |
Statistics (6)
Table 11-7 lists the details of the Statistics (6) informational event. For additional information, refer to the "Advanced Intelligent Network Feature Server Measurement Collection Started—Statistics (6)" section.
Table 11-7 Statistics (6) Details
Description |
Advanced Intelligent Network Feature Server Measurement Collection Started (AIN Feature Server Measurement Collection Started) |
Severity |
Information |
Threshold |
100 |
Throttle |
0 |
Datawords |
Start Time—STRING [40] |
Primary Cause |
Indicates that the 15-minute traffic measurement collection process has started on the Advanced Intelligent Network (AIN) feature server. |
Primary Action |
No action is necessary. |
Statistics (7)
Table 11-8 lists the details of the Statistics (7) informational event. For additional information, refer to the "Advanced Intelligent Network Feature Server Measurement Collection Finished—Statistics (7)" section.
Table 11-8 Statistics (7) Details
Description |
Advanced Intelligent Network Feature Server Measurement Collection Finished (AIN Feature Server Measurement Collection Finished) |
Severity |
Information |
Threshold |
100 |
Throttle |
0 |
Datawords |
End Time—STRING [40] |
Primary Cause |
Indicates that the 15-minute traffic measurement collection process has completed on the AIN feature server. |
Primary Action |
No action is necessary. |
Statistics (8)
Table 11-9 lists the details of the Statistics (8) warning event. To monitor and correct the cause of the event, refer to the "Message Send Failure—Statistics (8)" section.
Table 11-9 Statistics (8) Details
Description |
Message Send Failure |
Severity |
Warning |
Threshold |
100 |
Throttle |
0 |
Datawords |
Originating Process—STRING [40] Traffic Mgr Msg Type—STRING [40] Traffic Agent Msg Type—STRING [30] |
Primary Cause |
The originating process (for Call Agent (CA), Feature Server (FS) or Element Management System (EMS)) is not in active state, or is shut down. |
Primary Action |
Check for other alarms and events generated from this component (CA, FS, or EMS). |
Secondary Cause |
The hub is down. |
Secondary Action |
If the originating process is down, try to bring the process back into the normal state. |
Ternary Cause |
The platform is currently shutting down a process. |
Ternary Action |
If the hub process is down, try to bring it into a normal state. (Contact Cisco Technical Assistance Center (TAC).) |
Statistics (9)
Table 11-10 lists the details of the Statistics (9) warning event. To monitor and correct the cause of the event, refer to the "Measurement Table Structured Query Language Read Error—Statistics (9)" section.
Table 11-10 Statistics (9) Details
Description |
Measurement Table Structured Query Language Read Error (Measurement Table SQL Read Error) |
Severity |
Warning |
Threshold |
100 |
Throttle |
0 |
Datawords |
Measurement Table Na—STRING [40] |
Primary Cause |
There is no database connection or the connection is faulty. |
Primary Action |
Check to see if there are any other events generated that indicate there is a database problem. |
Secondary Cause |
The traffic measurement table(s) are corrupted. |
Secondary Action |
Correct any database-related problems. |
Ternary Cause |
Two processors or more are attempting to access the table at the same time. |
Ternary Action |
If all of the database-related problems are cleared, and this warning event report still occurs, contact Cisco TAC for technical support. |
Statistics (10)
Table 11-11 lists the details of the Statistics (11) warning event. To monitor and correct the cause of the event, refer to the "Measurement Table Structured Query Language Write Error—Statistics (10)" section.
Table 11-11 Statistics (10) Details
Description |
Measurement Table Structured Query Language Write Error (Measurement Table SQL Write Error) |
Severity |
Warning |
Threshold |
100 |
Throttle |
0 |
Datawords |
Measurement Table Na—STRING [40] |
Primary Cause |
There is no database connection or the connection is faulty. |
Primary Action |
Check to see if any other events are generated that indicate there is a database problem. |
Secondary Cause |
The traffic measurement table(s) are corrupted. |
Secondary Action |
Correct any database-related problems. |
Ternary Cause |
Two processors or more are attempting to access the table at the same time. |
Ternary Action |
If all of the database-related problems are cleared, and this warning event report still occurs, contact Cisco TAC for technical support. |
Statistics (11)
Table 11-12 lists the details of the Statistics (11) warning event. To monitor and correct the cause of the event, refer to the "Measurement Collection Application Programming Interface Failure—Statistics (11)" section.
Table 11-12 Statistics (11) Details
Description |
Measurement Collection Application Programming Interface Failure (Measurement Collection API Failure) |
Severity |
Warning |
Threshold |
100 |
Throttle |
0 |
Datawords |
Message Type—STRING [40] Platform Type—STRING [40] |
Primary Cause |
This report is issued when the traffic measurement subsystem on the call agent or the feature server encounters difficulties when it tries to collect measurement values from one of the processes. |
Primary Action |
Execute a status command for the affected call agent or feature server. |
Secondary Cause |
The originating process (for CA, FS or EMS) is not in active state or is shut down. |
Secondary Action |
If the status report indicates that the originating process is down, try to bring the process back into the normal (In Service) state. If you need assistance in restoring a process, contact Cisco TAC. |
Ternary Cause |
The platform is currently shutting down the originating process. |
Ternary Action |
If this event report is being issued every collection period, contact Cisco TAC for assistance. |
Subsequent Action |
Note Traffic measurements will not be available for the affected call agent or feature server for the measurement period in which this event report was issued. |
Statistics (12)
Table 11-13 lists the details of the Statistics (12) major alarm. To troubleshoot and correct the cause of the alarm, refer to the "Measurement Handshake Error—Schema Inconsistency—Statistics (12)" section.
Table 11-13 Statistics (12) Details
Description |
Measurement Handshake Error—Schema Inconsistency |
Severity |
Major |
Threshold |
5 |
Throttle |
0 |
Datawords |
Schemas Out of Synchronization—STRING [64] |
Primary Cause |
Counters were added or deleted from the schema in Oracle but not in the call agent. |
Primary Action |
Add or delete the counters on the CA. |
Secondary Cause |
Load is installed incorrectly. |
Secondary Action |
Reinstall the load. |
Statistics (13)
Table 11-14 lists the details of the Statistics (13) major alarm. To troubleshoot and correct the cause of the alarm, refer to the "Traffic and Measurements Module Application Programming Interface Failure—Statistics (13)" section.
Table 11-14 Statistics (13) Details
Description |
Traffic and Measurements Module Application Programming Interface Failure (TMM API Failure) |
Severity |
Major |
Threshold |
1 |
Throttle |
10 |
Datawords |
TMM Error—FOUR_BYTES |
Primary Cause |
Unable to initialize shared memory. |
Primary Action |
Reconfigure and restart system. |
Secondary Cause |
Unable to attach to shared memory. |
Secondary Action |
Restart offending process. |
Ternary Cause |
Shared memory table overflow. |
Ternary Action |
Reconfigure/restart or fix problematic application. |
Subsequent Cause |
Shared memory exhaustion. |
Subsequent Action |
Reconfigure and restart system. |
Statistics (14)
Table 11-15 lists the details of the Statistics (14) warning event. To monitor and correct the cause of the event, refer to the "MDII Trunk—Statistics (14)" section.
Table 11-15 Statistics (14) Details
Description |
MDII Trunk |
Severity |
Warning |
Threshold |
100 |
Throttle |
0 |
Datawords |
Trunk Group—FOUR_BYTES CIC—FOUR_BYTES |
Primary Cause |
Calls on the MDII trunk termination are not being successfully completed. |
Primary Action |
The Cisco BTS 10200 system may take this trunk out of service if it does not take the full trunk group out of service. |
Statistics (15)
Table 11-16 lists the details of the Statistics (15) minor alarm. To troubleshoot and correct the cause of the alarm, refer to the "Threshold Crossing Alert—Statistics (15)" section.
Table 11-16 Statistics (15) Details
Description |
Threshold Crossing Alert |
Severity |
Minor |
Threshold |
100 |
Throttle |
0 |
Datawords |
Description—STRING [256] |
Primary Cause |
A threshold crossing has occurred. |
Primary Action |
Reduce the provisioning workload. |
Statistics (16)
Table 11-17 lists the details of the Statistics (16) warning event. To monitor and correct the cause of the event, refer to the "Trunk Group Has Reached the MDII Alarm Threshold—Statistics (16)" section.
Table 11-17 Statistics (16) Details
Description |
Trunk Group has Reached the MDII Alarm Threshold |
Severity |
Warning |
Threshold |
100 |
Throttle |
0 |
Datawords |
Trunk Group - FOUR_BYTES |
Primary Cause |
The trunk group has reached the MDII alarm threshold. |
Primary Action |
Check the performance status of trunk group. |
Monitoring Statistics Events
This section provides the information you need for monitoring and correcting statistics events. Table 11-18 lists all of the statistics events in numerical order and provides cross-references to each subsection.
Test Report—Statistics (1)
The Test Report event is for testing the statistics event category. The event is informational and no further action is required.
Call Agent Measurement Collection Started—Statistics (2)
The Call Agent Measurement Collection Started event functions as an informational alert that the call agent measurement collection has started. The event is informational and no further action is required.
Call Agent Measurement Collection Finished—Statistics (3)
The Call Agent Measurement Collection Finished event functions as an informational alert that the call agent measurement collection is finished. The event is informational and no further action is required.
Plain Old Telephone Service/Centrex/Telecommunications Data Link Monitor Feature Server Measurement Collection Started—Statistics (4)
The Plain Old Telephone Service/Centrex/Telecommunications Data Link Monitor Feature Server Measurement Collection Started event functions as an informational alert that the POTS/Centrex (CTX)/telecommunications data link monitor (TDM) feature server measurement collection has started. The event is informational and no further action is required.
Plain Old Telephone Service/Centrex/Telecommunications Data Link Monitor Feature Server Measurement Collection Finished—Statistics (5)
The Plain Old Telephone Service/Centrex/Telecommunications Data Link Monitor Feature Server Measurement Collection Finished event functions as an informational alert that the POTS/CTX/TDM feature server measurement collection has finished. The event is informational and no further action is required.
Advanced Intelligent Network Feature Server Measurement Collection Started—Statistics (6)
The Advanced Intelligent Network Feature Server Measurement Collection Started event functions as an informational alert that the AIN feature server measurement collection has started. The event is informational and no further action is required.
Advanced Intelligent Network Feature Server Measurement Collection Finished—Statistics (7)
The Advanced Intelligent Network Feature Server Measurement Collection Finished event functions as an informational alert that the AIN feature server measurement collection has finished. The event is informational and no further action is required.
Message Send Failure—Statistics (8)
The Message Send Failure event serves as a warning that a message send has failed. The primary cause of the event is that the originating process (for CA, FS, or EMS) is not in active state, or is shut down. To correct the primary cause of the event, check for other alarms and events generated from this component (CA, FS, or EMS). The secondary cause of the event is that the hub is down. To correct the secondary cause of the event, is if the hub process is down, try to bring it into a normal state. The ternary cause of the event is that the platform is currently shutting down a process. To correct the ternary cause of the event, if the originating process is down, try to bring the process back into the normal state. If this event report is being issued on every collection interval and there are no other event reports being issued, contact the Cisco TAC to resolve the communication issue.
Issued when the traffic measurement subsystem in the CA fails to send messages to the EMS due to
•Originating process (for CA, FS, or EMS) is not in active state, or is shut down.
•Hub is down.
•Platform is currently shutting down a process.
If this event is being issued frequently (every collection interval), then there are communication difficulties between the Call Agent and the EMS, and there will be additional events being issued. These other events will indicate the nature of the communication difficulties. The repair procedures for the other event reports should be followed to correct the Call Agent/EMS communication issue:
•Check for other alarms and events generated from this component (CA, FS, or EMS).
•If the originating process is down, try to bring the process back into the normal state.
•If the hub process is down, try to bring it into a normal state.
If this event report is being issued on every collection interval and there are no other event reports being issued, contact the Cisco TAC to resolve the communication issue.
Measurement Table Structured Query Language Read Error—Statistics (9)
The Measurement Table Structured Query Language Read Error event serves as a warning that the measurement table had a Structured Query Language (SQL) read error. The primary cause of the event is that there is no database connection, or the connection is faulty. To correct the primary cause of the event, check to see if there are any other events generated that indicate there is a database problem. The secondary cause of the event is that the traffic measurement tables are corrupted. To correct the secondary cause of the event, correct any database-related problems. The ternary cause of the event is that two processors or more are attempting to access the table at the same time. To correct the ternary cause of the event, check to see if all of the database-related problems are cleared, and this warning event report still occurs, contact Cisco TAC for technical support.
Measurement Table Structured Query Language Write Error—Statistics (10)
The Measurement Table Structured Query Language Write Error event serves as a warning that the measurement table has had a SQL write error. The primary cause of the event is that there is no database connection, or the connection is faulty. To correct the primary cause of the event, check to see if there are any other events generated that indicate there is a database problem. The secondary cause of the event is that the traffic measurement tables are corrupted. To correct the secondary cause of the event, correct any database-related problems. The ternary cause of the event is that two processors or more are attempting to access the table at the same time. To correct the ternary cause of the event, check to see if all of the database-related problems are cleared. If this warning event still occurs, contact Cisco TAC for technical support.
Measurement Collection Application Programming Interface Failure—Statistics (11)
The Measurement Collection Application Programming Interface Failure event serves as a warning that the measurement of application programming interface (API) statistics has failed. The primary cause of the event is that the report is issued when the traffic measurement subsystem on the call agent or feature server encounters difficulties when it tries to collect measurements from one of the processes. To correct the primary cause of the event, execute a status command for the affected call agent or feature server. The secondary cause of the event is that the originating process (for CA, FS, or EMS) is not in active state, or is shut down. To correct the secondary cause of the event, check and see if the status report indicates that the originating process is down, and try to bring the process back into the normal (In Service) state. If you need assistance in restoring a process, contact Cisco TAC. The ternary cause of the event is that the platform is currently shutting down the originating process. To correct the ternary cause of the event, check and see if this event report is being issued every collection period. Contact Cisco TAC for assistance.
|
Note Traffic measurements will not be available for the affected call agent or feature server for the measurement period in which this event report was issued.
|
Measurement Handshake Error—Schema Inconsistency—Statistics (12)
The Measurement Handshake Error—Schema Inconsistency alarm (major) indicates that a measurement handshake error has occurred. To troubleshoot and correct the cause of the Measurement Handshake Error—Schema Inconsistency alarm, refer to the "Measurement Handshake Error—Schema Inconsistency—Statistics (12)" section.
Traffic and Measurements Module Application Programming Interface Failure—Statistics (13)
The Traffic and Measurements Module Application Programming Interface Failure alarm (major) indicates that the Traffic and Measurements module (TMM) API failed. To troubleshoot and correct the cause of the Traffic and Measurements Module Application Programming Interface Failure alarm, refer to the "Traffic and Measurements Module Application Programming Interface Failure—Statistics (13)" section.
MDII Trunk—Statistics (14)
The MDII Trunk event serves as a warning that the calls on the MDII trunk termination are not being successfully completed. The Cisco BTS 10200 system might take the MDII trunk out of service if it does not take the full trunk group out of service.
Threshold Crossing Alert—Statistics (15)
The Threshold Crossing Alert alarm (minor) indicates that a threshold crossing has occurred. To troubleshoot and correct the cause of the Threshold Crossing Alert alarm, refer to the "Threshold Crossing Alert—Statistics (15)" section.
Trunk Group Has Reached the MDII Alarm Threshold—Statistics (16)
The Trunk Group Has Reached the MDII Alarm Threshold event serves as a warning that the trunk group has reached the MDII alarm threshold. To correct the cause of the event, check the performance status of the trunk group.
Troubleshooting Statistics Alarms
This section provides the information you need for monitoring and correcting statistics alarms. Table 11-19 lists all of the statistics alarms in numerical order and provides cross-references to each subsection.
Measurement Handshake Error—Schema Inconsistency—Statistics (12)
The Measurement Handshake Error—Schema Inconsistency alarm (major) indicates that a measurement handshake error has occurred. The primary cause of the alarm is that counters have been added or deleted from the schema in Oracle but not in the Call Agent. To correct the primary cause of the alarm, add or delete the counters on the CA. The secondary cause of the alarm is that the software load has been installed incorrectly. To correct the secondary cause of the alarm, reinstall the software load.
Traffic and Measurements Module Application Programming Interface Failure—Statistics (13)
The Traffic and Measurements Module Application Programming Interface Failure alarm (major) indicates that the TMM API failed. The primary cause of the alarm is that the system is unable to initialize the shared memory. To correct the primary cause of the alarm, reconfigure and restart the system. The secondary cause of the alarm is that a process is unable to attach to the shared memory. To correct the secondary cause of the alarm, restart the offending process. The ternary cause of the alarm is that the shared memory table has overflowed. To correct the ternary cause of the alarm, reconfigure/restart or fix problematic application. The subsequent cause of the alarm is that the shared memory is exhausted. To correct the subsequent cause of the alarm, reconfigure and restart the system.
Threshold Crossing Alert—Statistics (15)
The Threshold Crossing Alert alarm (minor) indicates that a threshold crossing has occurred. To correct the cause of the alarm, reduce the provisioning workload.