Feedback
Table Of Contents
Diagnosing Cisco CallManager Problems
Diagnosing Lost or Distorted Audio Problems
Diagnosing Audio Problems from the Cisco IP Phone
Testing Speakerphone and Volume Levels
Diagnosing One-Way Audio or No Audio
Testing Cisco CallManager Configuration
Checking RTP Header Compression
Diagnosing Cisco CallManager Feature Issues
Diagnosing Codec and Region Mismatches
Diagnosing Location and Bandwidth Problems
Diagnosing Conference Bridge Problems
Diagnosing Slow Server Response
Determining Gateway Reorder Tone Problems
Diagnosing Gateway Registration Problems
Diagnosing Gatekeeper Problems
Inter-Cluster Trunks or H.225 Gateway
Diagnosing Cisco CallManager Problems
This chapter addresses some common problems that you may experience with Cisco CallManager and related devices. Each problem category provides suggestions for the troubleshooting tools you should use to help isolate the problem. This chapter provides general categories of potential problems and suggestions about how to troubleshoot those problems. It does not provide an exhaustive list of problems and resolutions.
If you encounter a problem that cannot be resolved using the tools and utilities described in this document, consult your Cisco Technical Assistance Center (TAC) for assistance. Be sure to have available the Cisco CallManager Administration Details, plus any diagnostic information (for example, traces) you have gathered before calling the TAC.
This chapter contains the following topics:
•
Diagnosing Cisco CallManager Feature Issues
•
•
•
•
Voice Quality
You may experience voice quality issues including lost or distorted audio signal during phone calls. Common problems include audio breaks (like broken words), or the presence of odd noises and audio distortion, such as echo, watery or robotic voice quality. One-way audio, that is, a conversation between two people where only one person can hear anything, is not actually a voice quality issue, but will be discussed later in this section.
You may experience audio problems with one or more of the following:
•
•
•
Diagnosing Lost or Distorted Audio Problems
One of the most common problems you may encounter is broken audio signal (often described as garbled speech, or lost syllables within a word or sentence). There are two common causes for this: packet loss and/or jitter. Packet loss means that audio packets do not arrive at their destination because they were dropped or arrived too late to be useful. Jitter is the variation in the arrival times of packets. In the ideal situation, all Voice over IP (VoIP) packets would arrive exactly at a rate of 1 every 20 microseconds (ms). Notice that this is not the length of time it takes for a packet to get from point A to point B, but is simply the variation in packet arrival times.
There are many sources of variable delay in a network. Some of these cannot be controlled, and some can. Variable delay cannot be eliminated entirely in a packetized voice network. Digital Signal Processors (DSP) on phones and other voice-capable devices are designed to buffer some of the audio, in anticipation of variable delay. This dejittering is done only when the audio packet has reached its destination and is now ready to be put into a conventional audio stream.
The Cisco IP Phone 7960 can buffer as much as one second of voice samples. The jitter buffer is adaptive, meaning if a burst of packets is received, the Cisco IP Phone 7960 can play them out in an attempt to control the jitter. The network administrator needs to minimize the variation between packet arrival times by applying Quality-of-Service (QoS) and other measures in advance (especially if calls cross a wide-area network).
When faced with a lost or distorted audio problem, the first thing to do is to try to isolate the path of the audio. Try to identify each network device (switches and routers) in the path of the call's audio stream. Keep in mind that the audio may be between two phones, or between a phone and a gateway, or it could have multiple legs (from a phone to a transcoding device and from there to another phone). Try to identify if the problem occurs only between two sites, only through a certain gateway, on a certain subnet, and so on. This will help narrow down which devices you need to look at more carefully.
Next, if is often best to disable silence suppression (also known as Voice Activation Detection or VAD) if this has not been done already. This mechanism does save bandwidth by not transmitting any audio when there is silence, but may cause noticeable or unacceptable clipping at the beginning of words.
You can disable this in Cisco CallManager Administration, under Service > Service Parameters. From there, select the server and the Cisco CallManager service. Then set SilenceSuppressionSystemWide to F (alternatively you can set SilenceSuppressionWithGateways to F, but this does not apply to H.323 gateways or MGCP gateways). When in doubt, turn both off by selecting the Value F for each.
Using a network analyzer, if a network analyzer is available, then a monitored call between two phones should have 50 packets per second (or 1 packet every 20 ms) when silence suppression is disabled. With proper filtering, it should be possible to identify if packets are being lost or delayed excessively.
Remember that delay by itself will not cause clipping, only variable delay will. Notice in the table below, which represents a perfect trace, the arrival times between the audio packets (which will have an RTP header), will be 20 ms. In a poor quality call (such as a call with a lot of jitter), the arrival times would vary greatly.
The following table illustrates a perfect trace.
Placing the packet analyzer into various points in the network will help narrow down where the delay is coming from. If no analyzer is available, other methods will be required. It is important to examine interface statistics of each device in the path of the audio. Another tool for tracking calls with poor voice quality is the Diagnostic Call Detail Records (CDR). See "Call Detail and Call Management Records" for more information about CDRs.
Diagnosing Audio Problems from the Cisco IP Phone
The Cisco IP Phone 7960 provides another tool for diagnosing possible audio problems. On an active call, you can press the "i" button twice rapidly and the phone will display an information screen that contains packet receive and transmit statistics, as well as average and maximum jitter counters.
Note
The most common sources for delay and packet loss are devices where a higher speed interface feeds into a lower speed interface. For example, a router may have a 100 Megabyte (MB) fast Ethernet interface connected to the LAN and a slow frame-relay interface, connected to the WAN. If the poor audio quality occurs only when communicating to the remote site, then the most likely causes of the problem include:
•
•
•
•
On the LAN, the most common problems are physical-level errors (such as CRC errors) caused by faulty cables, interfaces, or by incorrectly configured devices (such as a port speed or duplex mismatch). Make sure that the traffic is not crossing any shared-media device, such as a hub.
There could also be situations where the traffic is taking a slower path through the network than expected. If QoS has been configured correctly, then the possibility exists that there is no call admission control. Depending on your topology, this can be accomplished through the use of Locations in Cisco CallManager Administration configuration, or by using a Cisco IOS router as a gatekeeper. In any case, you should always know how many calls could be supported across your WAN.
If possible, test this by disabling silence suppression as described in the preceding sections, then place calls between the two sites. Do not place the calls on hold or on mute, because this will stop packets from being transmitted. With the maximum number of calls across the WAN, the calls should all have acceptable quality. Test to make sure that a fast busy is returned when trying to make one more call.
Diagnosing Crackling Sounds
Another poor quality symptom may be a crackling, which is sometimes caused by a defective power supply or some kind of strong electrical interference close to the phone. Try swapping the power supply and moving the phone.
Checking Your Loads
Gateway and phone loads should be verified. Check Cisco Connection Online (CCO) at www.cisco.com for the latest software loads, new patches, or release notes relating to the problem.
Echo
Echo occurs when the speech energy being generated and transmitted down the primary signal path is coupled into the receive path from the far end. The speaker then hears his or her own voice, delayed by the total echo path delay time.
Voice can reflect back. This can happen but go unnoticed in a traditional voice network because the delay is so low. To the user, it sounds more like a side-tone than an echo. In a VoIP network, it will always be noticeable, since packetization and compression contributes to the delay.
The important thing to remember is that the cause of the echo is always with analog components and wiring. For instance, IP packets cannot simply turn around and go back to the source at a lower audio level or on digital T1/E1 circuits. So, on a call from one Cisco IP Phone to another, there should never be any problem. The only exception may be if one party is using a speakerphone that has the volume set too high or some other situation where an audio loop is created.
Testing Speakerphone and Volume Levels
Make sure that the problem phones are not using the speakerphone, and that they have the headset volume set to reasonable levels (start with 50% of the maximum audio level). Most of the time, the problems will occur when attaching to the PSTN by way of a digital or analog gateway.
Testing the Gateway
Determine which gateway is being used. If a digital gateway is in use, then it may be possible to add additional padding in the transmit direction (towards the PSTN). Since lower signal strength will yield less reflected energy, this should clear up the problem.
Additionally, you can adjust the receive level so that any reflected audio is reduced even further. It is very important to remember to make small adjustments at a time. Too much attenuation of the signal will make the audio impossible to hear on both sides. Alternatively, you can contact the carrier and request to have the lines checked. On a typical T1/PRI circuit in North America, the input signal should be -15 dB. If the signal level is much higher (-5 dB, for example), then echo will be the likely result.
Keeping an Echo Log
You should keep a log of all calls that experience echo. Record the time of the problem, the source phone number, and the number called. Gateways have a fixed time of 16 ms of echo cancellation. If the delay in the reflected audio is longer than this, the echo cancellor will be unable to work properly. This should not be an issue for local calls, and long distance calls should have external echo cancellers built into the network at the Central Office. This is one of the reasons why it is important to note the external phone number of a call that experiences echo.
Checking Your Loads
Verify your gateway and phone loads. Check Cisco Connection Online at www.cisco.com for the latest software loads, new patches, or release notes that may relate to the problem.
Diagnosing One-Way Audio or No Audio
When one person cannot hear another person during a call, there is one-way audio. This can be caused by an improperly configured Cisco IOS Gateway, a firewall, or a routing or default gateway problem, among other things.
Diagnostic Call Detail Records (CDR) are useful for determining if a call is experiencing one-way audio because they log transmitted and received packets (see Diagnosing Lost or Distorted Audio Problems). You can also press the i button twice quickly on a Cisco IP Phone 7960 during an active call to view details about transmitted and received packets.
Checking MTP
If you are using Media Termination Point (MTP) in a call (to support supplementary services such as hold and transfer with H.323 devices that do not support H.323 version 2), check to see if the MTP allocated is working correctly. Cisco IOS routers support H.323 version 2 beginning in Cisco IOS releases 11.3(9)NA and 12.0(3)T. Starting with Cisco IOS release 12.0(7)T, the optional H.323 Open/Close LogicalChannel is supported, so that software-based MTP is no longer required for supplementary services.
You can use the MTP device, as well as Conference Bridge and Transcoder features to bridge two or more audio streams. If the MTP, Conference Bridge, or Transcoder is not working properly, one-way audio or audio loss might be experienced. Shut down MTP to find out if MTP is causing the problem.
Testing Cisco CallManager Configuration
There are a number of causes for one-way audio or loss of audio during a call. The most common cause is an improperly configured device. For instance, Cisco CallManager handles the call setup for a Cisco IP Phone. The actual audio stream occurs between the two Cisco IP Phones (or between the Cisco IP Phone and a gateway). It is entirely possible that the Cisco CallManager is able to signal to a destination phone (making it ring) when the phone originating the call does not have an IP route to the destination phone. A common cause for this is when the default gateway in the phone is improperly configured either manually or on the DHCP server.
If a call consistently has one-way audio, use a PC that is configured on the same subnet as the phone and has the same default gateway and try to ping the destination Cisco IP Phone. Then using a PC that is configured on the same subnet as the destination phone (with the same default gateway as the destination phone) ping the source phone. Both of those tests should work. Other things that can affect the audio traffic include a firewall or a packet filter (such as access lists on a router) that may be blocking the audio in one or both directions. If the one-way audio occurs only through a voice-enabled Cisco IOS Gateway, then check the configuration carefully. IP routing must be enabled (look at the configuration to make sure that no ip routing is not found near the beginning of the configuration).
Checking RTP Header Compression
Make sure that if you are using RTP header compression to save bandwidth across the WAN, that it is enabled on each router carrying voice traffic that attaches to the WAN circuit. There should not be a situation where the RTP header is compressed on one end but cannot be decompressed on the other side of the WAN. A Sniffer is a very useful tool when troubleshooting one-way audio problems, because you can verify whether the phone or gateway is actually sending or receiving packets.
Note
Diagnosing Phone Resets
Phones will power cycle or reset for two reasons: 1) TCP failure connecting to Cisco CallManager, or 2) failure to receive an acknowledgement to the phone's KeepAlive messages.
The following procedure shows the steps for troubleshooting phone resets.
Procedure
Step 1
Step 2
Step 3
Step 4
Step 5
•
•
Step 6
Diagnosing Dropped Calls
Premature termination of dropped calls can be the result of a phone or gateway resetting (see Diagnosing Phone Resets) or a circuit problem, such as incorrect PRI configuration.
The first step is to determine whether this problem is isolated to one phone or to a group of phones. Perhaps the affected phones are all on a particular subnet or location. The next step is to check the Event Viewer for phone or gateway resets.
You will see one Warning and one Error message for each phone that resets. This indicates that the phone cannot keep its TCP connection to the Cisco CallManager alive, so the Cisco CallManager resets the connection. This may be because a phone was turned off or there may be a problem in the network. If this is an intermittent problem, it may be useful to use Microsoft Performance to record phone registrations.
If the problem seems to be occurring only through a certain gateway, such as a Cisco Access DT-24+, then the best course of action is to enable tracing and/or view the Call Detail Records (CDR). The CDR files will give a Cause Of Termination (CoT) that may help determine the cause of the problem. See "Call Detail and Call Management Records" for more information about CDRs.
The disconnect cause values (origCause_value and destCause_value— depending on which side hung up the call), map to Q.931 disconnect cause codes (in decimal) and can be found at the following location:
http://www.cisco.com/univercd/cc/td/doc/product/software/ios113ed/dbook/disdn.htm.
If the call is going out of a gateway to the PSTN, then the CDR can be used to determine which side is hanging up the call. Much of the same information can be obtained by enabling tracing on the Cisco CallManager. Because the trace tool can affect Cisco CallManager performance, you will want to use this option only as a last resort or if your network is not yet in production.
Diagnosing Cisco CallManager Feature Issues
Problems may occur with features that are used in conjunction with Cisco CallManager, such as Conference Bridge or Media Termination Point. Some feature problems can be caused by configuration errors or lack of server resources.
Diagnosing Codec and Region Mismatches
If a user gets a reorder tone (busy signal) when going off-hook, it could be the result of codec disagreement between regions. Verify that both call ends support at least one common codec (for example, G.711). If they do not, you will need to use transcoders.
A region specifies the range of supported codecs that can be used with each of the other regions. Every device belongs to a region.
Note
For example, Region1<->Region2 = G.711, means that a call between a device in Region1 and a device in Region2 can use G.711 or any other supported codec that requires the same or less bandwidth as G.711 (any supported codecs within G.711, G.729, G.723, and so on).
Note
Cisco IP Phone 7960—G.711A-law/µ-law, G.729AnnexB
Cisco IP Phone SP12 series and VIP 30—G.711A-law/µ-law, G.723.1
Cisco Access Gateway DE30 and DT-24+—G.711A-law/µ-law, G.723.1
Diagnosing Location and Bandwidth Problems
If a user gets a reorder tone after dialing a number, it could be because the Cisco CallManager bandwidth allocation for the location of one of the call end devices has been exceeded. Cisco CallManager checks for 24 kilobit available bandwidth for each device before making a call. If less than 24k bandwidth is available, Cisco CallManager will not set up the call and the user will hear a reorder tone.
12:42:09.017 Cisco CallManager|Locations: Orig=1 BW=12 Dest=0 BW=-1 (-1 implies infinite bw available) 12:42:09.017 Cisco CallManager|StationD - stationOutputCallState tcpHandle=0x4f1ad98 12:42:09.017 Cisco CallManager|StationD - stationOutputCallInfo CallingPartyName=, CallingParty=5003, CalledPartyName=, CalledParty=5005, tcpHandle=0x4f1ad98 12:42:09.017 Cisco CallManager|StationD - stationOutputStartTone: 37=ReorderTone tcpHandle=0x4f1ad98Once the call is established, the Cisco CallManager will subtract bandwidth from the locations depending on the codec used in that call. If the call is using G.711, Cisco CallManager will subtract 80k; if the call is using G.723, Cisco CallManager will subtract 24k; if the call is using G.729, Cisco CallManager will subtract 24k.
Diagnosing Conference Bridge Problems
Use the following information to troubleshoot a No Conference Bridge available error message. This could indicate either a software or a hardware problem.
First, check to see if you have any available software or hardware Conference Bridge resources registered with Cisco CallManager. Use either Microsoft Performance or the Admin Serviceability Tool to check the number of Unicast AvailableConferences.
Note
The Cisco IP Voice Media Streaming application performs the conference bridge function. One software installation of Cisco IP Voice Media Streaming will support 16 Unicast Available Conferences (3 people/conference), as shown in the following trace.
Note
10:59:29.951 Cisco CallManager|UnicastBridgeControl - wait_capabilities_StationCapRes - Device= CFB_kirribilli - Registered - ConfBridges= 16, Streams= 48, tcpHandle=4f1273810:59:29.951 Cisco CallManager|UnicastBridgeManager - UnicastBridgeRegistrationReq - Device Registration Complete for Name= Xoð ô%ð - DeviceType= 50, ResourcesAvailable= 16, deviceTblIndex= 0One E1 port (WS-X6608-E1 card contains 8x E1 ports) provides five Unicast Available Conferences (max conference size = 6), as shown in the following trace.
11:14:05.390 Cisco CallManager|UnicastBridgeControl - wait_capabilities_StationCapRes - Device= CFB00107B000FB0 - Registered - ConfBridges= 5, Streams= 16, tcpHandle=4f19d6411:14:05.480 Cisco CallManager|UnicastBridgeManager - UnicastBridgeRegistrationReq - Device Registration Complete for Name= Xoð ô%ð - DeviceType= 51, ResourcesAvailable= 5, deviceTblIndex= 0The following hardware trace on the Cisco Catalyst 6000 8 Port Voice T1/E1 and Services Module, indicates that the E1 port 4/1 in the card has registered as a Conference Bridge with Cisco CallManager.
greece-sup (enable) sh port 4/1Port Name Status Vlan Duplex Speed Type----- ------------------ ---------- ---------- ------ ----- ----------4/1 enabled 1 full -Conf BridgePort DHCP MAC-Address IP-Address Subnet-Mask-------- ------- ----------------- --------------- ---------------4/1 disable 00-10-7b-00-0f-b0 10.200.72.31 255.255.255.0Port Call-Manager(s) DHCP-Server TFTP-Server Gateway-------- ----------------- --------------- --------------- -----------4/1 10.200.72.25 - 10.200.72.25 -Port DNS-Server(s) Domain-------- ----------------- -------------------------------------------4/1 - 0.0.0.0Port CallManagerState DSP-Type-------- ---------------- -------- 4/1 registered C549Port NoiseRegen NonLinearProcessing----- ---------- -------------------4/1 disabled disabledSecond, check the maximum number of users configured in your Ad Hoc or Meet-Me conference to determine if the problem occurred because this number was exceeded.
Transcoding Problems
If you have installed a hardware transcoder in the Cisco Catalyst 6000 8 Port Voice T1/E1 and Services Module, and it does not work as expected (you cannot make calls between two users with no common codec), check to see if you have any available Transcoder resources registered with Cisco CallManager (must be hardware). Use Microsoft Performance or the Admin Serviceability Tool to check the number of MediaTermPointsAvailable available.
Note
One E1 port (WS-X6608-E1 card contains 8x E1 ports) provides Transcoder/MTP resources for 16 calls, as shown in the following trace.
Note
11:51:09.939 Cisco CallManager|MediaTerminationPointControl - Capabilities Received - Device= MTP00107B000FB1 - Registered - Supports 16 callsThe following hardware trace on the Cisco Catalyst 6000 8 Port Voice T1/E1 and Services Module, indicates that the E1 port 4/2 in the card has registered as an MTP/transcoder with Cisco CallManager.
greece-sup (enable) sh port 4/2Port Name Status Vlan Duplex Speed Type----- ------------------ ---------- ---------- ------ ----- ----------4/2 enabled 1 full - MTPPort DHCP MAC-Address IP-Address Subnet-Mask-------- ------- ----------------- --------------- ---------------4/2 disable 00-10-7b-00-0f-b1 10.200.72.32 255.255.255.0Port Call-Manager(s) DHCP-Server TFTP-Server Gateway-------- ----------------- --------------- --------------- -----------4/2 10.200.72.25 - 10.200.72.25 -Port DNS-Server(s) Domain-------- ----------------- -------------------------------------------4/2 - 0.0.0.0Port CallManagerState DSP-Type-------- ---------------- --------4/2 registered C549Port NoiseRegen NonLinearProcessing----- ---------- -------------------4/2 disabled disabled
Note
To make a call between two devices using a low bit rate code (such as G.729 and G.723) that do not support the same codec, a transcoder resource is required.
Assume Cisco CallManager has been configured such that the codec between Region1 and Region2 is G.729. The following scenarios are possible:
•
•
•
•
MTP Resource Problems
An MTP resource problem could be the source of the transcoding problem if a call is established, but supplementary services are not available on an H.323 device that does not support H323v2. First, determine whether you have any available software or hardware MTP resources registered with Cisco CallManager. Use Microsoft Performance or the Admin Serviceability Tool to check the number of MediaTermPointsAvailable.
Note
Using MTP to support supplementary services with H.323 devices that do not support H.323v2 allows one MTP software application to support 24 calls as shown in the following trace.
Note
10:12:19.161 Cisco CallManager|MediaTerminationPointControl - Capabilities Received - Device= MTP_kirribilli. - Registered - Supports 24 callsOne E1 port (WS-X6608-E1 card contains 8x E1 ports) provides MTP resources for 16 calls, as shown in the following trace.
11:51:09.939 Cisco CallManager|MediaTerminationPointControl - Capabilities Received - Device= MTP00107B000FB1 - Registered - Supports 16 callsThe following hardware trace from the Cisco Catalyst 6000 8 Port Voice T1/E1 and Services Module, indicates that the E1 port 4/2 in the card has registered as an MTP/transcoder with Cisco CallManager.
greece-sup (enable) sh port 4/2Port Name Status Vlan Duplex Speed Type----- ------------------ ---------- ---------- ------ ----- ----------4/2 enabled 1 full - MTPPort DHCP MAC-Address IP-Address Subnet-Mask-------- ------- ----------------- --------------- ---------------4/2 disable 00-10-7b-00-0f-b1 10.200.72.32 255.255.255.0Port Call-Manager(s) DHCP-Server TFTP-Server Gateway-------- ----------------- --------------- --------------- -----------4/2 10.200.72.25 - 10.200.72.25 -Port DNS-Server(s) Domain-------- ----------------- -------------------------------------------4/2 - 0.0.0.0Port CallManagerState DSP-Type-------- ---------------- --------4/2 registered C549Port NoiseRegen NonLinearProcessing----- ---------- -------------------4/2 disabled disabledSecond, in the Gateway Configuration screen of Cisco CallManager Administration, check to see if the Media Termination Point Required box is checked.
Third, verify that Cisco CallManager has allocated the required number of MTP devices.
From the SDI file:
15:22:23.848 Cisco CallManager|MediaManager(40) started15:22:23.848 Cisco CallManager|MediaManager - wait_AuConnectRequest15:22:23.848 Cisco CallManager|MediaManager - wait_AuConnectRequest - Transcoder Enabled15:22:23.848 Cisco CallManager|MediaManager - wait_AuConnectRequest - party1(16777357), party2(16777358), proxies=1, connections=2, current proxies=015:22:23.848 Cisco CallManager|MediaManager - wait_AuConnectRequest - proxy connections15:22:23.848 Cisco CallManager|MediaManager - wait_AuConnectRequest - allocating MTP(ci=16777359)15:22:23.848 Cisco CallManager|MediaManager - wait_AllocateMtpResourceRes15:22:23.848 Cisco CallManager|MediaManager - wait_AllocateMtpResourceRes - start 2 connections15:22:23.848 Cisco CallManager|MediaManager - wait_AllocateMtpResourceRes - creating connection between party1(16777357) and party2(16777359)15:22:23.848 Cisco CallManager|MediaManager - wait_AllocateMtpResourceRes - creating connection between party1(16777358) and party2(16777359)15:22:23.848 Cisco CallManager|MediaCoordinator - wait_MediaCoordinatorAddResource - CI=16777359 count=115:22:23.848 Cisco CallManager|MediaCoordinator - wait_MediaCoordinatorAddResource - CI=16777359 count=2Diagnosing Dial Plan Problems
A Dial Plan is a list of numbers and groups of numbers that tells the Cisco CallManager what devices (such as phones and gateways) to send calls to when a certain string of digits is collected. It is analogous to a static routing table in a router. Please be certain your dial plan concepts, basic call routing, and planning have been carefully considered and properly configured before trying to troubleshoot a potential dial plan issue. Very often, the problem lies with planning and configuration. Refer to the route plan configuration chapters in the Cisco CallManager Administration Guide for more information.
Use the following tips to help troubleshoot dial plans problems:
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Diagnosing Partition Problems
Route partitions inherit the error handling capabilities for the Cisco CallManager software. That is, a console and SDI file trace are provided for logging information and error messages. These messages will be part of the digit analysis component of the traces. It is vital that you know how the Partitions and Calling Search Spaces are configured and what devices are in each partition and its associated calling search space to determine the source of the problem. Refer to the route plan chapters in the Cisco CallManager Administration Guide and the Cisco CallManager System Guide for more information.
The following trace is an example of a number dialed that is in the device's Calling Search Space. For more detailed explanations about SDI traces, review the case studies in this document.
08:38:54.968 Cisco CallManager|StationInit - InboundStim - OffHookMessageID tcpHandle=0x6b8802808:38:54.968 Cisco CallManager|StationD - stationOutputDisplayText tcpHandle=0x6b88028, Display= 500008:38:54.968 Cisco CallManager|StationD - stationOutputSetLamp stim: 9=Line instance=1 lampMode=LampOn tcpHandle=0x6b8802808:38:54.968 Cisco CallManager|StationD - stationOutputCallState tcpHandle=0x6b8802808:38:54.968 Cisco CallManager|StationD - stationOutputDisplayPromptStatus tcpHandle=0x6b8802808:38:54.968 Cisco CallManager|StationD - stationOutputSelectSoftKeys tcpHandle=0x6b8802808:38:54.968 Cisco CallManager|StationD - stationOutputActivateCallPlane tcpHandle=0x6b8802808:38:54.968 Cisco CallManager|Digit analysis: match(fqcn="5000", cn="5000", pss="RTP_NC_Hardwood:RTP_NC_Woodland:Local RTP", dd="")In the Digit Analysis component of the previous trace, the pss (Partition Search Space, also known as Calling Search Space) is listed for the device placing the call.
In the following trace, RTP_NC_Hardwood;RTP_NC_Woodland;Local_RTP are the partitions this device is allowed to call.
08:38:54.968 Cisco CallManager|Digit analysis: potentialMatches=PotentialMatchesExist08:38:54.968 Cisco CallManager|StationD - stationOutputStartTone: 33=InsideDialTone tcpHandle=0x6b8802808:38:55.671 Cisco CallManager|StationInit - InboundStim - KeypadButtonMessageID kpButton: 5 tcpHandle=0x6b8802808:38:55.671 Cisco CallManager|StationD - stationOutputStopTone tcpHandle=0x6b8802808:38:55.671 Cisco CallManager|StationD - stationOutputSelectSoftKeys tcpHandle=0x6b8802808:38:55.671 Cisco CallManager|Digit analysis: match(fqcn="5000", cn="5000", pss="RTP_NC_Hardwood:RTP_NC_Woodland:Local RTP", dd="5")08:38:55.671 Cisco CallManager|Digit analysis: potentialMatches=PotentialMatchesExist08:38:56.015 Cisco CallManager|StationInit - InboundStim - KeypadButtonMessageID kpButton: 0 tcpHandle=0x6b8802808:38:56.015 Cisco CallManager|Digit analysis: match(fqcn="5000", cn="5000", pss="RTP_NC_Hardwood:RTP_NC_Woodland:Local RTP", dd="50")08:38:56.015 Cisco CallManager|Digit analysis: potentialMatches=PotentialMatchesExist08:38:56.187 Cisco CallManager|StationInit - InboundStim - KeypadButtonMessageID kpButton: 0 tcpHandle=0x6b8802808:38:56.187 Cisco CallManager|Digit analysis: match(fqcn="5000", cn="5000", pss="RTP_NC_Hardwood:RTP_NC_Woodland:Local RTP", dd="500")08:38:56.187 Cisco CallManager|Digit analysis: potentialMatches=PotentialMatchesExist08:38:56.515 Cisco CallManager|StationInit - InboundStim - KeypadButtonMessageID kpButton: 3 tcpHandle=0x6b8802808:38:56.515 Cisco CallManager|Digit analysis: match(fqcn="5000", cn="5000", pss="RTP_NC_Hardwood:RTP_NC_Woodland:Local RTP", dd="5003")08:38:56.515 Cisco CallManager|Digit analysis: analysis results08:38:56.515 Cisco CallManager||PretransformCallingPartyNumber=5000The key thing to note is that PotentialMatchesExist is the result of Digit Analysis of the numbers that were dialed until the exact match is found and the call is routed accordingly.
The following trace describes what happens when the Cisco CallManager is attempting to dial the directory number 1001 and it is not in the Calling Search Space for that device. Again, the key thing to note is the digit analysis routine had potential matches until only the first digit was dialed. The route pattern associated with the digit 1 is in a partition that is not in the device's calling search space, RTP_NC_Hardwood;RTP_NC_Woodland;Local_RTP. Therefore, the phone received a reorder tone (busy signal).
08:38:58.734 Cisco CallManager|StationInit - InboundStim - OffHookMessageID tcpHandle=0x6b8802808:38:58.734 Cisco CallManager|StationD - stationOutputDisplayText tcpHandle=0x6b88028, Display= 500008:38:58.734 Cisco CallManager|StationD - stationOutputSetLamp stim: 9=Line instance=1 lampMode=LampOn tcpHandle=0x6b8802808:38:58.734 Cisco CallManager|StationD - stationOutputCallState tcpHandle=0x6b8802808:38:58.734 Cisco CallManager|StationD - stationOutputDisplayPromptStatus tcpHandle=0x6b8802808:38:58.734 Cisco CallManager|StationD - stationOutputSelectSoftKeys tcpHandle=0x6b8802808:38:58.734 Cisco CallManager|StationD - stationOutputActivateCallPlane tcpHandle=0x6b8802808:38:58.734 Cisco CallManager|Digit analysis: match(fqcn="5000", cn="5000", pss="RTP_NC_Hardwood:RTP_NC_Woodland:Local RTP", dd="")08:38:58.734 Cisco CallManager|Digit analysis: potentialMatches=PotentialMatchesExist08:38:58.734 Cisco CallManager|StationD - stationOutputStartTone: 33=InsideDialTone tcpHandle=0x6b8802808:38:59.703 Cisco CallManager|StationInit - InboundStim - KeypadButtonMessageID kpButton: 1 tcpHandle=0x6b8802808:38:59.703 Cisco CallManager|StationD - stationOutputStopTone tcpHandle=0x6b8802808:38:59.703 Cisco CallManager|StationD - stationOutputSelectSoftKeys tcpHandle=0x6b8802808:38:59.703 Cisco CallManager|Digit analysis: match(fqcn="5000", cn="5000", pss="RTP_NC_Hardwood:RTP_NC_Woodland:Local RTP", dd="1")08:38:59.703 Cisco CallManager|Digit analysis: potentialMatches=NoPotentialMatchesExist08:38:59.703 Cisco CallManager|StationD - stationOutputStartTone: 37=ReorderTone tcpHandle=0x6b88028Route partitions work by associating a partition name with every directory number in the system. The directory number can be called only if the calling device contains the partition within a list of partitions to which it is permitted to place calls—its partition search space. This provides for extremely powerful control over routing.
When a call is being placed, Digit Analysis attempts to resolve the dialed address only in those partitions that the partition search space specifies. Each partition name comprises a discrete subset of the global dial-able address space. From each listed partition, Digit Analysis retrieves the pattern that best matches the sequence of dialed digits. Then, from among the matching patterns, Digit Analysis chooses the best match. If two patterns equally match the sequence of dialed digits, Digit Analysis breaks the tie by selecting the pattern associated with the partition listed first in the partition search space.
Diagnosing Security Problems
Using partitions and calling search spaces, in addition to more common filtering based on sections of the @ macro (which stands for the North American Numbering Plan) in a route pattern, you can configure Cisco CallManager to create a secure dialing plan for users. Partitions and Calling Search Spaces are an integral part of security and are especially useful for multi-tenant environments and for creating an individual user level. Filtering, a subset of the Calling Search Space/Partition concept, can add additional granularity to the security plan.
This is an extension to the Diagnosing Dial Plan Problems. Be advised, usually the last thing you want to do when trying to fix a filtering problem is to run an SDI trace. There is simply not enough information and the potential for causing more harm is too great.
Run a show tech command on Cisco CallManager to diagnose Slow Server Response.
Diagnosing Slow Server Response
Slow response from the server could result if the duplex of the switch does not match the duplex port setting on the Cisco CallManager server. For optimal performance set both switch and server to 100/Full. We do not recommend using the Auto setting on either the switch or the server. You must restart the Cisco CallManager server for this change to take effect.
Determining Gateway Reorder Tone Problems
Users placing a call through the gateway might get a reorder tone if they are attempting to make a restricted call or to call a number that has been blocked. A reorder tone may occur if the dialed number is out of service or if the PSTN has an equipment or service problem. Check to be sure the device giving the reorder tone has registered. Also, check your dial plan configuration to ensure that the call can be successfully routed.
The following procedure shows the steps for troubleshooting reorder tones through gateways.
Procedure
Step 1
Step 2
Step 3
Step 4
Diagnosing Gateway Registration Problems
One of the most common issues encountered with gateways on a Cisco CallManager is a registration problem. Registration can fail for a variety of reasons.
This section describes two similar but different categories of gateways. The Cisco Access AS-X, AT-X and Cisco Access DT-24+ and DE-30+ belong to one category. These gateways are stand-alone units that are not directly connected to a Network Management Processor (NMP). The second category includes the Analog Access WS-X6624 and Digital Access WS-X6608. These gateways are blades installed in a Catalyst 6000 chassis with direct connectivity to the NMP for control and statusing.
First, check that the gateway is up and running. All of the gateways have a heartbeat LED that blinks 1-second-on, 1-second-off when the gateway software is running normally. If this LED is not blinking at all, or blinking very rapidly, then the gateway software is not running. Normally this will result in an automatic reset of the gateway. Also, it is normal for the gateway to reset itself if it cannot complete the registration process after about 2 to 3 minutes. So you may happen to look at the heartbeat LED while the device is resetting, but if the normal blinking pattern does not appear in 10 to 15 seconds, then the gateway has suffered a serious failure. On the Cisco Access Analog gateways, the green heartbeat LED is on the far right of the front panel. On the Cisco Access Digital gateways, it is the far left red LED on the top edge of the card. On the Cisco Analog Access WS-X6624, it is a green LED inside the blade (not visible from the front panel) on the far right card edge near the front. Finally, on the Digital Access WS-X6608 there is a separate heartbeat LED for each of the 8 spans on the blade. There are 8 red LEDs across the card (not visible from the front panel) about two thirds of the way towards the back.
The second thing to check is that the gateway has received its IP address. A standalone gateway must receive its IP address via DHCP or BOOTP. A Catalyst gateway may receive its IP address by DHCP, BOOTP, or by manual configuration through the NMP. If you have access to the DHCP server, the best way to check a stand-alone gateway is to verify that the device has an outstanding lease on an IP address. If the gateway shows up on your server, this is a good indication, but not definitive. Delete the lease at the DHCP server, and then reset the gateway. If the gateway reappears on the server with a lease within a couple of minutes, then everything is working fine in this area. If not, then either the gateway cannot contact the DHCP server (Is a router improperly configured and not forwarding DHCP broadcasts? Is the server running?), or cannot get a positive response (Is the IP address pool depleted?). If checking these suggestions does not yield the answer, use a sniffer trace to determine the specific problem.
For a Catalyst 6000 gateway, you should check to make sure the NMP can communicate with the gateway. You can check this by trying to ping its internal IP address from the NMP. The IP address is in the format:
127.1.module.portIn our example, we would enter:
Console (enable) ping 127.1.7.1127.1.7.1 is aliveIf pinging works, then the sh-port command will show IP address information. Make sure the IP address information and the TFTP IP address is correct as well. If the gateway is failing to obtain valid DHCP information, the tracy utility (supplied by Cisco TAC) can be used to determine the problem. After obtaining this utility from TAC, issue the following command from the Cat6000 Command Line Interface (CLI):
tracy_start mod port
In this example, the WS-X6624 is module 7 and it only has a single 860 processor, so it is port 1. The command we would issue is tracy_start 7 1.
The following output is actually from the 860-console port on the gateway board itself, however, the output of the tracy command is nothing more than a remote copy of the 860-console port.
| || || | | | | || | | | | | | | | || | | | | | |:.:| | | | | | |:..C i s c o S y s t e m sCAT6K Analog Gateway (ELVIS)APP Version : A0020300, DSP Version : A0030300, Built Jun 1 2000 16:33:01ELVIS>> 00:00:00.020 (XA) MAC Addr : 00-10-7B-00-13-DE00:00:00.050 NMPTask:got message from XA Task00:00:00.050 (NMP) Open TCP Connection ip:7f01010100:00:00.050 NMPTask:Send Module Slot Info00:00:00.060 NMPTask:get DIAGCMD00:00:00.160 (DSP) Test Begin -> Mask<0x00FFFFFF>00:00:01.260 (DSP) Test Complete -> Results<0x00FFFFFF/0x00FFFFFF>00:00:01.260 NMPTask:get VLANCONFIG00:00:02.870 (CFG) Starting DHCP00:00:02.870 (CFG) Booting DHCP for dynamic configuration.00:00:06.570 (CFG) DHCP Request or Discovery Sent, DHCPState = INIT_REBOOT00:00:06.570 (CFG) DHCP Server Response Processed, DHCPState = INIT_REBOOT00:00:06.780 (CFG) IP Configuration Change! Restarting now...00:00:10.480 (CFG) DHCP Request or Discovery Sent, DHCPState = INIT00:00:14:480 (CFG) DHCP Timeout Waiting on Server, DHCPState = INIT00:00:22:480 (CFG) DHCP Timeout Waiting on Server, DHCPState = INIT00:00:38:480 (CFG) DHCP Timeout Waiting on Server, DHCPState = INITIf this timeout message continues to scroll by, then there is a problem contacting the DHCP server. First thing to check is that the Catalyst 6000 gateway port is in the correct VLAN.
You will find this information in the information you retrieved using the sh port command. If the DHCP server is not on the same VLAN as the Catalyst 6000 gateway, then make sure the appropriate IP helper addresses have been configured to forward the DHCP requests to the DHCP server. It is possible for the gateway to get stuck in the INIT state after a VLAN number change until the gateway resets. When in this state, it would not hurt to try resetting the gateway. Every time the 860 gets reset, your tracy session will be lost, so you must close your existing session and re-establish a new one by issuing the following commands:
tracy_close mod port
tracy_start mod port
If all this checks out and you are still seeing the DHCPState = INIT messages, you should check to see if the DHCP server is functioning correctly. If so, start a sniffer trace to see whether the requests are being sent and the server is responding.
Once DHCP is working correctly, the gateway will have an IP address that will allow the use of the tracy debugging utility. This utility is a built-in feature of the NMP command set for the Catalyst gateways, and available as a helper application that runs on Windows 98/NT/2000 for the stand-alone gateways. To use the helper application tracy utility, you need to connect to the gateway by using the IP address to which it is assigned. This tracy application works on all the gateways, provides a separate trace window for each gateway (up to eight may be traced at once), and allows traces to be logged directly to a file you specify.
The next step is to verify that the TFTP server IP address was correctly provided to the gateway. This is normally provided by DHCP in either Option 66 (by name or IP address), Option 150 (IP address only), or si_addr (IP address only). If your server has multiple Options configured, si_addr will take precedence over Option 150, which will take precedence over Option 66. If Option 66 provides the DNS_NAME of the TFTP server, then the DNS server(s) IP address(es) must have been specified by DHCP, and the name entered in Option 66 must resolve to the correct TFTP server IP address. A Catalyst gateway could be configured by the NMP to disable DHCP, and the NMP operator must then enter all configuration parameters by hand at the console, including the TFTP server address.
Additionally, the gateways will always attempt to resolve the name CiscoCM1 via DNS. If successful, the CiscoCM1 IP address will take precedence over anything the DHCP server or NMP tells it for the TFTP server address, even if the NMP has DHCP disabled.
You can check the current TFTP server IP address in a gateway by using the tracy utility. Enter the following command to get the configuration task number:
TaskID: 0Cmd: show tlLook for a line with config or CFG and use the corresponding number as the taskID for the next line. For example, for the Cisco Access Digital gateway. In the examples that follow, we have bolded lines of text to make it easier for you to see the messages being explained. In the actual display output, text is not bolded. The examples are from an WS-X6624 model, the command to dump the DHCP information is:
TaskID: 6Cmd: show dhcpThe TFTP server IP address is then clearly shown. If it is not correct, verify that your DHCP options and other information it provides are correct.
Once the TFTP address is correct, the next step would be to ensure that the gateway is getting its configuration file from the TFTP server. If you see the following in the tracy output, your TFTP service may not be working correctly or the gateway might not be configured on the Cisco CallManager:
00:09:05.620 (CFG) Requesting SAA00107B0013DE.cnf File From TFTP Server00:09:18.620 (CFG) TFTP Error: Timeout Awaiting Server Response for .cnf File!The gateway will attempt to connect to the same IP address as the TFTP server if it does not get a configuration file. This is fine unless you are in a clustered environment in which the gateway needs to receive its list of redundant Cisco CallManagers. If the card is not getting its TFTP information correctly, check the TFTP service on the Cisco CallManager and make sure it is running. Also, check the TFTP trace on the Cisco CallManager.
Another common problem is that the gateway is not configured correctly on the Cisco CallManager. A typical error is entering an incorrect MAC address for the gateway. If this is the case, for a Catalyst 6000 gateway, you will probably get the following messages on the NMP console every two minutes:
2000 Apr 14 19:24:08 %SYS-4-MODHPRESET:Host process (860) 7/1 got reset asynchronously2000 Apr 14 19:26:05 %SYS-4-MODHPRESET:Host process (860) 7/1 got reset asynchronously2000 Apr 14 19:28:02 %SYS-4-MODHPRESET:Host process (860) 7/1 got reset asynchronouslyThis is what the tracy output would look like if the gateway is not in the Cisco CallManager database:
00:00:01.670 (CFG) Booting DHCP for dynamic configuration.00:00:05.370 (CFG) DHCP Request or Discovery Sent, DHCPState = INIT_REBOOT00:00:05.370 (CFG) DHCP Server Response Processed, DHCPState = BOUND00:00:05.370 (CFG) Requesting DNS Resolution of CiscoCM100:00:05.370 (CFG) DNS Error on Resolving TFTP Server Name.00:00:05.370 (CFG) TFTP Server IP Set by DHCP Option 150 = 10.123.9.200:00:05.370 (CFG) Requesting SAA00107B0013DE.cnf File From TFTP Server00:00:05.370 (CFG) TFTP Error: .cnf File Not Found!00:00:05.370 (CFG) Requesting SAADefault.cnf File From TFTP Server00:00:05.380 (CFG) .cnf File Received and Parsed Successfully.00:00:05.380 (CFG) Updating Configuration ROM...00:00:05.610 GMSG: GWEvent = CFG_DONE --> GWState = SrchActive00:00:05.610 GMSG: CCM#0 CPEvent = CONNECT_REQ --> CPState = AttemptingSocket00:00:05.610 GMSG: Attempting TCP socket with CCM 10.123.9.200:00:05.610 GMSG: CCM#0 CPEvent = SOCKET_ACK --> CPState = BackupCCM00:00:05.610 GMSG: GWEvent = SOCKET_ACK --> GWState = RegActive00:00:05.610 GMSG: CCM#0 CPEvent = REGISTER_REQ --> CPState = SentRegister00:00:05.680 GMSG: CCM#0 CPEvent = CLOSED --> CPState = NoTCPSocket00:00:05.680 GMSG: GWEvent = DISCONNECT --> GWState = Rollover00:00:20.600 GMSG: GWEvent = TIMEOUT --> GWState = SrchActive00:00:20.600 GMSG: CCM#0 CPEvent = CONNECT_REQ --> CPState = AttemptingSocket00:00:20.600 GMSG: Attempting TCP socket with CCM 10.123.9.200:00:20.600 GMSG: CCM#0 CPEvent = SOCKET_ACK --> CPState = BackupCCMAnother possible registration problem could be that the load information is incorrect or the load file is corrupt. The problem could also occur if the TFTP server is not working. In this case, tracy clearly shows that the TFTP server reported the file is not found:
00:00:07.390 GMSG: CCM#0 CPEvent = REGISTER_REQ --> CPState = SentRegister00:00:08.010 GMSG: TFTP Request for application load A002130000:00:08.010 GMSG: CCM#0 CPEvent = LOADID --> CPState = AppLoadRequest00:00:08.010 GMSG: ***TFTP Error: File Not Found***00:00:08.010 GMSG: CCM#0 CPEvent = LOAD_UPDATE --> CPState = LoadResponseIn this case, you can see that the gateway is requesting application load A0021300, although the correct load name would be A0020300. For a Catalyst 6000 gateway, the same problem can occur when a new application load needs to get its corresponding DSP load as well. If the new DSP load is not found, a similar message will appear.
ELVIS>> 00:00:00.020 (XA) MAC Addr : 00-10-7B-00-13-DE00:00:00.050 NMPTask:got message from XA Task00:00:00.050 (NMP) Open TCP Connection ip:7f01010100:00:00.050 NMPTask:Send Module Slot Info00:00:00.060 NMPTask:get DIAGCMD00:00:00.160 (DSP) Test Begin -> Mask<0x00FFFFFF>00:00:01.260 (DSP) Test Complete -> Results<0x00FFFFFF/0x00FFFFFF>00:00:01.260 NMPTask:get VLANCONFIG00:00:02.030 (CFG) Starting DHCP00:00:02.030 (CFG) Booting DHCP for dynamic configuration.00:00:05.730 (CFG) DHCP Request or Discovery Sent, DHCPState = INIT_REBOOT00:00:05.730 (CFG) DHCP Server Response Processed, DHCPState = BOUND00:00:05.730 (CFG) Requesting DNS Resolution of CiscoCM100:00:05.730 (CFG) DNS Error on Resolving TFTP Server Name.00:00:05.730 (CFG) TFTP Server IP Set by DHCP Option 150 = 10.123.9.200:00:05.730 (CFG) Requesting SAA00107B0013DE.cnf File From TFTP Server00:00:05.730 (CFG) .cnf File Received and Parsed Successfully.00:00:05.730 GMSG: GWEvent = CFG_DONE --> GWState = SrchActive00:00:05.730 GMSG: CCM#0 CPEvent = CONNECT_REQ --> CPState = AttemptingSocket00:00:05.730 GMSG: Attempting TCP socket with CCM 10.123.9.200:00:05.730 GMSG: CCM#0 CPEvent = SOCKET_ACK --> CPState = BackupCCM00:00:05.730 GMSG: GWEvent = SOCKET_ACK --> GWState = RegActive00:00:05.730 GMSG: CCM#0 CPEvent = REGISTER_REQ --> CPState = SentRegister00:00:06.320 GMSG: CCM#0 CPEvent = LOADID --> CPState = LoadResponse00:01:36.300 GMSG: CCM#0 CPEvent = TIMEOUT --> CPState = BadCCM00:01:36.300 GMSG: GWEvent = DISCONNECT --> GWState = Rollover00:01:46.870 GMSG: CCM#0 CPEvent = CLOSED --> CPState = NoTCPSocket00:01:51.300 GMSG: GWEvent = TIMEOUT --> GWState = SrchActive00:01:51.300 GMSG: CCM#0 CPEvent = CONNECT_REQ --> CPState = AttemptingSocket00:01:51.300 GMSG: Attempting TCP socket with CCM 10.123.9.200:01:51.300 GMSG: CCM#0 CPEvent = SOCKET_ACK --> CPState = BackupCCM00:01:51.300 GMSG: GWEvent = SOCKET_ACK --> GWState = RegActive00:01:51.300 GMSG: CCM#0 CPEvent = REGISTER_REQ --> CPState = SentRegister00:01:51.890 GMSG: CCM#0 CPEvent = LOADID --> CPState = LoadResponseThe difference here is that the gateway gets stuck in the LoadResponse stage and eventually times out. This problem can be resolved by correcting the load file name in the Device Defaults area of Cisco CallManager Administration.
Diagnosing Gatekeeper Problems
Before starting any gateway-to-gatekeeper troubleshooting, verify that there is IP connectivity within the network. Assuming that there is IP connectivity, use the information in this section to troubleshoot your gateway.
Inter-Cluster Trunks or H.225 Gateway
Refer to the Cisco CallManager Administration Guide and Cisco CallManager System Guide at the following location:
http://www.cisco.com/univercd/cc/td/doc/product/voice/c_callmg/3_1/index.htm
Admission Rejects
Admission Rejects (ARJ) are issued when Cisco CallManager has registered with the Gatekeeper but cannot send a phone call. Configuration issues on the gatekeeper should be the primary focus when the gatekeeper is issuing an ARJ. The following procedure is a general guideline for troubleshooting.
Procedure
Step 1
Step 2
Step 3
Registration Rejects
Registration Rejects (RRJ) are issued when Cisco CallManager cannot register with the Gatekeeper. Configuration issues on the gatekeeper should be the primary focus when the gatekeeper is issuing a RRJ.
The following procedure is a general guideline for troubleshooting.
Procedure
Step 1
Step 2
Step 3
