IPCC Troubleshooting Guide

Introduction

This document provides information to troubleshoot the Internet Protocol Contact Center (IPCC), which focuses on the Peripheral Gateway (PG) and the Cisco Intelligent Contact Management (ICM). Although this document contains some information about common problems with Cisco CallManager and Cisco Global Directory, this document makes no attempt to completely describe these components. Rather, this document concentrates on symptoms and methods to identify the source of problems that the PG sees.  The problems can relate to software or configuration.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• How to troubleshoot and support Cisco ICM PG

Components Used

The information in this document is based on these software and hardware versions:

• Cisco ICM version 4.6.2

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Symptoms and Troubleshooting Actions

Look at the PG logs for IPCC. When you see unspecified errors in the Peripheral Interface Manager (PIM), Open Peripheral Controller (OPC), or Computer Telephony Interface (CTI) Server logs, go directly to the JTapi Gateway (GW) log for a better text description of the problem. The JTAPI interface usually provides exceptions when things go wrong on third-party requests. These exceptions provide only string descriptions with no error code. As a result, the PIM/OPC/CTI Server logs many errors as unspecified errors.

Cisco IPCC PIM Does Not Go Active

PIM Not Enabled

Check for the existence of a PIM log. If there is no PIM log, check to make sure the peripheral is enabled in the Cisco ICM Setup. Sometimes, the peripheral is added, but you need to enable the peripheral.

Select Edit > Peripheral, and check the Enabled check box.

PIM Restarts

If the PIM process restarts, view the PIM log on the Cisco CallManager PG with the dumplog utility. If the log file indicates an error with the OPCHeartbeatTimeout, you must modify this registry setting. Use regedt32 to make the change.

Modify OPCHeartbeatTimeout in the registry under eagtpim dynamic data to 10. Here is the path:

HKEY_LOCAL_MACHINE\SOFTWARE\GeoTel\ICR\<cust_inst>\<PG_inst>\PG\CurrentVersion\
PIMS\<pim_inst>\EAGENTData\Dynamic

Note: This key appears over two lines here due to space limitations.

PIM in Idle State in PIM Log Window

If the PIM process is in an idle state, run these checks:

• Check the PIM log. You must see, "Attempt to Activate", at least once a minute.

• If the PIM is not active, use the dumplog utility to check the OPC log. Run opctest to see if the OPC process receives the configuration from the router.

• If the OPC process does not receive the configuration from the router, use the dumplog utility to view the pgagent log. The pgagent process must have an active path to the Central Controller. If pgagent does not have an active path, check the network connectivity and the DMP configuration in the PG setup. On the router, use the dumplog utility to view the ccagent log. Verify whether the PG device (DMP System ID) is enabled as a device on the router.

• Enable the PG in the router configuration through setup or in the registry under DMP registry.

• In a command window, use the tracert command to verify network connectivity between the router and the PG.

  Note: There can be a discrepancy between DNS and DHCP.

• Verify whether the IP address for the router is in the host file in the c:\winnt\system32\drivers\etc directory.

• Check if the Logical Controller ID configured in PG > Setup matches the ID for the PG Logical Interface Controller in Configure > ICM. Ensure that the Peripheral ID configured for the peripheral in PG > Setup matches the ID for the peripheral in Configure > ICM.

• Modify ICM setup to match the configuration.

JGW Process Does Not Go Active

Incorrect Version of Microsoft Java Virtual Machine (JVM) is Installed

• Go to a command prompt and type jview and press ENTER. Information on the installed Java version appears:

  Microsoft (R) Command-line Loader for Java version 5.00.3190

• If you do not see this output, or if the version is earlier than 3190, you must install the correct version of the Microsoft JVM. Run msjavx86.exe. This file is installed in the icr\bin directory during setup.

Java Classpath is Incorrect

• From a command prompt, go to the icr\bin directory and type jtapigw and press ENTER.  A response similar to this appears:

  18:43:17 Fail: Node Manager Required Arguments missing.
  18:43:17 Trace: at com/cisco/icr/ems.EMSFailMessage (ems.java:164)
  18:43:17 Trace: at com/cisco/icr/NodeManager.setStartupArgs
   (NodeManager.java:27)
  18:43:17 Trace: at MainWorkerThread.mainImplementation
   (MainWorkerThread.java:41)
  18:43:17 Trace: at MainWorkerThread.run (MainWorkerThread.java:19)

• Alternatively, this message appears:

  Java.lang.NoClassDefFoundError: com/cisco/icr/GWThreadGroup

• If you see the second message when you run jtapigw, check your Java classpath.  Use the registry editor to look at the value Classpath under the SOFTWARE\Microsoft\Java VM key.  Set the key like this:

  C:\WINNT\java\classes;.;c:\icr\bin\icrjavalib.zip

Note: The drive letter and Windows system directory can differ and the characters after classes and before c:\icr… are: semicolon, period, and semicolon.

Cisco JTAPI Client Not Installed on PG

• From a command prompt, go to the icr\bin directory, type jtapigw and press ENTER.  A response similar to this appears:

  18:43:17 Fail: Node Manager Required Arguments missing.
  18:43:17 Trace: at com/cisco/icr/ems.EMSFailMessage (ems.java:164)
  18:43:17 Trace: at com/cisco/icr/NodeManager.setStartupArgs (NodeManager.java:27)
  18:43:17 Trace: at MainWorkerThread.mainImplementation (MainWorkerThread.java:41)
  18:43:17 Trace: at MainWorkerThread.run (MainWorkerThread.java:19)

• Instead of the above, you can see this message:

  Java.lang.NoClassDefFoundError

• If you see something like the second message when you run jtapigw, verify the Cisco JTAPI client is installed on the PG. Check for the file CiscoJtapiVersion.class under c:\winnt\java\lib.

• If this file does not exist, you can install the file on the PG from the Cisco CallManager; http://<callmanager name>/main.asp. You can locate the file under the application tab.

JTAPI GW Log Complains About Incompatible JTAPI Version

• If you installed only JTAPI 4.1 service pack (SP) 4 with any hot fix less than 50 on the Cisco CallManager PG, you need to upgrade.

• If you have just run ICM > Setup to upgrade the PG, check to make sure that the date/time on the file \icr\bin\icrjavalib.zip shows an updated date. The date must be approximately the same as the date/time on the bldXXXXX.version file, within about a day.

Note: Setup cannot update this file if the file is in use when you run Setup. This situation can occur if you have an internet browser open because, the browser treats the zip file as a directory for the class path if the browser opens the zip. In order to avoid this problem, close all browser sessions before you run Setup.  If Setup cannot update the file, a message appears, and instructs you to reboot your PC in order to update the files. You must reboot.

JTAPIGW Cannot Connect to Cisco CallManager

• The PIM communicates with the JTAPI Gateway (JTAPIGW), and the JTAPIGW communicates with the Cisco CallManager. As the PIM tries to go active, the PIM tells JTAPIGW to initialize communications with the Cisco CallManager through JTAPI.

• You must see messages that indicate that the JTAPIGW has accepted a connection from the PIM and contacts getProvider(), for example:

  13:16:47 pg2A-jgw1 Trace:Calling getProvider () 172.24.79.128; 
    login=PGUser;passwd=<***edited***>
  13:16:52 pg2A-jgw1 Trace: Returned successfully from getProvider()

  Note: This example appears over multiple lines due to space limitations.

If you do not see the trace returned successfully, you can see other errors after the call to getProvider().  The trace to getProvider() shows the parameters used to initialize JTAPI.  The first parameter is the service name, which is the IP host name or IP address of the Cisco CallManager machine.  In this example, the IP address is used.  If a name is used, the PG must be able to resolve the name through a host file or DNS.  Make sure you can ping the name or address.  If you need to change the service name, re-run ICM > Setup and change the name in the Edit Peripheral dialog.

The trace of the call to getProvider() also shows the login name used. Notice that the trace does not show the password. The login name and password are taken from what the administrator enters under ICM > Setup. These must match a valid user and password configured in the directory and administered in the Cisco User Preferences Web page to have the ability to control each of the agent devices and route points. Check to make sure the name and password are correct in ICM > Setup. Configure the user in the directory to have permission to control only valid agent devices and route points.

JGW Log Indicates Unknown Host

The JTAPI GW process cannot resolve the address of the Cisco CallManager. Configure the service parameter in the PIM dialog box in Setup with the Cisco CallManager host name or IP address. If the host name configuration for Cisco CallManager is correct, make sure you can ping the Cisco CallManager. If not, use the IP Address of the Cisco CallManager, instead of the host name.

JGW Log Indicates Invalid Password or User

The JTAPI GW logs into the Global Directory with a username and password. The username and password in the PIM dialog box in Setup must match the username and password for the user configured in the global directory in Cisco CallManager admin Web page under ccmadmin > User > Global Directory.

If the user does not exist, add a new user. Make sure to check the CTI Enabled check box at the bottom of the page.

CTI Check Box Not Enabled on Global Directory User Page

A check box on the Cisco CallManager global directory user page can enable or disable the CTI privileges for a PIM or IP IVR user. You must check and update this check box in order for the PIM/JTAPI GW to go active. This check box ensures that two CTI devices cannot connect to the Cisco CallManager, which can cause problems (default limit is 400).

Cisco CallManager Service Does Not Run

• On Cisco CallManager version 3, this service shows in service control as "Cisco CallManager". Start the service.

• The Cisco CallManager service is normally set to restart if it exits abnormally, but you can configure this to "off" for possible issues with migration of devices on failover scenarios.

• Check the Event log to see if the Cisco CallManager service restarts. The system sometimes restarts if the system identifies a problem with adequate CPU use. The system reports errors or warnings in the event log that indicate a "slow SDL timer thread". With this type of error, Cisco CallManager restarts. This version of Cisco CallManager runs at normal priority so other applications that run on the system can interfere with the call signal.

• When the physical memory is less or the system encounters other timing issues, the Cisco CallManager can come up with an error that indicates it could not initialize after a 10 minute time-out and restart. There is a DCOM component service for the Cisco CallManager database layer (DBL) that has a problem initializing. Stop and start this DBL DCOM service through component services – DCOM components to solve this problem.

  Note: This is not the same as a system service like Cisco CallManager.

  Open a case with the Cisco Technical Assistance Center (TAC). This can probably be a problem the next time you restart the system, unless you resolve the underlying timing issue.

Directory Problems (Configuration, Not Running, Directory Password)

Directory Service Does Not Run

• Confirm that the directory service is up and runs properly.  By default, this is the DC Directory Server in service control on the Cisco CallManager machine. Try to start the machine. You can encounter errors.

• The Directory Service can go into a paused state if the system runs out of memory or disk space. Errors appear in the Microsoft Windows 2000 event log. Resolve resource issues and restart the directory service, if necessary.

Cisco CallManager Web User Page Does Not Work

Verify whether the Cisco Global Directory user Web page can actually view and configure users and assign permissions to control devices. Both the JTAPIGW and the Web page use the Cisco CallManager to access the directory server to access users and permissions. If the problem with JTAPIGW is due to a directory server problem, the user Web page can also have problems. Possible reasons are that the directory server does not run or the directory is not configured correctly, if at all.

Directory Server Not Installed

In order to use Cisco CallManager 3.0.5 and later, you must install a directory server. The AVVID DC Directory is the default which is available on the Spirian installation CDs. After you install the directory server, the installation of the Cisco CallManager configures the directory.

You must perform this installation correctly, and the directory server must be up and must run properly in order for the JTAPIGW to log into the Cisco CallManager and use JTAPI.

Make sure that the DC Directory Service and Cisco CallManager both run properly.

When you install Cisco CallManager, you must enter "ciscocisco" when you see the directory manager password prompt. If you enter anything else, you may have to remove the DC Directory Software (Add/Remove) and reinstall. If the removal process tells you that some files cannot be removed, you must manually remove or rename the current c:\dcdsrvr directory.

Directory Server Installed But Does Not Run

Check the control panel to confirm that the service cannot start. Next, verify whether the administrator is configured and the log in and password are correct for the service in the Properties field.

Directory Server Installed and Runs But Cannot Log in With the DCD Admin Tool

Start DC Directory Admin from your system Start menu. Log in with your user Directory Manager with the password "ciscocisco" (default) or whatever password the admin has configured. If you receive an error that indicates that the user is not configured, run one of the Cisco AVVID configuration files in the DCDSrvr\bin directory. If this is the primary Cisco CallManager, Publisher, run avvid_cfg.cmd from the DOS prompt. If this is a secondary Cisco CallManager, run avvid_scfg.cmd from the command prompt.

If you see errors that indicate this is already configured, the user does exist. If there are no errors, things must start to work properly now. Go back and check access from the Global Directory User pages on ccmadmin.

Note: The DC Directory goes into pause mode if the directory is low on system resources.

Agent Cannot Log In

This example uses a sample ICM configuration for a device target:

Device Target Sample
Enterprise Name	Agent9782755100
Global Address	Agent9782755100
ConfigParm	/devtype CiscoPhone /dn 9782755100

The next example uses a sample ICM configuration for an agent:

Agent Sample
Peripheral	CCMPG_PIM1
Peripheral Number	1234
Password	XXX

When you run ICM > Setup for the PG, you specify an agent extension length of "4". So, in the sample configuration, the extension for the sample device is the last 4 digits of the /dn parameter (for example, "5100").

Try to log in with CTITest.

If you cannot log an agent in with the soft phone, try the same operation through ctitest . Here is a sample list of ctitest commands that you can use to log in the sample agent to the sample device target. This list of commands assumes that the CTI Server listens on port 42027 on machine CTIServerA. This list also assumes that the device is an extension for the peripheral represented as ICM peripheral 5000.

config /hostA CTIServerA
config /portA 42067
config /service CLIENT_EVENTS+CLIENT_CONTROL
agent /periph 5001 /inst 9782755100
open
login 1234 XXX /inst 9782755100

PIM and CTI Server are Not Active in OPC

Use the opctest "status" command and confirm the IPCC PIM and the CTI Server show in PIM_ACTIVE and CTI_ACTIVE states. The title bars of the PIM and CTI Server log windows also indicates the process state.

CTI Client Cannot Connect

Check the settings to connect to CTI Server. For the desktop soft phone, the settings are in the .ini file (usually c:\program files\geotel\cti desktop\cticonfig.ini). The settings to check include:

• PeripheralID—This value must match peripheral ID for IPCC peripheral in Configure > ICM.

• SideAHost—This value must be the IP host name or address of CTI Server side A.

• SideBHost—This value must be IP host name or address of CTI Server side B.  If CTI Server is simplexed, you can leave this field blank.

• SideAPort—This value must match the port the CTI Server on side A listens for connections. This value is specified in ICM Setup for the CTI Server. CTI Server shows this port in the title bar and logs this value when CTI Server starts. Verify whether the client can ping the CTI Server.

CTI Client Gets Error Indicates Agent Needs to be Logged into ACD

Run the setup.exe that resides in the \icr\bin directory on the PG/CTI Server. Select the CTI Gateway component. Verify whether the Agent Login Required check box is unchecked. This check box selection is not applicable for IPCC or any third-party control applications. The purpose of this check box is to monitor applications other ACD agents.

PIM Log Shows One of the These Login Errors

Use procmon to the pim and "trace tp*" to turn on third-party tracing (case sensitive). This must show the login request. Verify whether the parameters are correct. The instrument is traced as "Device=". This value must match the /dn string in the device target configparam. The agent ID is traced as "AgentID=". This value must match the agent peripheral number in Configure/ICM.

• INVALID_PASSWORD

  Make sure the password is correct (the password may not be traced as clear text). If the password is incorrect the log must show an INVALID_PASSWORD_SPECIFIED error.

• INVALID_OBJECT

  Indicates that the configuration parameters in the Device Target contain an invalid device type. This error appears like this with spaces between keywords:

  /devtype CiscoPhone /dn 9782755100

• INVALID_DEVICE_TARGET

  Indicates something in the Device Target is invalid, most likely something in the configuration parameters field. With the dumplog utility, view the PIM log for the last time that the PIM restarted. The log validates the device targets and log errors when the device target configuration strings are invalid.

JGW Log Shows Log in Errors

Check the jgw log for any errors that occur upon login attempts. Use procmon to the PIM and "trace *TP*" to turn on third-party tracing (case sensitive). Look for the line, "MsgAddCallObserver: Addr: XXXX" where XXXX is the extension into which you try to log in. This extension must be a valid Cisco CallManager extension on a device which the PG user has permission to control. The extension must be the correct number of digits for the phone as the Cisco CallManager knows. In other words, the extension must be the number that you dial from another phone on the same Cisco CallManager to reach the phone in question.

Device Not in Provider Domain

If the jgw log shows an exception, which indicates that the device is not in the provider domain, the phone is not associated with the user with which JTAPI GW logs in. Make sure the extension at the far side of the Global Directory user device association list is correct. Also ensure that the device line number is not registered twice. Shared line appearance is a Cisco CallManager feature that IPCC does not support. You can inadvertently try to set up a shared line appearance with two phones that have the same line. If you change one line number, the other changes, and PG cannot log into the correct device. In order to resolve this problem, delete both lines and add them to the Cisco CallManager.

INVALID_SKILL_GROUP_SPECIFIED

In order to log in, an agent must be configured in Configure/ICM as a member of at least one skill group (Skill Group Member).

Agent Already Logged In to Another Phone

Make sure that the agent (as the Agent Peripheral Number represents) is not already logged into another device target. One way to check this is to run Monitor ICR and run the Free from Agent report for the agent in question. If the agent is logged in, this shows you the network target ID of the device target in which the agent is logged in. The agent data appears in the awdb only if you have configured ICM to send agent data for the peripheral to this AW.

• You could also query for this in isqlw against the Agent_Real_Time table in the awdb. First, find the skill target for the agent (for example, select * from Agent where PeripheralID = XXX and PeripheralNumber = YYY). Then, check whether the agent is logged in (for example, select * from Agent_Real_Time where SkillTargetID = XXX).

• You can also check for this when you connect to procmon to PIM and run dagent <agent peripheral number>.

Device Target Already Logged In

Make sure the device target (as the instrument specifies) does not already have another agent logged in.

• One way to check this is to run isqlw against the Agent_Real_Time table in the awdb. First, find the network target ID for the device target in question. For example, select * from Device_Target where ConfigParam like ‘%1003%’. Now, see if the device target is logged in. For example, select * from Agent_Real_Time where NetworkTargetID = XXX.

• You can also check for this when you connect to procmon to the PIM and dump the device target. There are two ways to dump the device target. The ddt command takes a network target ID as input and dumps the device target. The deadt command takes the /dn string from the device target configuration as input and dumps the device target. For example, if the device target /dn string is /dn 9782755100, you dump the device target as deadt 9782755100.

Device Not Associated With PG User

Go to the Cisco CallManager Web page, select User/Global Directory and find the userid that the PG uses. Check the “Associated Devices” and make sure the user has permission to control the device.

• If you cannot find the device on the user page (checked or unchecked), there can be a problem with the synchronization between the database (where Cisco CallManager stores the devices) and the directory server (which stores the devices and stores user profiles). Check to see if the directory server (DC Directory Server) runs.

• Check the Windows NT Event Viewer Application log and look for errors from the DC directory or metalink. If import errors occur, run avvid_recfg from c:\dcdsrvr\bin.

• Make sure Microsoft Java Virtual Machine (JVM) is installed on the Cisco CallManager machine. In order to test this, type jview from a command prompt. For Cisco CallManager 2.4, you must install JVM manually. For Cisco CallManager 3, the platform is Windows 2000 and JVM installation is automatic.

Phone Device Not Active

Check whether the phone is powered on, registered with Cisco CallManager, and able to make and receive calls from the phone without agent control.

Agent Cannot Make Call

Agent in Not Ready State

Make sure the agent is logged in and not in Available state.  If the agent is not Available, the agent cannot make a call. In order to make a call, first click Not Ready.

Incorrect Agent Desk Settings

If there is an error only when you dial certain numbers, check those numbers from a physical phone to make sure you can dial in successfully. If you have configured an ICM dialed number plan, check to see if the number you dial matches one of the wildcards in your dialed number plan. Then check to see if the agent desk settings for the agent permit the agent to dial the type of number that the dialed number plan entry identifies (for example, International).

Dialed Number Plan in PIM Blocks Access

The dialed number plan configured for each PIM can be incorrectly configured or correctly configured to prevent an agent from calling out to a certain number. The error in the PIM log must indicate a permission error. The numbers for agents and devices cannot overlap when the dialed number plan is used to make agent to agent calls.

Agent Makes Short Call, Has To Wait To Make New Call

The Router makes the agent unavailable when the agent makes a call or when a call is routed to the agent. This mechanism allows the Router to route another call to the agent before the PIM reports the call has arrived. Some networks take several seconds to actually route the call. The Router does not cancel the timer based on agent state.

If the actual time taken to route calls to the PIM from the routing client is relatively short, you can change the configurable time in the router. On one of the routers in a DOS command window, use rtsetting.exe. Look under Extrapolation > Agent. The default setting is 10 seconds. If the value is too short, the Router routes calls to agents that are about to receive a call. This causes the PIM to drop calls.

The default time-out on the PIM is 7 seconds. You can modify this value with the regedt32 command. Add the "AgentReserveTimout" key in this path:

HKEY_LOCAL_MACHINE\SOFTWARE\GeoTel\ICR\<cust_inst>\<pg_inst>\PG\CurrentVersion\
  PIMS\<pim_inst>\EAGENTData\Dynamic\

Note: This key will be added in the version 4.1.5 setup.

Note: This key appears over two lines here due to space limitations.

The PIM number must always be a few seconds less than the Router extrapolation timer to prevent the Router from sending new pre-call events to the PIM before the original event is processed. This causes problems in the PIM.

If the call arrives after the PIM time out, the call is considered a non-ACD call, and none of the context variables, service, or skill group info is assigned to the call.

Agent Cannot Go -- Not Ready, Busy, or Other

Agent In "Active" State

If the agent is on a call and clicks Not Ready, Busy, or Other, the agent state does not change right away. This is intentional. The agent remains in Talk or Held state until the completion of the call. The agent transitions to Not Ready, Work Ready, or Work Not Ready, depending on which button is pressed. If, after the call ends, the agent transitions immediately to Available, you must check the agent desk settings for the agent and see if Available After Incoming or Available After Outgoing are set. These settings override the tasks the agent performs with the buttons during a call.

Agent Desk Settings Prevent Transition

Check agent desk settings for the agent in Configure ICM and see if Idle Reason Required is checked. If the check box is checked, the agent cannot go into the Not Ready state without a reason code. Either modify the Desktop_Settings.cfg to match the agent desk setting in Configure ICM, or change the agent desk setting in Configure ICM.

If there is no agent desk setting assigned to the agent, the agent is able to log in and go ready, but the agent is not able to go not_ready or log out. The resolution is to close the agent application, assign an agent desk setting, and log in again.

Agent Has to Wait To Go Not Ready

The Router makes the agent unavailable when the agent makes a call or when a call is routed to the agent. This mechanism allows the Router to route another call to the agent before the PIM reports the call as received. Some networks take several seconds to actually route the call. The Router does not cancel the timer based on agent state.

If the actual time taken to route calls to the PIM from the route client is relatively short, you can change the configurable time in the router. On one of the routers in a DOS command window, use rtsetting.exe. Look under Extrapolation > Agent. The default is 10 seconds. If the value is too short, the Router routes calls to agents that are about to receive a call. This causes the PIM to drop calls.

Agent Cannot Go Ready

PRIVILEGE_VIOLATION_ON_SPECIFIED_DEVICE

There is an inconsistency in the data for the login request and the ready request. Possibly, the instrument, agent ID, or peripheral numbers do not match. Turn on CTI Server trace with procmon and set regset to 0xf8 to see the appropriate traces. You can also view this in the OPC or PIM logs, if third-party (TP) tracing is on.

Agent Cannot Logout

Agent Current State Prevents Making Call

If the agent is in a Work Ready, Work Not Ready, or Available state, the agent must first go to Not Ready before the agent logs out. Either modify Desktop_Settings.cfg to match the agent desk setting in Configure ICM, or change the agent desk setting in configure ICM.

Agent Desk Settings Prevents State Change

If the agent is in Not Ready state and still cannot log out, check the agent desk settings for the agent in Configure ICM and see if Logout Reason Required is checked.

Agent Shows Active Call or Agent Talking, But No Call At Phone

Log Agent Out and Back In

If the soft phone shows a call that is no longer physically present, the agent state can be stuck in Talking or Hold and the agent is not be able to log out. This can be due to a software bug in JTAPI or the PIM. In order to clear the condition, first attempt to clear the call from the soft phone if the release button is enabled. If this does not work, attempt to logout the agent. If the logout button does not work, exit, and restart the soft phone. If the condition persists, exit the soft phone, run Task Manager, run kill geodcs.exe and common~1.exe, and restart the soft phone. These processes can continue to run and remember the invalid agent state.

Agent in Incorrect State at the PIM

In procmon , check the state of the agent at the PIM. If you restart the agent desktop and the condition does not clear, there are more measures which you can take. CTI Server and OPC provide mechanisms to clear calls with the debug interface of procmon or opctest . This is a slightly preferred option to the other option which is to cycle the PG service or at least the close the PIM window.

Calls Clear Soon After Alert or Established

Registry Settings Incorrect

With regedt32, check these registry settings:

HKEY_LOCAL_MACHINE\SOFTWARE\GeoTel\ICR\<cust_inst>\<pg_inst>\PG\
  CurrentVersion\<pim_inst>\CallControl\MaxAlertingTimeAllowedForCall

and

HKEY_LOCAL_MACHINE\SOFTWARE\GeoTel\ICR\<cust_inst>\<pg_inst>\PG\
  CurrentVersion\<pim_inst>\CallControl\MaxConnectedTimeAllowedForCall

Note: These registry keys appear over two lines here due to space limitations.

Set these values to 300 and 28800 respectively.

Post Routing Does Not Work

Check the Route and Route Script

Use the AW Call Tracer tool to verify whether the call gets to the script and the script runs correctly. Run Script Editor and monitor the script. Look at the Router, OPC, and PIM logs for problems. Most route errors are traced unconditionally.

'Use DN Label Map' Checked for Peripheral's Routing Client

There is a setting for each routing client in Configure ICM labeled, "Use DN/Label Map." If this setting is set to "Yes" you need to configure a "Dialed Number Label" entry for each combination of dialed number and possible target label. This setting is not useful on PG routing clients and must be set to "No".

No Label Configured for Routed Device Target

Verify the Label Configured on the routing client. You must configure Label on each client even if the label is identical on each client.

CTI Route Point is Not Configured on Cisco CallManager

In order to use Post Routing you must configure a "CTI Route Point" on the Cisco CallManager and assign a line to the route point with the desired directory number (for example, "5000"). For agent calls to post routing, use the dialed number plan. An agent dialing to Cisco CallManager CTI Route Point confuses the IPCC soft phone in CTI Desktop version 4.1.9.

Device for CTI Route Point is Not on List of Devices Controlled by PG User

You must add the CTI Route Point device to the list of "Associated Devices" for the PG user on the Cisco CallManager user web page under global directory. If you create a new device, add the line(s) first, and then add the device to the user "Associated Devices" list. If you add more lines to an device that already exists in the user device list, you need to restart the JGW for the JGW to recognize the new lines. However, if you add a new device, add a line to the device, and then add the device to the user device list, the JGW must be able to recognize the new device (within about 30 seconds).

No Dialed Number Configured on Cisco ICM

Check the dialed number to make sure the number is configured for the peripheral routing client. Run procmon to JGW and turn on the trace as "trace *ROUTE*" (case sensitive). Check JGW log for errors that pertain to the dialed number. Upon startup, JGW attempts to register a route call back for the dialed number. When a call is made to the dialed number, the gateway receives a "RouteEvent".

Along with dialed number, verify whether the call type is created and mapped correctly to the script.

Check that Restart of JGW is not Required

If you have configured an ICM dialed number, have set up the CTI Route Point, and have added it to the user device list but you still do not receive route requests when the number is dialed, you may need to restart the JGW (or cycle the PG). You only need to restart if you have turned on the trace in JGW (trace *ROUTE*) and you see errors that show the address is not in the provider. Generally, the JGW must be able to recognize new CTI Route Points that are added to the user device list without the need to restart. Also, if lines are added to a CTI Route Point that already exists, the JGW does not recognize them without the need to restart. You must be able to avoid a restart if you add a new CTI Route Point for each dialed number instead of new lines to devices that already exist.

Note: This assumes DeviceListPolling is turned on in the JTAPI.ini file in the winnt\java\lib directory in the PIM. If DeviceListPolling is turned off, you must turn DeviceListPolling on. If DeviceListPolling is turned off, and you add any device to the user list, you must cycle the PG or at least JTAPI GW for the PG to see the new device.

Check OPC Logs for Route Dialogs

Use opctest to turn on route tracing "debug /routing" and check OPC logs for errors when calls are made to the route point. Check to see that route requests are being received and labels are returned. The route requests appear as "CSTA_ROUTE_REQUEST" and "ICR_NEW_CALL_REQ" messages. The returned labels appear as "ICR_CONNECT" messages. If errors occur, you can see "ICR_DIALOG_FAIL" messages instead of "ICR_CONNECT" messages. In this case, check the router log for errors.

Check Router Logs for Route Dialogs

Use rtsetting.exe to turn on route tracing and check router logs for errors when calls are made to the route point.

Make sure all required labels are configured. If your route script targets IPCC/EA agents, you must have labels configured for the Post Routing client for each targeted device target.

Routing Script Does Not Dequeue Call When Agent Becomes Available

Check the router log for errors. If there are none:

No Error in Router Log -- Queue Node is Queued to the Skill Group Base Priority

If the queue node queues to the base priority, nothing happens when the agent becomes available. There are two option to fix this issue:

• There is a router registry setting called AutoLoginBase (use rtsetting.exe). Change this setting to allow the call to be queued to the base skill group to work more or less as expected. There is no preference to primary over secondary skills when this type of queuing takes place.

• Queue to either the primary and/or secondary skill sets in the queue node explicitly.

Router Log Indicates Label Not Configured for Target

Configure label for the device target in question and all other targets to which this routing client can route. Use the AW bulk configuration tool for a more efficient way to do this over configure ICR.

Ring No Answer Heard When All Agents and Queue Ports Are Busy

Check Router Logs

• Route errors must be traced unconditionally.

• You can use the call tracer tool to test the route path.

• Use rtrtrace to turn on the route request trace and check router logs for errors when calls are made to the route point.

• Make sure all required labels are configured. If the route script targets IPCC/EA agents, you must have labels configured for each targeted device target. Each device target must have labels configured for each route client that tries to send calls. So, if a call is pre-routed from the network directly to an available agent, the network routing client must have a label for the associated device target. If the call is first queued at a VRU and then delivered to the agent, the VRU routing client must have a label for the associated device target.

Make Sure DN Label Mapping is Turned Off in the Routing Client in Configure ICM

Make sure that Use DN/Label map is not checked in the Routing Client tab within Configuration Manager/PG Explorer.

Check PIM Logs

• Use procmon to turn on the trace in the PIM (trace precall, trace *call_event*) and check logs. The pre-call message appears from the Router. You also see "DeliveredEvent" with "DevTgDevStr" set to the agent extension. If the call does not show up, ensure that the label is correct for the route client.

Arbitrary Transfers Get Inconsistent Results

IPCC does not support the option to place a call on hold and make a new call because the Cisco CallManager provides inconsistent results. This is considered a product enhancement and can be considered for a future release.

Alternate Does Not Work

When a consult call is switched/alternated/held/retrieved, Cisco CallManager breaks the consult association. This results in an arbitrary transfer scenario which is not supported. Agents can reconnect to the customer and start a new consult. The IPCC soft phone disables the alternate button until it is resolved, but third-party vendors can complain.

Conferenced Party Cannot Conference in Another Party

Cisco CallManager has a limitation that only the conference initiator can add more parties to the conference. Other parties cannot add more parties in the Cisco CallManager.

Agent Station Logged Out Unexpectedly

Agent Desk Settings Inactivity Timer

In the Agent Desk settings, there is a time setting to log out agents in the Not Ready state. The maximum inactivity time is 2 hours but you can configure the time to be less. Agents in the Available state are not be logged out while in the inactivity state. The agent transitions from Ready to Not Ready if the ring no answer timer expires (also a configurable agent desk setting).

CTI Server Heartbeat Time Out

CTI Server has a configured heartbeat time out. Older computers, overworked CTI Servers, or networks with bandwith problems can be the root cause. CTI Server logs must report an error in the log.

Agent Does Not Behave as Configured in Agent Desk Settings

The agent desk settings in Configure ICR(M) and the agent configuration file both have to agree on how the agent is handled.

There is a work timer in the peripheral configuration on ICM in Configuration Parameters. Set the parameters as \WORKTIMER 30 to set a 30-second delay on auto-available.

The desktop configuration file resides in:

\program files\geotel\cti desktop\Desk_Settings.cfg

The work mode on Incoming must be set to Required, not Required with Data in Desk_Settings.cfg and in the Configure ICR(M) agent desk settings. Required with Data supersedes the auto-available option.

Consult Transfers Fail

Look at the JTAPI GW log and see if there are any errors that indicate why the consult transfer fails. Check whether the agent software allows hold/retrieve or alternate operations on the consult call. When either call is held/retrieved, the call is no longer considered consultative, but an "arbitrary" transfer by the Cisco CallManager. The Cisco CallManager has problems with arbitrary transfers. Limit the user to reconnect or complete the transfer when in a consultative call.

Consult Party Hangs Up but Call Appearance Does Not Disappear

Cisco CallManager currently has problems with a disconnect event for a conference initiated consult when the conference is not completed. Disconnect the call a second time to clear the call appearance at the agent phone.

Translation Route to VRU Does Not Work

First, monitor the active script. Then check the Router, OPC, and PIM logs of the routing client and the VRU. Most errors are traced unconditionally, but you can turn the trace up to get a better picture of what happens.

Here is the translation route sequence:

• The Routing Client makes a new call request to Router.

• The Router returns a connect to the Routing Client with a label that must deliver the call to the IVR.

• The IVR must then send up a RequestInstruction that the VRU PG uses to look up the peripheral target.

• The Router matches peripheral targets of the request instruction with the translation routes peripheral targets it waits on.

• The Routing Script continues with run script or queue nodes as designed by the customer.

Monitor the active script to find the failure path. Look at the router trace for errors. Check to see whether the route client receives initial labels. Verify whether the VRU receives the call. Verify whether the VRU sends up a request instruction at the VRU PIM or OPC level.

Route Request Does Not Get to "Translation Route to VRU Node" in the Route Script

Monitor the script and verify whether the request gets to the translation route to VRU node.

First, in the route script, a select or route select node with a translation route selected is not enough to translate the route to the service controlled VRU. A Translation Route to VRU node is required.

Second, the monitor must show that the call gets to the translation route node. A failure here means that a translation route cannot be determined or the RequestInstruction route request message was not received from the IVR.

Translation Route Time Outs in Router Log

Translation route time out error indicates that the Router does not receive the request instruction. Verify the OPC and the VRU PIM for errors and to see whether the RequestInstruction arrives.

Turn up "translation routing" and "network VRU tracing" with the rtrtrace tool on the Router for a better indication of what occurs in the Router. In the VRU PG OPC, turn up service control reporting with opctest .

VRU PIM Log Indicates DNIS Not Found on Trunk Group X

The Request Instruction must indicate a valid trunk group that maps to a trunk group peripheral number in one of the trunk groups configured for the VRU PG. Cycle the VRU PG to receive the update of the trunk group peripheral number, if modified.

Check ICM Configuration

Make sure the DN Label Mapping is turned off in the IVR PG routing client. The IVR PG needs a network VRU assignment. The network VRU must be type 2. The IVR PG must have a network trunk group and trunk group assigned. Reference the network trunk group in the trunk group.

The NIC/post routing PG must have a label for each of the DNIS in the peripheral targets. (Make the labels the same as the DNIS for the request for the routing client in the translation route wizard. You can set this up in the prefix, select the prefix = DNIS option.)

The VRU routing client needs a label configured for the device target to which it routes when an agent becomes available.

Troubleshoot the Cisco IP IVR - ICM Interface

This Cisco IP IVR section covers how to troubleshoot configuration errors between the IP IVR and ICM and includes common problems with the setup for IVR PG post routing and translation routing. Refer to the Cisco IP IVR Troubleshooting Guide for more information on general IVR errors.

In general, check the MIVR logs under the appadmin > Engine > Trace files web page.

• IVR CTI Ports and CTI Route Points configured in Cisco CallManager, IVR, and ICM.

• IVR CTI Ports and CTI Route Points are associated with IVR user in Cisco CallManager global directory.

• Service Control check box is checked in IVR ICM configuration.

• Script names in IVR script definitions match network VRU script names in ICM.

• Trunk group number in VRU PG matches CTI port group number in IP IVR.

Translation Route Fails

Along with all of the other actions you use to troubleshoot, you can also try these things to help troubleshoot the IP IVR.

• Check the MIVR log. This log can generally point to problem areas.

• Use debug settings to turn up on Cisco IP IVR are SS_TEL and LIB_ICM.

• Turn on Cisco Jtapi log for IP IVR with jtprefs on the IP IVR. See Debug Tools. Stop and start IP IVR engine after you turn on the trace.

• Verify whether the CTI Port group number on the IP IVR JTAPI translation route port group matches the peripheral number in the trunk group configuration in the ICM.

Script Does Not Play or Plays Error Message

Check the IP IVR logs under engine-trace files to verify whether:

• Run Script is received.

• IP IVR can find script. Upload scripts with the Repository Admin Tool.

• IP IVR can find prompt. User defined prompts reside in \wfavvid\prompts\ user\en_us\ on the IP IVR.

JTAPI Status Shows Partial Service

This generally means some of the CTI Ports or CTI Route Points configured in the IP IVR have not been configured and/or associated with the IP IVR user on the Cisco CallManager.

This can also mean the scripts are not named correctly or have not been uploaded to the Repository Manager.

ICM Status on IP IVR Shows Partial Status

Generally, this condition indicates a partial configuration or mismatched configuration on one side or the other.

Stutter Prompt Heard When a Call is Dequeued from the Router

This is a misconfigured routing script that allows too little timeout in network VRU script configuration in Configure ICR.

Some of the scripts that are available with the IP IVR for ICM interface run a very long time, but the default time out on the ICM network script configuration is three minutes. If the script times out and the run script failure path plays another run script, these run scripts basically get queued at the IVR. When the scripts are dequeued, you hear many scripts play over each other.

Troubleshoot IVR Service Statistics

IVR statistics are important to IPCC service level reports. Therefore, some information on how to troubleshoot is included here. As an overview, the changes in Router and VRU PG where implemented calls in the VRU are counted as queued, instead of connected. When calls get routed, they are reported as answered. When the customer in queue disconnects calls, they are reported as abandoned. Refer to the readme.txt of hot fixes 53 and 54 for additional details. The Router sends down special queue events that indicate which state the call is in at the Router.

There is a special registry set up in the VRU PIM so you must voluntarily turn this feature on in order to ensure minimal disruption.

Enterprise Service Real Time Report 10 makes special use of this data when you add the VRU service(s) and Cisco CallManager PG Service(s) to one or more enterprise peripheral reports. Enterprise Service Real Time Report requires that VRU PG and Cisco CallManager PG services be grouped in an enterprise service for reporting purposes.

Other useful queue reports are the new call type reports for real time and historical records, and skill group real time grid now shows calls queued against the skill group.

No Service Statistics or Termination Call Detail Records Are Generated

VRU PIM does not generate CSTA events. Turn on Service Control Reporting in the VRU PG set up. This is in the registry key in ServiceControlQueueReporting under:

HKEY_LOCAL_MACHINE\SOFTWARE\GeoTel\ICR\<cust_inst>\<PG_inst>\PG\CurrentVersion\
  PIMS\<pim_inst>\VRUData\Config

Note: This registry key appears over two lines here due to space limitations.

VRU Reports All Calls as Connected, Not Queued as Required

ServiceControlQueueReporting Registry Key Not in Pace or Not Set to 1

Startup log for VRU PIM must complain if it does not exist.

Add the ServiceControlQueueReporting key and set the value to 1 in:

HKEY_LOCAL_MACHINE\SOFTWARE\GeoTel\ICR\<cust_inst>\<PG_inst>\PG\CurrentVersion\
  PIMS\<pim_inst>\VRUData\Config

Note: This key appears over two lines here due to space limitations.

Calls Are Counted Against Wrong Service or do not Appear in Service Reports

OPC Log Indicates the Service Mapping Not Found

The OPC log indicates the service mapping is not found when calls are counted against the wrong service or do not appear in service reports.

Cisco ICM Reporting Issues

Cisco ICM is not designed for easy correlation of data Call Type, Service, and Skill Group tables. The numbers generally have slightly different meanings in each group. There is only one service for a call, but there can be two skill groups if more than one agent is involved. The redirect on no answer (RONA) feature likely generates another post route without the generation of another termination record.

Agent Skill Group, Skill Group, Service, Call Type Numbers Do Not Balance

• Symptom: Calls handled or other statistic fields do not match between service, call type, and/or skill group reports.

• Condition: Call type, service, and skill groups are set up with a logical map to each other, but reports still do not exactly match.

• Troubleshoot: If call volume is less than 1 call per second, turn on trace settings in OPC, PIM, and JTAPI GW, as appropriate to CSTA, PIM, AGENT, and third-party events. Refer to the tools section of this document for instructions.

• Document the call flow:

  • Is the initial post route on the Cisco CallManager PG or VRU PG?

  • Is forward on no answer (FONA) configured and to what is FONA configured to redirect?

  • Is a default skill group configured with peripheral number 0 to separate out routed calls from non-routed and outbound calls?

Grab the historical data from the these tables for a day with the "select *" statements:

• Peripheral_Half_Hour

• Call_Type_Half_Hour

• Service_Half_Hour

• Skill_Group_Half_Hour

• Termination_Call_Detail

• Route_Call_Detail

Troubleshoot Cisco CallManager

When you collect the traces in Cisco CallManager, you can turn the flags from the Cisco CallManager Admin page under Services > Trace Flags. 0xCB05 is a good trace flag set up for SDL tracing of CTI errors. Set 0xCB05 under service parameters for debug purposes. Refer to AVVID TAC Cases: Collecting Troubleshooting Information for more information. Refer to the Cisco CallManager online documentation, including troubleshooting guides.

Turn on the Trace for Cisco CallManager

Refer to Set Up Cisco CallManager Traces for Cisco Technical Support for information on how to turn on the trace for Cisco CallManager.

Change Cisco CallManager IP Addresses

Refer to Changing the Cisco CallManager IP Addresses and change the server name.

1. Run Setup on Cisco CallManager PG and change JTAPI services for the Cisco CallManager PIM. If you have extension mobility, and/or phone services.

2. Stop CRA Engine.

3. In CRA - Change IP address under Engine Configuration.

4. Change IP under JTAPI.

5. Stop DC Directory Service on the server.

6. Change IP Address in Directory configuration.

7. In Cisco CallManager - change IP Address under System > Server.

8. Change IP Address in URLs under System > Enterprise Parameters.

9. Change IP Address in all URLs under Features > Phone Services.

10. Change Server IP Address - Network Properties.

11. Change DHCP Option 150 to new IP Address.

12. Change IP in the hotel profile in DC Directory, Cisco CallManager > System Profile > Hoteling.

13. Open SQL Enterprise Manager.

14. Change IP addresses in URLs in PlugIn table.

In order to back-up your configuration changes:

1. Open stiBackup configuration.

2. Change server IP addresses under all appropriate tabs.

Debug Tools

Procmon

Procmon is a command line tool that you can use to debug PIM and JTAPI GW processes.

• Usage: procmon <customer name> <node> process.

  • Procmon ipcc pg1a pim1

  • Procmon ipcc pg1a jgw1

  • Procmon ipcc cg1a ctisvr

Here are some useful trace settings for each of the processes:

• JTAPI GW (use procmon)

  • trace JT_TPREQUESTS (turns on third-party request traces)

  • trace JT_JTAPI_EVENT_USED (turns on traces for the JTAPI Events the PG uses)

  • trace JT_PIM_EVENT  (turns on traces for event messages sent to the PIM)

  • trace JT_ROUTE_MESSAGE (turns on routing client traces)

  • trace JT_LOW*  (traces based on the underlying JTAPI and CTI layers)

• PIM (use procmon)

  • trace  tp* (turns on third-party request traces)

  • trace precall (turns on precall event traces)

  • trace *event (turns on agent and call event traces)

  • trace csta* (turns on CSTA call event traces)

• CTI SERVER (use procmon)

  • regset EMSTraceMask 0xf8  (turns on useful CTI Server traces, likely to wrap around)

OPCtest

Opctest is a command line tool to debug OPC process on the PG.

• Usage: opctest /cust <customer name> /node <node>

  • opctest /cust ipcc /node pg1a

• Useful settings

  • debug /agent (turns on agent event traces)

  • debug /routing (turns on routing event traces)

  • debug /cstacer (turns on csta event traces)

  • debug /tpmsg (turns on third-party call request traces)

rttest

Rttest is a command line interface tool to debug the router process on the ICM. See rtrtrace for the GUI version.

• Usage: rttest /cust ipcc

rtsetting.exe

GUI tool to change router registry settings.

• There is an option to set settings back to default.

rtrtrace.exe

GUI tool to turn on various router traces on the ICM.

• Settings particularly useful for IPCC are:

  • Queuing – for problems dequeuing.

  • Service Control – for problems with VRU interface.

  • Translation Routing – for problems with translation routes.

dumplog

Dumps Cisco ICM binary files to text files. Change directories to the process logfiles directory.

• OPC, PIM, and JtapiGW process log files reside in icr\<customer_name>\<node>\logfiles\.

• On the PG, there is a batch file called cdlog where you type >cdlog <cust> <node>.

• Usage: dumplog process name

  • Dumplog /" (for help on different dumplog options)

  • Dumplog jgw1

  • Dumplog pim1

  • Dumplog opc

vrutrace

A tool to view the VRU PG capture file. Works similar to dumplog.

Call Tracer

Cisco ICM tool that you can use to debug routing scripts.  You can find this tool in the AW menu item on the AW.

jtprefs

This is a tool to turn on JTAPI traces for the JTAPI client on the IP IVR. The JTAPI traces on the IPCC PG are controlled with the procmon interface. This tool resides in program files\CiscoJtapiTools\.

Performance Monitor

A Microsoft Windows 2000 administrative tool that shows real time data for the Cisco CallManager, Cisco IP IVR, and the ICM. You can see calls in progress, registered devices, and process CPU utilization. You can find this tool under Start > Programs > Administrative Tools.

Log Files

Cisco ICM Log Files

Cisco ICM log files reside in \icr\<cust>\<node>\logfiles. Here, customer references the customer instance name and node references pg1a, ra for router, cg1a, and more. Use dumplog to view the log files.

Note: You can view event capture files with trace tools such as vrutrace. These files are in a different directory.

Cisco CallManager Log Files

Cisco CallManager log files normally reside in \program files\cisco\ccm\trace with trace directories of:

• Ccm - CallManager SDI logs.

• Dbl - Database Layer logs.

• Sdl - Call Signaling logs.

• Tftp - Logs for the tftp server.

You can modify the trace settings for these files from the Cisco CallManager admin page under trace settings. You can modify SDL trace settings under service parameters in Cisco CallManager.

IP IVR Log Files

IP IVR logfiles reside in \program files\wfavvid. You can also view IPIVR log files from the appadmin page under engine-trace files.

You can view Cisco JTAPI Client logs when you turn on JTAPI events with jtprefs.exe and restart the IP IVR engine.

Useful Profile Data

When you collect data to open cases, collect the data listed in this section, in addition to the log files.

Number of Agents

What is the number of agents configured?

Gateways Used

How many gateways are configured?

Software Versions of the Components

Cisco CallManager, JTAPI Client, ICM, Gateway IOS version, and IP IVR.

• You can find the Cisco CallManager version on the Cisco CallManager admin web page under Help > About > Details.

• In order to find the JTAPI Client version, simply type jview CiscoJtapiVersion in a command prompt in the directory \winnt\java\lib on the Cisco CallManager PG.

• You can also find the IP IVR version.

IVR Type

What type of IVR is in use?

Platforms

What types of platforms are in use/ CPU / and amount of physical memory.

Related Information

• Using Remote Process Monitor Console (Procmon)
• How to Use the Dumplog Utility
• Using the OPCTest Command Line Utility
• The Cisco ICM rttest Utility
• Turning up Tracing
• Reading a Drwtsn32 log file
• AVVID TAC Cases: Collecting Troubleshooting Information
• Changing the Cisco CallManager IP Addresses
• Technical Support & Documentation - Cisco Systems