This document describes the steps to take when one of these problems occurs with agents in a Cisco IP Contact Center (IPCC) Express environment:
Cisco Agent Desktop (CAD) agents who are logged in and processing calls possibly do not appear in Cisco Supervisor Desktop (CSD).
CAD agents appear in CSD but then quickly disappear.
CAD agents are unable to Chat with CSD users.
Messages appear in the Event Viewer logs on the UCCX server. This is an example:
Event Type: Warning Event Source: FCCServer Event Category: None Event ID: 3 Date: 1/21/2010 Time: 2:01:52 PM User: N/A Computer: TACUCCX1 Description: FCCS3008 Network communication error <COMM_FAILURE> sending message to application <Agent1>. The application will be logged out.
Readers of this document should have knowledge of these topics:
Cisco Customer Response Solutions (CRS)
Cisco Agent Desktop
Cisco Supervisor Desktop
The information in this document is based on these software and hardware versions:
Cisco CRS version 3.x and later
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
For more information on document conventions, refer to the Cisco Technical Tips Conventions.
Cisco Agent Desktop (CAD) agents who are logged in and processing calls possibly do not appear in Cisco Supervisor Desktop (CSD).
CAD agents appear in CSD but then quickly disappear.
CAD agents are unable to Chat with CSD users.
Messages appear in the Event Viewer logs on the UCCX server:
Event Type: Warning Event Source: FCCServer Event Category: None Event ID: 3 Date: 1/21/2010 Time: 2:01:52 PM User: N/A Computer: TACUCCX1 Description: FCCS3008 Network communication error <COMM_FAILURE> sending message to application <Agent1>. The application will be logged out.
CAD and CSD use a proprietary chat protocol in order to communicate agent states and chat messages in a client server architecture. These messages are sent and received by the Cisco Desktop Chat Service on the UCCX server and distributed to CAD and CSD users as necessary. If the Cisco Desktop Chat Service is unable to make a connection to the CAD and/or CSD applications over the ports used by this protocol, the previous symptoms are the result.
These ports must be open throughout the transit path in order for the Chat protocol to work correctly:
UCCX Server - TCP 59000
CAD - TCP 59020
CSD - TCP 59021
CAD (59020) ------ (59000) UCCX Server (59000) ------ (59021) CSD
Complete these steps in order to resolve this issue:
If CAD is installed on a Citrix or Microsoft Terminal Services environment, make sure that the PostInstall.exe utility is configured for this deployment type. Complete these steps:
On the Citrix or MTS server, open Program Files > Cisco > Desktop > bin > PostInstall.exe.
Confirm that the IP addresses shown are those of the UCCX server(s).
In the left panel, choose Thin Client Environment.
Choose Yes in the panel on the right.
Click Apply.
Restart all CAD and CSD instances.
If the UCCX version is 7.0(1), verify that the CAD application does not experience Cisco bug ID CSCtd75811 (registered customers only) .
When these symptoms are present, open Windows Task Manager on the Cisco Agent Desktop PC.
Click on the Processes tab.
Check the Show processes from all users box.
Sort the list of processes by Image Name.
If multiple instances of agent.exe are present, upgrade UCCX to a version which contains the fix to Cisco bug ID CSCtd75811 (registered customers only) .
If the UCCX version is 7.0(1), verify that the system path includes WebAdminLib.dll. To do so, choose Web CDA > Services Configuration > Enterprise Data > Fields. Complete the workaround mentioned in the CAD Troubleshooting Guide if this error is returned:
CDAUI1000 Error while trying to load data. Ensure that the system path includes the location of the WebAdminLib.dll.
Verify that the Agent is assigned to the correct team in Web CDA > Personnel > Agents. Agents can appear to be assigned to different teams in Web CDA than AppAdmin because Directory Services are not synchronized. Manually synchronize Directory Services from the Web CDA > Services Configuration or restart the Cisco Desktop Sync Service from AppAdmin Control Center.
Verify that all Desktop Services are running in AppAdmin Control Center. If any of these services show in a Stopped, Partial Service or Invalid state, contact Cisco Technical Assistance Center (TAC) for assistance. This is the list of services that should be running:
Cisco Desktop Call/Chat Service
Cisco Desktop Enterprise Service
Cisco Desktop LDAP Monitor Service
Cisco Desktop License and Resource Manager Service
Cisco Browser and IP Phone Agent Service - if you use CAD-BE or IPPA
Cisco Desktop Recording & Statistics Service
Cisco Desktop Sync Service
Test connectivity over the previously mentioned ports between the UCCX server and CAD and the UCCX server and CSD.
On the UCCX server, open a Command Prompt window.
Start the CAD or CSD application on the client PC.
Issue these commands and substitute the correct IP addresses:
Telnet <CSD IP Address> 59021 Telnet <CAD IP Address> 59020
Note: A failure message is denoted with the appearance of this message in the Command Prompt Window.
Connecting to 192.168.xxx.xxx Could not open a connection to host on port 59000:Connect failed
If no failure message is received, test connectivity from the CSD and CAD PCs to the UCCX server.
On the CAD or CSD PC, open a Command Prompt window.
Issue this command, and substitute the correct IP address of the master UCCX server:
Telnet <UCCX IP Address> 59000
If no failure message is received, monitor the open Telnet connection when these symptoms appear.
If a failure message appears immediately or while you monitor the open Telnet connection, examine these:
If antivirus software is running on the CAD or CSD PC, turn it off including real time scanning if applicable.
If Windows Firewall is enabled, turn it off in the Windows Firewall section of the Control Panel.
Verify that the above mentioned ports are not blocked via Group Policy.
Verify that any network firewall in the transit path allows traffic over the previously mentioned ports.
If CAD and CSD is reachable only over VPN, make sure the VPN headend and all other security devices allow traffic over the above mentioned ports.
Verify Cisco Security Agent (CSA) does not block the previously mentioned ports if it is installed.
Verify that the network ports in the transit path are free of errors.
If the CAD or CSD PC has mutliple NICs and IP addresses, disable the unused NIC.
Verify that no third party applications on the CAD or CSD PC conflict with the UCCX software and prevent communication on the previously mentioned ports. Perform these steps:
To do so, build a test PC with the base version of the operating system installed such as Windows Vista, XP and so forth. Avoid the use of a pre-packaged corporate image.
Install only CAD or CSD on this test PC.
Test to see if the previously mentioned symptoms exist.
If not, add each application on the production PC to the test PC incrementally, and test the CAD and CSD functionality between each installation.
Intermittently Agent begins to disappear and appear on the Supervisor Desktop under Team Agent state Report.
This problem is caused when multiple threads do not exit correctly under Agent desktop PC. When the CHAT communication on TCP Port 59000 and 59020 between Client and Server has been lost due to network interruption/outage. CAD is unable to recover from this and creates multiple Agent.exe instances.
Exit out affected Cisco Agent Desktop application and launch CAD again.
In PIM log, there is a PERERR_CM_REDIRECT_CALL_UNKNOWN_DESTINATION message. This happens when the agent does not pick up the call and the call goes back into queue, the agent becomes not ready then the Request operation failed error message appears.
Complete these steps in order to resolve this issue:
Set FilterOutRecoveryOnTimerExpiry to1 and restart the services on the PG at this location:
HKLM\SOFTWARE\Cisco Systems, Inc.\icm\<instance name>\PG1A\PG\CurrentVersion\JGWS\jgw1\JGWData\Dynamic
Set cvp time out with this parameter: setRnaTimeOut *:15
During the navigation of different pages of Cisco Desktop Administrator, one of these errors appears:
CDAUI1000 Error while trying to load data. Ensure that the system path includes the location of the WebAdminLib.dll
Or
The requested resource () is not available.
Complete these steps in order to resolve the issue:
In Control Panel, launch the System utility.
On the Advanced tab, click Environment Variables.
From the list of System Variables, choose Path and click Edit.
In the variable value field, locate the C:\Program Files\Cisco\Desktop\bin string. Use cut and paste to move it to the beginning of the variable value string. This results in the DLL that is accessed from the correct location.
Click OK and close System utility.
Stop the Cisco Unified Node Manager.
On the Unified CCX server, delete the teamadmin folder from these locations:
C:\Program Files\wfavvid\tomcat_appadmin\webapps
C:\Program Files\wfavvid\tomcat_appadmin\work\catalina\localhost
Restart the Unified Node Manager, and wait two minutes.
Note: After you complete these steps you should be able to access Cisco Desktop Administrator and navigate to the different pages.
You are unable to see all agents in the monitor screen and in the log, this error message appears:
2010-09-24 07:37:44.404 DEBUG [0xeec] FCCClientAPI::corbaInit: This client is not running in Citrix/MTS mode. A static port number will be used. 2010-09-24 07:37:44.388 DEBUG [0xeec] FCCClientAPI::corbaInit: Unable to open registry:HKEY_LOCAL_MACHINE\SOFTWARE\Spanlink\CAD\Chat Server API\Setup.Looking in LDAP.
This issue is documented in Cisco bug ID CSCsa19971 (registered customers only) .
Complete these steps in order to set the registry key on the Citrix server and resolve this error:
Choose Start > Run and type regedit.
Create the registry key: HKEY_LOCAL_MACHINE\SOFTWARE\Spanlink\Chat Server API\Setup .
In that key, create a DWORD value named OmniOrbUsePort.
Set the value to 0.
Restart the Citrix server.
The agents that up in CSD depends on the software communication. CAD software communicates to CSD about its state through UCCX. If communication breaks any where in this communication path, complete these steps in order to resolve the issue:
Note: This can occur because the sync is broken as the CSD shows no service after you choose the team.
Stop the node manager.
Restart the CTI manager in all the Cisco CallManager in the cluster.
Start the node manager on the master IPCCX and wait until it appears.
Start the node meager in the sub IPCCX.
This can happen if ODBC DSN is created without any passwords. When you test the ODBC DSN, you need to enter the password and then test the connection. In this example, CTIOS Server tried to login without a password and thus ODBC connection failed. Since the ODBC connection failed, the SQL Queries in the CTIOS Registry was not done at all. .
First make sure the ODBC connection is set up correctly and works, and then restart the CTIOS Server in order to resolve this issue.
The Cisco Supervisor Desktop shows six agents logged in under the Real Time Displays section of the application window, but only four of the six that are logged in show up in the Agents window.
Two agents were logged out of the Cisco chat service because ports were blocked from the workstation to UCCX server. You need to verify that the ports are not blocked in order to resolve this issue.
Agents intermittently disappear and re-appear from CSD. CAD loses connectivity to the call/chat service and does not recover properly. Multiple instances of agent.exe appear on the agent PC processes list in Task Manager. Netstat on the agent PC shows multiple connections to the server.
The solution is to apply a fixed version of the Splkstd.dll file to each of the CAD desktops as a workaround for this issue. There are two solutions for this issue:
Kill the agent.exe processes on the Task Manager and open CAD again, on the desktop computer of the agent. Then you see the agents on the Supervisor Desktop. This is a quick fix but not a permanent solution.
On the workstation of the agent, choose C: > Program Files > Cisco > Desktop > bin directory. Then complete these steps:
Rename the existing SplkStd.dll, for example SplkStd.old.
Extract the patched SplkStd.dll file to the renamed path.
Reboot the agent workstation.
Note: This is documented in the Cisco bug ID CSCtd75811 (registered customers only)
Agents do not appear on CSD at all.
Complete these steps in order to resolve this issue:
In order to navigate to CDA, choose Start > Programs > Cisco > Desktop > Admin.
Click on Call Center 1 and from the menu, choose Desktop Administrator > Side A.
On the next window if you have not previously configure a password, just click on login (no password needed).
Choose Services configuration > Synchronize Directory service and click on the Synchronize button.
Restart the Cisco Desktop service.
The real time display in Cisco Supervisor Desktop does not update under the root Skill group area for any group. You are able to see agent details if you drill down into a specific group.
Set the debug threshold on FCCserver and FCRasSvr in the 'C:/Program Files/Cisco/Desktop/config' folder on both the PGs. Ensure the debug threshold is set for the supervisor PC also.
Once the Agent and Supervisor login and statistical display problem is observed, collect the FCCserver, Rascalserver, and Supervisor log and debug.
In order to ensure that you can complete the sync, restart the Cisco Sync Service from Windows Services on both the PGs.
After the Cisco Sync Service has been restarted, check the logs:
Line 6781: 2012-06-25 09:42:37:111 INFO DESK1039 Error getting skill stats from enterprise server: <-20>. Line 6785: 2012-06-25 09:47:08:863 ERROR DESK3038 ChatAPI DropPresenceUser failed. Line 6889: 2012-06-25 09:47:15:570 ERROR LRMS2001 The LRM client failed to connect to any service. 2012-06-25 09:49:34:164 WARN FCCS3008 Network communication error <TRANSIENT> sending message to application <AGENT_DESKTOP_012918>. The application will be logged out.
Note: Enter the netstat -p tcp -b -n command on the CAD server side in order to show which .exe uses these ports.
From the logs it was determined that CHAT and Enterprise services in UCCE use different ports than those mentioned when connected to the CAD clients:
Chat TCP 5900 to connect to the Agent TCP 37350 to connect to the Supervisor.
Enterprise TCP 59004 to connect to the Agent client.
In order to resolve this issue, restart the Cisco Enterprise Service. If that does not fix the issue, restart the PG servers.
When more than 10 skills were assigned to an agent, sync replaced the data with that received in the last update from CTI. Supervisor would then only show what it had received in the most recent update from sync. So, in CSD kill Groups - Skill Agent Statistics Display report does not show correct agents for the skills selected on the left of the pan. This is documented in the Cisco bug ID CSCsm81553 (registered customers only) .
When a team or skill is chosen that contains no records, a supervisor would show all members of that team. Reports do work correctly after the sync patch is installed, when a valid team or skill is selected. This is documented in the Cisco bug ID CSCsm81587 (registered customers only) .