CRS Quick Tracing Guide for Version 3.x and 4.0.x

Available Languages

Updated:October 3, 2008

Document ID:69952

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Introduction

Prerequisites

Requirements

Components Used

Conventions

Before you Begin

Key Words

Traces to Turn On

General Trace Information

CRS 4.0.x Default Traces

Minimum Traces

Symptom Category

Application Administration Issues

Script Upload or Application Configuration Issues

Application Loading and Invocation Failures

ASR/TTS/VXML Issues

ASR

TTS

VXML

Desktop Issues/Call Control

Agent Desktop Issues and Call Control

General Issues with the Cisco Supervisor Desktop

Agent State Issues with the Cisco Supervisor Desktop (Example: Missing Agents)

Reporting Issues with Cisco Supervisor Desktop

IPIVR (ICM) Related Issues

Reporting Issues

Historical Reporting Issues

Real-Time Reporting Issues

Calls Stuck in Queue

Script Behavior Issues

Subsystems Down Issues

Upgrade or Installation Issues

Upgrade and BARS Issues

Installation Issues

VOIP Monitoring and Recording Issues

Related Information

Introduction

This document provides guidelines for quick tracing on a Cisco Customer Response Solutions (CRS) server in a Cisco Unified Contact Center Express environment. This document only serves as a guide, and does not cover all situations. In order to troubleshoot an issue in some situation, you may need to collect more data than what this document mentions.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

Cisco CallManager
Cisco CRS

Components Used

The information in this document is based on these software and hardware versions:

Cisco CallManager version 3.x or 4.x
Cisco CRS version 3.x or 4.x

Refer to the matrix in Cisco Customer Response Solutions (CRS) Software and Hardware Compatibility Guide for information on complete compatibility between Cisco CallManager and CRS, and CRS versions involved in this solution.

Cisco CRS provides a single, consistent, and easy-to-manage platform for these products:
- Cisco Unified Contact Center Express Edition (IPCC Express)
- Cisco IP Interactive Voice Response (IP IVR)
- Cisco IP Queue Manager (IP QM)

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Before you Begin

Every escalation has certain minimum requirements. Before an escalation, perform these steps:

Turn on the appropriate traces (see the Traces to Turn On section).
Reproduce the problem if possible.
Collect the logs for every the symptom category (see the Symptom Category Table section).
Examine the logs in detail and troubleshoot in depth before you escalate the issue.

Key Words

Here are some key words to look for in logs when you troubleshoot:

Exception
Error
Fail
Fault
OOS, OUT_OF_SERVICE or out of service.
RECOVERY_DETECTED, which means that the last restore did not complete. In order to try and resolve this problem, run the restore process again.
POTENTIAL_DEADLOCK_DETECTED, which means that if you see POTENTIAL_DEADLOCK_DETECTED in the logs, but do not see DEADLOCK_AVOIDED, try to resolve the problem with an engine restart.

Traces to Turn On

This section lists the traces that you must turn on before an escalation.

General Trace Information

All CRS server tracing denotes debug tracing. For example, when the CRS tracing indicates ICD_HDM, turn on CRS debug tracing for ICD_HDM.

When you set Java Telephony API (JTAPI) tracing, increase the number of log files to 100, and select all debug levels except MISC_DEBUGGING. Remember to restart the engine. Ensure that the Use Rotating Log Files option is checked.

When CRS tracing indicates CCM, you can infer that for both the CallManager service and the CTIManager service:

The Trace On option is checked
The Debug Trace Level is set to Detailed
All other traces are set to the defaults

In addition, you must gather these CCM logs from all CCM nodes:

C:\Program Files\Cisco\Trace\SDL\CCM (SDL trace files)
C:\Program Files\Cisco\Trace\SDL\CTI (SDL trace files)
C:\Program Files\Cisco\Trace\CCM (CCM trace files)
C:\Program Files\Cisco\Trace\CTI (CTI trace files)

CRS 4.0.x Default Traces

In order to match CRS 4.0.x default tracing, ensure that you turn on these debugs on the CRS 3.0.x system:

SS_CM
SS_RM
SS_RMCM
SS_TEL

Minimum Traces

Here is the minimum level of traces that is necessary:

Traces:
- Default Tracing for 4.0.x (for CRS 3.x see the CRS 4.0.x Default Tracing)
Logs:
- MIVR
- MCVD

Minimum traces are necessary for all issues. All other issues need the minimum traces, plus the specific traces mentioned in the various categories.

Symptom Category

Here is the symptom category table:

Main Category	Subcategory
Application Administration Issues	Script Upload or Application Configuration
Application Loading and Invocation Issue	Application Loading and Invocation Issue
ASR/TTS/VXML	ASR TTS VXML
Desktop Issues and Call Control	Agent Desktop Issues and Call Control General Issues with the Cisco Supervisor Desktop Agent State Issues with the Cisco Supervisor Desktop (Example: Missing Agents) Reporting Issues with Cisco Supervisor Desktop
IPIVR (ICM) Related Issues	IPIVR (ICM) Related Issues
Reporting Issues	Historical Reporting Issues Real Time Reporting Issues Calls Stuck in Queue
Script Behavior Issues	Script Behavior Issues
Subsystems Down Issues	Subsystems Down Issues (including machine restarts/engine startup problems)
Upgrade and Install Issues	Upgrade and BARS Issues Installation Issues
VOIP Monitoring and Recording Issues	VOIP Monitoring and Recording Issues

Application Administration Issues

These are the minimum levels of traces and logs necessary for application administration issues:

Traces
- CRS Server
  - MADM
    - ADM_CFG (ADM in 3.x)
    - LIB_CFG (CRS 4.x only)
    - LIB_AXL (CRS 4.x only – For JTAPI configuration items only).
Logs
- CRS Server
  - MADM (4.x only)
  - C:\Program Files\wfavvid\tomcat_appadmin\logs\* (CRS 3.x only)
- CCM - Primary JTAPI Provider
  - C:\Program Files\Cisco\Trace\AXL\* (For JTAPI configuration items only)

Script Upload or Application Configuration Issues

Gather the traces and logs listed in the Application Administration Issues section along with these traces and logs:

Traces
- CRS Server
  - MIVR
    - APP_MGR
    - SCRIPT_MGR (CRS 4.x only)
    - PROMPT_MGR (Only needed for prompt upload issues)
    - ENG
Logs
- CRS Server
  - MIVR
  - MADM (CRS 4.x only)
  - C:\Program Files\wfavvid\tomcat_appadmin\logs\* (CRS 3.x only)

Application Loading and Invocation Failures

Here are the traces and logs necessary:

Traces
- CRS Server
  - MIVR
    - APP_MGR
    - ENG
Logs
- CRS Server
  - MIVR

ASR/TTS/VXML Issues

ASR

These traces and logs are necessary to troubleshoot ASR issues:

Traces
- CRS Server
  - MIVR
    - SS_MRCP_ASR (Xdebugging1 CRS 4.x only)
    - TRACELEVEL.MIVR.SS_NUAN_ASR.XDEBUGGING=true in file C:\Program Files\wfavvid\system.properties (CRS 3.x only)
    - TRACELEVEL.MIVR.SS_NUAN_TTS.XDEBUGGING=true in file C:\Program Files\wfavvid\system.properties (CRS 3.x only)
    Note: The last two traces require a restart of the CRS engine in order to take effect.
Logs
- CRS Server
  - MIVR (for Debug Level)
- Speech Server (CRS 3.x only)
  - C:\Program Files\Cisco Systems\WFNuance SpeechServer\Logs\*.* - Collect from all the configured Speech Servers. These logs can be co-located with CRS.

TTS

These traces and logs are necessary to troubleshoot TTS issues:

Traces
- CRS Server
  - MIVR
    - SS_MRCP_TTS (Xdebugging1 CRS 4.x only)
    - TRACELEVEL.MIVR.SS_NUAN_ASR.XDEBUGGING=true in file C:\Program Files\wfavvid\system.properties (CRS 3.x only)
    - TRACELEVEL.MIVR.SS_NUAN_TTS.XDEBUGGING=true in file C:\Program Files\wfavvid\system.properties (CRS 3.x only)
    Note: The last two traces require a restart of the CRS engine in order to take effect.
Logs
- CRS Server
  - MIVR (for Debug Level)
- Speech Server (CRS 3.x only)
  - C:\Program Files\Cisco Systems\WFNuance SpeechServer\Logs\*.* - Collect from all the configured Speech Servers. These logs can be co-located with CRS.

VXML

These traces and logs are necessary to troubleshoot VXML issues:

Traces
- CRS Server
  - MIVR
    - SS_VB
    - STEP_VOICEBROWSER (in CRS 3.x Only)
Logs
- CRS Server
  - MIVR

Desktop Issues/Call Control

Agent Desktop Issues and Call Control

Issues with the Cisco Agent Desktop or Call Control require these traces and logs:

Traces
- CRS Server
  - JTAPI
  - MIVR
    - ICD_CTI
- Agent Desktop
  - Level=305 (level = 204 for CRS 3.x) (C:\program files\cisco\desktop\config\agent.ini (3.x = fastcalllocal.ini)). In CRS 3.x, you must close and re-open Cisco Agent Desktop in order to set tracing. In CRS 4.x traces are set dynamically.
Logs
- CRS Server
  - JTAPI
  - MIVR
- Agent Desktop
  - C:\program files\cisco\desktop\log\agent*.*
  - C:\program files\cisco\desktop\log\ciscojtapi*.* (This file exists only in CRS 3.x., and not in CRS 4.x.)

General Issues with the Cisco Supervisor Desktop

For general issues with Cisco Supervisor Desktop, you need these traces and logs:

Traces
- Cisco Supervisor Desktop
  - Level=305 (level = 204 for for CRS 3.x) C:\program files\cisco\desktop\config\supervisor.ini. In CRS 3.x you must close and re-open Cisco Supervisor Desktop to set tracing. In CRS 4.x, traces are set dynamically.
  - Level = 4
  - ICD_CTI
Logs
- Cisco Supervisor Desktop
  - C:\program files\cisco\desktop\log\agent*.*
  - C:\program files\cisco\desktop\log\supervisor*.*

Agent State Issues with the Cisco Supervisor Desktop (Example: Missing Agents)

For any agent issues with Cisco Supervisor Desktop, collect these traces along with the traces and logs mentioned in the General Issues with the Cisco Supervisor Desktop section:

Traces
- Cisco Supervisor Desktop
  - Level=4 for C:\program files\cisco\desktop\config\FCCServer.cfg

Reporting Issues with Cisco Supervisor Desktop

You need these traces and logs:

Traces
- CRS Sever
  - MIVR
    - ICD_RTDM Debug (all CRS versions) + Xdebugging 1 (CRS 4.x only)
    - Set flag TRACELEVEL.MIVR.ICD_RTDM.XDEBUGGING=true in file C:\Program Files\wfavvid\system.properties (CRS 3.x only)
    - ICD_CTI
Logs
- Cisco Supervisor Desktop
  - C:\program files\cisco\desktop\log\*.log
  - C:\program files\cisco\desktop\log\*.dbg
- CRS Server
  - MIVR

IPIVR (ICM) Related Issues

These traces and logs are necessary for issues related to IPIVR:

Traces
- CRS Sever
  - MIVR
    - SS_ICM
    - LIB_ICM
Logs
- CRS Server
  - MIVR

Reporting Issues

Historical Reporting Issues

Historical reporting issues require these traces and logs:

Traces
- Historical Reporting Client
  - C:\program files\cisco cra historical reporting\scheduler\sch.ini Change the log Level=4
  - C:\program files\cisco cra historical reporting\hrcConfig.ini Change the log Level=3
- CRS Server
  - MIVR
    - ICD_HDM
    - CRA_HRDM
Logs
- Historical Reporting Client
  - C:\program files\cisco cra historical reporting\logs\*
  - C:\program files\cisco cra historical reporting\scheduler\ciscosch.log
  - C:\program files\cisco cra historical reporting\scheduler\historicalreportscheduler.mdb
- CRS Server
  - MIVR
- Historical reporting server (not necessary for client login issues)
  - Export data from database to plain text files.

Real-Time Reporting Issues

Real-time reporting issues require these traces and logs:

Traces
- CRS Server
  - MIVR
    - ICD_RTDM Debug (all CRS versions) + Xdebugging 1 (CRS 4.x only)
    - Set flag TRACELEVEL.MIVR.ICD_RTDM.XDEBUGGING=true in file C:\Program Files\wfavvid\system.properties (CRS 3.x only)
Logs
- CRS Server
  - MIVR
- A screenshot that shows the problem with the timestamp

Calls Stuck in Queue

Calls that are stuck in the queue require these traces and logs:

Traces
- CRS Server
  - MIVR
    - ICD_RTDM Debug (all CRS versions) + Xdebugging 1 (CRS 4.x only)
    - Set flag TRACELEVEL.MIVR.ICD_RTDM.XDEBUGGING=true in file C:\Program Files\wfavvid\system.properties (CRS 3.x only)
    - ICD_CTI.
  - JTAPI
- CallManager
  - CCM
Logs

Note: Capture logs from the time when the call started to the time when the call got stuck.
- Reporting Client
  - Screenshots of Contacts and CSQ IP ICD Stats real-time reports.
  - A screenshot that shows the problem with the timestamp
- CRS Server
  - JTAPI
  - .aef script.
- CallManager
  - CCM

Script Behavior Issues

For any issues with the script behavior, these traces and logs are necessary:

Traces
- CRS Sever
  - MIVR
    - APP_MGR
    - ENG
    - SCRIPT_MGR (CRS 4.x only)
    - Select the related Step and Subsystem tracing.
      - Example, for DB step related issues:
        
        SS_DB
        
        STEPS_DB (DB_STEPS in CRS 3.x)
Logs
- CRS Server
  - MIVR

Subsystems Down Issues

Subsystems down issues include machine restarts and engine startup problems. You need these traces and logs:

Traces
- CRS Sever
  - MIVR
    - Corresponding Subsystem debug
    - ENG
Logs
- CRS Server
  - MIVR
  - If engine does not start, you need Windows Application and System Event Viewer logs
    
    Note: Always use the Save As option, and save the file in the .txt format. Do not save the file in the .evt format. Also, DO NOT use the Export option.

Upgrade or Installation Issues

Upgrade and BARS Issues

Issues with upgrade or Backup and Restore System (BARS) require these traces and logs:

Traces
- CRS 4.x default
Logs
- Logs from BARS Backup Server (typically the CallManager Publisher):
  
  Note: Zip these files and name the file, BARS.zip.
  - C:\Program Files\cisco\trace\BARS\*.*
  - C:\Program Files\common files\Cisco\logs\BARS\*.* (Including the \BACKUP and \RESTORE folders)
  - C:\winnt\system32\barbi.log
- From each CRS server
  
  Note: Zip these files and name the file CRSNodeX.zip, where X denotes the CRS node number.
  - Copy these files to the C:\ Drive:
    - C:\CalInstall.log
    - C:\CRSAutorun.log
    - C:\CRSMsiInstallLog.txt
    - C:\CRSMsdeInstallLog.txt
    - C:\CRSMsiUnregister.log ( if it exists)
    - C:\UpdateTool.log
    - C:\SQLInstallLog.txt
    - C:\jcuLog.txt
    - C:\SQLUtil.log
    - C:\Updatetool.log
  - Copy these files to a folder named CRSInstall
    - C:\Documents and Settings\<Logged on account, typically "Administrator">\Local Settings\Temp\CRSInstallation\*.* ("Administrator" in most cases)
  - Copy these files to a folder named WINNT
    - C:\Winnt\*.*
      
      Note: This set of files includes any files that were created on the date of the installation failure, at the root of WINNT, and obviously not the entire directory or all files.
  - Copy these files to a folder named MCVD:
    
    Note: These files may or may not exist, depending on the part in the process that encountered the error.
    - C:\Program Files\wfavvid\log\MCVD\*.*
- Copy these files to a folder named MIVR:
  
  Note: These files may or may not exist, depending on the part in the process that encountered the error.
  - C:\Program Files\wfavvid\log\MIVR\*.*
- Copy these files into a folder named MARC:
  - C:\Documents and Settings\<Logged on account, typically "Administrator">\Local Settings\Temp\log\MARC\*.*
- Windows Application and System Event Viewer logs
  
  Note: Always use the Save As option, and save the file in the .txt format. Do not save the file in the .evt format. Also, DO NOT use the Export option.

Installation Issues

Installation issues require these traces and logs:

Traces
- CRS 4.x default
Logs
- You need the same logs as the Upgrade and BARS Issues section, except that you do not need the Windows Application and System Event Viewer logs.

VOIP Monitoring and Recording Issues

These traces and logs are necessary:

Traces
- Cisco Supervisor Desktop
  - Level=8000 C:\program files\cisco\desktop\config\supervisor.ini)
- CRS Server
  - C:\Program Files\Cisco\Desktop\config\FCVoipmonsvr.cfg set Level = 2 (dynamic)
  - C:\Program Files\Cisco\Desktop\config\ FCRasSvr.cfg set Range = 1–4, 50,3000–8000
Logs
- Cisco Supervisor Desktop
  - C:\program files\cisco\desktop\log\agent*.*
  - C:\program files\cisco\desktop\log\supervisor*.*
- CRS Server
  - C:\Program Files\Cisco\Desktop\log\FCVoipmon*.*
  - C:\Program Files\Cisco\Desktop\log\ FCRas*.*

Sniffer traces are often necessary to determine whether you receive the appropriate audio streams.

To record: Sniffer cap from Agent Desktop and from the Recording server.
To monitor: Sniffer cap from Agent Desktop and from the Cisco Supervisor Desktop.