Troubleshooting
During power up, your Cisco Catalyst 6500 Series switch or Cisco 7600 Series Internet Router goes through a system check. This system check also includes all the modules (for example, the MWAM) that are installed.
If your MWAM is not powering up properly, begin troubleshooting using the procedures in this appendix. This appendix guides you through some initial checks and procedures that can solve some basic MWAM problems. It can also direct you to the appropriate document for detailed troubleshooting information and procedures to solve more complex problems.
This appendix contains general problem solving information about your Catalyst 6500 switch or Cisco 7600 router and specific problem solving information about your MWAM module in the following sections:
Tip 
If you cannot locate the source of the problem, contact a customer service representative for information on how to proceed. For technical support information, see the Cisco Information Packet publication that shipped with your product. Before you call, have the following information ready:
•Chassis type and serial number
•Maintenance agreement or warranty information
•Type of software and version number
•Date you received the new chassis
•Brief description of the problem
•Brief explanation of the steps you have taken to isolate the problem
For specific troubleshooting details about the Catalyst 6500 switch or Cisco 7600 router, refer to the Troubleshooting chapters in the following books:
•Catalyst 6500 Series Switch Installation Guide—Appendix E
•Cisco 7600 Series Router Installation Guide—Chapter 4
General Problem Solving
As you start your problem solving, you first look at the way your Catalyst 6500 switch or Cisco 7600 router is responding during power up. The key to problem solving is to isolate the problem to a specific subsystem by comparing what your switch or router is doing to what it should be doing.
The LEDs on the front panel of the switch or router enable you to determine the switch or router performance and operation. For a description of these LEDs, refer to the following books:
•Catalyst 6500 Series Switch Installation Guide—Appendix E
•Cisco 7600 Series Router Installation Guide—Chapter 4
When problem solving, consider the following:
•Power and cooling systems—Power LEDs, external power source, power cable, power supply and circuit breaker, and blower and fan. Also consider inadequate ventilation or air circulation.
•Modules—LEDs on the modules help identify a failure (see the "Front-Panel Features" section for details about the MWAM STATUS LED).
•Cables—External cables that connect the router to the network.
Troubleshooting the Power and Cooling Systems
In your Catalyst 6500 switch or Cisco 7600 router, both the power LED and the fans can help you troubleshoot a power problem. The following items will help you isolate the problem (refer to the "Troubleshooting the Power Supply" section in Appendix E, "Troubleshooting" of the Catalyst 6500 Series Switch Installation Guide or the "Troubleshooting the Power Supply" section in Chapter 4, "Troubleshooting" of the Cisco 7600 Series Router Installation Guide for details):
Check the following items to help isolate problems with the power supply installation:
•With the Catalyst 6500 switch or Cisco 7600 router connected to the power source, is the power LED on the front panel on?
–If not, check the DC input, DC source, and the power supply wiring.
–Check the power supply connection to the motherboard.
–If the power LED is still off, the problem might be a power supply failure.
•Does the Catalyst 6500 switch or Cisco 7600 router shut down after being on a short time?
–Check the fans. If the fans are not working, the switch or router will overheat and shut itself down.
–If the fans are not working, check the power supply connections to the fans.
–Ensure that the chassis intake and exhaust vents are clear.
Check the environmental site requirements in the:
–Catalyst 6500 Series Switch Installation Guide—Chapter 2
–Cisco 7600 Series Router Installation Guide—Chapter 2
•Does the switch or router partially boot, but the LEDs do not go on?
–Check for a power supply failure by inspecting the power LED on the front panel of the switch or router. If the power LED is on, the power supply is functional.
–If the power LED is not on, refer to the Cisco Information Packet for warranty information or contact customer service.
Environmental Monitoring
The Catalyst 6500 switch and the Cisco 7600 router both provide environmental monitoring of their chassis components for over temperature conditions. This environmental monitoring provides an early warning indication of possible component failure to ensure safe and reliable system operation and to avoid network interruptions.
Some causes of abnormally high switch or router temperatures are:
•Fan failure
•Air conditioner failure in the room
•Air blockage to cooling vents
Take steps to correct the problem. For information about environmental operating conditions, refer to the following books:
•Catalyst 6500 Series Switch Installation Guide—Chapter 2
•Cisco 7600 Series Router Installation Guide—Chapter 2
Troubleshooting Modules
Typical module problems can be caused by the module itself, cables, or cable connections. As a rule, just verifying the status LEDs of a particular module will help you to determine any faults.
Check for the following general symptoms to help isolate a problem:
•Module is not recognized by the switch or router.
–Make sure the module is firmly seated in its slot.
–Check the LEDs on the module. Each module has its own set of LEDs.
–Make sure you have a version of Cisco IOS software that supports the module.
•Module is recognized, but interface ports do not initialize.
–Make sure the module is firmly seated in its slot.
–Check external cable connections.
–Make sure you have a version of Cisco IOS software that supports the module.
•Switch or router does not boot properly, or constantly or intermittently reboots.
–Make sure the module is firmly seated in its slot.
–Check the switch or router chassis or software. Refer to the Cisco Information Packet publication that shipped with your switch or router for warranty information, or contact customer service.
•Switch or router powers on and boots only when a particular module is removed.
–Check the module. Refer to the Cisco Information Packet publication that accompanied your order for warranty information, or contact customer service.
•Switch or router powers on and boots only when a particular cable is disconnected.
–There may be a problem with the module or cable. Refer to the Cisco Information Packet publication for warranty information, or contact customer service.
MWAM Problem Solving
In a typical MWAM problem solving situation, you would first check the following:
•The module is firmly seated in its slot.
•The installation is verified.
•The STATUS LED is on.
•The version of Cisco IOS software that supports the MWAM is installed.
•The MWAM configuration is verified.
This section also contains information that can be useful for some common problems that may affect the proper operation of the MWAM, such as:
•DIMM Issue
•Performance Tuning
•Buffer Allocation for Complex 0 CPUs
Module Seating
Note All Catalyst 6500 series switches and Cisco 7600 series internet routers support hot swapping, which lets you install, remove, replace, and rearrange the modules without turning off the system power. When the system detects that a module has been installed or removed, it runs diagnostic and discovery tests automatically, acknowledges the presence or absence of the module, and resumes system operation with no operator intervention. However, DO NOT remove the MWAM until it has shut down (see the "Front-Panel Features" section and the "MWAM Shutdown" section for details).
To ensure the MWAM is securely seated, see the "MWAM Installation" section.
Verify the installation
Note As a rule, right after installing the MWAM, you verify the installation of the MWAM.
To verify the installation of the MWAM, see the "Verifying the Installation" section.
STATUS LED
Verify that the MWAM STATUS LED is on. Periodically, check the STATUS LED. If the STATUS LED changes to orange or red, the MWAM has not successfully completed the boot process or may have encountered an error. For more information about the orange or red STATUS LED states, see Table 1-1.
Determining the MWAM Cisco IOS Image Name
To determine the MWAM Cisco IOS image name, execute the following command at the Cisco Supervisor 2 (see the "Software Description" section for more details):
show cdp entry *
You can also execute the following command at the Supervisor to determine the MWAM Cisco IOS image name:
session slot x processor [2-6]
Then issue the following command:
show version
You can also enter the following command from the privileged EXEC mode (also available through show images at the processor complex):
show version
Verifying the MWAM Configuration
To determine the MWAM configuration, see the "Verifying the MWAM Configuration" section.
DIMM Issue
The following shows a possible DIMM issue (CSCin29212) that may be solved by re-inserting the DIMM memory:
*** Cache Error Exception***
Cache Err Reg = 0x14001f20
instruction reference, primary cache, , tag field error, error on internal request on
SysAD Bus,
PC = 0x800fb5d4, Cause Reg = 0x20008068, Status Reg = 0x400005
%MWAM-0-CORRECTABLE_ECC_ERR: A correctable ECC error has occurred,
A_BUS-L2_ERRORS: 0x0, A_BU
A-MEMIO-ERRORS: 0xFF, A_SCD_BUS_ERR_STATUS: 0x80DC0000
RRECTABLE_ECC_ERR: A correctable ECC error has occurred.
MWAM Performance Tuning
You can improve system performance (by 6 to 12 percent depending on the traffic type) by turning off VLAN accounting:
Router# configure terminal
Router(config)# no vlan accounting
Router#
Maximum Buffer Allocation for Complex 0 CPUs
When upgrading processor complex images, it may be necessary to set the maximum allowable small buffer allocation for processor complex 0 to 7000 and the maximum allowable middle sized buffer allocation to 3500.
MWAM System Error Messages
Error messages may be received and reported in the system log (SysLog). This section lists these messages from the MWAM SysLog Files.
Processor Complex Resetting
When upgrading MWAM software, you may see an error message similar to the following:
%SVCLC-SP-5-STRRECVD: mod 5: MWAM: No response from IOS complex <complex>, resetting complex.
If this occurs, you must upgrade the ROMMON image (see the "Upgrading the ROMMON Image" section.
Note After receiving the error message, proceed immediately to the procedure named Upgrading the ROMMON Image before the error occurs again. If you are unable to perform this procedure immediately, the processor complex continues to reset at three minute intervals until the ROMMON image upgrade process is completed. The ROMMON image upgrade process should take approximately 30 seconds to complete.
Crashinfo/Crashdump
Error Message %MWAM_FILESYSTEM-3-FC_CANNOT_FORMAT: Insufficient memory to format bootflash file system.
Explanation The router does not have enough working storage available in heap memory to format a file system for crashinfo data. The router will function normally without a bootflash file system, however, the router cannot save information for problem analysis if it crashes.
Recommended Action Try to format the file system again at a later time or shortly after a re-boot.
Error Message %MWAM_FILESYSTEM-3-FC_FLASH_WRITE_ERROR: Cannot write to bootflash file system:
Explanation The router cannot write to the bootflash.
Recommended Action If the problem occurred while executing a CLI command, try the command again. If problems persist, contact Cisco technical support.
Error Message %MWAM_FILESYSTEM-3-FC_INVALID_DIR_REQ: Cannot read or write complete %s file system.
Explanation A CLI command attempted to copy or overwrite the full file system.
Recommended Action Copy from or to an individual file.
Error Message %MWAM_FILESYSTEM-3-FC_DIR_IS_FULL: Cannot add %s to file system, directory is full.
Explanation The file system directory holds a limited number of entries, and an attempt was made to add a new file when the directory was already full. This can happen even when the file system still contains free space.
Recommended Action Delete an existing file or FORMAT the file system.
Error Message %MWAM_FILESYSTEM-6-BOOTFLASH_BUSY: Cannot access bootflash, filesystem is busy
Explanation The MWAM bootflash does not support reading and writing at the same time. The message means that bootflash is being accessed by some other operation, like copy, dir, format or squeeze.
Recommended Action Try the bootflash operation again after other operations have completed.
Flash Error Messages
Error Message %MWAM_FLASH-3-ERR: "%s"
Explanation An error occurred while programming/erasing the BootFlash
Recommended Action Copy the error message exactly as it appears on the console or in the system log. Research and attempt to resolve the error using the Output Interpreter "OUTPUT_INTERPRETER." Also perform a search of the Bug Toolkit "BUG_TOOLKIT." If you still require assistance, open a case with the Technical Assistance Center via the Internet "TAC_CASE_OPEN," or contact your Cisco technical support representative and provide the representative with the gathered information.
Error Message %MWAM_FLASH-3-OTHER_CPU_RELOAD: "%s"
Explanation The other CPU within this complex is reloading, flash programming operation can not proceed.
Recommended Action Wait for the system to complete the reload operation.
Error Message MWAM_FLASH-3-FLASH_LOCKED_BY_OTHER_CPU: "%s"
Explanation The other CPU within this complex is programming the shared flash, flash programming must complete before proceeding with reload
Recommended Action Wait for the other CPU to complete its flash programming operation, then reenter the CLI command.
NVRAM Error Messages
Error Message %MWAM_NVRAM-3-ERR: "%s"
Explanation An error occurred while saving/erasing the configuration from flash.
Recommended Action Copy the error message exactly as it appears, and report it to your technical support representative.
Error Message MWAM_NVRAM-5-INFO: "%s"
Explanation Information showing the state of the NVRAM configuration in the BootFlash.
Recommended Action If this message recurs, copy the error message exactly as it appears on the console or in the system log. Research and attempt to resolve the error using the Output Interpreter "OUTPUT_INTERPRETER." Also perform a search of the Bug Toolkit "BUG_TOOLKIT." If you still require assistance, open a case with the Technical Assistance Center via the Internet "TAC_CASE_OPEN," or contact your Cisco technical support representative and provide the representative with the gathered information.
Error Message %MWAM_NVRAM-5-KRUSTY_NVRAM_NOT_FOUND: "%s"
Explanation Information indicating there is no NVRAM pointer.
Recommended Action This indicates either NVRAM has not initialized properly, or it is corrupted.
FUR Error Messages
Error Message %MWAM_FUR-3-ERR: "%s\n"
Explanation An error occurred while upgrading or invalidating the Field-Upgradeable ROMMON.
Recommended Action Copy the error message exactly as it appears, and report it to your technical support representative.
Error Message %MWAM_FUR-1-INFO: "%s\n"
Explanation Information about the Field-Upgradable ROMMON.
Recommended Action If this message recurs, copy the error message exactly as it appears on the console or in the system log. Research and attempt to resolve the error using the Output Interpreter "OUTPUT_INTERPRETER." Also perform a search of the Bug Toolkit "BUG_TOOLKIT." If you still require assistance, open a case with the Technical Assistance Center via the Internet "TAC_CASE_OPEN," or contact your Cisco technical support representative and provide the representative with the collected information.
System Error Messages
Error Message %MWAM-0-ADDRESS_TRAP: "Address trap: %d occurred on physical address: 0x%llx at EPC: 0x%x, ERR_EPC: 0x%x"
Explanation A previously defined address trap has occurred.
Recommended Action Copy the error message exactly as it appears, and report it to your technical support representative.
Error Message %MWAM-0-UNCORRECTABLE_ECC_ERR: "An uncorrectable ECC error has occurred, A_BUS_L2_ERRORS: 0x%llx, A_BUS_MEMIO_ERRORS: 0x%llx, A_SCD_BUS_ERR_STATUS: 0x%llx"
Explanation An uncorrectable ECC error has occurred.
Recommended Action Copy the error message exactly as it appears, and report it to your technical support representative.
Error Message %MWAM-0-CORRECTABLE_ECC_ERR: "A correctable ECC error has occurred, A_BUS_L2_ERRORS: 0x%llx, A_BUS_MEMIO_ERRORS: 0x%llx, A_SCD_BUS_ERR_STATUS: 0x%llx"
Explanation A correctable ECC error has occurred.
Recommended Action Copy the error message exactly as it appears, and report it to your technical support representative.
Error Message %MWAM-0-MISDIRECTED_INTERRUPT: "A misdirected interrupt occurred with int_mask: 0x%x"
Explanation A misdirected interrupt occurred.
Recommended Action Copy the error message exactly as it appears, and report it to your technical support representative.
Feedback