Troubleshooting


If your system has problems during start up or operation, use the information in this chapter to help isolate the cause. Problems with the initial startup are often caused by a switching module that has become dislodged from the backplane or a power cord that is disconnected from the power supply.

Although temperature conditions above the maximum acceptable level rarely occur at initial startup, some environmental monitoring functions are included in this chapter because they also monitor power supply output voltages.

Information about troubleshooting software features and configuration problems is not discussed in this chapter.

The information is presented in these sections:

System Boot Verification

Problem Solving to the System Component Level

Using LEDs to Identify Startup Problems

Troubleshooting the Power Supply

Troubleshooting the Fan Assembly

Troubleshooting Backplane Modules

Troubleshooting Switching Modules

Troubleshooting Supervisor Engines

Contacting the Cisco Technical Assistance Center


Note This chapter covers only the chassis component hardware aspects of troubleshooting. For configuration issues, refer to the software configuration guide or command reference for your switch.


System Boot Verification

When the initial system boot is complete, verify the following:

That the power supplies are supplying power to the system

The LEDs should be green. Use the show environment Cisco IOS command to view power supply activity.

That the system fan assembly is operating

Listen for fan activity. The Fan tray LED should be green during operation.

That the system software boots successfully

Hook up a terminal and view the startup banner. Use an RJ-45-to-RJ-45 rollover cable to connect the console port to a PC with terminal emulation software set for 9600 baud, 8 data bits, no parity, and 1 stop bit. Watch for any messages after startup.

That the supervisor engine and all switching modules are installed properly in their slots, and that each initialized without problems

If all of these conditions are met and the hardware installation is complete, refer to the software configuration guide and command reference publications for your switch so that you can troubleshoot the software.

If any of these conditions is not met, use the procedures in this chapter to isolate and, if possible, resolve the problem.

Problem Solving to the System Component Level

The key to success when troubleshooting the system is to isolate the problem to a specific system component. Your first step is to compare what the system is doing to what it should be doing. Because a startup problem can usually be caused by a single component, it is more efficient to isolate the problem to a subsystem rather than troubleshoot each separate component in the system.

The switch consists of the following subsystems:

Power supply—Includes the power supplies and power supply fans. If you suspect problems with the power supplies, see the "Troubleshooting the Power Supply" section.

Fan assembly—The system fan assembly should be operating whenever system power is on. You should be able to hear the fan assembly when the system is operating. If you determine that the fan assembly is not operating, see the "Troubleshooting the Fan Assembly" section.

Supervisor engine—The supervisor engine contains the operating system software, so check the supervisor engine if you have trouble with the system software. Status LEDs on the supervisor engine indicate whether the supervisor engine has initialized correctly. If there is a problem with the supervisor engine, try reseating the supervisor engine in the chassis and restarting the switch. For more information about LED meanings, refer to the "LEDs" section. If the supervisor engine does not boot correctly after reseating and restarting, contact Cisco TAC. See the "Contacting the Cisco Technical Assistance Center" section.

Switching modules—The LEDs labeled STATUS on each switching module indicate whether the switching module has been initialized correctly. The supervisor engine must be operating properly before the switching module will initialize. If a switching module is improperly installed in the switch, it will not function, so if the supervisor engine is working but a switching module is not, try reseating the module. For more informations, see the "Troubleshooting Switching Modules" section. If you determine that the switching module is not operating, contact Cisco TAC as described in the "Contacting the Cisco Technical Assistance Center" section.

Using LEDs to Identify Startup Problems

All system states in the startup sequence are indicated using LEDs. By checking the LEDs, you can determine when and where the system failed in the startup sequence. If you have problems after the switch is on, refer to the configuration procedures in the software configuration guide for your switch.

After you connect the power cords to your Catalyst 4500 series switch, follow these steps to determine whether your system is operating properly:


Step 1 Check the power supply LEDs:

The LED labeled GOOD should turn green when power is applied to the supply. The LED should remain on during normal system operation.

If the LED labeled GOOD does not light, or if the LED labeled FAIL lights, see the "Troubleshooting the Power Supply" section.


Note If a power supply is installed and not connected to a power source, power supply LEDs will indicate a failure.


Step 2 Listen for the system fan assembly. If you do not immediately hear it begin to operate, see the "Troubleshooting the Fan Assembly" section.

Step 3 Check that the LEDs on the supervisor engine light as follows:

The LED labeled STATUS flashes orange once and stays orange during diagnostic boot tests.

It turns green when the module is operational (online).

If the system software is unable to start up, this LED stays orange.

If the LED labeled STATUS on the supervisor engine front panel is red or orange, connect a console to the management port and use the show environment command to check for possible problems. For more information, see the "Troubleshooting Supervisor Engines" section.

The Ethernet management port LED turns green when the module is operational (online) and a link is established with another network device. If no signal is detected, the LED labeled LINK turns off.

Step 4 Verify that the LEDs labeled STATUS on each switching module are green when the supervisor engine completes initialization.

This LED indicates that the supervisor engine and switching modules are receiving power, have been recognized by the supervisor engine, and contain a valid Flash code version. However, this LED does not indicate the state of the individual interfaces on the switching modules. If an LED labeled STATUS is red or orange, try reseating the switching module or supervisor engine and restarting the switch. If the problem persists, contact Cisco TAC. See the "Contacting the Cisco Technical Assistance Center" section.

Step 5 If the boot information and system banner are not displayed, verify that the terminal is set for 9600 baud, 8 data bits, no parity, and 1 stop bit and connected properly to the console port.


Troubleshooting the Power Supply

To help isolate a power subsystem problem, follow these steps:


Step 1 Check whether the power supply LED labeled GOOD is on or the LED labeled FAIL is on. (on the DC multi-input power supply, the LEDs are labeled INPUT 1, 2, or 3 or OUTPUT FAIL.)

Step 2 If the LED labeled GOOD is off or if the LED labeled FAIL is on, take the following steps:

Ensure that the power supply is flush with the back of the chassis.

Unplug the power cord, loosen and reinstall the power supply, tighten the captive installation screws, and then plug in the power cord.

Step 3 If the LED labeled GOOD remains off, there might be a problem with the AC source or the power cable. Connect the power cord to another power source if one is available.

Step 4 If the LED labeled GOOD fails to light after you connect the power supply to a new power source, replace the power cord.


Note If this unit has more than one power cord, repeat Step 1 through Step 4 for each power input.


Step 5 If the LED labeled GOOD still fails to light when the switch is connected to a different power source with a new power cord, the power supply is probably faulty. Replace the power supply.

Step 6 If the LED labeled FAN OK fails to light when the switch is connected to a power source with a known good power cord, there is a malfunction in the fan that cools the power supply. Replace the power supply.

Step 7 If a second power supply is available, install it in the second power supply bay.

Step 8 Check that the LED labeled GOOD is on for the additional power supply. Check that the LED labeled FAIL is off.

Step 9 If the LEDs are not on, repeat the previous procedure to troubleshoot the second power supply.

If you are unable to resolve the problem, or if you determine that either a power supply or backplane connector is faulty, contact the Cisco Technical Assistance Center (TAC) for instructions.


Troubleshooting Power Problems using Cisco IOS

Check for system messages related to the power supply, and refer to the system message guide for your software release. You may need to add a power supply or upgrade to a larger one for your current configuration or verify that the switches are set properly. Connect a terminal to the console port, and look for any of the following system messages:

C4K_CHASSIS-2-INLINEPOWEROFF

C4K_CHASSIS-2-INSUFFICIENTPOWERDETECTED

C4K_CHASSIS-2-INSUFFICIENTPOWERSHUTDOWN

C4K_CHASSIS-3-INSUFFICIENTPOWER

C4K_CHASSIS-3-INSUFFICIENTPOWERSUPPLIESDETECTED

C4K_CHASSIS-3-MIXINPOWERDETECTED

C4K_IOSMODPORTMAN-3-UNKNOWNPOWERSUPPLY

C4K_IOSMODPORTMAN-4-INLINEPOWEROVERMAX

C4K_IOSMODPORTMAN-4-INLINEPOWERSUPPLYBAD

C4K_IOSMODPORTMAN-4-POWERSUPPLYBAD

C4K_IOSMODPORTMAN-4-POWERSUPPLYFANBAD

C4K_SUPERVISOR-3-POWERSUPPLYSTATUSREADFAILED

C4K_SUPERVISOR-3-POWERSUPPLYSEEPROMREADFAILED

C4K_SUPERVISOR-3-POWERSUPPLYSEEPROMINVALID

C4K_SUPERVISOR-4-INLINEVOLTAGEOUTOFRANGE

You may also use the show environment status powersupply, show module all, and show power commands to monitor PS status, load, and activity.

The 1400 W DC multi-input supply can not be used with other power supply types, but other power supplies in this product line work with the other types during an upgrade. If you mix power supplies in a Catalyst 4500 series chassis, the switch detects the type of power supply in power supply bay 1 (PS1) and ignores the power supply in power supply bay 2 (PS2) while issuing system messages and showing the power supply in bay 2 as in the err-disable state in the output of the show power command. When the power supply in bay 1 is removed, the switch recognizes the power supply in bay 2, and you may then put a new matching power supply in bay 1. Both supplies should now resume normal function.

Troubleshooting the Fan Assembly


Note All fans must be operating or a failure will occur.


Environmental problems may initially appear to be problems with the fan tray. To help isolate a fan assembly problem, follow these steps:


Step 1 Check the status LED on the fan tray.

If the LED is off and the rest of the system is functioning, the fan tray is not getting power or is not seated correctly on the backplane.

If the LED is green, the fans are operating normally. There may be conditions impairing fan performance, but they are minimal in impact.

If the LED is red, one or more fans have failed.

Step 2 Connect a terminal and determine the fan tray status shown by the show environment status CLI command.

Step 3 Look for system messages reporting a temperature problem or problem with the fans. Individual messages may suggest different solutions. Connect a terminal to the console port, and look for any of the following system messages:

C4K_CHASSIS-2-INSUFFICIENTFANSDETECTED

C4K_CHASSIS-2-INSUFFICIENTFANSSHUTDOWN

C4K_IOSMODPORTMAN-4-CRITICALTEMP

C4K_IOSMODPORTMAN-4-FANTRAYBAD

C4K_IOSMODPORTMAN-4-FANTRAYPARTIALFAILURE

C4K_IOSMODPORTMAN-4-FANTRAYREMOVED

C4K_SUPERVISOR-3-FANTRAYREADFAILED

C4K_SUPERVISOR-3-FANTRAYSEEPROMREADFAILED

C4K_SUPERVISOR-3-FANTRAYSEEPROMINVALID

C4K_IOSMODPORTMAN-4-TEMPHIGH

C4K_IOSMODPORTMAN-4-TEMPUNDERCRITICAL

C4K_CHASSIS-2-OVERHEATINGSHUTDOWN

Step 4 Determine whether the airflow is restricted or if the ambient temperature in the room is too warm.

Step 5 Determine whether the power supply is functioning properly. See the "Troubleshooting the Power Supply" section.

Step 6 Verify that the fan assembly is properly seated in the backplane by loosening the captive installation screws, removing the fan assembly, and reinstalling it.

Step 7 Restart the system.

Step 8 Verify that all fans are operating. You should hear the fans at system start.

Step 9 If the system is still detecting a fan assembly failure, contact the Cisco TAC for assistance.


Troubleshooting Backplane Modules

The Cisco Catalyst 4500 Series redundancy scheme uses removable redundancy modules (also called mux-buffers), on the passive backplane to switch traffic to the active supervisor engine. There is one redundancy module per switching module. Redundancy modules and a redundant clock ship standard with every Cisco Catalyst 4507R and 4510R chassis. Spare redundancy modules (C4K-MUX=) and a clock module (C4K-CLOCK=) are available for serviceability.

The following conditions indicate that you may need to replace the redundancy modules and clock module:

The switch powers down and stays down for a few minutes to a few days for no clear reason.

The output-fail LED on the power supplies are red and no power is delivered to the chassis. The other LEDs on the power supply are green.

The Status LEDs on the switching modules and the supervisor engine are flashing green.

CPU Utilization LEDs are flashing green or off.

If you observe these conditions, contact the Cisco TAC for assistance in ordering replacement redundancy modules and a clock.

Troubleshooting Switching Modules

Each switching module has one LED labeled STATUS that provides information about the module and one numbered LED labeled LINK for each port on the module. Figure 5-1 shows the Gigabit Ethernet port and status LEDs. Figure 5-2 shows the 10/100BASE-T port LEDs. Table 5-1 describes the switching module LEDs and their expected behavior.

Figure 5-1 Gigabit Ethernet Port and Status LEDs

Figure 5-2 10/100BASE-T Port LEDs

Table 5-1 Switching Module LEDs 

LED
Color/State
Description

STATUS

 

Indicates the results of a series of self-tests and diagnostic tests performed by the switch.

 

Green

All the tests pass.

 

Red

A test other than an individual port test failed.

 

Orange

System boot, self-test diagnostics running, or the module is disabled.

LINK1

 

Indicates the status of the port.

 

Green

The port is operational (a signal is detected).

 

Orange

The link has been disabled by software.

 

Flashing orange

The link has been disabled due to a hardware failure.

 

Off

No signal is detected.

Port Status2

 

Indicates individual port status.

 

Green

The port is operational (a signal is detected).

 

Orange

The link has been disabled by software.

 

Flashing orange

The link has been disabled due to a hardware failure.

 

Off

No signal is detected.

1 Used on the WS-X4232-L3 Ethernet routing module.

2 LEDs labeled 1 through the number of ports on the switching module are the individual port link LEDs.


Troubleshooting Switching Modules Using Cisco IOS

Connect a terminal to the console port, and look for any of the following system messages:

C4K_CHASSIS-3-LINECARDMUXBUFFERTOSUPALIGNMENTWRONG

C4K_CHASSIS-3-LINECARDNOTVALIDINSLOT

C4K_CHASSIS-3-MODULENOTSUPPORTHALF

C4K_IOSINTF-5-STALEPHYPORT

C4K_IOSMODPORTMAN-4-INLINEPOWEROVERMAX

If you observe any of these messages, follow the suggestion for that message.

Some problems can be solved by resetting the switching module. Use the hw-module module <n> reset command to reset a switching module, or remove and re-insert the switching module, which resets, restarts, and power cycles the switching module. The show module and show diagnostics online module commands can also provide information useful in solving problems with ports on individual modules.

Not all software versions support all switching modules. If you are having trouble with a module, refer to the software release notes to be sure that it is supported by your software.

Troubleshooting Supervisor Engines

This section only addresses problems with hardware. Problems with features or configuration are not covered here. Refer for your software configuration guide and release notes for information on configuring features or identifying known problems.

Table 5-2 describes the supervisor engine LEDs. Check the LEDs on your supervisor and compare them to the described LED behaviors.

Table 5-2 Supervisor Engine LEDs 

LED
Color/State
Description

STATUS

 

Indicates the results of a series of self-tests.

 

Green

All diagnostic tests passed.

 

Red

A test failed.

 

Orange

System boot or diagnostic test is in progress.

 

Off

Module is disabled.

UTILIZATION

Green 1-100%

If the switch is operational, this display indicates the current traffic load over the backplane (as an approximate percentage).

LINK

 

Indicates the status of the 10/100BASE-T Ethernet management port or uplink ports.

 

Green

The link is operational.

 

Orange

The link is disabled by user.

 

Flashing orange

The power-on self-test indicates a faulty port.

 

Off

No signal is detected or there is a link configuration failure.

ACTIVE

 

Indicates whether the uplink port is active or not.

 

Green

The port is active.

 

Off

The port is not active.

ACTIVE

 

The LED to the right of the uplink ports is used to identify the active supervisor engine in switches with two supervisor engines.


Troubleshooting Supervisor Engines Using Cisco IOS

Connect a terminal to the console port, and look for any of the following system messages:

C4K_CHASSIS-3-LINECARDMUXBUFFERTOSUPALIGNMENTWRONG

C4K_SUPERVISOR-3-MUXBUFFERREADSUPERVISORSELECTIONFAILED

C4K_CHASSIS-3-TEMPERATURESENSORREADFAILED

C4K_HW-3-X2IDENTIFICATIONFAILURE

C4K_HW-3-X2OUIREGREADFAILURE

C4K_HWACLMAN-4-CLASSIFCAMPARITYERROR

C4K_HWACLMAN-4-CLASSIFCAMREPLYPARITYERROR

C4K_HWACLMAN-4-CLASSIFCAMREQPARITYERROR

C4K_HWNETFLOWMAN-3-NETFLOWSTOPPED

C4K_HWNETFLOWMAN-4-FATALERRORINTERRUPTSEEN

C4K_HWNETFLOWMAN-4-NONFATALPARITYERRORINTERRUPTSEEN

C4K_IOSMODPORTMAN-4-NFLABSENT

C4K_IOSMODPORTMAN-4-NFLIDPROMINVALID

C4K_IOSMODPORTMAN-4-NFLMISMATCH

C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT

C4K_REDUNDANCY-2-POSTFAIL_RESET

C4K_REDUNDANCY-2-INCOMPATIBLE_SUPERVISORS

C4K_REDUNDANCY-2-IOS_VERSION_CHECK_FAIL

C4K_REDUNDANCY-2-IOS_VERSION_INCOMPATIBLE

C4K_REDUNDANCY-2-NON_SYMMETRICAL_REDUNDANT_SYSTEM

C4K_REDUNDANCY-2-POSTFAIL

C4K_REDUNDANCY-2-POSTFAIL_RESET

C4K_REDUNDANCY-4-CONFIGSYNCFAIL

C4K_SUPERVISOR-2-SUPERVISORSEEPROMINVALID

C4K_SUPERVISOR-3-RETIMERDISABLEFAILED

C4K_SUPERVISOR-3-RETIMERINITFAILED

C4K_SUPERVISOR-3-SEEPROMREADFAILED

C4K_SUPERVISOR-4-INLINEVOLTAGEOUTOFRANGE

C4K_SUPERVISOR-7-SEEPROMWRITEFAILED

C4K_SWITCHMANAGER-3-SSOACTIVEPORTACKTIMEOUT

C4K_SYSMAN-2-POWERONSELFTESTFAIL

These system messages indicate a problem with the supervisor engine. Some problems will prevent a console connection and will not allow you to use messages in diagnosing a problem. If you are unable to establish a terminal connection and the STATUS LED is red, contact Cisco TAC immediately to order a replacement.

Problems with redundant supervisor systems are often due to mismatched active and standby supervisor engines. Redundancy requires that both supervisor engines be the same model and running the same Cisco IOS release. If one supervisor has a NetFlow service card, the other must as well.

Some problems with supervisor engines are due to backplane connections that are not fully seated. If removing and reinserting the supervisor engine and then restarting the switch does not solve the problem, call Cisco TAC and replace the supervisor engine.

Some problems can be solved by resetting the supervisor engine. Use the hw-module module <n> reset power-cycle command to reset a switching module, or remove and re-insert the switching module, which resets, restarts and power cycles the switching module. Pressing the reset button on the supervisor engine causes the software to reload, but does not power cycle the supervisor engine.


Note When you power-cycle or remove a supervisor engine in a redundant system the other supervisor engine becomes the active supervisor and the ports retain connectivity. In a non-redundant system, all of the switching modules lose connectivity until the supervisor engine is reinserted and completely restarted.


The show diagnostics power-on command may provide useful information for some supervisor engine problems.

Not all software versions support all supervisor engines. If you are having trouble with a supervisor engine, refer to the software release notes to be sure that it is supported by your software.

Contacting the Cisco Technical Assistance Center

If you are unable to solve a startup problem after using the troubleshooting suggestions in this chapter, contact a Cisco TAC representative for assistance and further instructions.

Before you call, have the following information ready to help the Cisco TAC assist you as quickly as possible:

Date you received the switch

Chassis serial number (refer to Figure 5-3 to Figure 5-6 for serial number locations)

Type of software and release number

Maintenance agreement or warranty information

Brief description of the problem

Brief explanation of the steps you have already taken to isolate and resolve the problem

See the "Obtaining Documentation and Submitting a Service Request" section for more information about contacting the TAC.

Serial Numbers

When you contact Cisco Technical Assistance, have the serial number of your switch available. Refer to Figure 5-3 to 5-6 to locate the serial number on your switch. You may also get the serial number from the output of a show version command.

Figure 5-3 Catalyst 4503 Serial Number Location

Figure 5-4 Catalyst 4506 Serial Number Location

Figure 5-5 Catalyst 4507R Serial Number Location

Figure 5-6 Catalyst 4510R Serial Number Location