Cisco 7600 Series Router SIP, SSC, and SPA Software Configuration Guide - Troubleshooting the Fast Ethernet and Gigabit Ethernet SPAs [Cisco 7600 Series Routers]

Table Of Contents

Troubleshooting the Fast Ethernet and Gigabit Ethernet SPAs

General Troubleshooting Information

Using debug Commands

Using show Commands

Performing Basic Interface Troubleshooting

Verifying the Interface Is Up

Verifying the Line Protocol Is Up

Verifying Output Hang Status

Verifying the CRC Counter

Verifying Late Collisions

Verifying the Carrier Signal

Understanding SPA Automatic Recovery

When Automatic Recovery Occurs

If Automatic Recovery Fails

Configuring the Interface for Internal and External Loopback

Configuring the Interface for Internal Loopback

Configuring the Interface for External Loopback

Verifying Loopback Status

Using the Cisco IOS Event Tracer to Troubleshoot Problems

Preparing for Online Insertion and Removal of a SPA

Troubleshooting the Fast Ethernet and Gigabit Ethernet SPAs

This chapter describes techniques that you can use to troubleshoot the operation of your Fast Ethernet or Gigabit Ethernet SPAs.

It includes the following sections:

•General Troubleshooting Information

•Performing Basic Interface Troubleshooting

•Understanding SPA Automatic Recovery

•Configuring the Interface for Internal and External Loopback

•Using the Cisco IOS Event Tracer to Troubleshoot Problems

•Preparing for Online Insertion and Removal of a SPA

The first section provides information about basic interface troubleshooting. If you are having a problem with your SPA, use the steps in the "Performing Basic Interface Troubleshooting" section to begin your investigation of a possible interface configuration problem.

To perform more advanced troubleshooting, see the other sections in this chapter.

General Troubleshooting Information

This section describes general information for troubleshooting SIPs and SPAs. It includes the following sections:

•Using debug Commands

•Using show Commands

Using debug Commands

Along with the other debug commands supported on the Cisco 7600 series router, you can obtain specific debug information for SPAs on the Cisco 7600 series router using the debug hw-module bay privileged EXEC command.

The debug hw-module bay command is intended for use by Cisco Systems technical support personnel.

Caution Because debugging output is assigned high priority in the CPU process, it can render the system unusable. For this reason, use debug commands only to troubleshoot specific problems or during troubleshooting sessions with Cisco technical support staff. Moreover, it is best to use debug commands during periods of lower network traffic and fewer users. Debugging during these periods decreases the likelihood that increased debug command processing overhead will affect system use.

For information about other debug commands supported on the Cisco 7600 series router, refer to the Cisco IOS Debug Command Reference and any related feature documents for the applicable Cisco IOS release.

Using show Commands

There are several show commands that you can use to monitor and troubleshoot the SIPs and SPAs on the Cisco 7600 series router. This chapter describes using the show interfaces command to perform troubleshooting of your SPA.

For more information about show commands to verify and monitor SIPs and SPAs, see the following chapters of this guide:

•Chapter 12, "Configuring the Fast Ethernet and Gigabit Ethernet SPAs"

Performing Basic Interface Troubleshooting

You can perform most of the basic interface troubleshooting using the show interfaces fastethernet, show interfaces gigabitethernet, or show interfaces tengigabitethernet command and examining several areas of the output to determine how the interface is operating.

The following example shows output from both the show interfaces fastethernet, show interfaces gigabitethernet and show interfaces tengigabitethernet commands with some of the significant areas of the output to observe shown in bold:
Router# show interfaces fastethernet 3/2/3
FastEthernet3/2/3 is up, line protocol is up 
Hardware is FastEthernet SPA, address is 000e.d623.e840 (bia 000e.d623.e840)
Internet address is 33.1.0.2/16
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec, 
reliability 255/255, txload 59/255, rxload 83/255
Encapsulation ARPA, loopback not set
Keepalive not supported
Full-duplex, 100Mb/sARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:11, output 00:00:08, output hang never
Last clearing of "show interface" counters 3d00h
Input queue: 0/75/626373350/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 32658000 bits/sec, 68032 packets/sec
5 minute output rate 23333000 bits/sec, 48614 packets/sec
	17792456686 packets input, 1067548381456 bytes, 0 no buffer
	Received 0 broadcasts (0 IP multicasts)
	0 runts, 0 giants, 0 throttles
	0 input errors, 0 CRC, 0 frame, 130043940 overrun, 0 ignored
	0 watchdog
	0 input packets with dribble condition detected
	12719598014 packets output, 763177809958 bytes, 0 underruns
	0 output errors, 0 collisions, 0 interface resets
	0 babbles, 0 late collision, 0 deferred
	0 lost carrier, 0 no carrier
	0 output buffer failures, 0 output buffers swapped out
Router# show interfaces gigabitethernet 2/0/1
GigabitEthernet2/0/1 is down, line protocol is down 
  Hardware is GigEther SPA, address is 000a.f330.2e40 (bia 000a.f330.2e40)
  Internet address is 2.2.2.1/24
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not supported
  Full-duplex, 1000Mb/s, link type is force-up, media type is SX
  output flow-control is on, input flow-control is on
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 03:18:49, output 03:18:44, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     1703 packets input, 638959 bytes, 0 no buffer
     Received 23 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 1670 multicast, 0 pause input
     1715 packets output, 656528 bytes, 0 underruns
     0 output errors, 0 collisions, 4 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
Router# show interfaces tengigabitethernet 7/0/0
TenGigabitEthernet7/0/0 is up, line protocol is up (connected)
  Hardware is TenGigEther SPA, address is 0000.0c00.0102 (bia 000f.342f.c340)
  Internet address is 15.1.1.2/24
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not supported
  Full-duplex, 10Gb/s
  input flow-control is on, output flow-control is on 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:10, output hang never
  Last clearing of "show interface" counters 20:24:30
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
  L2 Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes
  L3 in Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes mcast
  L3 out Switched: ucast: 0 pkt, 0 bytes mcast: 0 pkt, 0 bytes
     237450882 packets input, 15340005588 bytes, 0 no buffer
     Received 25 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     1676 packets output, 198290 bytes, 0 underruns
     0 output errors, 0 collisions, 4 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
To verify that your interface is operating properly, complete the steps in Table 13-1:
Table 13-1 Basic Interface Troubleshooting Steps
Action

Example
Step 1

From global configuration mode, enter the show interfaces fastethernet, show interfaces gigabitethernet or show interfaces tengigabitethernet command.
Router# show interfaces fastethernet 3/2/3
 
Router# show interfaces gigabitethernet 2/0/1
 
Router# show interfaces tengigabitethernet 7/0/0
Step 2

Verify that the interface is up.
Router# show interfaces fastethernet 3/2/3
FastEthernet3/2/3 is up, line protocol is up 
Router# show interfaces gigabitethernet 2/0/1
GigabitEthernet2/0/1 is up, line protocol is up 
Router# show interfaces tengigabitethernet 7/0/0
TenGigabitEthernet7/0/0 is up, line protocol is up 
(connected)
Step 3

Verify that the line protocol is up.
Router# show interfaces fastethernet 3/2/3
FastEthernet3/2/3 is up, line protocol is up 
Router# show interfaces gigabitethernet 2/0/1
GigabitEthernet2/0/1 is up, line protocol is up 
Router# show interfaces tengigabitethernet 7/0/0
TenGigabitEthernet7/0/0 is up, line protocol is up 
(connected)
Step 4

Verify that the interface duplex mode matches the remote interface configuration.
The following example shows that the local interface is currently operating in full-duplex mode:
Router# show interfaces fastethernet 3/2/3
[text omitted]
Keepalive not supported
Full-duplex, 100Mb/sARP type: ARPA, ARP Timeout 04:00:00
Router# show interfaces gigabitethernet 2/0/1
[text omitted]
Keepalive not supported
Full-duplex, 1000Mb/s, link type is force-up, media type 
is SX
Router# show interfaces tengigabitethernet 7/0/0
[text omitted]
Keepalive not supported
  Full-duplex, 10Gb/s
Step 5

Verify that the interface speed matches the speed on the remote interface.
The following example shows that the local interface is currently operating at 100 Mbps (Fast Ethernet and Gigabit Ethernet) or 10 Gbps (Ten Gigabit Ethernet):
Router# show interfaces fastethernet 3/2/3
[text omitted]
Keepalive not supported
Full-duplex, 100Mb/sARP type: ARPA, ARP Timeout 04:00:00
Router# show interfaces gigabitethernet 2/0/1
[text omitted]
Keepalive not supported
Full-duplex, 1000Mb/s, link type is force-up, media type 
is SX
Router# show interfaces tengigabitethernet 7/0/0
[text omitted]
Full-duplex, 10Gb/s
Step 6

Observe the output hang status on the interface.
ARP type: ARPA, ARP Timeout 04:00:00
Last input 03:18:49, output 03:18:44, output hang never
Step 7

Observe the CRC counter.
0 input errors, 0 CRC, 0 frame, 130043940 overrun, 0 
ignored
Step 8

Observe the late collision counter.
0 output errors, 0 collisions, 4 interface resets
0 babbles, 0 late collision, 0 deferred
Step 9

Observe the carrier signal counters.
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
For more information about the verification steps and possible responses to correct detected problems, see the following sections:

•Verifying the Interface Is Up

•Verifying the Line Protocol Is Up

•Verifying Output Hang Status

•Verifying the CRC Counter

•Verifying Late Collisions

•Verifying the Carrier Signal

Verifying the Interface Is Up

In the output from the show interfaces fastethernet, show interfaces gigabitethernet or show interfaces tengigabitethernet command, verify that the interface is up. If the interface is down, perform the following corrective actions:

•If the interface is administratively down, use the no shutdown interface configuration command to enable the interface.

•Be sure that the cable is fully connected.

•Verify that the cable is not bent or damaged. If the cable is bent or damaged, the signal will be degraded.

•Verify that a hardware failure has not occurred. Observe the LEDs to confirm the failure. See the other troubleshooting sections of this chapter, and refer to the Cisco 7600 Series Router SIP, SSC, and SPA Hardware Installation Guide. If the hardware has failed, replace the SPA as necessary.

Verifying the Line Protocol Is Up

In the output from the show interfaces fastethernet, show interfaces gigabitethernet or show interfaces tengigabitethernet command, verify that the line protocol is up. If the line protocol is down, the line protocol software processes have determined that the line is unusable.

Perform the following corrective actions:

•Replace the cable.

•Check the local and remote interface for misconfiguration.

•Verify that a hardware failure has not occurred. Observe the LEDs to confirm the failure. See the other troubleshooting sections of this chapter, and refer to the Cisco 7600 Series Router SIP, SSC, and SPA Hardware Installation Guide. If the hardware has failed, replace the SPA as necessary.

Verifying Output Hang Status

In the output from the show interfaces fastethernet, show interfaces gigabitethernet or show interfaces tengigabitethernet command, observe the value of the output hang field.

The output hang provides the number of hours, minutes, and seconds since the last reset caused by a lengthy transmission. When the number of hours in the field exceeds 24 hours, the number of days and hours is shown. If the field overflows, asterisks are printed. The field shows a value of never if no output hangs have occurred.

Verifying the CRC Counter

In the output from the show interfaces fastethernet, show interfaces gigabitethernet or show interfaces tengigabitethernet command, observe the value of the CRC counter. Excessive noise will cause high CRC errors accompanied by a low number of collisions.

Perform the following corrective actions if you encounter high CRC errors:

•Check the cables for damage.

•Verify that the correct cables are being used for the SPA interface.

Verifying Late Collisions

In the output from the show interfaces fastethernet, show interfaces gigabitethernet or show interfaces tengigabitethernet command, observe the value of the late collision counter.

Perform the following corrective actions if you encounter late collisions on the interface:

•Verify that the duplex mode on the local and remote interface match. Late collisions occur when there is a duplex mode mismatch.

•Verify the length of the Ethernet cables. Late collisions result from cables that are too long.

Verifying the Carrier Signal

In the output from the show interfaces fastethernet, show interfaces gigabitethernet or show interfaces tengigabitethernet command, observe the value of the carrier signal counters. The lost carrier counter shows the number of times that the carrier was lost during transmission. The no carrier counter shows the number of times that the carrier was not present during transmission.

Carrier signal resets can occur when an interface is in loopback mode or shut down.

Perform the following corrective actions if you observe the carrier signal counter incrementing outside of these conditions:

•Check the interface for a malfunction.

•Check for a cable problem.

Understanding SPA Automatic Recovery

When Fast Ethernet or Gigabit Ethernet SPAs encounter thresholds for certain types of errors and identify a fatal error, the SPAs initiate an automatic recovery process.

You do not need to take any action unless the error counters reach a certain threshold, and multiple attempts for automatic recovery by the SPA fail.

The Gigabit Ethernet SPAs might perform automatic recovery for the following types of errors:

•SPI4 TX/RX out of frame

•SPI4 TX train valid

•SPI4 TX DIP4

•SPI4 RX DIP2

When Automatic Recovery Occurs

If the SPI4 errors occur more than 25 times within 10 milliseconds, the SPA automatically deactivates and reactivates itself. Error messages are logged on the console indicating the source of the error and the status of the recovery.

If Automatic Recovery Fails

If the SPA attempts automatic recovery more than five times in an hour, then the SPA deactivates itself and remains deactivated.

To troubleshoot automatic recovery failure for a SPA, perform the following steps:

Step 1 Use the show hw-module bay slot/bay oir command to verify the status of the SPA. The status is shown as "failed" if the SPA has been powered off due to five consecutive failures.

Step 2 If you verify that automatic recovery has failed, perform OIR of the SPA. For information about performing OIR, see the "Preparing for Online Insertion and Removal of a SPA" section.

Step 3 If reseating the SPA after OIR does not resolve the problem, replace the SPA hardware.

Configuring the Interface for Internal and External Loopback

Loopback support is useful for testing the interface without connectivity to the network, or for diagnosing equipment malfunctions between the interface and a device. The Fast Ethernet and Gigabit Ethernet SPAs support both an internal and an external loopback mode. The external loopback mode requires the use of a loopback cable and implements a loopback through the transceiver on the SPA.

You can also configure an internal loopback without the use of a loopback cable that implements a loopback at the PHY device internally on a Fast Ethernet or Gigabit Ethernet interface port, or at the MAC device internally on a Fast Ethernet or Gigabit Ethernet interface port. By default, loopback is disabled.

Configuring the Interface for Internal Loopback

Different Fast Ethernet and Gigabit Ethernet interfaces use different loopback commands.

To enable internal loopback at the PHY device for an interface on a SPA, use one of the following commands beginning in interface configuration mode:

Command or Action

Purpose

Router(config-if)# loopback

Enables an interface for internal loopback on the Gigabit Ethernet SPA.

Router(config-if)# loopback internal

Enables an interface for internal loopback on the Gigabit Ethernet SPA.

Router(config-if)# loopback mac

Enables an interface for internal loopback at the MAC controller level on the Fast Ethernet SPA.

Router(config-if)# loopback driver

Enables an interface for internal loopback at the transceiver level on the Fast Ethernet SPA.

Configuring the Interface for External Loopback

Before beginning external loopback testing, remember that the external loopback mode requires the use of a loopback cable.

External loopback cannot be configured on Fast Ethernet SPAs. To enable external loopback on Gigabit Ethernet SPAs, use the following commands beginning in interface configuration mode:

Command

Purpose

Router(config-if)# loopback external

Enables an interface for external loopback on the Gigabit Ethernet SPA.

Verifying Loopback Status

To verify whether loopback is enabled on an interface port on a SPA, use the show interfaces fastethernet, show interfaces gigabitethernet or show interfaces tengigabitethernet in privileged EXEC command and observe the value shown in the "loopback" field.

The following example shows that loopback is disabled for interface port 3 on the Fast Ethernet SPA installed in bay 2 of the SIP that is located in slot 3 of the Cisco 7600 series router:
Router# show interfaces fastethernet 3/2/3
FastEthernet3/2/3 is up, line protocol is up 
Hardware is FastEthernet SPA, address is 000e.d623.e840 (bia 000e.d623.e840)
Internet address is 33.1.0.2/16
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec, 
reliability 255/255, txload 59/255, rxload 83/255
Encapsulation ARPA, loopback not set
The following example shows that loopback is disabled for interface port 0 (the first port) on the Gigabit Ethernet SPA installed in the top (0) bay of the SIP that is located in slot 3 of the Cisco 7600 series router:
Router# show interfaces gigabitethernet 3/0/0
GigabitEthernet3/0/0 is up, line protocol is up 
  Hardware is GigMac 1 Port 10 GigabitEthernet, address is 0008.7db3.8dfe (bia )
  Internet address is 10.0.0.2/24
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec, rely 255/255, load 1/255
  Encapsulation ARPA, loopback not set
The following example shows that loopback is disabled for interface port 0 (the first port) on the Ten Gigabit Ethernet SPA installed in the top (0) bay of the SIP that is located in slot 7 of the Cisco 7600 series router:
Router# show interfaces tengigabitethernet 7/0/0
TenGigabitEthernet7/0/0 is up, line protocol is up (connected)
  Hardware is TenGigEther SPA, address is 0000.0c00.0102 (bia 000f.342f.c340)
  Internet address is 15.1.1.2/24
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
Using the Cisco IOS Event Tracer to Troubleshoot Problems

Note This feature is intended for use as a software diagnostic tool and should be configured only under the direction of a Cisco Technical Assistance Center (TAC) representative.

The Event Tracer feature provides a binary trace facility for troubleshooting Cisco IOS software. This feature gives Cisco service representatives additional insight into the operation of the Cisco IOS software and can be useful in helping to diagnose problems in the unlikely event of an operating system malfunction or, in the case of redundant systems, Route Processor switchover.

Event tracing works by reading informational messages from specific Cisco IOS software subsystem components that have been preprogrammed to work with event tracing, and by logging messages from those components into system memory. Trace messages stored in memory can be displayed on the screen or saved to a file for later analysis.

The SPAs currently support the "spa" component to trace SPA OIR-related events.

For more information about using the Event Tracer feature, refer to the following URL:

http://www.cisco.com/en/US/products/sw/iosswrel/ps1829/products_feature_guide09186a0080087164.html

Preparing for Online Insertion and Removal of a SPA

The Cisco 7600 series router supports online insertion and removal (OIR) of the SIP, in addition to each of the SPAs. Therefore, you can remove a SIP with its SPAs still intact, or you can remove a SPA independently from the SIP, leaving the SIP installed in the router.

This means that a SIP can remain installed in the router with one SPA remaining active, while you remove another SPA from one of the SIP subslots. If you are not planning to immediately replace a SPA into the SIP, then be sure to install a blank filler plate in the subslot. The SIP should always be fully installed with either functional SPAs or blank filler plates.

For more information about activating and deactivating SPAs in preparation for OIR, see the "Preparing for Online Insertion and Removal of SIPs and SPAs" topic in the "Troubleshooting a SIP" chapter in this guide.