Troubleshooting the Serial SPAs

This chapter describes techniques that you can use to troubleshoot the operation of your serial SPAs.

It includes the following sections:

The first section provides information about basic interface troubleshooting. If you are having a problem with your SPA, use the steps in the General Troubleshooting Information to begin your investigation of a possible interface configuration problem.

To perform more advanced troubleshooting, see the other sections in this chapter.

For more information about troubleshooting serial lines, see the Internetwork Troubleshooting Handbook at http:/​/​www.cisco.com/​en/​US/​docs/​internetworking/​troubleshooting/​guide/​tr1901.html.

General Troubleshooting Information

This section describes general information for troubleshooting SIPs and SPAs. It includes the following sections:

Interpreting Console Error Messages

To view the explanations and recommended actions for Cisco ASR 1000 Series Routers error messages, including messages related to Cisco ASR 1000 Series Routers SIPs and SPAs, refer to the System Messages for Cisco IOS XE document.

System error messages are organized in the documentation according to the particular system facility that produces the messages. The SIP and SPA error messages for serial SPAs use the following facility names:

  • Cisco ASR 1000 Series SIP—ASR1000_SIP
  • 2-Port and 4-Port Channelized T3 SPA—SPA_CHOC_DSX
  • 2-Port and 4-Port Clear Channel T3/E3 Serial SPA—SPA_T3E3

Using debug Commands


Caution

Because debugging output is assigned high priority in the CPU process, it can render the system unusable. For this reason, use debug commands only to troubleshoot specific problems or during troubleshooting sessions with Cisco technical support staff. Moreover, it is best to use debug commands during periods of lower network traffic and fewer users. Debugging during these periods decreases the likelihood that increased debug command processing overhead will affect system use.

Along with the other debug commands supported on the Cisco ASR 1000 Series Routers, you can obtain specific debug information for SPAs on the Cisco ASR 1000 Series Routers using the debug hw-module subslot privileged EXEC command.

The debug hw-module subslot command is intended for use by Cisco technical support personnel. For information about other debug commands supported on the Cisco ASR 1000 Series Routers, refer to the Cisco IOS Debug Command Reference and any related feature documents for your Cisco IOS software release.

Using show Commands

There are several show commands that you can use to monitor and troubleshoot the SIPs and SPAs on the Cisco ASR 1000 Series Routers. This chapter describes using the show interfaces and show controllers commands to perform troubleshooting of your SPA.

Performing Basic Interface Troubleshooting

You can perform most of the basic interface troubleshooting using the show interfaces serial command and examining several areas of the output to determine how the interface is operating.

The output of the show interfaces serial EXEC command displays information specific to serial interfaces.


Note
The output of the show interfaces serial command will vary depending on the type of serial SPA.

This section describes how to use the show interfaces serial command to diagnose serial line connectivity problems in a wide-area network (WAN) environment. The following sections describe some of the important fields of the command output:

Serial Lines: show interfaces serial Status Line Conditions

You can identify five possible problem states in the interface status line of the show interfaces serial display:

  • Serial x is down, line protocol is down

  • Serial x is up, line protocol is down

  • Serial x is up, line protocol is up (looped)

  • Serial x is up, line protocol is down (disabled)

  • Serial x is administratively down, line protocol is down

The following example shows the interface statistics on the first port of a 4-Port Clear Channel T3/E3 SPA installed in subslot 0 of the SIP located in chassis slot 5:

Router# show interfaces serial
Serial5/0/0 is up, line protocol is up 
  Hardware is SPA-4T3E3
  Internet address is 110.1.1.2/24
  MTU 4470 bytes, BW 44210 Kbit, DLY 200 usec, 
     reliability 255/255, txload 234/255, rxload 234/255
  Encapsulation HDLC, crc 16, loopback not set
  Keepalive set (10 sec)
  Last input 00:00:05, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 40685000 bits/sec, 115624 packets/sec
  5 minute output rate 40685000 bits/sec, 115627 packets/sec
     4653081241 packets input, 204735493724 bytes, 0 no buffer
     Received 4044 broadcasts (0 IP multicast)
     0 runts, 0 giants, 0 throttles
              0 parity
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     4652915555 packets output, 204728203520 bytes, 0 underruns
     0 output errors, 0 applique, 4 interface resets
     0 output buffer failures, 0 output buffers swapped out
     2 carrier transitions

The following table shows the interface status conditions, possible problems associated with the conditions, and solutions to those problems.

Table 1 Serial Lines: show interfaces serial Status Line Conditions

Status Line Condition

Possible Problem

Solution

Serial x is up, line protocol is up

—

This is the proper status line condition. No action is required.

Serial x is down, line protocol is down

The router is not sensing a carrier detect (CD) signal (that is, the CD is not active).

The line is down or is not connected on the far end.

Cabling is faulty or incorrect.

Hardware failure has occurred in the channel service unit/data service unit (CSU/DSU).

  1. Check the CD LEDs to see whether the CD is active, or insert a breakout box on the line to check for the CD signal.

  2. Verify that you are using the proper cable (see your hardware installation documentation).

  3. Insert a breakout box and check all control leads.

  4. Contact your leased-line or other carrier service to see whether there is a problem.

  5. Swap faulty parts.

  6. If you suspect faulty router hardware, change the serial line to another port. If the connection comes up, the previously connected interface has a problem.

Serial x is up, line protocol is down

A local or remote router is misconfigured.

Keepalives are not being sent by the remote router.

A leased-line or other carrier service problem has occurred (noisy line or misconfigured or failed switch).

A timing problem has occurred on the cable.

A local or remote CSU/DSU has failed.

Router hardware (local or remote) has failed.

  1. Put the line in local loopback mode and use the show interfaces serial command to determine whether the line protocol comes up.

    Note   

    If the line protocol comes up, a failed remote device is the likely problem.This solution will only work with High-Level Data Link Control (HDLC) encapsulation. For Frame Relay (FR) and Point-to-Point Protocol (PPP) encapsulation, a looped interface will always have the line protocol down. In addition, you may need to change the encapsulation to HDLC to debug this issue.

  2. If the problem appears to be on the remote end, repeat Step 1 on the remote interface.

  3. Verify all cabling. Make certain that the cable is attached to the correct interface, the correct CSU/DSU, and the correct remote termination point.

  4. Enable the debug serial interface EXEC command.

  5. If the line protocol does not come up in local loopback mode, and if the output of the debug serial interface EXEC command shows that the keepalive counter is not incrementing, a router hardware problem is likely. Swap router interface hardware.

  6. If the line protocol comes up and the keepalive counter increments, the problem is not in the local router.

  7. If you suspect faulty router hardware, change the serial line to an unused port. If the connection comes up, the previously connected interface has a problem.

Note   

First enable per-interface debugging using the command debug interface serial x, and depending on the encapsulation, enable the corresponding debug:

For HDLC: debug serial interface For PPP: debug ppp negotiation For FR: debug frame-relay lmi

Caution   

Because debugging output is assigned high priority in the CPU process, it can render the system unusable. For this reason, use debug commands only to troubleshoot specific problems or during troubleshooting sessions with Cisco technical support staff. Moreover, it is best to use debug commands during periods of lower network traffic and fewer users. Debugging during these periods decreases the likelihood that increased debug command processing overhead will affect system use.

Serial x is up, line protocol is up (looped)

A loop exists in the circuit. The sequence number in the keepalive packet changes to a random number when a loop is initially detected. If the same random number is returned over the link, a loop exists.

  1. Use the show running-config privileged EXEC command to look for any loopback interface configuration command entries.

  2. If you find a loopback interface configuration command entry, use the no loopback interface configuration command to remove the loop.

  3. If you do not find the loopback interface configuration command, examine the CSU/DSU to determine whether they are configured in manual loopback mode. If they are, disable manual loopback.

  4. Reset the CSU or DSU, and inspect the line status. If the line protocol comes up, no other action is needed.

  5. If the CSU or DSU is not configured in manual loopback mode, contact the leased-line or other carrier service for line troubleshooting assistance.

Serial x is up, line protocol is down (disabled)

A high error rate has occurred due to a remote device problem.

A CSU or DSU hardware problem has occurred.

Router hardware (interface) is bad.

  1. Troubleshoot the line with a serial analyzer and breakout box. Examine the output of show controller T1 or show controller T3 or show controller serial x depending on whether the SPA is an 8-Port Channelized T1/E1 Serial SPA, 4-Port Channelized T3 SPA, or 4-Port T3/E3 Serial SPA.

  2. Loop CSU/DSU (DTE loop). If the problem continues, it is likely that there is a hardware problem. If the problem does not continue, it is likely that there is a telephone company problem.

  3. Swap out bad hardware, as required (CSU, DSU, switch, local or remote router).

Serial x is administratively down, line protocol is down

The router configuration includes the shutdown interface configuration command.

A duplicate IP address exists.

  1. Check the router configuration for the shutdown command.

  2. Use the no shutdown interface configuration command to remove the shutdown command.

  3. Verify that there are no identical IP addresses using the show running-config privileged EXEC command or the show interfaces EXEC command.

  4. If there are duplicate addresses, resolve the conflict by changing one of the IP addresses.

Serial Lines: Increasing Output Drops on Serial Link

Output drops appear in the output of the show interfaces serial command when the system is attempting to hand off a packet to a transmit buffer but no buffers are available.

Symptom: Increasing output drops on serial link

The possible problem that might cause this symptom could be that input rate to serial interface exceeds bandwidth available on serial link. The solution to this problem is as follows:

  1. Minimize periodic broadcast traffic, such as routing and Service Advertising Protocol (SAP) updates, by using access lists or by other means. For example, to increase the delay between SAP updates, use the ipx sap-interval interface configuration command.

  2. Increase the output hold queue size in small increments (for instance, 25 percent), using the hold-queue out interface configuration command.

  3. Implement priority queuing on slower serial links by configuring priority lists. For information on configuring priority lists, see the Cisco IOS configuration guides and command references for your Cisco IOS software release.


Note

Output drops are acceptable under certain conditions. For instance, if a link is known to be overused (with no way to remedy the situation), it is often considered more preferable to drop packets than to hold them. This is true for protocols that support flow control and can retransmit data (such as TCP/IP and Novell Internetwork Packet Exchange [IPX]). However, some protocols, such as DECnet and local-area transport, are sensitive to dropped packets and accommodate retransmission poorly, if at all.

Serial Lines: Increasing Input Drops on Serial Link

Input drops appear in the output of the show interfaces serial EXEC command when too many packets from that interface are still being processed in the system.

Symptom: Increasing number of input drops on serial link

The possible problem that might cause this symptom could be input rate exceeds the capacity of the router, or input queues exceed the size of output queues. The solution to this problem is as follows:

  1. Increase the output queue size on common destination interfaces for the interface that is dropping packets. Use the hold-queue number out interface configuration command. Increase these queues by small increments (for instance, 25 percent) until you no longer see drops in the show interfaces command output. The default output hold queue limit is 40 packets.

  2. Reduce the input queue size, using the hold-queue number in interface configuration command, to force input drops to become output drops. Output drops have less impact on the performance of the router than do input drops. The default input hold queue is 75 packets.

Serial Lines: Increasing Input Errors in Excess of 1 Per Cent of Total Interface Traffic

If input errors appear in the show interfaces serial command output, there are several possible sources of those errors. The most likely sources, along with possible solutions, are summarized in the following table.


Note

Any input error value for cyclic redundancy check (CRC) errors, framing errors, or aborts above 1 percent of the total interface traffic suggests some kind of link problem that should be isolated and repaired.

Symptom: Increasing number of input errors in excess of 1 percent of total interface traffic

Table 2 Serial Lines: Increasing Input Errors in Excess of 1 Percent of Total Interface Traffic

Possible Problem

Solution

The following problems can result in this symptom:

  • Faulty telephone company equipment

  • Noisy serial line

  • Incorrect clocking configuration

  • Incorrect cable or cable that is too long

  • Bad cable or connection

  • Bad CSU or DSU

  • Bad router hardware

  • Data converter or other device being used between router and DSU
Note   

We strongly recommend against the use of data converters when you are connecting a router to a WAN or a serial network.

  1. Use a serial analyzer to isolate the source of the input errors. If you detect errors, there likely is a hardware problem or a clock mismatch in a device that is external to the router.

  2. Use the loopback and ping tests to isolate the specific problem source.

  3. Look for patterns. For example, if errors occur at a consistent interval, they could be related to a periodic function, such as the sending of routing updates.

Serial Lines: Troubleshooting Serial Line Input Errors

The following table describes the various types of input errors displayed by the show interfaces serial command, possible problems that might be causing the errors, and solutions to those problems.

Table 3 Serial Lines: Troubleshooting Serial Line Input Errors

Input Error Type(Field Name)

Possible Problem

Solution

CRC errors (CRC)

CRC errors occur when the CRC calculation does not pass (indicating that data is corrupted) for one of the following reasons:

  • The serial line is noisy.

  • The serial cable is too long, or the cable from the CSU/DSU to the router is not shielded.

  • Serial clock transmit external (SCTE) mode is not enabled on the DSU.

  • The CSU line clock is incorrectly configured.

  • A ones density problem has occurred on the T1 link (incorrect framing or coding specification).

  1. Ensure that the line is clean enough for transmission requirements. Shield the cable, if necessary.

  2. Make sure that the cable is within the recommended length (no more than 50 feet [15.24 meters], or 25 feet [7.62 meters] for a T1 link).

  3. Ensure that all devices are properly configured for a common line clock. Set SCTE on the local and remote DSU. If your CSU/DSU does not support SCTE, see the to invert the transmit clock signal.

  4. Make certain that the local and remote CSU/DSU are configured for the same framing and coding scheme as that used by the leased-line or other carrier service (for example, Extended Superframe Format/binary eight-zero substitution [ESF/B8ZS]).

  5. Contact your leased-line or other carrier service, and have it perform integrity tests on the line.

Framing errors (frame)

 A framing error occurs when a packet does not end on an 8-bit byte boundary for one of the following reasons:

  • The serial line is noisy.

  • The cable is improperly designed, the serial cable is too long, or the cable from the CSU or DSU to the router is not shielded.

  • SCTE mode is not enabled on the DSU, the CSU line clock is incorrectly configured, or one of the clocks is configured for local clocking.

  • A ones density problem has occurred on the T1 link (incorrect framing or coding specification).

  1. Ensure that the line is clean enough for transmission requirements. Shield the cable, if necessary. Make certain that you are using the correct cable.

  2. Make sure that the cable is within the recommended length (no more than 50 feet [15.24 meters], or 25 feet [7.62 meters] for a T1 link).

  3. Ensure that all devices are properly configured to use a common line clock. Set SCTE on the local and remote DSU.

  4. Make certain that the local and remote CSU/DSU are configured for the same framing and coding scheme as that used by the leased-line or other carrier service (for example, ESF/B8ZS).

  5. Contact your leased-line or other carrier service, and have it perform integrity tests on the line.

Aborted transmission (abort)

Aborts indicate an illegal sequence of 1 bit (more than seven in a row).

The following are possible reasons for this to occur:

  • SCTE mode is not enabled on the DSU.
  • The CSU line clock is incorrectly configured.
  • The serial cable is too long, or the cable from the CSU or DSU to the router is not shielded.
  • A ones density problem has occurred on the T1 link (incorrect framing or coding specification).
  • A packet was terminated in the middle of transmission (typical cause is an interface reset or a framing error or a buffer overrun).
  • A hardware problem has occurred (bad circuit, bad CSU/DSU, or bad sending interface on remote router).
  1. Ensure that all devices are properly configured to use a common line clock. Set SCTE on the local and remote DSU.
  2. Shield the cable, if necessary. Make certain that the cable is within the recommended length (no more than 50 feet [15.24 meters], or 25 feet [7.62 meters] for a T1 link). Ensure that all connections are good.
  3. Check the hardware at both ends of the link. Swap faulty equipment, as necessary.
  4. Lower data rates and determine whether aborts decrease.
  5. Use local and remote loopback tests to determine where aborts are occurring.
  6. Contact your leased-line or other carrier service, and have it perform integrity tests on the line.

Serial Lines: Increasing Interface Resets on Serial Link

Interface resets that appear in the output of the show interfaces serial EXEC command are the result of missed keepalive packets.

Symptom: Increasing interface resets on serial link

The following table outlines the possible problems that might cause this symptom and describes solutions to those problems.

Table 4 Serial Lines: Increasing Interface Resets on Serial Link

Possible Problem

Solution

The following problems can result in this symptom:

  • Congestion on link (typically associated with output drops)
  • Bad line causing CD transitions
  • Possible hardware problem at the CSU, DSU, or switch

When interface resets are occurring, examine other fields of the show interfaces serial command output to determine the source of the problem. Assuming that an increase in interface resets is being recorded, examine the following fields:

  1. Check the Carrier Transitions field in the show interfaces serial command display. If carrier transitions are high while interface resets are being registered, the problem is likely to be a bad link or a bad CSU or DSU. Contact your leased-line or carrier service, and swap faulty equipment, as necessary.
  2. Examine the Input Errors field in the show interfaces serial command display. If input errors are high while interface resets are increasing, the problem is probably a bad link or a bad CSU/DSU. Contact your leased-line or other carrier service, and swap faulty equipment, as necessary.

If there is a high number of output drops in the show interfaces serial output, see the Serial Lines: Increasing Output Drops on Serial Link.

Serial Lines: Increasing Carrier Transitions Count on Serial Link

Carrier transitions appear in the output of the show interfaces serial EXEC command whenever there is an interruption in the carrier signal (such as an interface reset at the remote end of a link).

Symptom: Increasing carrier transitions count on serial link

The following table outlines the possible problems that might cause this symptom and describes solutions to those problems.

Table 5 Serial Lines: Increasing Carrier Transitions Count on Serial Link

Possible Problem

Solution

The following problems can result in this symptom:

  • Line interruptions due to an external source (such as physical separation of cabling, red or yellow T1 alarms, or lightning striking somewhere along the network)
  • Faulty switch, DSU, or router hardware
  1. Check hardware at both ends of the link (attach a breakout box or a serial analyzer, and test to determine the source of problems).
  2. If an analyzer or breakout box is incapable of identifying any external problems, check the router hardware.
  3. Swap faulty equipment, as necessary.

Using Bit Error Rate Tests

BERT (Bit-Error Rate Testing) is used for analyzing quality and for problem resolution of digital transmission equipments. BERT tests the quality of an interface by directly comparing a pseudorandom or repetitive test pattern with an identical locally generated test pattern.

The BERT operation is data-intensive. Regular data cannot flow on the path while the test is in progress. The path is reported to be in alarm state when BERT is in progress and restored to a normal state after BERT has been terminated.

BERT is built into most of the serial SPAs. With BER tests, you can test cables and signal problems in the field.

  • For the 2-Port and 4-Port Channelized T3 SPA, you can configure individual T1 lines to run BER tests, but only one BER test circuit exists for all 28 T1 lines. Hence, only one BER test can be run on a single T3 port at any given time.
  • For the 8-port Channelized T1/E1 SPA, there are six framer-assisted BERT engines, and six BER tests may run concurrently.
  • For the 1-Port Channelized OC-3/STM-1 SPA and 1-Port Channelized OC-12/STM-4 SPA, you can run a maximum of 27 concurrent tests across all paths on the SPA.

Note
BERT operation is data-intensive. Regular data cannot flow on the path while the test is in progress. The path is reported to be in alarm state when BERT is in progress and restored to a normal state after BERT has been terminated.

There are two categories of test patterns that can be generated by the onboard BER test circuitry: pseudorandom and repetitive. Pseudorandom test patterns are exponential numbers and conform to the CCITT/ITU O.151 and O.153 specifications; repetitive test patterns are all zeros, all ones, or alternating zeros and ones.

A description of the test patterns follows:

  • Pseudorandom test patterns:
    • 2^15 (per CCITT/ITU O.151)
    • 2^20 (per CCITT/ITU O.153)
    • 2^23 (per CCITT/ITU O.151)
    • QRSS (quasi-random signal sequence) (per CCIT/ITU O.151)
  • Repetitive test patterns:
    • All zeros (0s)
    • All ones (1s)
    • Alternating zeros (0s) and ones (1s)

The following table provides information about some additional BERT patterns supported by some of the channelized SPAs on the Cisco ASR 1000 Series Aggregation Services Routers:

Table 6 Additional BERT Pattern Support on the Channelized SPAs

Pattern

1-Port Channelized OC-3/STM-1 SPA

1-Port Channelized OC-12/STM-4 SPA

2-Port and 4-Port Channelized T3 SPA

1-in-8

Yes

Yes

Yes

3-in-24

Yes

Yes

Yes

2^9

Yes

Yes

Yes

2^11

Yes

Yes

Yes

2^15-inverted

Yes

Yes

Yes

2^20-QRSS

Yes

Yes

Yes

2^23-inverted

Yes

Yes

Yes

55Octet

Yes

No

Yes

55Daly

Yes

No

Yes

DS0-1, DS0-2, DS0-3, and DS0-4

Yes

No

Yes

Single Bit Error Injection

No

Yes

No

Both the total number of error bits received and the total number of bits received are available for analysis. You can set the testing period from 1 minute to 14,400 minutes (240 hours), and you can also retrieve the error statistics anytime during the BER test.

When running a BER test, your system expects to receive the same pattern that it is transmitting. To help ensure this:

  • Use a loopback at a location of your choice in the link or network. To see how to configure a loopback, go to the Using loopback Commands.
  • Configure remote testing equipment to transmit the same BER test pattern at the same time.

Configuring a BER Test

To send a BER test pattern on an interface, use the bert pattern command. For more information, see the Configuring the 1-Port Channelized OC-3/STM-1 SPA and 1-Port Channelized OC-12/STM-4 SPA chapter.

Viewing a BER Test

You can view the results of a BER test with the show controllers command.

You can view the results of a BER test at the following times:

  • After you terminate the test using the no bert command.
  • After the test runs completely.
  • Anytime during the test (in real time).
Router# show controllers serial T3 1/0/0
T3 1/0/0 is up.
C2T3 H/W Version: 3, C2T3 ROM Version: 0.79, C2T3 F/W Version: 0.29.0
T3 1/0/0 T1 1
No alarms detected.
Clock Source is internal.
BERT test result (running)
   Test Pattern: 2^15, Status: Sync, Sync Detected: 1
   Interval: 5 minute(s), Time Remain: 5 minute(s)
   Bit Errors (Since BERT Started): 6 bits,
   Bits Received (Since BERT start): 8113 Kbits
   Bit Errors (Since last sync): 6 bits
   Bits Received (Since last sync): 8113 Kbits

Interpreting BER Test Results

The following table explains the output of the show controllers command.

Table 7 Interpreting BER Test Results

Field

Description

BERT test result (running)

Indicates the current state of the test. In this case, “running” indicates that the BER test is still in progress. After a test is completed, “done” is displayed.

Test Pattern: 2^15, Status: Sync, Sync Detected: 1

Indicates the test pattern you selected for the test (2^15), the current synchronization state (Sync), and the number of times synchronization has been detected during this test (1).

Interval: 5 minute(s), Time Remain: 5 minute(s)

Indicates the time the test takes to run and the time remaining for the test to run.

If you terminate a BER test, you receive a message similar to the following:

Interval: 5 minute(s), Time Remain: 2 minute(s) (unable to complete)

“Interval: 5 minutes” indicates the configured run time for the test. “Time Remain: 2 minutes” indicates the time remaining in the test prior to termination. “(unable to complete)” signifies that you interrupted the test.

Bit Errors (Since BERT Started): 6 bits

Bits Received (Since BERT start): 8113 Kbits

Bit Errors (Since last sync): 6 bits

Bits Received (Since last sync): 8113 Kbits

These four lines show the bit errors that have been detected versus the total number of test bits that have been received since the test started and since the last synchronization was detected.

Using loopback Commands

Loopback support is useful for testing the interface without connectivity to the network, or for diagnosing equipment malfunctions between the interface and a device. The 2-Port and 4-Port T3/E3 Serial SPA supports both an internal and an external loopback mode. The external loopback mode requires the use of a loopback cable and implements a loopback through the transceiver on the SPA.

You can also configure an internal loopback without the use of a loopback cable that implements a loopback at the PHY device internally. By default, loopback is disabled.

To configure loopback, use the following commands:

Command

Purpose

Router# configure terminal

Enters global configuration mode.

T3/E3

Router(config)# interface serial slot/subslot/port

T1/E1

Router(config)# controller slot/subslot/port

Selects the interface to configure and enters interface configuration mode.

  • slot/subslot/port—Specifies the location of the T3/E3 interface.
  • slot/subslot/port—Specifies the location of the T1/E1 controller.

T3/E3

Router(config-if)# loopback {local | dte | network {line | payload} | remote}

T1/E1

Router(config-controller)# loopback {local [line | payload] | diag}

Specifies the loopback mode.

  • local—Loops back after going through the framer toward the terminal.
  • dte—Loops back after the LIU towards the terminal.
  • network—Loops back towards the network.
  • remote—Sends FEAC to set remote in loopback.
  • line—Loops back toward network before going through framer.
  • payload—Loops back toward network after going through framer.
  • diag—Loops back after going through the HDLC controller towards the terminal.

Verifying Loopback Mode

The following example shows that local loopback is set:

Router# show interfaces serial 2/0/0
Serial2/0/0 is up, line protocol is up (looped)
  Hardware is SPA-4T3E3
  MTU 4470 bytes, BW 44210 Kbit, DLY 200 usec, 
     reliability 255/255, txload 225/255, rxload 221/255
  Encapsulation FRAME-RELAY, crc 16, loopback set (local)
  Keepalive set (10 sec)
  LMI enq sent  13281, LMI stat recvd 13280, LMI upd recvd 0, DTE LMI up
  LMI enq recvd 1, LMI stat sent  0, LMI upd sent  0
  LMI DLCI 1023  LMI type is CISCO  frame relay DTE
  FR SVC disabled, LAPF state down
  Broadcast queue 0/256, broadcasts sent/dropped 0/0, interface broadcasts 0
  Last input 00:00:07, output 00:00:00, output hang never
  Last clearing of "show interface" counters 1d12h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 612756
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 38446000 bits/sec, 109217 packets/sec
  5 minute output rate 39023000 bits/sec, 110854 packets/sec
     14601577951 packets input, 642478074437 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicast)
     0 runts, 0 giants, 0 throttles
              0 parity
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     14545044296 packets output, 639982568049 bytes, 0 underruns
     0 output errors, 0 applique, 1 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
   rxLOS inactive, rxLOF inactive, rxAIS inactive
   txAIS inactive, rxRAI inactive, txRAI inactive

Using the Cisco IOS Event Tracer to Troubleshoot Problems

This feature is intended for use as a software diagnostic tool and should be configured only under the direction of a Cisco Technical Assistance Center (TAC) representative.

The Event Tracer feature provides a binary trace facility for troubleshooting Cisco IOS software. This feature gives Cisco service representatives additional insight into the operation of the Cisco IOS software and can be useful in helping to diagnose problems in the unlikely event of an operating system malfunction or, in the case of redundant systems, Route Processor switchover.

Event tracing works by reading informational messages from specific Cisco IOS software subsystem components that have been preprogrammed to work with event tracing, and by logging messages from those components into system memory. Trace messages stored in memory can be displayed on the screen or saved to a file for later analysis.

The SPAs currently support the “spa” component to trace SPA OIR-related events.

For more information about using the Event Tracer feature, refer to the following URL:

http:/​/​www.cisco.com/​en/​US/​docs/​ios/​12_0s/​feature/​guide/​evnttrcr.html

Preparing for Online Insertion and Removal of a SPA

The Cisco ASR 1000 Series Aggregation Services Routers support online insertion and removal (OIR) of the SIP, in addition to each of the SPAs. Therefore, you can remove a SIP with its SPAs still intact, or you can remove a SPA independently from the SIP, leaving the SIP installed in the router.

This means that a SIP can remain installed in the router with one SPA remaining active, while you remove another SPA from one of the SIP subslots. If you are not planning to immediately replace a SPA into the SIP, then be sure to install a blank filler plate in the subslot. The SIP should always be fully installed with either functional SPAs or blank filler plates.

For more information about activating and deactivating SPAs in preparation for OIR, see the Troubshooting the SIP chapter.