Cisco 7500 Series Routers

Document ID: 15101

Contents

Introduction
Before You Begin
      Conventions
      Prerequisites
      Components Used
What Is An Output Stuck?
Packet Memory on the Route/Switch Processor (RSP)
How Does the Route/Switch Processor (RSP) Detect an "Output Stuck/Frozen"
Troubleshooting
Information to Collect if You Open a TAC Case
Related Information

Introduction

This document explains the causes of "%RSP-3-RESTART: interface [xxx] output stuck/frozen/not transmitting" error messages, and how to troubleshoot them.

Note: The information in this document is based on Cisco IOS® Software Release 11.1 and later.

Before You Begin

Conventions

For more information on document conventions, see the Cisco Technical Tips Conventions.

Prerequisites

There are no specific prerequisites for this document.

Components Used

This document is not restricted to specific software and hardware versions.

The information presented in this document was created from devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If you are working in a live network, ensure that you understand the potential impact of any command before using it.

What Is An Output Stuck?

If an Interface Processor (IP) or a Versatile Interface Processor (VIP) stops forwarding traffic for a period of time, an "output stuck/frozen" or "not transmitting" may occur. This means that the interface card is stuck and corrective action has to be taken to restart the interface card.

The following messages on the console indicate an output stuck/frozen:

• %RSP-3-RESTART: interface Serial4/0/1:12, output stuck
  !--- In Cisco IOS Software releases up to 12.1(5) 
  
  

• %RSP-3-RESTART: interface Serial4/0/1:12, not transmitting 
  !--- In Cisco IOS Software releases 12.1(5) and later 
  
  

• %RSP-3-RESTART: interface Serial4/0/1:11, output frozen 

• %RSP-3-RESTART: cbus complex

Packet Memory on the Route/Switch Processor (RSP)

The following notions are important in order to understand the causes of output stuck:

• Packet memory (MEMD) buffers - MEMD size is fixed at 2 MB on the RSP7000, RSP1, RSP2, and RSP4. On the RSP8, MEMD size is 8 MB. MEMD is distributed between all interfaces at bootup, when there is an online insertion and removal (OIR), a microcode reload, a maximum transmission unit (MTU) change, or a Cbus complex. You can check the status of MEMD buffers with the show controllers cbus command.

  When MEMD is allocated, the following structures are created:

  • A local free queue (lfreeq). This is assigned to each interface, and is used for packets received on this interface.

  • A global free queue (gfreeq). This is also allocated, and an interface can fall back to that queue within some limits.

  • A transmit queue (txqueue or TxQ). This is assigned to each interface, and is used for packets going out from this interface.

• The transmit accumulator (txacc) represents the number of elements on the output interface transmit queue (txqueue). When the transmit accumulator (txacc) = transmit limit (txlimit), all buffers are freed. When the transmit accumulator (txacc) = 0, the queue is full, and no more queueing is allowed.

Here's an example of show controller cbus command output:

Router#show controller cbus 
MEMD at 40000000, 8388608 bytes (unused 697376, recarves 6, lost 0) 
RawQ 48000100, ReturnQ 48000108, EventQ 48000110 
BufhdrQ 48000130 (21 items), LovltrQ 48000148 (15 items, 2016 bytes) 
IpcbufQ 48000158 (24 items, 4096 bytes) 
IpcbufQ_classic 48000150 (8 items, 4096 bytes) 
3570 buffer headers (48002000 - 4800FF10) 
pool0: 8 buffers, 256 bytes, queue 48000138 
pool1: 2940 buffers, 1536 bytes, queue 48000140 
pool2: 550 buffers, 4512 bytes, queue 48000160 
pool3: 4 buffers, 4544 bytes, queue 48000168 
slot2: VIP2, hw 2.11, sw 22.20, ccb 5800FF40, cmdq 48000090, vps 8192 
software loaded from system 
IOS (tm) VIP Software (SVIP-DW-M), Version 12.0(11)S3, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1) 
ROM Monitor version 122.0 
Mx Serial(4), HW Revision 0x3, FW Revision 1.45 
Serial2/0/0, applique is V.35 DCE 
received clockrate 2015232 
gfreeq 48000140, lfreeq 480001D0 (1536 bytes) 
rxlo 4, rxhi 336, rxcurr 16, maxrxcurr 293 
txq 48001A00, txacc 48001A02 (value 294), txlimit 294

How Does the Route/Switch Processor (RSP) Detect an "Output Stuck/Frozen"

One of the tasks that the RSP background process performs is to check whether or not an interface is stuck. To understand how the RSP checks this, you need to understand what happens when a packet is transmitted to the packet memory (MEMD) TxQ:

• Packets sent from the RSP process level go through the process level holdQ before attempting to get queued to the appropriate MEMD buffer TxQ. The output holdQ is the "output queue" displayed in the show interfaces command:

  output queue 2/40, 0 drops

  Here, two packets are sitting on the process level holdQ. Note: Unless your router is process-switching a lot of packets, or congestion is present, the number of packets present in the output queue is most likely very low (zero or one).

  Router#show interfaces ethernet 1/0 
     Ethernet1/0 is up, line protocol is up 
       Hardware is cxBus Ethernet, address is 0000.0c6f.3b20 (bia 0000.0c6f.3b20)    
       Internet address is 10.200.40.23/22 
       MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec, rely 255/255, load 1/255    
       Encapsulation ARPA, loopback not set 
       Keepalive set (10 sec) 
       ARP type: ARPA, ARP Timeout 04:00:00 
       Last input 00:00:01, output 00:00:03, output hang never 
       
  !-- Last Output Timestamp 
     
       Last clearing of "show interface" counters never 
       Queueing strategy: fifo 
       Output queue 2/40, 0 drops; input queue    0/75, 0 drops  
       
  !-- Process Level HoldQ 
  
       5 minute input rate 0 bits/sec, 0 packets/sec 
       5 minute output rate 0 bits/sec, 0 packets/sec 
          199558 packets input, 20862085 bytes, 0 no buffer 
          Received 167627 broadcasts, 0 runts, 0 giants, 0 throttles    
          0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored    
          0 input packets with dribble condition detected 
          36178 packets output, 3592819 bytes, 0 underruns 
          0 output errors, 4 collisions, 1 interface resets 
          0 babbles, 0 late collision, 0 deferred 
          0 lost carrier, 0 no carrier 
          0 output buffer failures, 0 output buffers swapped out 
    Router#

• The RSP detects an output stuck only when there is a packet in the process level holdQ. If there are no packets in the process level holdQ, no time check is done, and there is no output stuck, even if the TxQ (the transmit queue in MEMD) is full and nothing is moving out of the interface.

• If the TxAcc is a positive number (meaning you have space to queue a packet on the TxQ), the packet is transmitted and the last output time is recorded (see show interfaces above).

• Once the RSP sees a packet in the process level holdQ, it checks the last_output timestamp of the outbound interface to know the last time a packet was sent on this interface. The RSP compares the current time and the last_output time, and if "CurrentTime-Last_Output" is greater than 1 minute, the RSP considers it an "output stuck" and clears the interface. In fact, the RSP background process calls the reset function for this specific interface. The following message is displayed on the console:

  %RSP-3-RESTART: interface Serial4/0/1:12, output stuck
  !--- In Cisco IOS Software releases earlier than 12.1(5)
  
  

  or

  %RSP-3-RESTART: interface Serial4/0/1:12, not transmitting
  !--- in Cisco IOS Software release 12.(5)and later
  
  

• If the result of this time check is greater than two minutes, the RSP declares an "output frozen" and takes more drastic measures (in this case, a Cbus complex of the RSP). The following messages are displayed on the console:

  %RSP-3-RESTART: interface Serial4/0/1:11, output frozen
  %RSP-3-RESTART: cbus complex

  See What Causes a "%RSP-3-RESTART: cbus complex"? for more information about Cbus complexes.

Troubleshooting

The "output stuck" message indicates that the RSP has detected the problem and has corrected it automatically. This error mechanism is part of the system's error recovery features, and is not a problem in itself.

An output frozen has more impact on your system, because it triggers a Cbus complex. A Cbus complex cleans everything, so you need to collect show technical-support output and, if applicable, show controller vip [slot #] tech output when the problem occurs; otherwise, the information is lost.

Attempting to debug an "output stuck" issue can be quite challenging, so we have attempted to add new support functionalities which gather data automatically as issues occur. The new debug cbus output-stuck command has been introduced in Cisco IOS Software version 12.1(5), and helps to troubleshoot output stuck/frozen issues on the Cisco 7500 platform.

An output stuck may be a hardware or software issue. Upgrading to the latest Cisco IOS software version in your release train should fix all known output stuck-related bugs. You may also use the Bug Toolkit (registered customers only) tool to find out if output stuck bugs are present in the software release you are currently running. If the upgrade doesn't solve the problem and you have spare parts, try swapping the affected card.

Information to Collect if You Open a TAC Case

Related Information

• Performance Tuning Basics
• What Causes a "%RSP-3-RESTART: cbus complex"?
• Technical Support - Cisco Systems