Troubleshooting "Ignored" Errors on an ATM Port Adapter

Introduction

This document explains the conditions under which a router increments the ignored error in show interface atm command output. It also explains how to troubleshoot this problem.

When a cell arrives, it is first stored in the framer's cell first-in first-out (FIFO). Then, it moves to ATM segmentation and reassembly's (SAR's) cell buffer and is reassembled using the host packet buffer across the Peripheral Component Interconnect (PCI) bus. After a packet is completed, the host driver is informed and processes it. While aborts and overruns point to lack of receive FIFO buffers at the framing and SAR chips respectively, ignored errors indicate a lack of packet memory buffers. Typically, ignored errors occur when one or more relatively slow output interfaces hold all of the input buffers allocated to the PA-A3. ignored errors also increment when the CPU experiences very high utilization and does not have any available cycles during which to replenish an interface's receive ring of packet buffers.

This sample output of the show interface ATM command was captured on a Cisco 7500 series router with a PA-A3 port adapter, which also is known as the Enhanced ATM PA:

router#show interface atm1/0/0
ATM1/0/0 is up, line protocol is up
       Hardware is cyBus ENHANCED ATM PA
   MTU 4470 bytes, sub MTU 4470, BW 44209 Kbit, DLY 190 usec,
   rely 255/255, load 1/255
   Encapsulation ATM, loopback not set, keepalive not set
   Encapsulation(s): AAL5 AAL3/4
   4096 maximum active VCs, 1 current VCCs
   VC idle disconnect time: 300 seconds
   Last input never, output 00:03:14, output hang never
   Last clearing of "show interface" counters never
   Queueing strategy: fifo
   Output queue 0/40, 0 drops; input queue 0/75, 0 drops
   5 minute input rate 0 bits/sec, 0 packets/sec
   5 minute output rate 0 bits/sec, 0 packets/sec
      588 packets input, 7430 bytes, 0 no buffer
      Received 0 broadcasts, 0 runts, 0 giants
      0 input errors, 0 CRC, 0 frame, 0 overrun, 45 ignored, 0 abort
      5 packets output, 560 bytes, 0 underruns
      0 output errors, 0 collisions, 0 interface resets
      0 output buffers copied, 0 interrupts, 0 failures

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on the Cisco 7200 and 7500 Series routers.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

For more information on document conventions, refer to the Cisco Technical Tips Conventions.

Packet Buffering on 7200 Series Routers

On Cisco 7200 series routers, the transmit buffers are derived from the receive buffers owned by other port adapters. This design also applies to 7500 series routers when the packet is locally switched to another PA on the same Versatile Interface Processor (VIP).

Cisco IOS® Software limits the number of receive host buffers per interface. Originally, the PA-A3 was allocated 400 host buffers when used with an NPE-150 having 1 MB of SRAM. Use the show controller atm command to display the number of interface host buffers.

7200#show controller atm 3/0
Interface ATM3/0 is up
      Hardware is ENHANCED ATM PA - DS3 (45Mbps
) Lane client mac address is 0030.7b1e.9054
 Framer is PMC PM7345 S/UNI-PDH, SAR is LSI ATMIZER II
 Firmware rev: G119, Framer rev: 1, ATMIZER II rev: 3
   idb=0x61499630, ds=0x6149E9C0, vc=0x614BE940
   slot 3, unit 2, subunit 0, fci_type 0x005B, ticks 73495
   400 rx buffers: size=512, encap=64, trailer=28, magic=4
 Curr Stats:
   rx_cell_lost=0, rx_no_buffer=0, rx_crc_10=0
   rx_cell_len=0, rx_no_vcd=0, rx_cell_throttle=0, tx_aci_err=0
  [output omitted]

The 7200 series generally stores packets in private particle buffers, which are allocated from the I/O memory pool. Private particle buffers minimize contention for buffer resources. The private pools are static and are allocated with a fixed number of buffers at Cisco IOS software initialization. New buffers cannot be created on demand for these pools. PA-A3 is considered to be High Bandwidth Port Adapter for 7200. Please refer to Cisco 7200 Series Port Adapter Hardware Configuration Guidelines when you install the module.

If the packet cannot be processed completely at interrupt time, the interface driver "coalesces" the packet into a contiguous buffer in DRAM in the public pool and prepares the packet for process switching.

The show buffers command displays both the public and the private pools.

7200#show buffers
Buffer elements: 
     499 in free list (500 max allowed) 
     886005 hits, 0 misses, 0 created 
[output omitted] 
Private particle pools: 
ATM2/0 buffers, 512 bytes (total 400, permanent 400):
     0 in free list (0 min, 400 max allowed)
      400 hits, 0 fallbacks
      400 max cache size, 0 in cache
      14 buffer threshold, 1 threshold transitions

In addition to public and private interface pools, Cisco IOS also creates special buffer control structures, called rings, in I/O memory. IOS and interface drivers use these rings to control which buffers are used to receive and transmit packets to the media. Rings are actually a common control structure used by many types of media controllers to manage the memory for packets being received or waiting to be transmitted. The rings themselves are a circular linked list of pointers to particles. IOS creates these rings on behalf of the media controllers and then manages them jointly with the interface drivers.

Each interface has a pair of rings:

• A receive ring for receiving packets.

• A transmit ring for transmitting packets.

These rings have fixed sizes determined by several factors, and in some case by user configuration.

On the 7200 series platform, the transmit ring packet buffers come from the receive ring of the originating interface for a switched packet or from a public pool if the packet was originated by IOS. They are deallocated from the transmit ring and returned to their original pool after the payload data is transmitted.

Packet Buffering on the VIP

On Cisco 7500 series routers, incoming packets can be switched locally at the VIP or switched by the Route Switch Processor (RSP). The VIP stores packets in particles in PCI memory in SRAM. The amount of PCI memory varies with the model of VIP. For example, a VIP with 512 kB PCI memory can support a single PA-A3 with a few virtual circuits (VCs) which have occasional bursts. More PCI memory enables the PA-A3 to sustain longer bursts of packets. Refer to Versatile Interface Processor configuration guides for more information.

Root Cause of Ignored Errors

The ignored errors indicate that the PA-A3 is running out of interface host packet buffers. These buffers are displayed in the output of the show buffers command.

Typically, the PA-A3 runs out of interface host buffers when it feeds a relatively slower egress interface or VC. In this configuration, the PA-A3 can overload the egress interface through a mismatch of interface rates. Since the slower-speed egress interface cannot return buffers as fast as the PA-A3 is dequeueing them to the output hold queue, a delay in buffer return causes the ingress PA-A3 to run out of input buffers. When this condition occurs, the PA-A3's ignored counter increments. This problem is more visible on older Network Processing Engines (NPEs) like the NPE-150.

In other words, a slower egress interface slows the rate at which the receive credits of the ingress ATM interface are replenished. This packet flow breaks the assumption that the outbound interface returns the buffer at the rate of buffer management time.

However, the ignored counter can increment even when there are available host buffers. Such errors increment when the port adapter's driver begins to throttle one or more VCs and stops accepting new packets. The purpose of such drops is to prevent one "aggressive" VC from allocating too many packet buffers and ultimately starving the other VCs of buffer resources.

Use the show controllers atm command to determine whether the throttling condition is active. The show controllers atm displays two important values:

• rx_ threshold—Defines the maximum number of receive particles that the PA-A3 can hold after which the microcode begins to regulate receive particle usage among VCs.

• rx_count—Defines the current number of receive particles held by the PA-A3 driver.

In this sample output of show controllers atm, rx_threshold equals rx_count.

Control data:
  rx_max_spins=73, max_tx_count=35, tx_count=8
   rx_threshold=1600, rx_count=1600, tx_threshold=4608
   tx bfd write indx=0x349, rx_pool_info=0x609EE860  

When the rx_count crosses the rx_threshold, the next packet received by PA-A3 is checked to see if one VC occupies too many packet buffers. If so, the PA-A3 discards this incoming packet until the total number of receive particles held by this violating VC fall below its quota. By default, a VC is assigned a per-VC receive credit limit derived from supporting a burst up to 10 ms for 64-byte packets or at least up to two MTU-sized packets. The receive limit also is sized to support the maximum burst size (MBS) worth of cells from the transmit side. Use the show atm pvc command to view the computed value.

Troubleshoot

When troubleshooting incrementing ignored errors, please gather this information before contacting Cisco Technical Support:

• What other interfaces reside in the router? Are these interfaces high-speed or low-speed interfaces?

• What type of network processing engine or network services engine is installed?

• What is the pattern of traffic through the interfaces? Does a fast ATM interface feed a slow egress interface?

• What is the amount of traffic (and the number of packets) being presented to the router when the ignored counter increments?

• Are other input error counters incrementing in addition to the ignored counter?

• Determine whether your router has sufficient memory to support the configured port adapters and features. Execute these commands several times and look for a pattern:

  • show process cpu

  • show memory summary

Workarounds

Ignored errors can be resolved by tuning values on either the egress interface or VC, or the ingress PA-A3 interface.

This list describes tuning techniques for the egress interface:

• Limit the number of buffers that can be held by the egress interfaces by using FIFO queueing with limited queue sizes. Refer to this example using a serial interface:

  interface Serial1/0
    no fair-queue
     hold-queue x out  

  By default, the output hold queue is set to 40. If you increased the hold queue, reset the value to 40, but avoid configuring a value lower than the default. To do basic buffer tuning, refer to Buffer tuning for all Cisco Routers.

• Use the tx-ring-limit command to reduce the size of the transmit ring on the outbound interface. The size of the transmit-ring needs to be small enough to avoid introducing latency due to queueing, and it needs to be large enough to avoid drops and a resulting impact to the TCP based flow. Please refer to Understanding and Tuning the tx-ring-limit Value for guidelines.

• If the PA-A3 is feeding an egress VC, increase the egress VC's cell rates. Please refer to Troubleshooting Input and Output Errors on PA-A3 ATM Port Adapters for a scenario and guidelines.

• Implement output ATM traffic shaping to reduce the amount of buffering on the egress interface. Please refer to Understand the VBR-nrt Service Category and ATM Traffic Shaping for ATM VCs.

• If the ignore counters increment due to high CPU utilization, refer to Troubleshooting High CPU Utilization on the Cisco Router.

• Consider an upgrade to PA-A6 which provides performance improvements over the PA-A3 ATM port adapter. The PA-A6 provides line rate performance using 128-byte packet sizes on the Cisco 7200 series routers using the NPE-400 and NSE-1 processing engines.

Solution 1: Tune the Receive Limit (rx-limit)

Cisco bug ID CSCdp96197 (registered customers only) introduced the ability to tune the receive ring limit with the rx-limit command. This technique is effective only when the problem is caused by one or few VCs violating their individual buffer allocation.

The rx-limit command allows you to specify the percent of total receive buffers allocated to a particular VC. A larger rx-limit value effectively allows the VC (acting as the input logical interface) to store more buffers waiting in the egress queues of slower interfaces or slower egress VCs.

Verify your configuration with the show atm vc <vcd> or show atm pvc <vpi>/<vci> command, as illustrated here:

7200#show atm pvc 1/100
 ATM1/0.1: VCD: 14, VPI: 1, VCI: 100 
UBR, PeakRate: 149760 
AAL5-LLC/SNAP, etype:0x0, Flags: 0xC20, VCmode: 0x0 
OAM frequency: 0 second(s), OAM retry frequency: 1 second(s), 
OAM retry frequency: 1 second(s) 
OAM up retry count: 3, OAM down retry count: 5 
OAM Loopback status: OAM Disabled 
OAM VC state: Not Managed 
ILMI VC state: Not Managed 
Rx Limit: 25 percent
InARP frequency: 15 minutes(s) 
Transmit priority 4 
InPkts: 0, OutPkts: 0, InBytes: 0, OutBytes: 0 
InPRoc: 0, OutPRoc: 0 
InFast: 0, OutFast: 0, InAS: 0, OutAS: 0 
InPktDrops: 0, OutPktDrops: 0 
CrcErrors: 0, SarTimeOuts: 0, OverSizedSDUs: 0, 
LengthViolation: 0, CPIErrors: 0 
Out CLP=1 Pkts: 0 
OAM cells received: 0 
F5 InEndloop: 0, F5 InSegloop: 0, F5 InAIS: 0, F5 InRDI: 0 
F4 InEndloop: 0, F4 InSegloop: 0, F4 InAIS: 0, F4 InRDI: 0 
OAM cells sent: 0 
F5 OutEndloop: 0, F5 OutSegloop: 0, F5 OutRDI: 0 
F4 OutEndloop: 0, F4 OutSegloop: 0, F4 OutRDI: 0 
OAM cell drops: 0 
Status: UP

Solution 2: NPE/NSE With More Host Buffers

Increase the number of host buffers allocated to the PA-A3, based on the network processing engine or network services engine installed in the 7200 series router. Cisco bug ID CSCdt74722 (registered customers only) increases the number of host buffers available for use by the PA-A3 to receive incoming data. Additional host buffers are a static allocation. This means that Cisco IOS Software does not perform any dynamic recarving based on the addition or removal of port adapters.

New generation ATM port adapter PA-A6 ATM provides support for up to 8191 VCs compared to 4096 VCs for the PA-A3 ATM port adapter. The PA-A6 ATM port adapter also provides performance improvements over the PA-A3 ATM port adapter. The PA-A6 provides line rate performance using 128-byte packet sizes on the Cisco 7200 series routers using the NPE-400 and NSE-1 processing engines.

Note: The PA-A6 ATM port adapter is not currently supported on the Cisco 7500 series routers. It is also not currently available on the Cisco 7600 FlexWAN.

This table provides the default number of particles in the private interface pool for the PA-A3 and PA-A6 ATM port adapters.

Network Processing or Services Engine	Particle Size	Default Particles ( Input Buffer Value)
NPE-225 and below	512	1200
NPE-300 and NSE-1	512	2400
NPE-400	512	4000
NPE-G1	512	4000

Use the show controller atm command and the show buffer command to view the number of receive buffers allocated to the PA-A3.

router#show controller atm 5/0
Interface ATM5/0 is up 
Hardware is ENHANCED ATM PA - OC3 (155000Kbps) 
Framer is PMC PM5346 S/UNI-155-LITE, SAR is LSI ATMIZER II 
Firmware rev: G127, Framer rev: 0, ATMIZER II rev: 3 
  idb=0x62948598, ds=0x6294FEA0, vc=0x6297F940 
  slot 5, unit 2, subunit 0, fci_type 0x0056, ticks 120012 
   1200 rx buffers: size=512, encap=64, trailer=28, magic=4 
[output omitted]


router# show buffer
[output omitted] 
Private particle pools: 
Serial4/0 buffers, 512 bytes (total 192, permanent 192): 
     0 in free list (0 min, 192 max allowed) 
     192 hits, 0 fallbacks 
     192 max cache size, 128 in cache 
     10 buffer threshold, 0 threshold transitions 
Serial4/1 buffers, 512 bytes (total 192, permanent 192): 
     0 in free list (0 min, 192 max allowed) 
     192 hits, 0 fallbacks 
     192 max cache size, 128 in cache 
     10 buffer threshold, 0 threshold transitions 
Serial4/2 buffers, 512 bytes (total 192, permanent 192): 
     0 in free list (0 min, 192 max allowed) 
     192 hits, 0 fallbacks 
     192 max cache size, 128 in cache 
     10 buffer threshold, 0 threshold transitions 
Serial4/3 buffers, 512 bytes (total 192, permanent 192): 
     0 in free list (0 min, 192 max allowed) 
     192 hits, 0 fallbacks 
     192 max cache size, 128 in cache 
     10 buffer threshold, 0 threshold transitions 
ATM5/0 buffers, 512 bytes (total 1200, permanent 1200):
    0 in free list (0 min, 1200 max allowed)
    1200 hits, 1 misses

In the show buffers command output, 0 in free list indicates that all of the private particle pool buffers are being held by the interface driver. Use the packet and drop counters in the show interface output to gauge whether your PA-A3 or PA-A6 has sufficient packet memory.

Related Information

• ATM Technology Support
• Technical Support & Documentation - Cisco Systems