Troubleshooting Interface and VC Counter Issues on ATM Port Adapters

Introduction

This document describes how ATM port adapters report traffic statistics and resolve issues with incorrect packet or byte counters displayed in the output of the show interface atm or show atm vc commands.

Prerequisites

Requirements

This document requires an understanding of the variable and fixed overhead fields on an ATM interface, including the well-known 5-byte cell header. Refer to the following technical tips for more information on this overhead:

• Measuring the Utilization of ATM PVCs

• Implementing Network Management on ATM Interfaces

• Understanding Maximum Transmission Unit (MTU) on ATM Interfaces

• What Bytes Are Counted by IP to ATM CoS Queueing?

Components Used

This document is not restricted to specific software and hardware versions.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Understanding Statistics Reporting Mechanisms

Updating virtual circuit (VC) and ATM interface/subinterface statistics is handled differently depending on the platform. In general, port adapters for the Cisco 7x00 Series Routers use the following common mechanism for reporting statistics:

1. The framer strips the layer-1 framing bits.

2. The segmentation and reassembly (SAR) mechansim reassembles the packet, sets appropriate error bits, if any, and passes the packet to the host driver.

3. Once the packet is put into memory via direct memory access (DMA) to the host, receive interrupt is generated.

4. The driver processes the packet and does the accounting.

The driver is responsible for updating the counters for all packets, good or bad. The platform-specific software block is responsible for calculating the input and output bit rates, received load, etc.

The Cisco 7500 Series uses a distributed architecture which requires communication between the PA-A3 interface driver and the Versatile Interface Processor (VIP) CPU, as well as between the VIP CPU and the RSP CPU. The peripheral component interconnect (PCI) host driver on the PA-A3 collects per-VC statistics on each packet and sends them to the VIP driver. The Route/Switch Processor (RSP) sends a command to the VIP to retrieve statistics via a periodic Cisco IOS® process. When the system is initialized, it creates a special background process that handles the autonomous statistics from the VIP as a scheduled process rather than at the interrupt level to minimize system interruption.

When enabled, the debug atm events command displays output similar to the following when the VIP CPU sends a message to the RSP to report VC statistics.

received CCB_CMD_ATM_GET_VC_STATS command vcd #

The VIP sends updated statistics information to the RSP at 12-second intervals. Thus, the value displayed in the show command output may not be the instantaneous value.

Use the debug atm event command at the VIP console to isolate the problem. The debug output can be used to indicate whether the VIP is sending erroneous VC statistics to the RSP or whether the correct information is being corrupted during communication between the VIP CPU and the RSP CPU. For more information, refer to Understanding debug atm event Output on ATM Router Interfaces.

Caution: Before issuing debug commands, refer to Important Information on Debug Commands. The debug atm events command may print a large amount of disruptive debug output on a production router depending on the number of VCs for which it needs to report statistics as well as the amount of VC-related events.

Note: On the Cisco 12000 Series, Engine 0 and Engine 1 line cards send updates every 10 seconds, while other Engine models such as Engine 2 send updates at a faster rate. The 4xOC3 ATM line card uses Engine 0 architecture.

Layer-2 vs Layer-3 Counters

In the show interface command output for the main interface, the "input packets" field counts the number of packets that have been received and switched successfully to an egress interface.

In the show atm vc {vcd#} command output for a virtual circuit (VC), the "InPkts" field counts the number of packets that have been received properly and passed on to the IOS switching engine. If the IOS switching engine cannot process the packets and drops them in the interface hold queue, it counts these packets as drops only and does not increment the input packets counter. Thus, the value of the VC's "InPkts" count will equal the sum of the main interface's "input packets" counter and input queue drops counter. The show atm vc {vcd#} command output also displays an "InPktDrops" field to count the number of packet drops at the VC level. The separate input drop counts allow for determining whether the drop happened at the VC level or at the interface level.

The show interface atm command output for a subinterface represents a sum of the per-VC counters on that subinterface. The following sample output of the show interface atm command for a subinterface on a PA-A3 illustrates that only layer-2 information, such as ATM Adaptation Layer 5 (AAL5) counters and Operations, Administration and Maintenance (OAM) cell counts, is displayed.

7206#show int atm 4/0.1 
ATM4/0.1 is administratively down, line protocol is down 
  Hardware is ENHANCED ATM PA 
  MTU 4470 bytes, BW 149760 Kbit, DLY 80 usec, 
     reliability 0/255, txload 1/255, rxload 1/255 
  Encapsulation ATM 
  0 packets input, 0 bytes 
  0 packets output,0 bytes 
  0 OAM cells input, 0 OAM cells output 
  AAL5 CRC errors : 0 
  AAL5 SAR Timeouts : 0 
  AAL5 Oversized SDUs : 0

Since the subinterface counter reflects layer-2 information only, the counters on the main interface and the subinterface may be different. Note that this difference enhances the ability to determine where packets are being dropped. For example, an arriving packet may pass layer-2 checks, such as the AAL5 cyclic redundancy check (CRC), and be forwarded to the main interface where an inbound IP ACL specifies a drop action for the source or destination IP address. This packet would increment the drop counter only on the VC and the subinterface, not on the main interface.

Common Problems When Reporting Interface and VC Counter Statistics on ATM Port Adapters

This section describes some of the most common problems seen when reporting interface and VC counter statistics on ATM Port Adapters. Several symptoms are discussed and solutions for each are provided. The most common symptoms include:

• Calculated interface rates above the physical line rate.

• Negative counter for input queue.

• Double accounting or double the expected counter values.

• Incorrect "InBytes" value on PVC with QoS service policy.

• Incorrect or no statistics on ATM subinterfaces.

Most of these problems are software issues that have been resolved in various releases of the Cisco IOS software.

Calculated Interface Rates Above the Physical Line Rate

This symptom has been seen and resolved in the following Cisco bug IDs:

Cisco Bug ID	Description
CSCdt49209	When 64-bit SNMP counters were introduced in Cisco IOS Software Release 12.0(15)S, ATM interfaces reported calculated output interface rates above the physical line rate. This problem does not affect traffic flow.
CSCdv13285	When using the encapsulation aal5mux ppp to terminate PPP over ATM (PPPoA) sessions, a Cisco 7200 Series router with Cisco Express Forwarding (CEF) enabled may report an extremely high input data rate. The root cause of this problem is counting erroneous PPP echo-request or echo-reply packets at 65000 bytes.

Negative Counter for Input Queue

All interfaces on Cisco routers use an input queue to store packets that fail to be matched against a route cache entry with fast switching or against an entry in the CEF table. Such packets are queued in the incoming interface's input queue for processing. Some packets are always processed, but with the appropriate configuration and in stable networks, the rate of processed packets should never congest the input queue. If the input queue is full, the packet will be dropped.

In rare cases, the input queue counter displayed in the show interface atm output may become a negative value, as illustrated below:

7206_B#show int atm 1/0 
 ATM1/0 is up, line protocol is up 
   Hardware is ENHANCED ATM PA 
   Description: DNEC.678475.ATI 1/40 
   MTU 4470 bytes, sub MTU 4470, BW 44209 Kbit, DLY 190 usec, 
      reliability 255/255, txload 6/255, rxload 1/255 
   Encapsulation ATM, loopback not set 
   Keepalive not supported 
   Encapsulation(s): AAL5 
   4096 maximum active VCs, 170 current VCCs 
   VC idle disconnect time: 300 seconds 
   0 carrier transitions 
   Last input 00:00:00, output 00:00:00, output hang never 
   Last clearing of "show interface" counters 01:31:25 
   Input queue: -6/75/0/0 (size/max/drops/flushes); Total output drops: 0

This problem is reported and resolved in the following Cisco bug IDs:

Cisco Bug ID	Symptoms and Workarounds
CSCdj73443	To support faster switching speeds of process-switched packets between 600 and 1524 bytes in size, a pool of (contiguous) large buffers in SRAM was added. When a buffer was allocated from this special pool, the input queue count was not incremented. The input queue count eventually became a large positive number and packets outside the 600 - 1524 byte range were rejected due to a full input queue. This problem was resolved by removing a large SRAM contiguous buffer pool.
CSCdm44539	Negative input queue counts resulted from decrementing the input queue counter by two ATM and other interface types, including serial.

Double Accounting or Double the Expected Counter Values

In some cases, enabling a Cisco IOS feature or changing the IOS switching path leads to a doubling of packet counters or calculated bit rates. Such "double accounting" issues have been reported and resolved for various interface types and various features.

This problem is reported and resolved in the following Cisco bug IDs:

Cisco Bug ID

Symptoms and Workarounds

CSCds23924

Input police function, as part of a QoS service policy, is invoked twice. Results include double accounting of input packets as well as inflated conformed packet values and excess drops. The most important aspect of this fix, however, is QoS feature reordering. As a result of the reordering, we now have: Input rate-limiting committed access rate (CAR) applies to packets that are destined for the router. Used to only apply to CEF-switched packets. IP precedence set by input CAR or QPPB (QoS Policy Propagation via BGP) can be used for vc selection in vc-bundling. IP precedence/DSCP and QoS group set by input CAR or QPPB can be used for Modular QoS CLI (MQC) "output" packet classification.

For example, Cisco bug ID CSCds23924 resolves double accounting with input CAR or class-based policing resulting from executing this feature twice as the packets follow the CEF switching path. (CEF defines an IOS switching mechanism, which forwards packets from ingress to egress routed interface.) Results include double accounting of input packets as well as an inflated conformed packets value and excess drops.

With the PA-A3, enabling IP accounting doubled the calculated output bit rate as displayed in the show interface atm command. This problem is caused by IP accounting not being supported by distributed Cisco Express Forwarding (dCEF). Thus, enabling IP accounting changes the path of packet inside the router and leads to an inflated output bit rate. This problem is documented via Cisco bug ID CSCdv59172.

Incorrect "InBytes" Value on PVC with QoS Service Policy

On the Cisco 7500 Series, applying a QoS service policy to an ATM VC may lead to an incorrect "InBytes" value, as displayed in the output of the show atm vc {vcd#} command. The problem appears only when packets are distribute-switched between PVCs on the same physical interface which has dCEF enabled.

This problem is resolved via Cisco bug ID CSCdu17025.

Incorrect or No Statistics on ATM Subinterfaces

The PA-A3 interface driver is responsible for updating the VC counters and sending them to the common or platform-independent block of ATM code. The counters displayed in the show atm pvc x/y or show interface atm.subint command are displayed as reported by the common ATM code, which adds all the VC counters on that subinterface.

If you see correct VC counters and a non-incrementing (or zero) value for subinterface counters, the ATM common code may not be adding up all the VC counters. To troubleshoot this problem, capture the following:

• show interface atm x/y/z.a of the subinterface where the problem is seen.

• show atm pvc {vpi/vci} of the VCs configured under that subinterface.

This problem is reported and resolved in the following Cisco bug IDs:

Cisco Bug ID	Description
CSCdu41673	Subinterface counters are 64-bit counters. The VIP was sending only 32-bit counters when updating the VC statistics. This problem is resolved by ensuring that the VIP also updates the 64-bit counters when sending statistics information to the RSP.
CSCdt60738	A router with a Network Services Engine (NSE-1) displays different output packet values on a main interface versus a subinterface.

Note: Calculated bit rates are available only on a main interface.

Troubleshooting Steps

Cisco recommends doing the following to troubleshoot incorrect counters on the PA-A3 or other ATM interfaces before contacting Cisco TAC:

• Capture several outputs of the counter. Is the counter tracking output or input data?

• On which physical or logical interface is the problem seen? Possible answers include:

  • Input or output queue

  • Subinterface

  • VC

• The ATM driver is only responsible for reporting the input and output byte counts. Determine whether the problem is caused by the PA-A3 or is platform-specific. Begin by determining whether the "packets input" and "packets output" counters, as well as the input and output byte counters, are correct.

  • If yes, investigate a platform-specific issue.

  • If no, investigate a PA-specific issue.

Related Information

• Measuring the Utilization of ATM PVCs
• Implementing Network Management on ATM Interfaces
• Understanding Maximum Transmission Unit on ATM Interfaces
• What Bytes Are Counted by IP to ATM CoS Queueing?
• ATM Technology Support
• Technical Support & Documentation - Cisco Systems