Nexus 7000 Series F1-Module Input Discards

Introduction

This document describes how to troubleshoot input discards on the Cisco Nexus 7000 Series F1-Module.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on these software and hardware versions:

• Cisco Nexus 7000 Series Switches
• Cisco Nexus 7000 F1-Series, 32-Port, 1 and 10 Gigabit Ethernet Modules
• Cisco Nexus Operating System (NX-OS) Versions 5.X and later

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Troubleshoot Input Discards

When you observe input discards on an F1 Series linecard, it usually means that you have oversubscribed a port on egress. On most linecards, this scenario results in output discards on the egress interface; however, when the arbitration of the packet is F1-to-F1, and the traffic is credited, you can see input discards on the ingress port.

Switch#show interface eth 1/8
Ethernet1/8 is up
  Hardware: 1000/10000 Ethernet, address: 503d.e5df.a785 (bia 503d.e5df.a785)
  .
  .
  Load-Interval #2: 5 minute (300 seconds)
    input rate 168 bps, 0 pps; output rate 3.78 Kbps, 3 pps
  RX
    15539560971 unicast packets  3466668 multicast packets  0 broadcast packets
    15542893003 input packets  8720803713147 bytes
    4384352384 jumbo packets  0 storm suppression packets
    0 runts  0 giants  0 CRC  0 no buffer
    0 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  4029156 input discard
    0 Rx pause
  TX
    7409231138 unicast packets  125221759 multicast packets  127954348 broadcast packets
    7662272650 output packets  2001593436247 bytes
    472864528 jumbo packets
    0 output error  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble  0 output discard
    0 Tx pause
  1 interface resets

On the F1 Series linecards, there is both credited and uncredited traffic. The credited traffic is known unicast. All other traffic, such as multicast, broadcast, and unknown unicast, is characterized as uncredited.

The credited traffic requires a credit from the egress ASIC before the packet is sent across the fabric to the egress linecard. On an M1 Series linecard, the Octopus ASIC is used for arbitration, so the packet can move across the fabric to the egress module before the state of the egress port ASIC is known. If the egress port ASIC is overburdened, then the packet arrives before it is known, so it is dropped and logged as an output discard.

The F1 Series linecards have a Switch on a Chip (SOC) that functions as the arbitration ASIC as well as the port ASIC. This means that the linecard knows if it does not have the bandwidth that is required in order to process a packet, and it does not give a credit to the ingress port ASIC, which causes the packet to be dropped and logged as an input discard.

Identify the Oversubscribed Egress Port

Once you notice an increase in input discards, you must discover the port that is oversubscribed on egress. You can use these commands in order to identify the oversubscribed egress port:

Attach module X
Show hardware internal qengine asic Y memory vq-head-tail
Show hardware internal qengine sw vqi-map

The initial action that you must take is to determine the interface on which the input discards increase. For this example, the interface is Eth1/8.

Note: It is important that the input discards are increasing, or you will not see them in the vq-head-tail command output. 

You must then determine the ASIC on which the port resides. On the F132 linecard, there are two ports per ASIC, which begins with ASIC 0. For example, ports 1 and 2 are on ASIC 0, ports 3 and 4 are on ASIC 1, and ports 5 and 6 are on ASIC 2. For this example, the Eth1/8 interface is located on ASIC 3.

Note: Ensure that you attach to the module on which you see the input discards before you run these commands.

Here is a sample output:

Switch# attach module 1
module-1# show hardware internal qengine asic 3 memory vq-head-tail

+------------------------------------------------------------------
| VQ head tail for Orion Xbar Driver
| Inst 3
|

INDEX     THRESHOLD      HEAD       TAIL      PACKET COUNT     Q-LENGTH
_____     _________     ______     ______     ____________     ________
23        1             5936       10086      1084             2168
136       0             6702       6702       0                0
4096      0             3607       3607       0                0

In this example, Index 23 has a very high packet count and Q-length. This indicates that the index for this Virtual Queuing Index (VQI) receives too much traffic, and it does not send credits so that traffic is sent to it on egress. Therefore, it drops packets on ingress.

In order to determine the VQI itself, divide the Index by 4 (a constant) and leave the remainder. Here is an example for Index 23:

23/4 = 5 (with a remainder of 3), so the VQI for Index 23 is 5.

Enter the show hard int qengine sw vqi-map command in order to determine the interface to which this VQI maps:

module-1# show hard int qengine sw vqi-map
Supervisor VQI info:
--------------------
    sup 0 slot          : 4
    sup 1 slot          : 5
    sup xbar mask       : 0x000003ff

        | sup0 | sup1 |   sup0    |   sup1    |          |
    vqi |  vqi |  vqi | fpoe base | fpoe base | num fpoe | lb_type
    ----+------+------+-----------+-----------+----------+-----------
     32 |   32 |   32 |        36 |        44 |        1 | non-spread
     33 |   33 |   33 |        37 |        45 |        1 | non-spread
     34 |   34 |   34 |        32 |        40 |        4 | spread
     35 |   35 |   35 |        32 |        40 |        4 | spread

VQI property map:
-----------------
 vqi | asic | ldi | sl | sup | sprd | xbar | fpoe | #  | hdr  | xbar | vqi | lcl  
     | inst |     |    | vqi | type | mask | base | dl | type | asic | typ | pqi
-----+------+-----+----+-----+------+------|------+----+------+------+-----+----
   0 |    0 |   0 |  0 |  no |  rr  | 0155 |    0 |  1 |  v5  |  scz |   0 |   0
   1 |    0 |   1 |  0 |  no |  rr  | 0155 |    0 |  1 |  v5  |  scz |   0 |   1
   2 |    1 |   2 |  0 |  no |  rr  | 0155 |    1 |  1 |  v5  |  scz |   0 |   2
   3 |    1 |   3 |  0 |  no |  rr  | 0155 |    1 |  1 |  v5  |  scz |   0 |   3
   4 |    2 |   4 |  0 |  no |  rr  | 0155 |    2 |  1 |  v5  |  scz |   0 |   4
   5 |    2 |   5 |  0 |  no |  rr  | 0155 |    2 |  1 |  v5  |  scz |   0 |   5    

In the VQI property map section of the output, identify the VQI (vqi) that you previously calculated, the slot (sl), and the local Port Queuing Index (PQI) (lcl pqi) to which it is mapped. Here are the values from this output:

• vqi = 5
• sl = 0 (module 1)
• lcl pqi = 5 (port 6)

Note: In this example, the vqi and lcl pqi values are the same, but this is usually not the case. 

As shown, the VQI of 5 is at slot 0, which is Module 1 when you count from zero. The LCL PQI is 5, which is at port 6. Thus, the Eth1/6 interface is oversubscribed on egress, which causes input drops on the ingress interfaces for traffic that is destined to that port on egress.

Additional VQI Mapping Information

The VQI and Local Destination Index (LDI) allocations are determined when the module is brought online. The VQI is (currently) fixed at 12 Gb/s and is allocated differently based on the module type. The mapping that is used in this example for F1 does not apply to all of the modules. Ensure that you enter the show system internal ethpm info interface ethernet command in order to confirm the VQI and LDI that is assigned to your port.  

For example, here is the information for port 17 from multiple modules:

• M132 (port Eth3/17)
  

  N7KA# show system internal ethpm info interface ethernet 3/17 | i VQI
   LTL(0x90), VQI(0x64), LDI(0x6), IOD(0x50)

• M148 (port Eth5/17)
  

  N7KA# show sys int ethpm info interface ethernet 5/17 | i VQI
   LTL(0x30), VQI(0x7), LDI(0x3), IOD(0xe1)

• F132 (port Eth4/17)
  

  N7KA# show sys int ethpm info interface ethernet 4/17 | i VQI
   LTL(0x10), VQI(0x1c), LDI(0x10), IOD(0x26)

• F248 (port Eth6/17)
  

  N7KA# show system internal ethpm info interface ethernet 6/17 | i VQI
   LTL(0x60), VQI(0x3d), LDI(0x11), IOD(0x11d)

Here is the output from the show hardware internal qengine vqi-map command for these interfaces:

N7KA# show hardware internal qengine vqi-map
VQI   SUP  SLOT  LDI  EQI  FPOE  NUM  XBAR   IN   ASIC  ASIC  SV  FEA_
NUM   VQI  NUM   NUM  NUM  BASE  DLS  MASK   ORD  TYPE  IDX   ID  TURE
----  ---  ----  ---  ---  ----  ---  -----  ---  ----  ----  --  ----
7     no   4     3    3    32    4    0x3ff  0    0     0     0   0x0  <--- port 5/17
28    no   3     16   0    168   1    0x155  0    ORI   8     0   0x81 <--- port 4/17
61    no   5     17   2    44    1    0x155  0    CLP   4     0   0x80 <--- port 6/17
100   no   2     6    2    20    4    0x3ff  0    0     1     0   0x0  <--- port 3/17

(shows only VQIs 0x64, 0x7, 0x1c, 0x3d)