Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Unicast Traffic Flow And Buffering

Multicast Traffic Flow And Buffering

What Causes Input Discards?

Troubleshooting Scenarios

Scenerio 1. Input Discards

Step 1. Identify Ports With Input Discards

Step 2. ASIC Identification

Step 3. Identify Egress Congested Port

Scenerio 2. Input Discards With HOLB

HOLB Mitigation: Enable VOQ Limit

HOLB Mitigation: Traffic Classification

Related Information

Introduction

This document describes how to troubleshoot input discards on the Cisco Nexus 5600/6000 series switches.

Prerequisites

Requirements

Cisco recommends that you have basic knowledge of Cisco Nexus 6000 Series configuration.

Components Used

The information in this document is based on these software and hardware versions:

- Cisco Nexus 6001

- 7.1(3)N1(1)

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

Input discards are an indication of an oversubscribed egress port. It also means that you are likely dropping unicast traffic on that specific port. This article helps you to understand how unicast and multicast traffic is buffered on this platform and how input discards could occur along with the mitigation steps.

Unicast Traffic Flow And Buffering

Unicast traffic is queued at egress buffer pool first and then ingress buffer after egress queue is full as shown in the image.

There is 16MB ingress shared buffer and 9MB egress shared buffer. The buffers are shared between 12 x 10 gig ports or 3 x 40 gig ports. Shared buffer is good for burst absorption.

Here is a visual depiction of the memory allocation for your reference (Bigsur is the name of the ASIC/Unified Port Controller) as shown in the image.

Multicast Traffic Flow And Buffering

- Multicast packets are buffered and dropped at egress

- Drop the multicast packet close to congestion point in order to avoid Head of Line Blocking (HOLB)

- Maintain lossless fabric for unicast as shown in the image.

In majority of the cases, egress drops is always due to multicast/broadcast/Unknown unicast traffic.

What Causes Input Discards?

A congested egress port causes the egress buffers in order to fill up first and then it causes the back pressure on the ingress. This is only for unicast traffic. Once the ingress buffers are full then you could potentially drop traffic on ingress which results in input discards.

This explanation is at a very high level and easy to digest but there is a bit more to it especially when you look at different class of traffic, queues etc. There is a concept of Virtual Output Queue (VOQ) which is frequently used on the Nexus platform. VOQ is an allocation of ingress buffers for every IEEE 802.1p Class of Service (CoS) per egress port. So there is 8 VOQ per egress port.

Congestion on one egress port in one CoS eventually bleeds into the congestion of its corresponding VOQ on the ingress port. Once the limit is reached then traffic gets dropped. It however, does not affect traffic destined for other CoSs or other egress interfaces, thus avoiding HOLB, which would otherwise cause congestion to spread. The traffic flow from the ingress to egress port and the various blocks in play is as shown in the image.

Troubleshooting Scenarios

Scenerio 1. Input Discards

Lab Setup:

Line rate traffic egressing e1/3 and possible over-subscription:

nexus6001# sh int e1/3

Ethernet1/3 is up

Dedicated Interface

Hardware: 1000/10000 Ethernet, address: 002a.6a56.7a8a (bia 002a.6a56.7a8a)

Description: spirent 4/11

MTU 1500 bytes, BW 1000000 Kbit,, BW 1000000 Kbit, DLY 10 usec

reliability 255/255, txload 251/255, rxload 25/255

Encapsulation ARPA, medium is broadcast

Port mode is trunk

full-duplex, 1000 Mb/s

Beacon is turned off

Input flow-control is off, output flow-control is off

Switchport monitor is off

EtherType is 0x8100

Last link flapped 11:39:20

Last clearing of "show interface" counters 00:00:15

0 interface resets

30 seconds input rate 98683696 bits/sec, 8223 packets/sec

30 seconds output rate 986853640 bits/sec, 82019 packets/sec

Load-Interval #2: 5 minute (300 seconds)

input rate 98.68 Mbps, 8.22 Kpps; output rate 986.85 Mbps, 82.01 Kpps

RX

124003 unicast packets  0 multicast packets  0 broadcast packets

124003 input packets  186004500 bytes

0 jumbo packets  0 storm suppression bytes

0 runts  0 giants  0 CRC  0 no buffer

0 input error  0 short frame  0 overrun   0 underrun  0 ignored

0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop

0 input with dribble  0 input discard

0 Rx pause

TX

1236745 unicast packets  9 multicast packets  0 broadcast packets

1236754 output packets  1860065401 bytes

0 jumbo packets

0 output error  0 collision  0 deferred  0 late collision

0 lost carrier  0 no carrier  0 babble 0 output discard

0 Tx pause

nexus6001# sh int e1/4

Ethernet1/4 is up

Dedicated Interface

Hardware: 1000/10000 Ethernet, address: 002a.6a56.7a8b (bia 002a.6a56.7a8b)

Description: spirent 4/12

MTU 1500 bytes, BW 1000000 Kbit,, BW 1000000 Kbit, DLY 10 usec

reliability 255/255, txload 25/255, rxload 251/255

Encapsulation ARPA, medium is broadcast

Port mode is access

full-duplex, 1000 Mb/s

Beacon is turned off

Input flow-control is off, output flow-control is off

Switchport monitor is off

EtherType is 0x8100

Last link flapped 10:53:31

Last clearing of "show interface" counters 00:00:04

0 interface resets

30 seconds input rate 986840376 bits/sec, 82236 packets/sec

30 seconds output rate 98421072 bits/sec, 8223 packets/sec

Load-Interval #2: 5 minute (300 seconds)

input rate 986.84 Mbps, 82.23 Kpps; output rate 98.42 Mbps, 8.22 Kpps

RX

326332 unicast packets  0 multicast packets  0 broadcast packets

326332 input packets  489496500 bytes

0 jumbo packets  0 storm suppression bytes

0 runts  0 giants  0 CRC  0 no buffer

0 input error  0 short frame  0 overrun   0 underrun  0 ignored

0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop

0 input with dribble  863 input discard >>>>>>

0 Rx pause

TX

32633 unicast packets  2 multicast packets  0 broadcast packets

32635 output packets  48819096 bytes

0 jumbo packets

0 output error  0 collision  0 deferred  0 late collision

0 lost carrier  0 no carrier  0 babble 0 output discard

0 Tx pause

In a simulated setup as here, you know the cause of the oversubscription but in a production setup where traffic profile burst and it can be a challenge to spot out the congested egress ports though these commands.

The steps listed here helps you to identify the congested egress ports.

Step 1. Identify Ports With Input Discards

Input discards seen on port e1/4:

nexus6001# sh int e1/4 | in i disc

0 input with dribble  3024 input discard

0 lost carrier  0 no carrier  0 babble 0 output discard

nexus6001# sh queuing int e1/4

Ethernet1/4 queuing information:

TX Queuing

qos-group  sched-type  oper-bandwidth

0       WRR            100

RX Queuing

qos-group 0 >>>> Drops in QOS 0

q-size: 100160, q-size-40g: 100160, HW MTU: 1500 (1500 configured)

drop-type: drop, xon: 0, xoff: 0

Statistics:

Pkts received over the port             : 9612480

Ucast pkts sent to the cross-bar        : 9587016

Mcast pkts sent to the cross-bar        : 0

Ucast pkts received from the cross-bar  : 961249

Pkts sent to the port                   : 961261

Pkts discarded on ingress               : 3024 >>>>>>

Per-priority-pause status               : Rx (Inactive), Tx (Inactive)

Step 2. ASIC Identification

- Map the interface to the internal ASIC (UPC) number from this output.

- Find out the ingress ASIC ID from the ingress port ID on which you noticed drops.

nexus6001# sh hard internal bigsur all-ports

Bigsur Port Info:

Port     |asic|inst|inst|

name     |idx |slot|asic|eport|logi|flag|adm|opr|if_index|diag|ucVer

---------+----+----+----+-----+----+----+---+---+--------+----+-----

sup1     |0   |0   |0   | 0 - |48  |b3  |en |dn |15010000|pass| 0.00

sup0     |0   |0   |0   | 1 - |49  |b3  |en |dn |15020000|pass| 0.00

1gb1/1   |1   |0   |1   | 2 - |0   |b3  |en |up |1a000000|pass| 0.00

1gb1/2   |1   |0   |1   | 3 - |1   |b3  |en |up |1a001000|pass| 0.00

1gb1/3   |1   |0   |1   | 0 - |2   |b3  |en |up |1a002000|pass| 0.00

1gb1/4   |1** |0   |1   | 1 - |3   |b3  |en |up |1a003000|pass| 0.00 >>>>** is the asic number

1gb1/5   |1   |0   |1   | 6 - |4   |b3  |en |up |1a004000|pass| 0.00

1gb1/6   |1   |0   |1   | 7 - |5   |b3  |en |up |1a005000|pass| 0.00

1gb1/7   |1   |0   |1   | 4 - |6   |b3  |en |up |1a006000|pass| 0.00

1gb1/8   |1   |0   |1   | 5 - |7   |b3  |en |up |1a007000|pass| 0.00

1gb1/9   |1   |0   |1   |10 - |8   |b3  |en |up |1a008000|pass| 0.00

1gb1/10  |1   |0   |1   |11 - |9   |b3  |en |up |1a009000|pass| 0.00

1gb1/11  |1   |0   |1   | 8 - |10  |b3  |en |up |1a00a000|pass| 0.00

xgb1/12  |1   |0   |1   | 9 - |11  |b3  |en |dn |1a00b000|pass| 0.00

xgb1/13  |2   |0   |2   | 2 - |12  |b3  |en |dn |1a00c000|pass| 0.00

xgb1/14  |2   |0   |2   | 3 - |13  |b3  |en |dn |1a00d000|pass| 0.00

xgb1/15  |2   |0   |2   | 0 - |14  |b3  |en |dn |1a00e000|pass| 0.00

xgb1/16  |2   |0   |2   | 1 - |15  |b3  |en |dn |1a00f000|pass| 0.00

xgb1/17  |2   |0   |2   | 6 - |16  |b3  |en |dn |1a010000|pass| 0.00

xgb1/18  |2   |0   |2   | 7 - |17  |b3  |en |dn |1a011000|pass| 0.00

xgb1/19  |2   |0   |2   | 4 - |18  |b3  |en |dn |1a012000|pass| 0.00

xgb1/20  |2   |0   |2   | 5 - |19  |b3  |en |dn |1a013000|pass| 0.00

xgb1/21  |2   |0   |2   |10 - |20  |b3  |en |dn |1a014000|pass| 0.00

xgb1/22  |2   |0   |2   |11 - |21  |b3  |en |dn |1a015000|pass| 0.00

xgb1/23  |2   |0   |2   | 8 - |22  |b3  |en |dn |1a016000|pass| 0.00

xgb1/24  |2   |0   |2   | 9 - |23  |b3  |en |dn |1a017000|pass| 0.00

xgb1/25  |3   |0   |3   | 2 - |24  |b3  |en |dn |1a018000|pass| 0.00

xgb1/26  |3   |0   |3   | 3 - |25  |b3  |en |dn |1a019000|pass| 0.00

xgb1/27  |3   |0   |3   | 0 - |26  |b3  |en |dn |1a01a000|pass| 0.00

xgb1/28  |3   |0   |3   | 1 - |27  |b3  |en |dn |1a01b000|pass| 0.00

xgb1/29  |3   |0   |3   | 6 - |28  |b3  |en |dn |1a01c000|pass| 0.00

xgb1/30  |3   |0   |3   | 7 - |29  |b3  |en |dn |1a01d000|pass| 0.00

xgb1/31  |3   |0   |3   | 4 - |30  |b3  |en |dn |1a01e000|pass| 0.00

xgb1/32  |3   |0   |3   | 5 - |31  |b3  |en |dn |1a01f000|pass| 0.00

xgb1/33  |3   |0   |3   |10 - |32  |b3  |en |dn |1a020000|pass| 0.00

xgb1/34  |3   |0   |3   |11 - |33  |b3  |en |dn |1a021000|pass| 0.00

xgb1/35  |3   |0   |3   | 8 - |34  |b3  |en |dn |1a022000|pass| 0.00

xgb1/36  |3   |0   |3   | 9 - |35  |b3  |en |dn |1a023000|pass| 0.00

xgb1/37  |4   |0   |4   | 2 - |36  |b3  |en |dn |1a024000|pass| 0.00

xgb1/38  |4   |0   |4   | 3 - |37  |b3  |en |dn |1a025000|pass| 0.00

xgb1/39  |4   |0   |4   | 0 - |38  |b3  |en |dn |1a026000|pass| 0.00

xgb1/40  |4   |0   |4   | 1 - |39  |b3  |en |dn |1a027000|pass| 0.00

xgb1/41  |4   |0   |4   | 6 - |40  |b3  |en |dn |1a028000|pass| 0.00

xgb1/42  |4   |0   |4   | 7 - |41  |b3  |en |dn |1a029000|pass| 0.00

xgb1/43  |4   |0   |4   | 4 - |42  |b3  |en |dn |1a02a000|pass| 0.00

xgb1/44  |4   |0   |4   | 5 - |43  |b3  |en |dn |1a02b000|pass| 0.00

xgb1/45  |4   |0   |4   |10 - |44  |b3  |en |dn |1a02c000|pass| 0.00

xgb1/46  |4   |0   |4   |11 - |45  |b3  |en |dn |1a02d000|pass| 0.00

xgb1/47  |4   |0   |4   | 8 - |46  |b3  |en |dn |1a02e000|pass| 0.00

xgb1/48  |4   |0   |4   | 9 - |47  |b3  |en |dn |1a02f000|pass| 0.00

40gb2/1  |5   |1   |0   | 2 - |0   |b3  |dis|dn |1a0f0000|pass| 0.00

40gb2/2  |5   |1   |0   | 1 - |1   |b3  |dis|dn |1a0f1000|pass| 0.00

40gb2/3  |6   |1   |1   | 2 - |2   |b3  |dis|dn |1a0f2000|pass| 0.00

40gb2/4  |6   |1   |1   | 1 - |3   |b3  |dis|dn |1a0f3000|pass| 0.00

Done.

Step 3. Identify Egress Congested Port

- Identify the congested egress port with VOQ counters.

- Use the ASIC number in counters voq asic-num in order to find out which egress port contributes towards the drops.

nexus6001# sh plat soft qd info counters voq asic-num 1

+----------+------------------------------+------------------------+-----------+

|      port|                      TRANSMIT|               TAIL DROP|  HEAD DROP|

+----------+------------------------------+------------------------+-----------+

Eth1/3

QUEUE-3                       3222876464                  8545008           0

Eth1/4

QUEUE-3                        323451170                        0           0

Eth1/6

QUEUE-3                           871362                        0           0

SUP_HI

QUEUE-0                             2041                        0           0

+----------+------------------------------+------------------------+-----------+

Scenerio 2. Input Discards With HOLB

Lab Setup:

All ports are in vlan 100.

You can see input discards on both e1/4 and e1/1 which depends on the traffic rate on the ingress interfaces towardse1/3.

nexus6001# sh int e1/4 | in discard|rate

30 seconds input rate 592103840 bits/sec, 49341 packets/sec

30 seconds output rate 166412120 bits/sec, 13863 packets/sec

input rate 592.10 Mbps, 49.34 Kpps; output rate 834.82 Mbps, 69.55 Kpps

0 input with dribble  15245 input discard

0 lost carrier  0 no carrier  0 babble 0 output discard

nexus6001# sh int e1/1 | in discard|rate

30 seconds input rate 986839872 bits/sec, 82236 packets/sec

30 seconds output rate 99790992 bits/sec, 8310 packets/sec

input rate 986.84 Mbps, 82.23 Kpps; output rate 500.88 Mbps, 41.73 Kpps

0 input with dribble  110632 input discard

0 lost carrier  0 no carrier  0 babble 0 output discard

With the use of the same process as documented for Scenerio 1. You can find the egress congested port.

nexus6001# sh plat so qd info counters voq asic-num 1 <snip>

+----------+------------------------------+------------------------+-----------+

|      port|                      TRANSMIT|               TAIL DROP|  HEAD DROP|

+----------+------------------------------+------------------------+-----------+

Eth1/3

QUEUE-3                       3893719464                164782171           0

The flow that must be affected is towards 10.10.10.50. The flow between 10.10.10.101 and 10.10.10.102 must be clean.

This is however not the case. A stuck or slow-draining egress port can cause all buffers on one or more ingress ports that sends traffic to the egress port to be exhausted which thereby affects all traffic on these ingress ports. This is the classic HOLB problem.

Spirent traffic generators shows that the flows are dropped. The port numbers are Spirent port numbers is as shown in the image.

HOLB Mitigation: Enable VOQ Limit

In order to avoid this scenario, the VOQs (only for unicast traffic) can be configured with a set threshold.

nexus6001(config)# hard unicast voq-limit

After the configuration, the flows towards non congested ports is not affected.

Spirent Traffic generator view after the VOQ limit configuration is as shown in the image.

Though this configuration shows a clear advantage in order to prevent drops due to HOLB. Why is this not the default config?

Typically, traffic in a production environment could burst in nature. By the disablement of the VOQ threshold, you allow the ingress buffers to absorb a traffic micro burst without the need to get dropped.

Unless the situation warrants the need to enable VOQ limit, it is recommended to use the default which is to leave it disabled.

HOLB Mitigation: Traffic Classification

There is another method to mitigate HOLB with the use of QoS configuration. Since ingress discards only affects a specific VOQ which in turn is a specific QOS class, you can map the affected traffic to non-congested port to a different QOS group. From this output, the ingress discards affects QOS Group 0 class.

nexus6001# sh queuing int e1/4

Ethernet1/4 queuing information:

TX Queuing

qos-group  sched-type  oper-bandwidth

0       WRR            100

RX Queuing

qos-group 0 >>>> Drops in QOS 0

q-size: 100160, q-size-40g: 100160, HW MTU: 1500 (1500 configured)

drop-type: drop, xon: 0, xoff: 0

Statistics:

Pkts received over the port             : 9612480

Ucast pkts sent to the cross-bar        : 9587016

Mcast pkts sent to the cross-bar        : 0

Ucast pkts received from the cross-bar  : 961249

Pkts sent to the port                   : 961261

Pkts discarded on ingress               : 3024 >>>>>>

Per-priority-pause status               : Rx (Inactive), Tx (Inactive)

The configuration here maps important traffic to QoS-Group 2.

1. Define ACL for traffic that must not be dropped. Objective is to classify this traffic into a different QOS group so it does not get affected.

ip access-list SINGLEFLOW

statistics per-entry

10 permit ip 10.10.10.101/32 10.10.10.102/32

2. QoS Classification:

class-map type qos match-all FIX_AFFECTED_FLOW

match access-group name SINGLEFLOW

policy-map type qos QOS_POLICY_FIX_AFFECTED_FLOW

class FIX_AFFECTED_FLOW

set qos-group 2

3. Network QoS configuration:

class-map type network-qos QOSGRP2

match qos-group 2

policy-map type network-qos NQOS-GRP2

class type network-qos QOSGRP2

class type network-qos class-default

4. Apply the various policies. The network QoS is system wide while the classification policy can be applied to a single interface.

system qos

service-policy type network-qos NQOS-GRP2

interface Ethernet1/1

service-policy type qos input QOS_POLICY_FIX_AFFECTED_FLOW

5. The drops are not seen for QoS group 2 class:

nexus6001(config-if)# sh queuing int e1/1

Ethernet1/1 queuing information:

TX Queuing

qos-group  sched-type  oper-bandwidth

0       WRR            100

2       WRR              0

RX Queuing

qos-group 0

q-size: 100160, q-size-40g: 100160, HW MTU: 1500 (1500 configured)

drop-type: drop, xon: 0, xoff: 0

Statistics:

Pkts received over the port             : 525111

Ucast pkts sent to the cross-bar        : 327510

Mcast pkts sent to the cross-bar        : 0

Ucast pkts received from the cross-bar  : 0

Pkts sent to the port                   : 0

Pkts discarded on ingress               : 197868 >>>>

Per-priority-pause status               : Rx (Inactive), Tx (Inactive)

qos-group 2

q-size: 100160, q-size-40g: 100160, HW MTU: 1500 (1500 configured)

drop-type: drop, xon: 0, xoff: 0

Statistics:

Pkts received over the port             : 131413

Ucast pkts sent to the cross-bar        : 132096

Mcast pkts sent to the cross-bar        : 0

Ucast pkts received from the cross-bar  : 0

Pkts sent to the port                   : 0

Pkts discarded on ingress               : 0 >>> No Drops

Per-priority-pause status               : Rx (Inactive), Tx (Inactive)

Related Information

- Nexus 6000 Series Switches QoS Configuration Example

- Technical Support & Documentation - Cisco Systems