Troubleshoot FEX Performance Issues on Nexus 5000/6000 Series

Introduction

This document describes how to troubleshoot performance on the Fabric Extenders (FEX) that can attach to Nexus 5000 or 6000 Series Switches.

Note: None of the commands introduced in this document are disruptive.  You must have a Nexus 2000 switch connected to a 5000 or 6000 series switch.

Background Information

Navigate the CLI

Attach to the FEX

Attach to the FEX to run show commands on the FEX command line:

Nexus# attach fex fex
fex>

Enter Debug Exec Mode

Enter the debug mode on the FEX in order to run advanced commands and specify the FEX asic name. Refer to Table 1. for the FEX asic names.

fex# dbgexec [prt/woo/red/pri]

Exit Debug Exec Mode

In order to exit the Debug Exec Mode use CTRL+C keyboard sequence:

fex> [CTRL+C]

Exit the FEX

In order to exit the fex, use the command exit:

fex# exit

Terminology

Host Interface (HI)

Hi's are the ports that face the servers on the FEX.These are commonly known as front ports.Every front port on a FEX has a HI number. This number is usually different than the port number, but it is used to troubleshoot commands to refer to a port. Each asic tabulates front ports differently.

Network Interface (NI)

NI's are the FEX Control Ports on the FEX that connect back to the parent switch. These are also refered to as Network Uplinks. These also have a unique NI number dependent on the model.

FEX Fabric Port

These ports are the parent switch side of the unique link to the FEX. These ports are configured with switchport mode fex-fabric and a fex association commands. 

FEX ASIC Names 

Each FEX is designed with a different ASIC. The abbreviation of the ASIC name is used in the debugging mode to run commands.

Most models of the FEX have one ASIC, however the 2148 has 6, each with 8 front ports. These are referred to as rmon in troubleshoot commands. 

The ASIC names and associated abreviations are listed for reference:

 
Table 1.

FEX Model	ASIC Name	Abreviation
N2K-C2148T-1GE	redwood	rw
N2K-C2224TP-1GE N2K-C2248TP-1GE	portola	prt
N2K-C2232PP-10GE N2K-C2232TM-10GE	woodside	woo
N2K-C2248TP-E-1GE	princeton	pri
B22	woodside	woo
N2K-C2232TM-E-10GE	woodside	woo
N2K-C2248PQ-10GE	woodside/belmont	woo
N2K-C2348UPQ-10GE	tiburon	tib

Front Port Mapping

To interpet interface counter output it can be necessary to convert the front port number to a HI number. The conversion is dependent on the FEX chassis model.

N2K-C2148T-1GE

In this example, front port 26 (chassis-id/1/26) has been assigned rmon 3 HI 0:

switch#  attach fex chassis_id

fex-[chassis_id]#  show platform software redwood sts

N2K-C2224TP-1GE / N2K-C2248TP-1GE

In this example, front port 10 (135/1/10) has been assigned HI 9:

switch#  attach fex chassis_id

fex-[chassis_id]#  dbgexec portola

prt> fp

N2K-C2232PP-10GE / N2K-C2232TM-10GE

In this example, front port 20 (135/1/20) has been assigned HI 19:

switch#  attach fex chassis_id

fex-[chassis_id]# show platform software woodside sts

N2K-C2248TP-E-1G

N2K-C2248PQ-10GE & N2K-C2348UPQ-10GE 

In this example, HI28 maps to Front Port 29:

Verify SFP

This command shows the Small Form-Factor Pluggable (SFP) information for the port.

fex# show platform software woodside sfp rmon 0 HI5

In this example, you see that the SFP in HI5 is a 10G-Base-SR (LC) made by CISCO-AVAGO:

Note:  If you run this command on a FEX that uses copper ports, then you will notice the command errors. This is expected as there is no SFP to query. The prompt will return to no SFP found when that port is fiber, but does not currently contain an SFP.

Find Loss

Show commands can be run at the FEX prompt for the HI and NI ports in order to view interface counters on the FEX side of the FEX Fabric port links.

View HI Port Counters

This command shows port counter verification, similar to a show int:

fex-128# show platform software woodside rmon 0 HI0
+----------------------+----------------------+-----------------+--------------------------+----------------------+-----------------+
| TX                        |         Current        |       Diff          |           RX                  |       Current          |        Diff         |
+----------------------+----------------------+-----------------+--------------------------+----------------------+-----------------+
| TX_PKT_LT64     |                           0|                     0|    RX_PKT_LT64     |                           0|                     0|
| TX_PKT_64         |                           0|                     0|    RX_PKT_64          |                           0|                     0|
| TX_PKT_65         |                           0|                     0|    RX_PKT_65          |                           0|                     0|
| TX_PKT_128       |                           0|                     0|    RX_PKT_128        |                           0|                     0|
| TX_PKT_256       |                           0|                     0|    RX_PKT_256        |                           0|                     0|

Note: rmon 0 is used only when the FEX has one host asic. The 2224, 2248 and 2232 models have only one asic. The 2148 model has six asics, so rmon 0 through 5 will be used. See the Front Port Mapping section for further details.

View NI Port Counters

This command will show you the port counters for the network uplinks similar to a show int. This command shows you the FEX side of the link. This command does not show you the parent switch side of the link.

 fex-128# show platform software woodside rmon 0 NI0
+----------------------+----------------------+-----------------+--------------------------+----------------------+-----------------+
| TX                        |         Current        |       Diff          |           RX                  |       Current          |        Diff         |
+----------------------+----------------------+-----------------+--------------------------+----------------------+-----------------+
| TX_PKT_LT64     |                           0|                     0|    RX_PKT_LT64     |                           0|                     0|
| TX_PKT_64         |                           0|                     0|    RX_PKT_64          |                           0|                     0|
| TX_PKT_65         |                           0|                     0|    RX_PKT_65          |                           0|                     0|
| TX_PKT_128       |                           0|                     0|    RX_PKT_128        |                           0|                     0|
| TX_PKT_256       |                           0|                     0|    RX_PKT_256        |                           0|                     0|  

View Historical Drops

Historical Drops can be viewed with the drops command. This shows you all drops on the FEX since it was turned on.

This command also shows you drops to the FEX CPU that will not represent data traffic drops with DROP8 counters. These can be safely ignored.

 

Note: tail drop [8] and TAIL_DROP8 represents tail drops to the FEX CPU and is not relevant to troubleshoot performance as this happens under normal conditions.

prt> drops
PRT_SS_CNT_TAIL_DROP1 : 3 SS0
PRT_SS_CNT_TAIL_DROP1 : 6 SS1
PRT_SS_CNT_TAIL_DROP1 : 1 SS2
PRT_SS_CNT_TAIL_DROP1 : 25 SS3
PRT_SS_CNT_TAIL_DROP1 : 2 SS5
PRT_SS_CNT_TAIL_DROP8 : 142 SS0
PRT_SS_CNT_TAIL_DROP8 : 73 SS1
PRT_SS_CNT_TAIL_DROP8 : 11 SS2
PRT_SS_CNT_TAIL_DROP8 : 62048 SS3
PRT_SS_CNT_TAIL_DROP8 : 4613 SS4
PRT_SS_CNT_TAIL_DROP8 : 552 SS5

View Recent Drops and Interrupts

Interrupts sent to the CPU include tail-drops, which are drops due to congestion and lack of buffer space. These can be viewed with the show new_ints command: 

Note: 6.0 and later code uses show new_ints all

This example shows that frames tail drop in the SS1 buffer:

prt> show new_ints
|-----------------------------------------------|
| SS1 : ssx_int_norm_td 
|--+---------+----------------------------------+
| 1 | 00001c98   |    tail drop[1]     | frames are being tail dropped.
| 2 | 00005cac   |    tail drop[2]      | frames are being tail dropped.
| 8 | 0000012e   |    tail drop[8]     | frames are being tail dropped.

This example shows that NI 3 receives symbol errors:

 | NI3 : nix_xe_INT_xg  
 |--+---------+----------------------------------+ 
 |2 |00000005 | rx_local_fault                  | Link is in local fault state 
 |3 |00000007 | rx_remote_fault              | Link is in remote fault state 
 |4 |00000004 | rx_code_violation           | MAC received unexpected XGMII control characters. 
 |5 |00000004 | rx_err_symbol                 | MAC received an XGMII error character. 
 |16|00000001 | rx_local_fault_edge      | Local fault state has changed. 
 |17|00000001 | rx_remote_fault_edge  | Remote fault state has changed. 
 |-----------------------------------------------| 

This example shows that the FEX tail drops frames that ingress NI3:

 | SS4 : ssx_int_err 
 |--+---------+----------------------------------+ 
 |0 |00031aa9 | wo_cr[0]               | frames rcvd without credit for pausable classes. Pause is missing. 
 |1 |00014e21 | wo_cr[1]               | frames rcvd without credit for pausable classes. Pause is missing. 
 |2 |00018a9f  | wo_cr[2]                | frames rcvd without credit for pausable classes. Pause is missing. 
 |3 |00025efb  | wo_cr[3]                | frames rcvd without credit for pausable classes. Pause is missing. 
 |-----------------------------------------------| 

View the Port Traffic Rate in Real Time

 

The rate command outputs real time traffic rate statistics for a port. Unlike show int, its not an average, its the raw current data rate that second. In this example, NI 3 currently receives 2.96kbps in the Network to Host direction. A show int on the corresponding parent Nexus switch shows 2.96Kbps in the TX direction on the FEX Fabric uplink connected to NI 3.

Mitigate Loss

Tail drops are caused by buffer exhaustion. Typically the buffer becomes exhausted when multiple servers burst to the HIFs at once, or the host egress buffers cannot empty their outbound traffic fast enough to replenish the credits on the NIFs.

There are several options available to mitigate that loss.

Reposition Servers 

Move any servers with bursty traffic flows such as storage arrays and video endpoints off of the FEX and connect them directly to the base ports of the parent switch. This will prevent the bursty servers from exhausting the buffer and starving out traffic from less chatty hosts.

Nexus 5000 and 6000 Series Switches have larger buffers than FEX models, to connect bursty servers to the base ports mitigates the loss because the base port buffers are able to handle a much greater burst. 

Add Additional Uplinks

Some models of FEX can unlock additional buffer space when more uplinks from the FEX to the parent switch are added. This can potentially cease drops on the Network Uplinks.

Table 2.

  

Model	Buffer Increase When Adding Uplinks
2148	none
2224	buffer increase up to 2 uplinks
2248TP	buffer increase up to 4 uplinks
2232	buffer increase up to 4 uplinks
2248TP-E	none
2248PQ	none

Share HI Buffers

Most models of FEX can benefit from sharing the HI buffer across all host ports. If drops are seen on the HI, sharing the buffer might mitigate those drops.

Modify the FEX queue limit globally:

5k(config)# no fex queue-limit (globally applies to all fexes on that 5k)

Modify the FEX queue limit on individual FEX:

Fex queue 

5k(config)# fex 100
5k(config-fex)# no hardware [model] queue-limit

Nexus 6000 FEX Load Balance Enhancement

The Nexus 6000 has an additional option to change the load balancing algorithm from HIF to NIF. By default, even if packets arrive on different HIF ports, they might still be queued to the same NIF. With uplink-load-balance-mode enabled, they are distributed across multiple NIFs, and allow for more even usage of NIF egress buffers.

6k(config)# hardware N2248PQ uplink-load-balance-mode