Nexus 7000 F1 Module ELAM Procedure

Introduction

This document describes the steps used in order to perform an ELAM on a Cisco Nexus 7000 (N7K) F1 module, explains the most relevant outputs, and describes how to interpret the results.

Tip: Refer to the ELAM Overview document for an overview on ELAM.

Topology

In this example, a host on VLAN 10 (10.1.1.101 with MAC address 0050.56a1.1a01), port Eth3/18 sends an Internet Control Message Protocol (ICMP) request to a host that is also on VLAN 10 (10.1.1.102 with MAC address 0050.56a1.1aef), port Eth3/26. ELAM is used in order to capture this single frame from 10.1.1.101 to 10.1.1.102. It is important to remember that ELAM allows you to capture only a single frame.

In order to perform an ELAM on the N7K, you must first connect to the appropriate module (this requires the network-admin privilege):

N7K# attach module 3
Attaching to module 3 ...
To exit type 'exit', to abort type '$.' 
module-3# 

Determine the Ingress Forwarding Engine

Traffic is expected to ingress the switch on port Eth3/18. When you check the modules in the system, you see that Module 3 is an F1 module. It is important to remember that the N7K is fully-distributed, and that the modules, not the supervisor, make the forwarding decisions for dataplane traffic.

N7K# show module 3
Mod  Ports  Module-Type                Model              Status
---  -----  -------------------------- ------------------ ----------
3    32     1/10 Gbps Ethernet Module  N7K-F132XP-15      ok

For F1 modules, perform the ELAM on the Layer 2 (L2) Forwarding Engine (FE) with internal codename Orion. The N7K F1 has 16 FEs per module, so you must determine the Orion ASIC that is used for the FE on port Eth3/18. Enter this command in order to verify:

module-3# show hardware internal dev-port-map 
(some output omitted)
--------------------------------------------------------------
CARD_TYPE:         DCE 32 port 10G 
>Front Panel ports:32
--------------------------------------------------------------
 Device name             Dev role              Abbr num_inst:
--------------------------------------------------------------
> Orion Fwding Driver    DEV_LAYER_2_LOOKUP     L2LKP  16
+--------------------------------------------------------------+
+-----------+++FRONT PANEL PORT TO ASIC INSTANCE MAP+++--------+
+--------------------------------------------------------------+
FP port |  PHYS | MAC_0 | L2LKP | QUEUE |SWICHF 
...
   18      8       8       8       8       1 

In the output, you can see that port Eth3/18 is on Orion (L2LKP) instance 8. 

module-3# elam asic orion instance 8 
module-3(orion-elam)#

Configure the Trigger

The Orion ASIC has a very limited set of ELAM triggers when compared to the other FEs on the N7K platform. This is because the F1 is an L2-only module. Therefore, it makes switching decisions based on the MAC address information (or SwitchID in FabricPath environments).

With Nexus Operating Systems (NX-OS), you can use the question mark character in order to separate the ELAM trigger:

module-3(orion-elam)# trigger di field ?
  da           Destination mac-address
  mim_da       Destination mac-in-mac-address
  mim_sa       Source mac-in-mac-address
  sa           Source mac-address
  vlan

For this example, the frame is captured based on the source and destination MAC addresses on the ingress decision block.

Note: The F1 module does not require separate DBUS and RBUS triggers.

Here is the trigger:

module-3(orion-elam)# trigger di field sa 0050.56a1.1a01 da 0050.56a1.1aef

Start the Capture

The F1 module is different from the other N7K modules, because the ELAM begins immediately after the trigger is configured. In order to check the status of the ELAM, enter the status command:

module-3(orion-elam)# status
Armed

Once the frame that matches the trigger is received by the FE, the ELAM status shows as Triggered:

module-3(orion-elam)# status
Triggered

Interpret the Results

In order to display the ELAM results, enter the show capture command. Here is the excerpt from the ELAM data that is most relevant to this example (some output is omitted):

module-3(orion-elam)# show capture 
dc3v4_si[11:0]           :                   17
vlanx                    :                    a
di                       :                   1e or 1f
res_eth_da               :           5056a11aef
res_eth_sa               :           5056a11a01

Note: With the F1 module, the ELAM data that is used in order to make the forwarding decision and the data that contains the forwarding result are combined into the same output. Also, note that the MAC address format in the ELAM output does not include prepending zeros.

Destination MAC (res_eth_da) 5056a11aef = 0050.56a1.1aef
Source MAC      (res_eth_sa) 5056a11a01 = 0050.56a1.1a01

With this output, you can verify the source Local Target Logic (LTL) (dc3v4_si), the destination LTL (di), the VLAN (vlanx), and the source and destination MAC addresses (5056a11a01 and 5056a11aef, respectively).

The source LTL (dc3v4_si) represents the port on which the frame is received. The F1 ELAM displays two results for the destination LTL (1e or 1f). This occurs because the ELAM parser cannot read the least-significant bit of the ELAM data, which produces an ambiguous result. Therefore, Cisco recommends that you validate the hardware MAC address entry for the destination address, and verify it with the destination LTL in the ELAM.

N7K# show system internal pixm info ltl 0x17
Type            LTL
---------------------------------
PHY_PORT       Eth3/18

The output shows that the source LTL of 0x17 maps to port Eth3/18. This confirms that the frame is received on port Eth3/18.

module-3# show hardware mac address-table fe 8
  address 0050.56a1.1aef vlan 10 vdc 1

(some output omitted)

FE | Valid| PI|  BD  |      MAC      |  Index|
   |      |   |      |               |       | 
---+------+---+------+---------------+-------+ 
 8    1     0   34    0050.56a1.1aef  0x0001f   


N7K# show system internal pixm info ltl 0x1f
Type            LTL
---------------------------------
PHY_PORT       Eth3/26

With this output, you can verify that Orion instance 8 (the FE that makes the forwarding decision for Eth3/18) has a hardware MAC address entry of 0x1f for the destination MAC address 0050.56a1.1aef. This index is also the destination LTL (di) within the F1 ELAM data. 

Additionally, you can verify that LTL 0x1f maps to port Eth3/26. This confirms that the frame is sent from port Eth3/26.

Additional Verification

In order to verify how the switch allocates the LTL pool, enter the show system internal pixm info ltl-region command. The output from this command is useful in order to understand the purpose of an LTL if it is not matched to a physical port. A good example of this is a Drop LTL:

N7K# show system internal pixm info ltl 0x11a0
0x11a0 is not configured

N7K# show system internal pixm info ltl-region
 LTL POOL TYPE                          SIZE        RANGE
 =====================================================================
 DCE/FC Pool                            1024       0x0000 to 0x03ff
 SUP Inband LTL                           32       0x0400 to 0x041f
 MD Flood LTL                              1       0x0420
 Central R/W                               1       0x0421
 UCAST Pool                             1536       0x0422 to 0x0a21
 PC Pool                                1720       0x0a22 to 0x10d9
 LC CPU Pool                              32       0x1152 to 0x1171
 EARL Pool                                72       0x10da to 0x1121
 SPAN Pool                                48       0x1122 to 0x1151
 UCAST VDC Use Pool                       16       0x1172 to 0x1181
 UCAST Generic Pool                       30       0x1182 to 0x119f
 LISP Pool                                 4       0x1198 to 0x119b
 Invalid SI                                1       0x119c to 0x119c
 ESPAN SI                                  1       0x119d to 0x119d
 Recirc SI                                 1       0x119e to 0x119e
 Drop DI                                   2       0x119f to 0x11a0
 UCAST (L3_SVI_SI) Region                 31       0x11a1 to 0x11bf
 UCAST (Fex/GPC/SVI-ES)       3648       0x11c0 to 0x1fff
 UCAST Reserved for Future Use Region   2048       0x2000 to 0x27ff
======================> UCAST MCAST BOUNDARY <======================
 VDC OMF Pool                             32       0x2800 to 0x281f