Chapter 1. New and Changed Information

New and Changed Information

Chapter 2. Configuring FCoE NPV

FCoE NPV Overview

FCoE NPV Benefits

FCoE NPV Features

Fibre Channel Slow Drain Device Detection and Congestion Avoidance

VNP Ports

Licensing Requirements for FCoE NPV

Guidelines and Limitations for Configuring FCoE NPV

Configuring FCoE NPV

Configuring QoS for no-drop support

Configuring VLAN-VSAN Mapping

Explicit vFC Configuration

Implicit vFC Configuration

Configuring the FCoE NPV Core Switch

Configuring the FCoE NPV Edge Switch

Configuring a Pause Frame Timeout Value

Verifying the FCoE NPV Configuration

FCoE NPV Core Switch and FCoE NPV Edge Switch Configuration Example

FCoE NPV Core Switch and FCoE NPV Edge Switch with Implicit vFC Configuration Example

Chapter 3. Configuring FCoE VLANs and Virtual Interfaces

Information About Virtual Interfaces

Information About Shutting Down LAN Traffic

Notes About the shutdown lan Command

Examples of Shutdown LAN Traffic

Examples of Verifying Shutdown LAN Traffic

Guidelines and Limitations for FCoE VLANs and Virtual Interfaces

Mapping a VSAN to a VLAN

Creating a Virtual Fibre Channel Interface

Associating a Virtual Fibre Channel Interface to a VSAN

Creating an Implicit Virtual Fibre Channel Port Channel Interface

Configuring virtual Fibre Channel â€“ Port Channel Interface

Verifying the Virtual Interface

Mapping VSANs to VLANs Example Configuration

SAN Boot with vPC

SAN Boot with vPC Configuration Example

Chapter 4. FCoE Over FEX

Overview

FCoE Over FEX with vPC

LAN Shutdown

FCoE Over FEX Topologies

Straight Through FEX with Host Topology

Straight Through FEX with Host VPC Topology

Dual-Homed FEX Topology (Active/Active FEX Topology)

Guidelines and Limitations for FCoE Over FEX

Information Before Configuring FCoE Over FEX

Configuring Straight Through FEX with Host

Binding vFC to FEX Interface Explicitly

Binding VFC to FEX Interface Implicitly

Binding vFC to MAC Address

Configuring Straight Through FEX with Host vPC

Configuring Dual-Homed FEX

Preface

Audience

Document Conventions

Related Documentation for Cisco Nexus 9000 Series Switches

Documentation Feedback

Obtaining Documentation and Submitting a Service Request

Copyright Page

Chapter 1. New and Changed Information

This chapter provides release-specific information for each new and changed feature in the Cisco Nexus 9000 Series NX-OS FCoE Configuration Guide.

- New and Changed Information

New and Changed Information

This table summarizes the new and changed features for the Cisco Nexus 9000 Series NX-OS FCoE Configuration Guide and where they are documented.

New and Changed Features

Feature

Description

Changed in Release

Where Documented

N9K-C93180LC-EX

Support was added for the N9K-C93180LC-EX switch

7.0(3)I6(1)

Information Before Configuring FCoE Over FEX

Dual-homed FEX topology (Active/Active FEX topology)

Introduced this feature.

7.0(3)I5(2)

Click Here

Fibre Channel Slow Drain Device Detection

Added Fiber Channel slow drain device detection support in the N9K-X9732C-EX line card.

7.0(3)I5(2)

Fibre Channel Slow Drain Device Detection and Congestion Avoidance

FCoE NPV

Added support for the N9K-X9732C-EX line card.

7.0(3)I5(2)

Guidelines and Limitations for Configuring FCoE NPV

shutdown lan command support for FEX HIF ports and port-channels

To avoid having the FCoE link go down when the system detects a vPC inconsistency, use the shutdown lan command to shutdown the LAN traffic on port-channels and individual Ethernet ports.

7.0(3)I5(1)

LAN Shutdown

Fibre Channel Slow Drain Device Detection

Enables slow drain device detection and congestion avoidance when slow devices are attached to the network fabric resulting with end devices not accepting frames at a configured rate that leads to traffic congestion on the links

7.0(3)I5(1)

Fibre Channel Slow Drain Device Detection and Congestion Avoidance

FCoE Over FEX support

Enables Fibre Channel traffic to be carried on a FEX port. The FEX is connected to Cisco Nexus 9000 device (in FCoE NPV mode) through a Fabric Port Channel (FPC).

7.0(3)I5(1)

FCoE Over FEX

MAC bound vFC support

Binds the virtual Fibre Channel interface to a host MAC address before a virtual Fibre Channel interface can be used.

7.0(3)I5(1)

Creating a Virtual Fibre Channel Interface

Pause frame timeout support

Enables a pause frame timeout value on a port to empty the buffer space in the ISL link and helps to reduce the fabric slowdown and the congestion on other unrelated flows using the same link.

7.0(3)I5(1)

Configuring a Pause Frame Timeout Value

shutdown lan command support

Added support for the shutdown lan command.

7.0(3)I4(1)

Information About Shutting Down LAN Traffic

FCoE VLANs and virtual interfaces support

Added support for FCoE VLANs and virtual interfaces.

7.0(3)I4(1)

Configuring FCoE VLANs and Virtual Interfaces

FCoE NPV support

Initial support

7.0(3)I3(1)

Chapter 2. Configuring FCoE NPV

This chapter describes how to configure Fiber Channel over Ethernet (FCoE) N-port Virtulization (NPV) on Cisco NX-OS devices.

- FCoE NPV Overview

- VNP Ports

- Licensing Requirements for FCoE NPV

- Guidelines and Limitations for Configuring FCoE NPV

- Configuring FCoE NPV

- Verifying the FCoE NPV Configuration

- FCoE NPV Core Switch and FCoE NPV Edge Switch Configuration Example

- FCoE NPV Core Switch and FCoE NPV Edge Switch with Implicit vFC Configuration Example

FCoE NPV Overview

Fiber Channel over Ethernet (FCoE) N-port Virtulization (NPV) is an enhanced form of FCoE Initialization Protocol (FIP) snooping that provides a secure method to connect FCoE-capable hosts to an FCoE-capable FCoE forwarder (FCF) device.

FCoE NPV enables:

- The switch to act as an N-port virtualizer (NPV) connected to the core switch (FCF).

- The core switch (FCF) to view the NPV switch as another host.

- The multiple hosts connected to the NPV switch are presented as virtualized N-ports on the core switch (FCF) .

- FCoE NPV Benefits

- FCoE NPV Features

- Fibre Channel Slow Drain Device Detection and Congestion Avoidance

FCoE NPV Benefits

FCoE NPV provides the following:

- FCoE NPV provides the advantages of NPV to FCoE deployments (such as preventing domain ID sprawl and reducing Fiber-Channel Forwarder (FCF) table size).

- FCoE NPV provides a secure connect between FCoE hosts and the FCoE FCF.

- FCoE NPV does not have the management and troubleshooting issues that are inherent to managing hosts remotely at the FCF.

- FCoE NPV implements FIP snooping as an extension to the NPV function while retaining the traffic-engineering, VSAN-management, administration, and trouble shooting aspects of NPV.

FCoE NPV Features

The following are the FCoE NPV features:

- Automatic load balancing

- The server interfaces (Host logins) are distributed in a round robin fashion among the available multiple uplinks (NP ports or external-interfaces).

- You can enable disruptive automatic load balancing to load balance the existing server interfaces (hosts) to newly added NP uplink interfaces.

Example:

switch(config)# npv auto-load-balance disruptive

- Traffic mapping.

- You can specify the NP uplinks that a server interface can use to connect to core switches.

- Load balancing does not start automatically if the current mapped uplink goes down.

- Example:

switch(config)# npv traffic-map server-interface 2/1 external-interface 2/1

- FCoE forwarding in the FCoE NPV bridge.

- FCoE NPV supports the Data Center Bridging Exchange Protocol (DCBX).

- FCoE frames received over VNP ports are forwarded only if the L2_DA matches one of the FCoE MAC addresses assigned to hosts on the VF ports. Otherwise, they are discarded.

Note

FCoE NPV over port channel VNP ports use automatic traffic mapping only for FIP negotiations. FCoE traffic distribution over port channel VNP ports is based on the computed hash value.

Fibre Channel Slow Drain Device Detection and Congestion Avoidance

The data traffic between the end devices in Fibre Channel over Ethernet (FCoE) uses link level and per-hop based flow control. When the slow devices are attached to the fabric, the end devices do not accept the frames at a configured rate. The presence of the slow devices leads to traffic congestion on the links. The traffic congestion affects the unrelated flows in the fabric that use the same inter-switch links (ISLs) for its traffic, even though the destination devices do not experience the slow drain.

Beginning with Cisco NX-OS 7.0(3)I5(1), slow drain device detection and congestion avoidance is supported on N9K-C93180YC-EX. The enhancements are mainly on the edge ports that are connected to the slow drain devices to minimize the congestion condition in the edge ports.

Beginning with Cisco NX-OS 7.0(3)I5(2), slow drain device detection and congestion avoidance is supported on the N9K-X9732C-EX line card.

Beginning with CiscoNX-OS 7.0(3)I6(1), slow drain device detection and congestion avoidance is supported on the N9K-C93180LC-EX.

Note

Slow drain device detection and congestion avoidance is not supported on FEX ports.

VNP Ports

Connectivity from an FCoE NPV bridge to the FCF is supported only over point-to-point links. These links can be individual Ethernet interfaces or port channel interfaces. For each FCF connected to an Ethernet/port-channel interface, a vFC interface must be created and bound to it. These vFC interfaces must be configured as VNP ports.

On the VNP port, the FCoE NPV bridge emulates an FCoE-capable host with multiple enodes, each with a unique enode MAC address. By default, the VNP port is enabled in trunk mode.

Multiple VSANs can be configured on the VNP port. The FCoE VLANs that correspond to the VNP port VSANs must be configured on the bound Ethernet interface.

Note

VNP ports on the Cisco Nexus 9000 Series device emulate an FCoE capable host with multiple Ethernet nodes, each with unique Fabric Provided MAC-Addresses (FPMA).

Licensing Requirements for FCoE NPV

The following table shows the licensing requirements for FCoE NPV:

Product

License Requirement

Cisco NX-OS

FCoE NPV requires an FCoE NPV license (FCOE_NPV_PKG). For a complete explanation of the Cisco NX-OS licensing scheme and how to obtain and apply licensees, see the Cisco NX-OS Licensing Guide.

Guidelines and Limitations for Configuring FCoE NPV

Configuring FCoE NPV has the following configuration guidelines and limitations:

- Enabling FCoE NPV requires:

- Downloading and installing FCOE_NPV_PKG (license).

- Installing the FCoE feature set using the install feature-set fcoe-npv command.

- Enabling the NPV feature set using the feature-set fcoe-npv command.

- On the N9k-C92160YC and N9K-C93180YC-EX platforms, 6 QSFP+ ports (40G) can be used as either uplink ports or server ports.

- It is mandatory to do a TCAM Reservation (as mentioned here: Configuring QoS for no-drop support) for FCoE NPV to work.

- show commands with the internal keyword are not supported.

- FCoE NPV does not convert server FLOGI to FDISC.

- FCoE NPV supports a VNP port bound to a ethernet interface, port-channel, or break-out interface.

- FCoE NPV is supported on the following Cisco Nexus 9300 Series devices and Cisco Nexus 9500 Series devices (NX-OS 7.0(3)I3(1) and later):

- Nexus 9332PQ switch

- Nexus 9372PX switch

- Nexus 9372PX-E switch

- Nexus C9396PX switch

- Nexus C9504 and Nexus C9508 switches with the following line cards:

- X9432PQ

- X9464PX

- X9536PQ

- X9564PX

- X9636PQ

- FCoE NPV is supported on the following Cisco Nexus 9200 and Cisco Nexus 9300 Series devices (NX-OS 7.0(3)I5(1) and later):

- N9K-C92160YC-X

- N9K-C9272Q

Note

The N9k-9272Q switch has 72 ports. Ports 1 through 36 only operate in 40Gb mode. Ports 37 through 71 support breakout to 4 x 10-Gb connectivity. Port 72 does not support breakout.

- N9K-C9236C

- N9K-C93180YC-EX

- FCoE NPV is supported on the following Cisco Nexus 9500 Series devices (NX-OS 7.0(3)I6(1) and later):

- N9K-C93180LC-EX

- FCoE NPV is supported on the following Cisco Nexus 9500 Series devices (NX-OS 7.0(3)I5(2) and later):

- N9K-X9732C-EX

- FCoE NPV is not supported on the following Cisco Nexus 9500 Series devices (NX-OS 7.0(3)I5(2) and later):

- N9K-X97160YC-EX

- N9K-X9736C-EX

- FCoE NPV supports the following FEX (NX-OS 7.0(3)I5(1) and later):

- N2K-C2348UPQ

- N2K-C2232PP

- N2K-B22IBM-P

- N2K-B22HP-P

- Starting from Release 7.0(3)I6(1), the N9K-C93180LC-EX supports FCoE NPV on FEX with only one FEX model: the N2K-C2348UPQ.

- The following switches do not support FEX (NX-OS 7.0(3)I5(1) and later):

- N9K-C92160YC-X

- N9K-C9272Q

- N9K-C9236C

- On the following models, the 40G GEM (Gigabit Ethernet Expansion Module) ports cannot be used to connect FCoE hosts (F ports). They can be used as FCoE uplink ports (VNP Ports):

- Cisco Nexus 9332PQ switch

- Cisco Nexus 9372PX switch

- Cisco Nexus 9372PX-E switch

- Cisco Nexus 9396PX switch

- N9K-X9464PX line cards

- N9K-X9564PX line cards

- Application Leaf Engine (ALE) 40G ports cannot be used to connect FCoE hosts (F ports) and do not support Converged Network Adapters (CNA).

- FCoE NPV does not support nested NPV.

- FCoE NPV supports FLOGI/FDISC (nested NPIV).

To support multiple FLOGI from a single port, you must enable the NPIV feature to accommodate hosts/servers that send FDISC followed by FLOGI.

Examples of commands to enable/disable and display status of the NPIV feature:

- switch(config)# feature npiv
switch# show feature | include npiv
npiv                   1          enabled
switch#

- switch# show npv status

npiv is enabled

disruptive load balancing is disabled

External Interfaces:
====================
Interface: vfc-po100, State: Trunking
VSAN:    1, State: Waiting For VSAN Up
VSAN:    2, State: Up
VSAN:    3, State: Up, FCID: 0x040000
Interface: vfc1/49, State: Down

Number of External Interfaces: 2

- switch(config)# no feature npiv
switch# show feature | include npiv
npiv                   1          disabled
switch#

- When configuring QoS policies for an active-active FEX topology that is enabled for FCoE, you must configure the QoS policies on the FEX HIF port on both VPC peers to avoid unpredictable results (7.0(3)I5(1) and later releases).

Note

Only the following support an active-active FEX topology:

- N2K-C2232PP

- N2K-C2348UPQ

- NB22IBM

- NB22HP

Configuring FCoE NPV

- Configuring QoS for no-drop support

- Configuring VLAN-VSAN Mapping

- Explicit vFC Configuration

- Implicit vFC Configuration

- Configuring the FCoE NPV Core Switch

- Configuring the FCoE NPV Edge Switch

- Configuring a Pause Frame Timeout Value

Configuring QoS for no-drop support

A type qos ingress policy is used to mark ingress FCoE frames. The type qos policy has two classes. One for FCoE packets and the other for non- FCoE packets (default class). The type qos ingress policy must be applied to the interfaces that handle FCoE traffic (such as, all ethernet/port-channel interfaces bound to vFCs).

Note

Check to ensure that the port qos region has hardware TCAM space reserved.

This step is mandatory for FCoE NPV to work.

- Reserve TCAM space for the port qos region.

You may need to acquire TCAM space reserved for other regions (such as the l3qos region).

- Save the configuration.

- Reload the line cards or switch.

- Confirm the port qos region TCAM space.

- Example for TCAM carving on Network Forwarding Engine (NFE):

hardware access-list tcam region qos 256
hardware access-list tcam region fex-qos 256
hardware access-list tcam region fcoe-ingress 256
hardware access-list tcam region fcoe-egress 256

- Example for TCAM carving on Application Spine Engine (ASE) and Leaf Spine Engine (LSE):

hardware access-list tcam region ing-racl 1536
hardware access-list tcam region ing-redirect 256

- Example for TCAM carving on N9K-C92160YC-X, N9K-C9272Q, N9K-C9236C, or N9K-C93180YC-EX:

hardware access-list tcam region ing-racl 1536
hardware access-list tcam region ing-redirect 256

Example:

switch# show hardware access-list tcam region | i "IPV4 Port QoS \[qos\] size"
IPV4 Port QoS [qos] size = 0 /*** Value is 0; No reserved TCAM space.***/

switch# config
switch(config)# hardware access-list tcam region qos 256

Warning: Please reload all linecards for the configuration to take effect

switch# copy running-config startup-config

switch# reload

switch# show hardware access-list tcam region | i "IPV4 Port QoS \[qos\] size"
IPV4 Port QoS [qos] size = 256

The following are examples of configurations of required QOS policies.

- Configuring type qos class map.

switch(config)# class-map type qos match-any fcoe
switch(config-cmap-qos)# match protocol fcoe
switch(config-cmap-qos)# match cos 3
switch(config-cmap-qos)# exit
switch(config)#

- Configuring type qos policy map.

switch(config)# policy-map type qos fcoe_qos_policy
switch(config-pmap-qos)# class fcoe
switch(config-pmap-c-qos)# set qos-group 3
switch(config-pmap-c-qos)# exit
switch(config-pmap-qos)# exit
switch(config)#

- Configuring type network-qos policy map.

switch(config)# policy-map type network-qos fcoe_nq
switch(config-pmap-nqos)# class type network-qos c-nq3
switch(config-pmap-nqos-c)# pause pfc-cos 3
switch(config-pmap-nqos-c)# mtu 9216
switch(config-pmap-nqos-c)# class type network-qos c-nq2
switch(config-pmap-nqos-c)# mtu 1500
switch(config-pmap-nqos-c)# class type network-qos c-nq1
switch(config-pmap-nqos-c)# mtu 1500
switch(config-pmap-nqos-c)# class type network-qos c-nq-default
switch(config-pmap-nqos-c)# mtu 1500
switch(config-pmap-nqos-c)# exit
switch(config-pmap-nqos)# exit
switch(config)#

- Configuring type input queuing policy map.

switch(config)# policy-map type queuing fcoe-in-policy
switch(config-pmap-que)# class type queuing c-in-q3
switch(config-pmap-c-que)# bandwidth percent 50
switch(config-pmap-c-que)# class type queuing c-in-q-default
switch(config-pmap-c-que)# bandwidth percent 50
switch(config-pmap-c-que)# exit
switch(config-pmap-que)# exit
switch(config)#

- Configuring type output queuing policy map.

switch(config)# policy-map type queuing fcoe-out-policy
switch(config-pmap-que)# class type queuing c-out-q3
switch(config-pmap-c-que)# priority level 1
switch(config-pmap-c-que)# class type queuing c-out-q-default
switch(config-pmap-c-que)# bandwidth remaining percent 100
switch(config-pmap-c-que)# class type queuing c-out-q1
switch(config-pmap-c-que)# bandwidth remaining percent 0
switch(config-pmap-c-que)# class type queuing c-out-q2
switch(config-pmap-c-que)# bandwidth remaining percent 0

- Configuring the system wide QOS policy.

switch(config)# system qos
switch(config-sys-qos)# service-policy type queuing input fcoe-in-policy
switch(config-sys-qos)# service-policy type queuing output fcoe-out-policy
switch(config-sys-qos)# service-policy type network-qos fcoe_nq
switch(config-sys-qos)# exit
switch(config)#

Note

When Fex is connected, apply the QOS policy to the system level. Also, apply the QOS policy for the HIF port to honor pause frames in FCOE traffic.

switch(config)# system qos
switch(config-sys-qos)# service-policy type queuing input fcoe-in-policy
switch(config-sys-qos)# service-policy type queuing output fcoe-out-policy
switch(config-sys-qos)# service-policy type network-qos fcoe_nq
switch(config-sys-qos)# service-policy type qos input fcoe_qos_policy

Note

The qos policy needs to be attached to an HIF interface or the port-channel of an HIF interface:

- HIF interface

interface "HIF port"
service-policy type qos input pmap-fcoe-qos

- Port-channel of an HIF interface

interface port-channel
service-policy type qos input fcoe_qos_policy

- Applying the ingress qos policy to each Ethernet / port-channel interface that is bound to vFC interface for FCoE.

switch(config)# interface ethernet 2/1
switch(config-if)# switchport mode trunk
switch(config-if)# mtu 9216    /* Or maximum allowed value */
switch(config-if)# service-policy type qos input fcoe
switch(config-if)# exit
switch(config)#

Configuring VLAN-VSAN Mapping

VSANs and VLANs are required and the VSANs need to be mapped to the VLANs.

One VLAN can be mapped to only one VSAN and vice versa. The VSANs can then be added to F and NP vFC interfaces (described in a subsequent section).

- Example of VSAN creation:

switch(config)#
switch(config)# vsan database
switch(config-vsan-db)# vsan 10
switch(config-vsan-db)#

- Example VLAN configuration and binding to FCoE VSAN:

switch(config)# vlan 10
switch(config-vlan)# fcoe vsan 10
switch(config-vlan)# exit
switch(config)#

Note

On the following Cisco Nexus 9200 and 9300 platforms , all the ports ( both 10G and 40G) can be configured as server interfaces:

- Cisco Nexus 92160YC-X switches

- Cisco Nexus 9272Q switches

- Cisco Nexus 9236C switches

- Cisco Nexus 93180YC-EX switches

- N9K-X9732C-EX line cards

Explicit vFC Configuration

An explicit vFC interface is a vFC interface where the bound ethernet/port-channel interface is explicitly configured. (The interface ID range is 1-8192.)

- Example of explicit vFC bound to interface ethernet:

switch# configure terminal
switch(config)# interface vfc 21
switch(config-if)# bind interface ethernet 2/1

- Example of explicit vFC bound to interface port-channel:

switch# configure terminal
switch(config)# interface vfc 100
switch(config-if)# bind interface port-channel 100

- Example of explicit vFC bound to break-out port:

switch# configure terminal
switch(config)# interface vfc 111
switch(config-if)# bind interface ethernet 1/1/1

- Example of NP interface configuration using explicit vFC:

switch# configure terminal
switch(config)# interface vfc21
switch(config-if)# switchport mode NP
switch(config-if)# switchport trunk allowed vsan 10 /* optional; for restricting VSANs */

- Example of NP interface configuration using explicit bound ethernet interface:

switch# configure terminal
switch(config)# interface vfc152
switch(config-if)# bind interface port-channel152
switch(config-if)# switchport mode NP
switch(config-if)# switchport trunk allowed vsan 2
switch(config-if)# switchport trunk mode on
switch(config-if)# no shutdown

- Example of F interface configuration using explicit vFC:

switch# configure terminal
switch(config)# interface vfc15
switch(config-if)# bind interface ethernet 1/5
switch(config-if)# switchport mode F /* Default mode is F */
switch(config-if)# switchport trunk allowed vsan 10
switch (config-if)# exit
switch (config)# vsan database
switch(config-vsan-db)# vsan 10 interface vfc15
switch(config-vsan-db)# exit

Implicit vFC Configuration

An implicit vFC interface is a vFC interface that has an ID with the format slot/port or unit/slot/port or port-channelid. When this vFC is created , the ethernet interface slot/port or unit/slot/port or port-channelid is automatically (implicitly) bound to the interface. The running config would display the bound ethernet/port-channel interface. If the ethernet /port-channel interface does not exist or is bound to another explicit vFC interface, the vFC creation fails with an error.

- Example of implicit vFC bound to interface ethernet:

switch# configure terminal
switch(config)# interface vfc 2/1

- Example of implicit vFC bound to interface port-channel:

switch# configure terminal
switch(config)# interface vfc-port-channel 100

- Example of implicit vFC bound to break-out port:

switch# configure terminal
switch(config)# interface vfc 1/1/1

- Example of NP interface configuration using implicit vFC:

switch# configure terminal
switch(config)# interface vfc1/1/1
switch(config-if)# switchport mode NP
switch(config-if)# switchport trunk allowed vsan 10 /* optional; for restricting VSANs */

- Example of F interface configuration using implicit vFC:

switch# configure terminal
switch(config)# interface vfc1/1/1
switch(config-if)# switchport mode F /* Default mode is F */
switch(config-if)# switchport trunk allowed vsan 10
switch (config-if)# exit
switch (config)# vsan database
switch(config-vsan-db)# vsan 10 interface vfc1/1/1
switch(config-vsan-db)# exit

Configuring the FCoE NPV Core Switch

Perform the following steps to configure an FCoE NPV core switch.

SUMMARY STEPS

1.    configure terminal

2.    (Optional) switchto vdc vdc-name

3.    feature npiv

4.    interface ethernet slot/port

5.    switchport mode trunk

6.    exit

7.    interface vfc vfc-id

8.    bind interface ethernet slot/port

9.    exit

10.    vsan database

11.    vsan vsan-id

12.    vsan vsan-id interface vfc vfc-id

13.    exit

14.    vlan vlan-id

15.    fcoe vsan vsan-id

16.    exit

DETAILED STEPS

Command or Action

Purpose

Step 1

configure terminal

Enters configuration mode.

Step 2

switchto vdc vdc-name

(Optional)
Switch to storage VDC

Note:
This step is required only when a Cisco Nexus 7000 Series switch is used as the core switch.

Step 3

feature npiv

Enable NPIV

Step 4

interface ethernet slot/port

Enters interface configuration mode.

Step 5

switchport mode trunk

Set physical interface mode to trunk

Step 6

exit

Exits the interface mode.

Step 7

interface vfc vfc-id

Enters interface configuration mode.

Step 8

bind interface ethernet slot/port

Set vFC port mode of P1 to VF

Important:
The bind interface ethernet command is not required for an implicit vFC configuration.

Step 9

exit

Exits the interface configuration mode.

Step 10

vsan database

Enters VSAN configuration mode.

Step 11

vsan vsan-id

Create VSAN

Step 12

vsan vsan-id interface vfc vfc-id

add vFC into VSAN

Step 13

exit

Exits the VSAN configuration mode.

Step 14

vlan vlan-id

Enters VLAN configuration mode.

Step 15

fcoe vsan vsan-id

Create FCoE VLAN and map FCoE VLAN to VSAN

Step 16

exit

Exits the VLAN configuration mode.

Configuring the FCoE NPV Edge Switch

Perform the following steps to configure an FCoE NPV edge switch.

SUMMARY STEPS

1.    install feature-set fcoe-npv

2.    feature-set fcoe-npv

3.    vsan database

4.    vsan vsan-id

5.    exit

6.    vlan vlan-id

7.    fcoe vsan vsan-id

8.    exit

9.    interface ethernet slot/port

10.    switchport mode trunk

11.    exit

12.    interface vfc vfc-id

13.    switchport mode NP

14.    bind interface ethernet slot/port

15.    exit

16.    interface ethernet slot/port

17.    switchport mode trunk

18.    exit

19.    interface vfc vfc-id

20.    switchport trunk allowed vsan vsan-id

21.    bind interface ethernet slot/port

22.    exit

23.    vsan database

24.    vsan vsan-id interface vfc vfc-id

25.    vsan vsan-id interface vfc vfc-id

26.    exit

DETAILED STEPS

Command or Action

Purpose

Step 1

install feature-set fcoe-npv

Installs FCoE NPV.

Step 2

feature-set fcoe-npv

Enable FCoE NPV.

Note:
When enabling FCoE NPV for Cisco NX-OS 7.0(3)I4(1) and later releases, the following BCM settings are required per FCoE VLAN:

LEARN_DISABLE=1
L2_NON_UCAST_DROP=1
L2_MISS_DROP=1

- Ethernet VLANs do not require these BCM settings.

Step 3

vsan database

Enters VSAN configuration mode.

Step 4

vsan vsan-id

Create VSAN.

Step 5

exit

Exits the VSAN configuration mode.

Step 6

vlan vlan-id

Enters VLAN configuration mode.

Step 7

fcoe vsan vsan-id

Create FCoE VLAN and map it to VSAN

Step 8

exit

Exits the VSAN configuration mode.

Step 9

interface ethernet slot/port

Enters interface configuration mode.

Step 10

switchport mode trunk

Set switch side physical interface to trunk mode

Step 11

exit

Exits the VSAN configuration mode.

Step 12

interface vfc vfc-id

Enters interface configuration mode.

Step 13

switchport mode NP

Set vFC port mode of P2 to VNP

Step 14

bind interface ethernet slot/port

Set vFC port mode of P2 to VNP

Important:
The bind interface ethernet command is not required for an implicit vFC configuration.

Step 15

exit

Exits the interface configuration mode.

Step 16

interface ethernet slot/port

Enters interface configuration mode.

Step 17

switchport mode trunk

Set server side physical interface to trunk mode

Step 18

exit

Exits the interface configuration mode.

Step 19

interface vfc vfc-id

Enters interface configuration mode.

Step 20

switchport trunk allowed vsan vsan-id

Configure vFC port P3 to allow VSAN 100

Step 21

bind interface ethernet slot/port

Configure vFC port P3 to allow VSAN 100

Important:
The bind interface ethernet command is not required for an implicit vFC configuration.

Step 22

exit

Exits the interface configuration mode.

Step 23

vsan database

Enters VSAN configuration mode.

Step 24

vsan vsan-id interface vfc vfc-id

Add port VSAN 100 to VF port.

Step 25

vsan vsan-id interface vfc vfc-id

Add VNP port to VSAN 100.

Note:
This step is optional. The default port VSAN is 1 and is preferable for the VNP port.

Step 26

exit

Exits the VSAN configuration mode.

Configuring a Pause Frame Timeout Value

You can enable or disable a pause frame timeout value on a port. The system periodically checks the ports for a pause condition and enables a pause frame timeout on a port if it is in a continuous pause condition for a configured period of time. This situation results in all frames that come to that port getting dropped in the egress. This function empties the buffer space in the ISL link and helps to reduce the fabric slowdown and the congestion on the other unrelated flows using the same link.

Note

- Beginning with Cisco NX-OS 7.0(3)I5(1), configuring a pause frame timeout value is supported on N9K-C93180YC-EX.

- Beginning with Cisco NX-OS 7.0(3)I5(2), configuring a pause frame timeout value is supported on N9K-X9732C-EX.

- Beginning with Cisco NX-OS 7.0(3)I6(1), configuring a pause frame timeout value is supported on N9K-C93180LC-EX

When a pause condition is cleared on a port or when a port flaps, the system disables the pause frame timeout on that particular port.

The pause frame timeout is disabled by default. We recommend that you retain the default configuration for the ISLs and configure a value that does not exceed the default value for the edge ports.

For a faster recovery from the slow drain device behavior, you should configure a pause frame timeout value because it drops all the frames in the edge port that face the slow drain whether the frame is in the switch for a congested timeout or not. This process instantly clears the congestion in the ISL.

Use the no system default interface pause timeout milliseconds mode edge command to disable the pause frame timeout value on the edge ports. The default pause timeout value is 500 milliseconds.

SUMMARY STEPS

1.    switch# configure terminal

2.    switch# system default interface pause timeout milliseconds mode edge

3.    switch# system default interface pause mode edge

4.    switch# no system default interface pause timeout milliseconds mode edge

5.    switch# no system default interface pause mode edge

DETAILED STEPS

Command or Action

Purpose

Step 1

switch# configure terminal

Enters global configuration mode.

Step 2

switch# system default interface pause timeout milliseconds mode edge

Configures a new pause frame timeout value in milliseconds and the port mode for the device.

Note:
Timeout value is specified in multiples of 100 (range is 100-500).

Note:
The system default interface pause timeout milliseconds mode core command is not supported.

Step 3

switch# system default interface pause mode edge

Configures the default pause frame timeout value in milliseconds and the port mode for the device.

Step 4

switch# no system default interface pause timeout milliseconds mode edge

Disables the pause frame timeout for the device.

Step 5

switch# no system default interface pause mode edge

Disables the default pause frame timeout for the device.

This example shows how to configure a pause frame timeout value:

switch# configure terminal
switch(config)# system default interface pause timeout 500 mode edge
switch(config)# system default interface pause mode edge
switch(config)# no system default interface pause timeout 500 mode edge
switch(config)# no system default interface pause mode edge
switch(config)# end

This example shows how to display pause frame timeout information:

switch#(config-if)# attach module 1
module-1# sh creditmon interface ethernet 1/35

Ethernet1/35: PORT is EDGE, xoff_hits=2
flush-status        : OFF
total_xoff_hits     : 2
(cntr) pause frames : 832502
(cntr) pause quanta : 1962909 milli-seconds
(cntr) force drops  : 94320764
(cntr-pg) to_drops  : 0
DBG_xoff_hit_cnt    : 0
DBG_xoff_hit_time   : 274
DBG_port_fc_mode    : 2
DBG_force_tmo_val   : 300 milli-seconds
CFG_congestion_tmo  : 0 milli-seconds

This example shows how to display pause frame timeout information:

switch(config-if)# attach module 1
module-1#
module-1# sh creditmon interface all
Ethernet1/1: PORT is NONE, xoff_hits=0
Ethernet1/2: PORT is NONE, xoff_hits=0
Ethernet1/3: PORT is NONE, xoff_hits=0
Ethernet1/4: PORT is NONE, xoff_hits=0
Ethernet1/5: PORT is NONE, xoff_hits=0
Ethernet1/6: PORT is NONE, xoff_hits=0
Ethernet1/7: PORT is NONE, xoff_hits=0
Ethernet1/8: PORT is NONE, xoff_hits=0
Ethernet1/9: PORT is NONE, xoff_hits=0
Ethernet1/10: PORT is NONE, xoff_hits=0
Ethernet1/11: PORT is NONE, xoff_hits=0
Ethernet1/12: PORT is NONE, xoff_hits=0
Ethernet1/13: PORT is NONE, xoff_hits=0
Ethernet1/14: PORT is NONE, xoff_hits=0
Ethernet1/15: PORT is NONE, xoff_hits=0
Ethernet1/16: PORT is NONE, xoff_hits=0
Ethernet1/17: PORT is NONE, xoff_hits=0
Ethernet1/18: PORT is NONE, xoff_hits=0
Ethernet1/19: PORT is NONE, xoff_hits=0
Ethernet1/20: PORT is NONE, xoff_hits=0
Ethernet1/21: PORT is NONE, xoff_hits=0
Ethernet1/22: PORT is NONE, xoff_hits=0
Ethernet1/23: PORT is NONE, xoff_hits=0
Ethernet1/24: PORT is NONE, xoff_hits=0
Ethernet1/25: PORT is NONE, xoff_hits=0
Ethernet1/26: PORT is NONE, xoff_hits=0
Ethernet1/27: PORT is NONE, xoff_hits=0
Ethernet1/28: PORT is NONE, xoff_hits=0
Ethernet1/29: PORT is NONE, xoff_hits=0
Ethernet1/30: PORT is NONE, xoff_hits=0
Ethernet1/31: PORT is NONE, xoff_hits=0
Ethernet1/32: PORT is NONE, xoff_hits=0
Ethernet1/33: PORT is NONE, xoff_hits=0
Ethernet1/34: PORT is NONE, xoff_hits=0
Ethernet1/35: PORT is NONE, xoff_hits=0
Ethernet1/36: PORT is NONE, xoff_hits=0
Ethernet1/37: PORT is NONE, xoff_hits=0
Ethernet1/38: PORT is NONE, xoff_hits=0
Ethernet1/39: PORT is NONE, xoff_hits=0
Ethernet1/40: PORT is NONE, xoff_hits=0
Ethernet1/41: PORT is NONE, xoff_hits=0
Ethernet1/42: PORT is NONE, xoff_hits=0
Ethernet1/43: PORT is NONE, xoff_hits=0
Ethernet1/44: PORT is NONE, xoff_hits=0
Ethernet1/45: PORT is NONE, xoff_hits=0
Ethernet1/46: PORT is NONE, xoff_hits=0
Ethernet1/47: PORT is NONE, xoff_hits=0
Ethernet1/48: PORT is NONE, xoff_hits=0
Ethernet1/49: PORT is NONE, xoff_hits=0
Ethernet1/49/2: PORT is NONE, xoff_hits=0
Ethernet1/49/3: PORT is NONE, xoff_hits=0
Ethernet1/49/4: PORT is NONE, xoff_hits=0
Ethernet1/50: PORT is NONE, xoff_hits=0
Ethernet1/50/2: PORT is NONE, xoff_hits=0
Ethernet1/50/3: PORT is NONE, xoff_hits=0
Ethernet1/50/4: PORT is NONE, xoff_hits=0
Ethernet1/51: PORT is NONE, xoff_hits=0
Ethernet1/51/2: PORT is NONE, xoff_hits=0
Ethernet1/51/3: PORT is NONE, xoff_hits=0
Ethernet1/51/4: PORT is NONE, xoff_hits=0
Ethernet1/52: PORT is NONE, xoff_hits=0
Ethernet1/52/2: PORT is NONE, xoff_hits=0
Ethernet1/52/3: PORT is NONE, xoff_hits=0
Ethernet1/52/4: PORT is NONE, xoff_hits=0
Ethernet1/53: PORT is NONE, xoff_hits=0
Ethernet1/53/2: PORT is NONE, xoff_hits=0
Ethernet1/53/3: PORT is NONE, xoff_hits=0
Ethernet1/53/4: PORT is NONE, xoff_hits=0
Ethernet1/54: PORT is NONE, xoff_hits=0
Ethernet1/54/2: PORT is NONE, xoff_hits=0
Ethernet1/54/3: PORT is NONE, xoff_hits=0
Ethernet1/54/4: PORT is NONE, xoff_hits=0

module-1#

This example shows syslog messages that are displayed when a pause frame timeout occurs:

2016 Sep 14 09:58:05 N93180YC-2 %TAHUSD-SLOT1-2-TAHUSD_SYSLOG_CRIT:
Ethernet1/35, slowdrain xoff hit  ON (total xoff-hits:1)

2016 Sep 14 09:59:21 N93180YC-2 %TAHUSD-SLOT1-2-TAHUSD_SYSLOG_CRIT:
Ethernet1/35, slowdrain xoff hit OFF (last  xoff-time:76 seconds)

Verifying the FCoE NPV Configuration

To display FCoE/VPC configuration information, perform one of the following:

Command

Purpose

show fcoe

Displays status of Fibre Channel over Ethernet (FCoE) parameters on the switch.

show fcoe database

Displays content of the Fibre Channel over Ethernet (FCoE) database.

show int vfc vfc-id

Displays vFC interface information.

To display NPV configuration information, perform one of the following:

Command

Purpose

show npv status

Displays N Port Virtualization (NPV) current status

show npv traffic-map

Displays N Port Virtualization (NPV) traffic map.

show npv external-interface-usage server-interface if

Displays external vFC interfaces (NP interfaces) designated/allocated to the server vFC interface if through automatic or static allocation.

show npv external-interface-usage

Displays external vFC interfaces (NP interfaces) designated/allocated to all available server vFC interfaces through automatic or static allocation.

show npv flogi-table interface if

Displays host FLOGI table that lists server interface; VSAN; fcid allocated to the initiator connected to the server interface; PWWN and NWWN of initiator; and external interface/gateway on NPV switch designated to the server interface.

show npv flogi-table vsan vsan

Displays information about the aging time set for the MAC address entries.

show npv flogi-table

Displays information about N Port Virtualization (NPV) FLOGI session.

show fcoe-npv issu-impact

Displays information about VNP ports where FKA is disabled.

FCoE NPV Core Switch and FCoE NPV Edge Switch Configuration Example

Configuring FCoE NPV Core Switch and FCoE NPV Edge Switch

- Configure NPV core switch:

- Enable NPIV

npv-core(config)# feature npiv

- Set physical interface mode to trunk

npv-core(config)# interface Eth 1/20
npv-core(config)# switchport mode trunk

- Set vFC port mode of P1 to VF

npv-core(config)# interface vfc1201
npv-core(config)# bind interface Eth1/20

- Create VSAN and add vFC into VSAN

npv-core(config)# vsan database
npv-core(config-vsan-db)# vsan 100
npv-core(config-vsan-db)# vsan 100 interface vfc1201

- Create FCoE VLAN & map it to VSAN

npv-core(config)# vlan 100
npv-core(config-vlan)# fcoe vsan 100

- Configure FCoE NPV switch:

- Install FCoE NPV

npv(config)# install feature-set fcoe-npv

- Enable FCoE NPV

npv(config)# feature-set fcoe-npv

- Create VSAN

npv(config)# vsan database
npv(config-vsan-db)# vsan 100

- Create FCoE VLAN and map it to VSAN

npv(config)# vlan 100
npv(config-vlan)# fcoe vsan 100

- Set switch side physical interface to trunk mode

npv(config)# interface Eth1/20
npv(config-if)# switchport mode trunk

- Set vFC port mode of P2 to VNP

npv(config)# interface vfc1201
npv(config-if)# switchport mode NP
npv(config-if)# bind interface Eth1/20

- Set server side physical interface to trunk mode

npv(config)# interface Eth1/1
npv(config-if)# switchport mode trunk

- Configure vFC port P3 to allow VSAN 100

npv(config)# interface vfc11
npv(config-if)# switchport trunk allowed vsan 100
npv(config-if)# bind interface Eth1/1

- Add both VNP and VF ports into VSAN 100

npv(config)# vsan database
npv(config-vsan-db)# vsan 100 interface vfc1201
npv(config-vsan-db)# vsan 100 interface vfc11

FCoE NPV Core Switch and FCoE NPV Edge Switch with Implicit vFC Configuration Example

Configuring FCoE NPV Core Switch and FCoE NPV Edge Switch with Implicit vFC

- Configure NPV core switch:

- Enable NPIV

npv-core(config)# feature npiv

- Set physical interface mode to trunk

npv-core(config)# interface Eth 1/20
npv-core(config)# switchport mode trunk

- Set vFC port mode of P1 to VF (implicit vFC)

npv-core(config)# interface vfc 1/20

- Create VSAN and add vFC into VSAN

npv-core(config)# vsan database
npv-core(config-vsan-db)# vsan 100
npv-core(config-vsan-db)# vsan 100 interface vfc 1/20

- Create FCoE VLAN & map it to VSAN

npv-core(config)# vlan 100
npv-core(config-vlan)# fcoe vsan 100

- Configure FCoE NPV switch:

- Install FCoE NPV

npv(config)# install feature-set fcoe-npv

- Enable FCoE NPV

npv(config)# feature-set fcoe-npv

- Create VSAN

npv(config)# vsan database
npv(config-vsan-db)# vsan 100

- Create FCoE VLAN and map it to VSAN

npv(config)# vlan 100
npv(config-vlan)# fcoe vsan 100

- Set switch side physical interface to trunk mode

npv(config)# interface Eth 1/20
npv(config-if)# switchport mode trunk

- Set vFC port mode of P2 to VNP (implicit vFC)

npv(config)# interface vfc 1/20
npv(config-if)# switchport mode NP

- Set server side physical interface to trunk mode

npv(config)# interface Eth 1/1
npv(config-if)# switchport mode trunk

- Configure vFC port P3 to allow VSAN 100 (implicit vFC)

npv(config)# interface vfc 1/1
npv(config-if)# switchport trunk allowed vsan 100

- Add both VNP and VF ports into VSAN 100

npv(config)# vsan database
npv(config-vsan-db)# vsan 100 interface vfc 1/20
npv(config-vsan-db)# vsan 100 interface vfc 1/1

Chapter 3. Configuring FCoE VLANs and Virtual Interfaces

This chapter contains the following sections:

- Information About Virtual Interfaces

- Guidelines and Limitations for FCoE VLANs and Virtual Interfaces

- Verifying the Virtual Interface

- Mapping VSANs to VLANs Example Configuration

- SAN Boot with vPC

Information About Virtual Interfaces

Cisco Nexus devices support Fibre Channel over Ethernet (FCoE), which allows Fibre Channel and Ethernet traffic to be carried on the same physical Ethernet connection between the switch and the servers.

The Fibre Channel portion of FCoE is configured as a virtual Fibre Channel interface. Logical Fibre Channel features (such as interface mode) can be configured on virtual Fibre Channel interfaces.

A virtual Fibre Channel interface must be bound to an interface before it can be used. The binding is to a physical Ethernet interface (when the converged network adapter (CNA) is directly connected to the Cisco Nexus device), a MAC address (when the CNA is remotely connected over a Layer 2 bridge), or an EtherChannel when the CNA connects to the Fibre Channel Forwarder (FCF) over a virtual port channel (vPC).

- Information About Shutting Down LAN Traffic

Information About Shutting Down LAN Traffic

Converged Network Adapters (CNA) enable both FCoE and LAN traffic (Unified I/O) to co-exist over a physical link.

In vPC configurations with CNAs, network parameters need to be consistent across peer switches. If the system detects an inconsistency, the secondary vPC leg goes down. Since vPC legs carry both FCOE and LAN traffic, the FCoE link goes down also.

To avoid having the FCoE link go down in this situation, you can use the shutdown lan command to shutdown only the LAN traffic on port-channels and individual Ethernet ports.

Note

When vPC triggers the vPC secondary leg to be brought down, only the Ethernet VLANs are brought down for the secondary vPC leg. FCoE/storage VLANs of the secondary vPC leg remain up.

- Notes About the shutdown lan Command

- Examples of Shutdown LAN Traffic

- Examples of Verifying Shutdown LAN Traffic

Notes About the shutdown lan Command

- The shutdown lan command is only configurable on port-channel interfaces, FEX HIF ports, or on individual Ethernet interfaces that vFC interfaces are bound upon.

- The shutdown lan command is only configurable on port-channel interfaces or on individual Ethernet interfaces that are in an operational trunking state.

- The shutdown lan command cannot be enabled on the secondary vPC leg, if the vPC enabled shutdown lan is applied on the secondary vPC leg.

- A vPC enabled shutdown LAN is not operable if the shutdown lan command is applied on the secondary vPC leg.

- The shutdown lan command is not configurable on port-channel members.

- The shutdown lan command default is no shutdown lan (shutdown lan is disabled).

- The shutdown lan command has a prerequisite that the Link Layer Discovery Protocol (LLDP) feature be enabled.

- A port with a shutdown LAN configuration enabled cannot be added to a port channel.

- The shutdown LAN enable/disable configuration is on a per interface basis.

- The shutdown LAN trigger can be configured or by a vPC secondary leg bringdown request. (Under secondary interface, use the vpc peer-link bringdown command.)

- If a shutdown LAN is configured on an interface, a no shutdown lan command on the interface does not bring up the LAN VLANs. In this case, the no shutdown lan command has no impact.

- A shutdown LAN is triggered when a Type-1 inconsistency occurs in a VPC network.

Examples of Shutdown LAN Traffic

- Shutdown the LAN traffic on port-channel.

switch(config)#interface port-channel 955
switch(config-if)# shutdown lan

- Shutdown the LAN traffic on individual Ethernet port.

switch(config)#interface Ethernet 2/5
switch(config-if)# shutdown lan

Examples of Verifying Shutdown LAN Traffic

- Verifying when the shutdown lan command is issued on port-channel 955 with Ethernet interface 2/5 as member.

switch# sh interface port-channel 955 | grep LAN
All LAN VLANs are administratively shut

switch# sh interface ethernet 2/5 | grep LAN
All LAN VLANs are administratively shut

switch# sh run interface port-channel 955 | grep shut
shutdown lan

switch# sh run interface e2/5 | grep shut
shutdown lan

- Verifying when the vPC triggers shutdown LAN on the secondary vPC leg (port-channel 231 with Ethernet 2/31 as member).

switch# sh interface port-channel 231 | grep LAN
All LAN VLANs are administratively shut

Guidelines and Limitations for FCoE VLANs and Virtual Interfaces

FCoE VLANs and Virtual Fiber Channel (vFC) interfaces have these guidelines and limitations:

- Each vFC interface must be bound to an FCoE-enabled Ethernet or EtherChannel interface or to the MAC address of a remotely connected adapter. FCoE is supported on 10-Gigabit and 40-Gigabit Ethernet interfaces.

The Ethernet or EtherChannel interface that you bind to the vFC interface must be configured as follows:

- The Ethernet or EtherChannel interface must be a trunk port (use the switchport mode trunk command).

- The FCoE VLAN that corresponds to a vFC’s VSAN must be in the allowed VLAN list.

- You must not configure an FCoE VLAN as the native VLAN of the trunk port.

Note

The native VLAN is the default VLAN on a trunk. Any untagged frames transit the trunk as native VLAN traffic.

- You should use an FCoE VLAN only for FCoE.

- Do not use the default VLAN, VLAN1, as an FCoE VLAN.

- You must configure the Ethernet interface as PortFast (use the spanning-tree port type edge trunk command).

Note

You are not required to configure trunking on the server interface even if the switch interface is configured with trunking enabled. All non-FCoE traffic from the server is passed on the native VLAN.

- The vFC interface can be bound to Ethernet port channels with multiple member ports connected to FCoE Initialization Protocol (FIP) snooping bridges.

- For VF mode, each vFC interface is associated with only one VSAN.

For VNP mode, each vFC interface is associated with multiple VSANs.

- You must map any VSAN with associated vFC interfaces to a dedicated FCoE-enabled VLAN.

- FCoE is not supported on private VLANs.

- If the converged access switches (in the same SAN fabric or in another) need to be connected to each other over Ethernet links for a LAN alternate path, then you must explicitly configure such links to exclude all FCoE VLANs from membership.

- You must use separate FCoE VLANs for FCoE in SAN-A and SAN-B fabrics.

- FCoE connectivity to pre-FIP CNAs over virtual port channels (vPCs) is not supported.

- FCoE VLANs do not support Multiple Spanning Trees (MST). Creating an MST instance for an FCoE VLAN might cause SAN traffic disruption.

Note

Virtual interfaces are created with the administrative state set to down. You must explicitly configure the administrative state to bring the virtual interface into operation.

Mapping a VSAN to a VLAN

A unique, dedicated VLAN must be configured at every converged access switch to carry traffic for each VSAN in the SAN (for example, VLAN 1002 for VSAN 1, VLAN 1003 for VSAN 2, and so on).

SUMMARY STEPS

1.    switch# configure terminal

2.    switch(config)# vlan vlan-id

3.    switch(config-vlan)# fcoe [vsan vsan-id]

4.    switch(config-vlan)# exit

5.    (Optional) switch(config-vlan)# show vlan fcoe

6.    (Optional) switch(config)# copy running-config startup-config

DETAILED STEPS

Command or Action

Purpose

Step 1

switch# configure terminal

Enters global configuration mode.

Step 2

switch(config)# vlan vlan-id

Enters VLAN configuration mode. The VLAN number range is from 1 to 4096.

Step 3

switch(config-vlan)# fcoe [vsan vsan-id]

Enables FCoE for the specified VLAN. If you do not specify a VSAN number, a mapping is created from this VLAN to the VSAN with the same number.

Configures the mapping from this VLAN to the specified VSAN.

Step 4

switch(config-vlan)# exit

Exits VLAN configuration mode. You must exit this mode to execute the configured commands on your Cisco Nexus device.

Step 5

switch(config-vlan)# show vlan fcoe

(Optional)
Displays information about the FCoE configuration for a VLAN.

Step 6

switch(config)# copy running-config startup-config

(Optional)
Saves the change persistently through reboots and restarts by copying the running configuration to the startup configuration.

This example shows how to map VLAN 200 to VSAN 2:

switch(config)# vlan 200

switch(config-vlan)# fcoe vsan 2

Creating a Virtual Fibre Channel Interface

You can create a virtual Fibre Channel interface. You must bind the virtual Fibre Channel interface to a host MAC address before it can be used.

SUMMARY STEPS

1.    switch# configure terminal

2.    switch(config)# interface vfc vfc-id

3.    switch(config-if)# bind {interface {ethernet slot/port | port-channel channel-number} | mac-address MAC-address}

4.    (Optional) switch(config-if)# no bind {interface {ethernet slot/port | port-channel channel-number} | mac-address MAC-address}

5.    (Optional) switch(config)# no interface vfc vfc-id

DETAILED STEPS

Command or Action

Purpose

Step 1

switch# configure terminal

Enters global configuration mode.

Step 2

switch(config)# interface vfc vfc-id

Creates a virtual Fibre Channel interface (if it does not already exist) and enters interface configuration mode.

The virtual Fibre Channel interface ID range is from 1 to 8192.

Step 3

switch(config-if)# bind {interface {ethernet slot/port | port-channel channel-number} | mac-address MAC-address}

Binds the virtual Fibre Channel interface to the specified interface.

Note:
If this is a 10G breakout port, the slot/port syntax is slot/QSFP-module/port.

Step 4

switch(config-if)# no bind {interface {ethernet slot/port | port-channel channel-number} | mac-address MAC-address}

(Optional)
Unbinds the virtual Fibre Channel interface from the specified interface.

Note:
If this is a 10G breakout port, the slot/port syntax is slot/QSFP-module/port.

Step 5

switch(config)# no interface vfc vfc-id

(Optional)
Deletes a virtual Fibre Channel interface.

- This example shows how to bind a virtual Fibre Channel interface to an Ethernet interface:

switch# configure terminal
switch(config)# interface vfc 4
switch(config-if)# bind interface ethernet 1/4

- This example shows how to bind a virtual Fibre Channel interface to create a vPC:

switch# configure terminal
switch(config)# interface vfc 3
switch(config-if)# bind interface port-channel 1

- This example shows how to bind a virtual Fibre Channel interface to a MAC address:

switch# configure terminal
switch(config)# interface vfc 2
switch(config-if)# bind mac-address 00:0a:00:00:00:36

Note

A MAC bound vFC can be configured to a host sitting behind a FSB.

Note

When both MAC bound vFC and port-bound vFC are configured for the same interface, the port-bound vFC takes precedence.

Note

As a best practice, you should have either a MAC bound vFC or a port- bound vFC for a physical ethernet port or a port-channel, but not have both.

- This example shows how to delete a virtual Fibre Channel interface:

switch# configure terminal
switch(config)# no interface vfc 4

Associating a Virtual Fibre Channel Interface to a VSAN

A unique, dedicated VLAN must be configured at every converged access switch to carry traffic for each Virtual Fabric (VSAN) in the SAN (for example, VLAN 1002 for VSAN 1, VLAN 1003 for VSAN 2, and so on).

SUMMARY STEPS

1.    switch# configure terminal

2.    switch(config)# vsan database

3.    switch(config-vsan)# vsan vsan-id interface vfc vfc-id

4.    (Optional) switch(config-vsan)# no vsan vsan-id interface vfc vfc-id

DETAILED STEPS

Command or Action

Purpose

Step 1

switch# configure terminal

Enters global configuration mode.

Step 2

switch(config)# vsan database

Enters VSAN configuration mode.

Step 3

switch(config-vsan)# vsan vsan-id interface vfc vfc-id

Configures the association between the VSAN and virtual Fibre Channel interface.

The VSAN number must map to a VLAN on the physical Ethernet interface that is bound to the virtual Fibre Channel interface.

Step 4

switch(config-vsan)# no vsan vsan-id interface vfc vfc-id

(Optional)
Disassociates the connection between the VSAN and virtual Fibre Channel interface.

This example shows how to associate a virtual Fibre Channel interface to a VSAN:

switch# configure terminal

switch(config)# vsan database

switch(config-vsan)# vsan 2 interface vfc 4

Creating an Implicit Virtual Fibre Channel Port Channel Interface

You can create a virtual Fibre Channel (vFC), and implicitly bind it to an Ethernet interface or a port-channel using a single command. For this, the vFC identifier must match the Ethernet interface or port-channel identifier. The Ethernet interface can be a module (slot or port).

Configuring virtual Fibre Channel Interface

Before You Begin

- Ensure you have installed the correct license for FCoE.

- Ensure you have enabled FCoE.

Step 1

Enter global configuration mode:
switch# configure terminal

Step 2

Create a vFC (if it does not already exist):
switch(config) # interface vfc {id | slot/port | chassis/slot/port

Step 3

Bring up the vFC interface:
switch(config-if) # no shutdown

Step 4

Exit the interface configuration mode:
switch(config-if) # exit

Configuring virtual Fibre Channel Interface

This example shows how to implicitly bind a virtual Fibre Channel interface to an Ethernet interface:

switch# configure terminal
switch(config)# interface vfc 1/1/1
switch(config-if)# no shutdown
.
.
.
.
Switch# show interface vfc 1/1/1
vfc101/1/1 is trunking
Bound interface is Ethernet1/1/1
Hardware is Ethernet
Port WWN is 20:00:00:2a:6a:15:d2:7b
Admin port mode is F, trunk mode is on
snmp link state traps are enabled
Port mode is TF
Port vsan is 600
Trunk vsans (admin allowed and active) (1,500,600)
Trunk vsans (up)                       (600)
Trunk vsans (isolated)                 ()
Trunk vsans (initializing)             (1,500)
1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
49 frames input, 5516 bytes
0 discards, 0 errors
49 frames output, 5772 bytes
0 discards, 0 errors
last clearing of "show interface" counters Fri Oct 30 06:19:33 2015
Interface last changed at Fri Oct 30 06:19:33 2015

Configuring virtual Fibre Channel – Port Channel Interface

Step 1

Enter global configuration mode:
switch# configure terminal

Step 2

Create a vFC that implicitly binds to the Ethernet port-channel based on its number:
The port number range is from 1 to 4096.

switch(config) # interface vfc-port-channel port number

Step 3

Bring up the vFC port:
switch(config-if) # no shutdown

Step 4

Exit from the current interface configuration mode:
switch(config-if) # exit

Configuring virtual Fibre Channel - Port Channel Interface

The example shows how you can create a vFC-port-channel that implicitly binds to Ethernet port-channel:

switch# configure terminal
switch(config)# interface vfc-port-channel 300
switch(config-if)# no shutdown
.
.
.
.
switch# show interface vfc-port-channel 258

vfc-po258 is trunking
Bound interface is port-channel258
Hardware is Ethernet
Port WWN is 21:01:8c:60:4f:59:31:3f
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 600
Trunk vsans (admin allowed and active) (1,100,500,600)
Trunk vsans (up)                       (600)
Trunk vsans (isolated)                 ()
Trunk vsans (initializing)             (1,100,500)
1 minute input rate 3448 bits/sec, 431 bytes/sec, 4 frames/sec
1 minute output rate 9064 bits/sec, 1133 bytes/sec, 4 frames/sec
977735 frames input, 77172556 bytes
0 discards, 0 errors
977733 frames output, 205924892 bytes
0 discards, 0 errors
last clearing of "show interface" counters Thu Oct 29 06:35:41 2015
Interface last changed at Thu Oct 29 06:35:41 2015

Verifying the Virtual Interface

To display configuration information about virtual interfaces, perform one of the following tasks:

Command

Purpose

switch# show interface vfc vfc-id

Displays the detailed configuration of the specified Fibre Channel interface.

switch# show interface brief

Displays the status of all interfaces.

switch# show vlan fcoe

Displays the mapping of FCoE VLANs to VSANs.

This example shows how to display a virtual Fibre Channel interface bound to an Ethernet interface:

switch(config-if)# sh int vfc 172

vfc172 is trunking (Not all VSANs UP on the trunk)
Bound interface is Ethernet1/72
Hardware is Ethernet
Port WWN is 20:ab:e0:0e:da:4a:5d:9d
Admin port mode is F, trunk mode is on
snmp link state traps are enabled
Port mode is TF
Port vsan is 200
Speed is auto
Trunk vsans (admin allowed and active) (1,10,100,200)
Trunk vsans (up)                       (200)
Trunk vsans (isolated)                 ()
Trunk vsans (initializing)             (1,10,100)
799 fcoe in packets
80220 fcoe in octets
2199 fcoe out packets
2219828 fcoe out octets
Interface last changed at Thu Sep 15 08:52:51 2016

This example shows how to display a virtual Fibre Channel interface bound to a MAC address:

switch(config-if)# sh int vfc 132

vfc132 is trunking (Not all VSANs UP on the trunk)
Bound MAC is 000e.1e1b.c1c9
Hardware is Ethernet
Port WWN is 20:83:00:2a:10:7a:89:bf
Admin port mode is F, trunk mode is on
snmp link state traps are enabled
Port mode is TF
Port vsan is 2101
Speed is auto
Trunk vsans (admin allowed and active) (1,2001-2003,2101-2103)
Trunk vsans (up)                       (2101)
Trunk vsans (isolated)                 ()
Trunk vsans (initializing)             (1,2001-2003,2102-2103)
Interface last changed at Wed Sep 14 12:14:29 2016

This example shows how to display the status of all the interfaces on the switch (some output has been removed for brevity):

switch# show interface brief

-------------------------------------------------------------------------------

Interface  Vsan   Admin  Admin   Status          SFP    Oper  Oper   Port

Mode   Trunk                          Mode  Speed  Channel

Mode                                 (Gbps)

-------------------------------------------------------------------------------

fc3/1      1      auto   on      trunking         swl    TE      2    --

fc3/2      1      auto   on      sfpAbsent        --     --           --

...

fc3/8      1      auto   on      sfpAbsent        --     --           --

-------------------------------------------------------------------------------

Interface                Status     IP Address        Speed    MTU   Port

Channel

-------------------------------------------------------------------------------

Ethernet1/1              hwFailure  --                --       1500  --

Ethernet1/2              hwFailure  --                --       1500  --

Ethernet1/3              up         --                10000    1500  --

...

Ethernet1/39             sfpIsAbsen --                --       1500  --

Ethernet1/40             sfpIsAbsen --                --       1500  --

-------------------------------------------------------------------------------

Interface                Status     IP Address        Speed    MTU

-------------------------------------------------------------------------------

mgmt0                    up         172.16.24.41      100      1500

-------------------------------------------------------------------------------

-------------------------------------------------------------------------------

Interface  Vsan   Admin  Admin   Status          SFP    Oper  Oper   Port

Mode   Trunk                          Mode  Speed  Channel

Mode                                 (Gbps)

-------------------------------------------------------------------------------

vfc 1        1      F      --      down             --     --           --

...

This example shows how to display the mapping between the VLANs and VSANs on the switch:

switch# show vlan fcoe

VLAN      VSAN      Status

--------  --------  --------

15        15        Operational

20        20        Operational

25        25        Operational

30        30        Non-operational

Mapping VSANs to VLANs Example Configuration

The following example shows how to configure the FCoE VLAN and a virtual Fibre Channel interface:

SUMMARY STEPS

1.    Enable the associated VLAN and map the VLAN to a VSAN.

2.    Configure the VLAN on a physical Ethernet interface.

3.    Create a virtual Fibre Channel interface and bind it to a physical Ethernet interface.

4.    Associate the virtual Fibre Channel interface to the VSAN.

5.    (Optional) Display membership information for the VSAN.

6.    (Optional) Display the interface information for the virtual Fibre Channel interface.

DETAILED STEPS

Step 1

Enable the associated VLAN and map the VLAN to a VSAN.

switch(config)# vlan 200

switch(config-vlan)# fcoe vsan 2

switch(config-vlan)# exit

Step 2

Configure the VLAN on a physical Ethernet interface.

switch# configure terminal

switch(config)# interface ethernet 1/4

switch(config-if)# spanning-tree port type edge trunk

switch(config-if)# switchport mode trunk

switch(config-if)# switchport trunk allowed vlan 1,200

switch(config-if)# exit

Step 3

Create a virtual Fibre Channel interface and bind it to a physical Ethernet interface.

switch(config)# interface vfc 4

switch(config-if)# bind interface ethernet 1/4

switch(config-if)# exit

Note:
By default, all virtual Fibre Channel interfaces reside on VSAN 1. If the VLAN to VSAN mapping is to a VSAN other than VSAN 1, then proceed to Step 4.

Step 4

Associate the virtual Fibre Channel interface to the VSAN.

switch(config)# vsan database

switch(config-vsan)# vsan 2 interface vfc 4

switch(config-vsan)# exit

Step 5

(Optional) Display membership information for the VSAN.

switch# show vsan 2 membership
vsan 2 interfaces
vfc 4

Step 6

(Optional) Display the interface information for the virtual Fibre Channel interface.

switch# show interface vfc 4

vfc4 is up
Bound interface is Ethernet1/4
Hardware is Virtual Fibre Channel
Port WWN is 20:02:00:0d:ec:6d:95:3f
Port WWN is 20:02:00:0d:ec:6d:95:3f
snmp link state traps are enabled
Port WWN is 20:02:00:0d:ec:6d:95:3f
APort WWN is 20:02:00:0d:ec:6d:95:3f
snmp link state traps are enabled
Port mode is F, FCID is 0x490100
Port vsan is 931
1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
0 frames input, 0 bytes 0 discards, 0 errors
0 frames output, 0 bytes 0 discards, 0 errors
Interface last changed at Thu Mar 11 04:44:42 2010

SAN Boot with vPC

Cisco Nexus 9000 Series devices support the SAN boot of initiators on Link Aggregation Control Protocol (LACP) based vPC. This limitation is specific to LACP-based port channels. The host-facing vFC interfaces are bound to port channel members instead of the port channel itself. This binding ensures that the host-side vFC comes up during a SAN boot as soon as the link on the CNA/Host Bus Adapter (HBA) comes up, without relying on the LACP-based port channel to form first.

Note

Cisco Nexus 9000 Series devices support the SAN boot of channel mode on also.

Note

LACP suspend-individual should be removed from the port-channel, otherwise the physical interface will be suspended when LACP BPDU is not received from the host.

- SAN Boot with vPC Configuration Example

SAN Boot with vPC Configuration Example

In this example, virtual Fibre Channel interface 1 is bound to physical Ethernet interface 1/2/4 on Fabric A and on interface 1/2/3 on Fabric B. The interface is also associated with virtual port channel 1 on both fabrics.

switch(config)# interface port-channel 1
switch(config-if)# no lacp suspend-individual
switch(config)# interface vfc 1
switch(config-if)# bind interface eth 1/2/4
switch(config)# interface eth 1/2/4
switch(config-if)# channel-group 1 mode active
switch(config-if)# interface port-channel 1
switch(config-if)# vpc 1
switch(config-if)#
switch(config)# interface port-channel 1
switch(config-if)# no lacp suspend-individual
switch(config)# interface vfc 1
switch(config-if)# bind interface eth 1/2/3
switch(config)# interface eth 1/2/3
switch(config-if)# channel-group 1 mode active
switch(config-if)# interface port-channel 1
switch(config-if)# vpc 1

Chapter 4. FCoE Over FEX

- Overview

- Guidelines and Limitations for FCoE Over FEX

Overview

The Fibre Channel over Ethernet (FCoE) over Fabric Extenders (FEX) feature allows Fibre Channel traffic to be carried on a FEX port. The FEX is connected to a Cisco Nexus 9000 device that is in FCoE NPV mode through a Fabric Port Channel (FPC). FCoE over FEX enables the provisioning of FCoE on host connections.

For more information about FEX, see the Cisco Nexus 2000 Series NX-OS Fabric Extender Configuration Guide for Cisco Nexus 9000 Series Switches.

- FCoE Over FEX with vPC

- LAN Shutdown

- FCoE Over FEX Topologies

FCoE Over FEX with vPC

FCoE over FEX with virtual Port Channel (vPC) allows Fibre Channel traffic to be carried over a FEX using a virtual Port Channel (vPC).

LAN Shutdown

The LAN shutdown feature detects the capability of the FCoE host to support Data Center Bridging (DCBX). DCBX allows the switch to send the LAN Logical Link status (LLS) messages in a type-length-value (TLV) format. The LAN shutdown feature enables bring up and bring down of LAN links on a unified link carrying both FCoE and LAN traffic. When you use the shutdown lan command, only the LAN traffic stops while the FCoE traffic continues.

The shutdown lan command is supported for FEX HIF ports and port-channels (7.0(3)I5(1) and later).

FCoE Over FEX Topologies

FCoE over FEX is supported in the following topologies:

- Straight Through FEX with Host Topology

- Straight Through FEX with Host VPC Topology

- Dual-Homed FEX Topology (Active/Active FEX Topology)

Straight Through FEX with Host Topology

The straight through FEX with host topology is supported with NX-OS 7.0(3)I5(1) and later.

Straight Through FEX with Host VPC Topology

The straight through FEX with host VPC topology is supported with NX-OS 7.0(3)I5(1) and later.

Dual-Homed FEX Topology (Active/Active FEX Topology)

The dual-homed FEX topology is supported with NX-OS 7.0(3)I5(2) and later with Cisco Nexus 9300 and 9300-EX Series switches.

The following topology shows that each FEX is dual-homed with two Cisco Nexus 9000 Series switches. The FEX-fabric interfaces for each FEX are configured as a vPC on both peer switches. The host interfaces on the FEX appear on both peer switches.

Note

The host interfaces configuration should be the same on both switches.

Dual-Homed FEX Topology

Operational vPC is mandatory for the dual-homed FEX topology. In the Dual-Homed FEX Active/Active Topology, the vPC is already operational. FEX 101 is dual-homed to both parent switches: N9k-1 and N9k-2 on FEX-fabric interfaces Ethernet 1/1.

Note

Only the following support an active-active FEX topology:

- N2K-C2232PP

- N2K-C2348UPQ

- NB22IBM

- NB22HP

Note

A port channel within the same FEX is supported on Cisco Nexus 2200 Series Fabric Extenders.

Guidelines and Limitations for FCoE Over FEX

- FCoE over FEX is supported on the following FEXs:

- N2K-C2348UPQ

- N2K-C2232PP

- N2K-B22IBM-P

- N2K-B22HP-P

- Starting in Release 7.0(3)I6(1), N9K-C93180LC-EX supports FCoE over FEX only on the N2K-C2348UPQ. Other FEX models are not supported with this device.

- FCoE over FEX is not supported on the following:

- N9K-C92160YC-X

- N9K-C9272Q

- N9K-C9236C

- N9K-X9732C-EX

- N9K-C93180LC-EX

- FEX HIF ports only support vFC in F mode. NP mode vFCs are not supported over FEX HIF ports.

- A Fabric Port Channel cannot exceed a maximum of eight member ports.

Note

For information about scalability, see the Cisco Nexus 9000 Series NX-OS Verified Scalability Guide.

- Information Before Configuring FCoE Over FEX

Information Before Configuring FCoE Over FEX

Note

This information is applicable only for Cisco Nexus 9500 switches and the following specific Cisco Nexus 9300 switches (not applicable to other Cisco Nexus 9000 switches):

- N9K-C93180LC-EX

- N9K-C9332PQ

- N9K-C9372PX

- N9K-C9372PX-E

- N9K-C9396PX

- Enable the Configuration

Using the fcoe enable-fex command results in disabling any load-balancing (based on SID, DID, OXID) on the switch.

Note

The fcoe enable-fex command is supported only in T2-based Cisco Nexus 9000 FCoE NPV switches (N9K-C9332PQ, N9K-C9372PX-E, and N9K-C9396PX).

switch(config)# fcoe enable-fex
Warning: Enabling FCoE over FEX may cause traffic drop. Continue [y/n]?

- Disable the Configuration

Using the no fcoe enable-fex command results in enabling any load-balancing (based on SID, DID, OXID) on the switch.

switch(config)# no fcoe enable-fex
Warning: Enabling FCoE over FEX may cause traffic drop. Continue [y/n]?

Configuring Straight Through FEX with Host

Note

Considerations for FEX Fabric Port Channel (FPC):

- Priority flow control (PFC) requires to be enabled explicitly for Pause to work.

interface "port-channel"
priority-flow-control mode on

- If the switch is connected to a host that does not have DCBX support, PFC needs to be enabled explicitly on HIF interfaces.

interface "hif interface"
priority-flow-control mode on

- The "lldp" feature should be enabled on the switch with the feature lldp command.

- First attach the interfaces to the fex-fabric port-channel and then enable the priority-flow-control mode with the priority-flow-control mode on command.

Before You Begin

- Ensure configuration of FEX. For more information, see the Cisco Nexus 2000 Series NX-OS Fabric Extender Configuration Guide for Cisco Nexus 9000 Series Switches.

- Ensure the configuration of FCoE NPV. For more information, see the section: Configuring FCoE NPV.

SUMMARY STEPS

1.    configure terminal

2.    interface port-channel <port_num>

3.    switchport

4.    switchport mode fex-fabric

5.    fex associate <fex_id>

6.    mtu 9216

7.    no shutdown

8.    exit

9.    interface ethernet slot/port

10.    switchport

11.    switchport mode fex-fabric

12.    fex associate <fex_id>

13.    mtu 9216

14.    channel-group <port_number>

15.    no shutdown

16.    exit

17.    interface ethernet chassis_id/slot/port_number

18.    switchport mode trunk

19.    service-policy type qos input fcoe

20.    no shutdown

21.    exit

DETAILED STEPS

Command or Action

Purpose

Step 1

configure terminal

Example:

switch# configure terminal
switch(config)#

Enters global configuration mode.

Step 2

interface port-channel <port_num>

Example:

switch(config)#interface port-channel 101

Creates port-channel and enters the interface configuration mode.

Step 3

switchport

Example:

switch(config-if)#switchport

Sets Layer 2 switching port.

Step 4

switchport mode fex-fabric

Example:

switch(config-if)#switchport mode fex-fabric

Sets the interface type to be a uplink port for a Fabric extender (FEX).

Step 5

fex associate <fex_id>

Example:

switch(config-if)#fex associate 101

Associates a Fabric Extender (FEX) to a Fabric interface.

Step 6

mtu 9216

Example:

switch(config-if)#mtu 9216

Configures the MTU value to that of jumbo frames to carry FCoE traffic through the NIF ports.

Step 7

no shutdown

Example:

switch(config-if)#no shutdown

Sets the port-channel to up (administratively).

Step 8

exit

Example:

switch(config-if)#exit

Exit the interface configuration mode.

Step 9

interface ethernet slot/port

Example:

switch(config)#interface Ethernet1/1

Enters interface configuration mode.

Step 10

switchport

Example:

switch(config-if)#switchport

Sets Layer 2 switching port.

Step 11

switchport mode fex-fabric

Example:

switch(config-if)#switchport mode fex-fabric

Sets the interface type to be an uplink port for a fabric extender (FEX).

Step 12

fex associate <fex_id>

Example:

switch(config-if)#fex associate 101

Associates a Fabric Extender (FEX) to a Fabric interface.

Step 13

mtu 9216

Example:

switch(config-if)#mtu 9216

Configures the MTU value to that of jumbo frames to carry FCoE traffic through the NIF ports.

Step 14

channel-group <port_number>

Example:

switch(config-if)channel-group 101

Sets the Fabric interface a member of the Fabric port channel.

Step 15

no shutdown

Example:

switch(config-if)#no shutdown

Sets the port-channel to up (administratively).

Step 16

exit

Example:

switch(config-if)#exit

Exit the interface configuration mode.

Step 17

interface ethernet chassis_id/slot/port_number

Example:

switch(config)interface ethernet 101/1/1

Configures a FEX satellite interface or HIF (host interface) port and enters interface configuration mode.

Step 18

switchport mode trunk

Example:

switch(config-if)#switchport mode trunk

Sets interface type to be a trunk port.

Step 19

service-policy type qos input fcoe

Example:

switch(config-if)# service-policy type qos input fcoe

Sets the QoS policy on the HIF port channel to a no drop policy.

Note:
For more information about queuing policy configurations, see the section: Configuring QoS for no-drop support.

Step 20

no shutdown

Example:

switch(config-if)#no shutdown

Sets the port-channel to up (administratively).

Step 21

exit

Example:

switch(config-if)#exit

Exit the interface configuration mode.

The following is an example of FEX bring up in straight through mode with a host.

install feature-set fex
feature-set fex

fex 101
pinning max-links 1
description "2232PP-1"

interface port-channel101
switchport
switchport mode fex-fabric
fex associate 101
mtu 9216

interface Ethernet1/1
switchport
switchport mode fex-fabric
fex associate 101
mtu 9216
channel-group 101
no shutdown

interface Ethernet101/1/1
switchport mode trunk
service-policy type qos input fcoe-qo-policy
no shutdown

Binding vFC to FEX Interface Explicitly

SUMMARY STEPS

1.    configure terminal

2.    interface vfc < vfc-id>

3.    bind interface ethernet [chassid-id/slot/port]

4.    no shutdown

5.    end

DETAILED STEPS

Command or Action

Purpose

Step 1

configure terminal

Example:

switch# configure terminal
switch(config)#

Enters global configuration mode.

Step 2

interface vfc < vfc-id>

Example:

N9k(config)# interface vfc 1

Creates a virtual Fibre Channel Interface and enters interface configuration mode.

The chassis-id range is from 101 to 199.

Step 3

bind interface ethernet [chassid-id/slot/port]

Example:

N9k(config-if)# bind interface ethernet101/1/1

Explicitly binds the virtual fibre channel interface to the specified interface. Use the no form of the command to unbind.

Chassis id range is from 101 to 199.

Step 4

no shutdown

Example:

switch(config-if)#no shutdown

Bring up the interface (administratively).

Step 5

end

Example:

N9k(config-if)#end

Return to exec mode.

interface vfc1
bind interface ethernet101/1/1
switchport trunk mode on
no shutdown

Binding VFC to FEX Interface Implicitly

SUMMARY STEPS

1.    configure terminal

2.    interface vfc < chassis-id>/<slot>/<port>

3.    no shutdown

4.    end

DETAILED STEPS

Command or Action

Purpose

Step 1

configure terminal

Example:

switch# configure terminal
switch(config)#

Enters global configuration mode.

Step 2

interface vfc < chassis-id>/<slot>/<port>

Example:

switch(config)# interface vfc 101/1/1

Creates a virtual fibre Channel Interface and enters interface configuration mode; implicitly binding it to the underlying ethernet interface (ethernet chassid-id/slot/port).

The chassis-id range is from 101 to 199.

Step 3

no shutdown

Example:

switch(config-if)#no shutdown

Bring up the interface (administratively).

Step 4

end

Example:

switch(config-if)#end

Return to exec mode.

interface vfc101/1/1
switchport trunk mode on
no shutdown

Binding vFC to MAC Address

A MAC address bound vFC can also be created for a FEX host interface (HIF) port.

Note

A MAC bound vFC can be configured to a host sitting behind a FSB.

When both MAC bound vFC and port-bound vFC are configured for the same interface, the port-bound vFC takes precedence.

As a best practice, you should have either a MAC bound vFC or a port- bound vFC for a physical ethernet port or a port-channel. However, you cannot have both.

SUMMARY STEPS

1.    configure terminal

2.    interface vfc <number>

3.    bind mac-address <mac-address>

DETAILED STEPS

Command or Action

Purpose

Step 1

configure terminal

Example:

Enters global configuration mode.

Step 2

interface vfc <number>

Creates a virtual fibre Channel Interface.

Step 3

bind mac-address <mac-address>

Binds MAC address.

The following example shows how to bind a virtual Fibre Channel interface to a MAC address:

switch# configure terminal
switch(config)# interface vfc 2
switch(config-if)# bind mac-address 00:0a:00:00:00:36

Configuring Straight Through FEX with Host vPC

Before You Begin

- Ensure configuration of vPC between two Cisco Nexus 9000 Series switches. For more information, see the Cisco Nexus 9000 Series NX-OS Interfaces Configuration Guide.

- Ensure configuration of FEX. For more information, see the Cisco Nexus 2000 Series NX-OS Fabric Extender Configuration Guide for Cisco Nexus 9000 Series Switches.

- Ensure the configuration of FCoE NPV. For more information, see the section: Configuring FCoE NPV.

SUMMARY STEPS

1.    configure terminal

2.    interface port-channel <port_num>

3.    switchport

4.    switchport mode fex-fabric

5.    fex associate <fex_id>

6.    mtu 9216

7.    no shutdown

8.    exit

9.    interface ethernet slot/port

10.    switchport

11.    switchport mode fex-fabric

12.    fex associate <fex_id>

13.    mtu 9216

14.    channel-group <port_number>

15.    no shutdown

16.    exit

17.    interface ethernet chassis_id/slot/port_number

18.    switchport mode trunk

19.    channel group <host_port_num>

20.    no shutdown

21.    exit

22.    interface port-channel <host_port_number>

23.    switchport

24.    switchport mode trunk

25.    service-policy type qos input fcoe

26.    vpc 3

27.    no shutdown

28.    exit

DETAILED STEPS

Command or Action

Purpose

Step 1

configure terminal

Example:

switch# configure terminal
switch(config)#

Enters global configuration mode.

Step 2

interface port-channel <port_num>

Example:

switch(config)#interface port-channel 101

Creates port-channel and enters the interface configuration mode.

Step 3

switchport

Example:

switch(config-if)#switchport

Sets Layer 2 switching port.

Step 4

switchport mode fex-fabric

Example:

switch(config-if)#switchport mode fex-fabric

Sets the interface type to be a uplink port for a Fabric extender (FEX).

Step 5

fex associate <fex_id>

Example:

switch(config-if)#fex associate 101

Associates a Fabric Extender (FEX) to a Fabric interface.

Step 6

mtu 9216

Example:

switch(config-if)#mtu 9216

Configures the MTU value to that of jumbo frames to carry FCoE traffic through the NIF ports.

Step 7

no shutdown

Example:

switch(config-if)#no shutdown

Sets the port-channel to up (administratively).

Step 8

exit

Example:

switch(config-if)#exit

Exit the interface configuration mode.

Step 9

interface ethernet slot/port

Example:

switch(config)#interface Ethernet1/1

Enters interface configuration mode.

Step 10

switchport

Example:

switch(config-if)#switchport

Sets Layer 2 switching port.

Step 11

switchport mode fex-fabric

Example:

switch(config-if)#switchport mode fex-fabric

Sets the interface type to be an uplink port for a fabric extender (FEX).

Step 12

fex associate <fex_id>

Example:

switch(config-if)#fex associate 101

Associates a Fabric Extender (FEX) to a Fabric interface.

Step 13

mtu 9216

Example:

switch(config-if)#mtu 9216

Configures the MTU value to that of jumbo frames to carry FCoE traffic through the NIF ports.

Step 14

channel-group <port_number>

Example:

switch(config-if)channel-group 101

Sets the Fabric interface a member of the Fabric port channel.

Step 15

no shutdown

Example:

switch(config-if)#no shutdown

Sets the port-channel to up (administratively).

Step 16

exit

Example:

switch(config-if)#exit

Exit the interface configuration mode.

Step 17

interface ethernet chassis_id/slot/port_number

Example:

switch(config)interface ethernet 101/1/1

Configures a FEX satellite interface or HIF (host interface) port and enters interface configuration mode.

Step 18

switchport mode trunk

Example:

switch(config-if)#switchport mode trunk

Sets interface type to be a trunk port.

Step 19

channel group <host_port_num>

Example:

switch(config-if)# channel group 1

Makes the HIF port a member of a port-channel

Step 20

no shutdown

Example:

switch(config-if)#no shutdown

Sets the port-channel to up (administratively).

Step 21

exit

Example:

switch(config-if)#exit

Exit the interface configuration mode.

Step 22

interface port-channel <host_port_number>

Example:

switch(config)#interface port-channel 1

Creates a HIF port-channel.

Step 23

switchport

Example:

switch(config-if)#switchport

Sets Layer 2 switching port.

Step 24

switchport mode trunk

Example:

switch(config-if)#switchport mode trunk

Sets the interface to be a trunk port.

Step 25

service-policy type qos input fcoe

Example:

switch(config-if)# service-policy type qos input fcoe

Sets the QoS policy on the HIF port channel to a no drop policy.

Note:
For more information about queuing policy configurations, see the section: Configuring QoS for no-drop support.

Step 26

vpc 3

Configure VPC on the HIF port-channel. The VPC id on both the peers should be same for this Host VPC.

Step 27

no shutdown

Example:

switch(config-if)#no shutdown

Sets the port-channel to up (administratively).

Step 28

exit

Example:

switch(config-if)#exit

Exit the interface configuration mode.

The following is an example of FEX bring up in straight through mode with a host vPC.

- Configuration on Peer-1

install feature-set fex
feature-set fex

fex 101
pinning max-links 1
description "2232PP-1"

interface port-channel101
switchport mode fex-fabric
fex associate 101

interface port-channel101
switchport
switchport mode fex-fabric
fex associate 101
mtu 9216

interface Ethernet1/1
switchport
switchport mode fex-fabric
fex associate 101
mtu 9216
channel-group 101
no shutdown

interface Ethernet101/1/1
switchport mode trunk
channel-group 1
no shutdown

interface port-channel1
switchport
switchport mode trunk
service-policy type qos input fcoe
vpc 3

interface vfc-po1      /*** Implicit binding with VFC bound to port-channel ***/
bind interface port-channel1
switchport trunk mode on
no shutdown

interface vfc101/1/1   /*** Implicit binding with VFC bound to member port ***/
bind interface ethernet101/1/1
switchport trunk mode on
no shutdown

- Configuration on Peer-2

install feature-set fex
feature-set fex

fex 102
pinning max-links 1
description "2348UPQ-2"

interface port-channel102
switchport
switchport mode fex-fabric
fex associate 102
mtu 9216

interface Ethernet102/1/1
switchport mode trunk
channel-group 1
no shutdown

interface port-channel1
switchport
switchport mode trunk
service-policy type qos input fcoe
vpc 3

interface vfc1   /*** Explicit binding with VFC bound to port-channel ***/
bind interface port-channel1
switchport trunk mode on
no shutdown

interface vfc2   /*** Explicit binding with VFC bound to member port ***/
bind interface ethernet102/1/1
switchport trunk mode on
no shutdown

Note

The vFC binding should be either to the portchannel or to the member host interface.

Configuring Dual-Homed FEX

SUMMARY STEPS

1.    configure terminal

2.    fex fex-chassis_ID

3.    fcoe

4.    interface port-channel <port_num>

5.    switchport

6.    switchport mode fex-fabric

7.    fex associate <fex_id>

8.    mtu 9216

9.    vpc 1

10.    no shutdown

11.    exit

12.    interface ethernet slot/port

13.    switchport

14.    switchport mode trunk

15.    service-policy type qos input fcoe-qo-policy

16.    no shutdown

17.    exit

DETAILED STEPS

Command or Action

Purpose

Step 1

configure terminal

Example:

switch# configure terminal
switch(config)#

Enters global configuration mode.

Step 2

fex fex-chassis_ID

Example:

switch# fex 101
switch(config)#

Enters the configuration mode for the specified FEX. The range for fex-chassis_ID is from 101 to 199.

Step 3

fcoe

Example:

switch# fcoe
switch(config)#

Configures the FEX to send FCoE traffic only to this switch.

Step 4

interface port-channel <port_num>

Example:

switch(config)#interface port-channel170

Creates port-channel and enters the interface configuration mode.

Step 5

switchport

Example:

switch(config-if)#switchport

Sets Layer 2 switching port.

Step 6

switchport mode fex-fabric

Example:

switch(config-if)#switchport mode fex-fabric

Sets the interface type to be a uplink port for a Fabric extender (FEX).

Step 7

fex associate <fex_id>

Example:

switch(config-if)#fex associate 170

Associates a Fabric Extender (FEX) to a Fabric interface.

Step 8

mtu 9216

Example:

switch(config-if)#mtu 9216

Configures the MTU value to that of jumbo frames to carry FCoE traffic through the NIF ports.

Step 9

vpc 1

Configure VPC on the HIF port-channel.

Step 10

no shutdown

Example:

switch(config-if)#no shutdown

Sets the port-channel to up (administratively).

Step 11

exit

Example:

switch(config-if)#exit

Exit the interface configuration mode.

Step 12

interface ethernet slot/port

Example:

switch(config)#interface Ethernet170/1/18

Enters interface configuration mode.

Step 13

switchport

Example:

switch(config-if)#switchport

Sets Layer 2 switching port.

Step 14

switchport mode trunk

Example:

switch(config-if)#switchport mode trunk

Sets the interface to be a trunk port.

Step 15

service-policy type qos input fcoe-qo-policy

Example:

switch(config-if)# service-policy type qos input fcoe

Sets the QoS policy on the HIF port channel to a no drop policy.

Note:
For more information about queuing policy configurations, see the section: Configuring QoS for no-drop support.

Step 16

no shutdown

Example:

switch(config-if)#no shutdown

Sets the port-channel to up (administratively).

Step 17

exit

Example:

switch(config-if)#exit

Exit the interface configuration mode.

Note:
The same configuration should be performed on the other side for the Active-Active FEX to be up on both the sides.

Note

FEX should be FCoE pinned to one of the peers.

- Peer 1

fex 170
pinning max-links 1
description "2232PP-3 AA"
fcoe

interface port-channel170
switchport
switchport mode fex-fabric
fex associate 170
mtu 9216
vpc 1

interface Ethernet170/1/18
switchport mode trunk
service-policy type qos input fcoe-qo-policy
no shutdown

interface vfc1718     /*** Explicit binding ***/
bind interface Ethernet170/1/18
switchport trunk mode on
no shutdown

interface vfc170/1/18 /*** Implicit binding ***/
bind interface Ethernet170/1/18
switchport trunk mode on
no shutdown

- Peer 2

fex 170
pinning max-links 1
description "2232PP-3 AA"

interface port-channel170
switchport
switchport mode fex-fabric
fex associate 170
mtu 9216
vpc 1

interface Ethernet170/1/18
switchport mode trunk
service-policy type qos input fcoe-qo-policy
no shutdown

Note

The vFC binding should be either to the port channel or to the member host interfaces.

Preface

This preface includes the following sections:

- Audience

- Document Conventions

- Related Documentation for Cisco Nexus 9000 Series Switches

- Documentation Feedback

- Obtaining Documentation and Submitting a Service Request

Audience

This publication is for network administrators who install, configure, and maintain Cisco Nexus switches.

Document Conventions

Command descriptions use the following conventions:

Convention

Description

bold

Bold text indicates the commands and keywords that you enter literally as shown.

Italic

Italic text indicates arguments for which the user supplies the values.

[x]

Square brackets enclose an optional element (keyword or argument).

[x | y]

Square brackets enclosing keywords or arguments separated by a vertical bar indicate an optional choice.

{x | y}

Braces enclosing keywords or arguments separated by a vertical bar indicate a required choice.

[x {y | z}]

Nested set of square brackets or braces indicate optional or required choices within optional or required elements. Braces and a vertical bar within square brackets indicate a required choice within an optional element.

variable

Indicates a variable for which you supply values, in context where italics cannot be used.

string

A nonquoted set of characters. Do not use quotation marks around the string or the string will include the quotation marks.

Examples use the following conventions:

Convention

Description

screen font

Terminal sessions and information the switch displays are in screen font.

boldface screen font

Information you must enter is in boldface screen font.

italic screen font

Arguments for which you supply values are in italic screen font.

< >

Nonprinting characters, such as passwords, are in angle brackets.

[ ]

Default responses to system prompts are in square brackets.

!, #

An exclamation point (!) or a pound sign (#) at the beginning of a line of code indicates a comment line.

Related Documentation for Cisco Nexus 9000 Series Switches

The entire Cisco Nexus 9000 Series switch documentation set is available at the following URL:

Click Here

Documentation Feedback

To provide technical feedback on this document, or to report an error or omission, please send your comments to nexus9k-docfeedback@cisco.com. We appreciate your feedback.

Obtaining Documentation and Submitting a Service Request

For information on obtaining documentation, using the Cisco Bug Search Tool (BST), submitting a service request, and gathering additional information, see What's New in Cisco Product Documentation at: Click Here .

Subscribe to What's New in Cisco Product Documentation, which lists all new and revised Cisco technical documentation as an RSS feed and delivers content directly to your desktop using a reader application. The RSS feeds are a free service.

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS" WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.

This product includes cryptographic software written by Eric Young (eay@cryptsoft.com).

This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit. ( Click Here )

This product includes software written by Tim Hudson (tjh@cryptsoft.com).

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: Click Here . Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R)