FCoE with vPC Configuration Example
Beginning with Cisco NX-OS Release 4.1(3)N1(1), the Cisco Nexus 5000 Series switch supports vPCs which can be configured to increase bandwidth and increased load-balancing to the Ethernet fabric. This appendix includes a sample configuration on how to configure FCoE when using vPCs on the Cisco Nexus 5000 Series switch and includes the following sections:
Figure 4-1 shows the topology used in the examples described in this appendix.
Figure 4-1 Nexus 5000 FCoE and vPC Lab Topology
The configuration example includes the following parameters:
switchname: n5k-tme-1
switchname: n5k-tme-2
mgmt ip: 172.25.182.66
mgmt ip: 172.25.182.67
The configuration example includes the following hardware:
- Dell Server PE2950
- QLogic QLE8142 (Schultz) Generation-2 CNA
- 2 Cisco Nexus 5010 switches running Cisco NX-OS Release 4.1(3)N1(1)
The configuration example includes the following considerations and requirements:
1. Generation 2 CNAs that support DCBX are required.
2. Single host CNA port channel connection to a separate switch. FCoE interfaces will not be brought up if the port channel on a single switch contains more than one member port in a port channel or vPC.
3. Cisco NX-OS Release 4.1(3)N1(1) or a later release.
4. FC Features Package (FC_FEATURES_PKG) is necessary for running FCoE. If this is not installed, there will be a temporary license that will last 90 days.
This appendix includes the following sections:
Cisco Nexus 5000 Series Switch vPC Configuration Example
This example presumes that the basic configuration has been completed on the switch (for example, IP Address (mgmt0), switchname, and password for the administrator).
This example shows how to configure the basic vPC configuration. For more information on configuring vPC, refer to the Cisco Nexus 5000 Series vPC Quick Configuration Guide.
Note The configuration must be done on both peer switches in the vPC topology.
Step 1 Enable the vPC feature on both peer switches.
Enter configuration commands, one per line. End with CNTL/Z.
tme-n5k-1(config)# feature vpc
Enter configuration commands, one per line. End with CNTL/Z.
tme-n5k-2(config)# feature vpc
Step 2 Configure the vPC domain and peer-keep alive destinations:
tme-n5k-1(config)# vpc domain 2
tme-n5k-1(config-vpc-domain)# peer-keepalive destination 192.165.200.229
tme-n5k-2(config)# vpc domain 2
tme-n5k-2(config-vpc-domain)# peer-keepalive destination 192.165.200.230
Note In this set up, switch tme-n5k-1 has the mgmt IP address of 192.165.200.229 and switch tme-n5k-2 has the mgmt IP address of 192.165.200.230.
Step 3 Configure the port channel interface that will be used as the vPC peer-link:
tme-n5k-1(config)# int port-channel 1
tme-n5k-1(config-if)# vpc peer-link
Note The spanning tree port type is changed to network port type on vPC peer-link. This will enable STP Bridge Assurance on vPC peer-link provided that the STP Bridge Assurance (which is enabled by default) is not disabled.
tme-n5k-2(config)# int port-channel 1
tme-n5k-2(config-if)# vpc peer-link
Step 4 Verify that the peer-keepalive can be reached:
tme-n5k-1(config)# show vpc peer-keepalive
vPC keep-alive status : peer is alive
--Destination : 172.25.182.167
--Receive status : Success
--Last update from peer : (0) seconds, (975) msec
tme-n5k-2(config)# show vpc peer-keepalive
--PC keep-alive status : peer is alive
--Destination : 172.25.182.166
--Receive status : Success
--Last update from peer : (0) seconds, (10336) msec
Step 5 Add member ports to the vpc-peer link port channel and bring up the port channel interface:
tme-n5k-1(config-if-range)# int po 1
tme-n5k-1(config-if)# switchport mode trunk
tme-n5k-1(config-if)# no shut
tme-n5k-1(config-if)# exit
tme-n5k-1(config)# int eth 1/39-40
tme-n5k-1(config-if-range)# switchport mode trunk
tme-n5k-1(config-if-range)# channel-group 1
tme-n5k-1(config-if-range)# no shut
tme-n5k-1(config-if-range)#
tme-n5k-2(config-if-range)# int po 1
tme-n5k-2(config-if)# switchport mode trunk
tme-n5k-2(config-if)# no shut
tme-n5k-2(config-if)# exit
tme-n5k-2(config)# int eth 1/39-40
tme-n5k-2(config-if-range)# switchport mode trunk
tme-n5k-2(config-if-range)# channel-group 1
tme-n5k-2(config-if-range)# no shut
tme-n5k-2(config-if-range)#
tme-n5k-1(config-if-range)# show int po1
Hardware: Port-Channel, address: 000d.ecde.a92f (bia 000d.ecde.a92f)
MTU 1500 bytes, BW 20000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Input flow-control is off, output flow-control is off
Switchport monitor is off
Members in this channel: Eth1/39, Eth1/40
Last clearing of "show interface" counters never
1 minute input rate 1848 bits/sec, 0 packets/sec
1 minute output rate 3488 bits/sec, 3 packets/sec
tme-n5k-1(config-if-range)#
tme-n5k-2(config-if-range)# show int po1
Hardware: Port-Channel, address: 000d.ecdf.5fae (bia 000d.ecdf.5fae) MTU 1500 bytes, BW 20000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Input flow-control is off, output flow-control is off
Switchport monitor is off
Members in this channel: Eth1/39, Eth1/40
Last clearing of "show interface" counters never
minute input rate 1848 bits/sec, 0 packets/sec
minute output rate 3488 bits/sec, 3 packets/sec
tme-n5k-2(config-if-range)#
Step 6 Create the vPC and add member interfaces:
tme-n5k-1(config)# int po 11
tme-n5k-1(config-if)# vpc 11
tme-n5k-1(config-if)# switchport mode trunk
tme-n5k-1(config-if)# no shut
tme-n5k-1(config-if)# int eth 1/1
tme-n5k-1(config-if)# switchport mode trunk
tme-n5k-1(config-if)# channel-group 11
tme-n5k-1(config-if)# spanning-tree port type edge trunk
Warning Edge port type (portfast) should only be enabled on ports connected to a single host. Connecting some devices such as hubs, concentrators, switches, or bridges to this interface when edge port type (portfast) is enabled, can cause temporary bridging loops. Caution should be used in this type of configuration
tme-n5k-2(config)# int po 11
tme-n5k-2(config-if)# vpc 11
tme-n5k-2(config-if)# switchport mode trunk
tme-n5k-2(config-if)# no shut
tme-n5k-2(config-if)# int eth 1/1
tme-n5k-2(config-if)# switchport mode trunk
tme-n5k-2(config-if)# channel-group 11
tme-n5k-2(config-if)# spanning-tree port type edge trunk
Warning Edge port type (portfast) should only be enabled on ports connected to a single host. Connecting some devices such as hubs, concentrators, switches, or bridges to this interface when edge port type (portfast) is enabled, can cause temporary bridging loops. Caution should be used in this type of configuration.
Note To run FCoE over a vPC topology, the port channel can only have a sinlge member interface.
Note The vPC number configured under the port channel interface must match on both Nexus 5000 switches. The port channel interface number does not have to match on both switches.
Step 7 Verify that the vPC interfaces are up and operational:
tme-n5k-1(config-if)# show vpc statistics vpc 11
vPC Status: Up, vPC number: 11
Hardware: Port-Channel, address: 000d.ecde.a908 (bia 000d.ecde.a908)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Input flow-control is off, output flow-control is off
Switchport monitor is off
Members in this channel: Eth1/1
Last clearing of "show interface" counters never
minute input rate 4968 bits/sec, 8 packets/sec
minute output rate 792 bits/sec, 1 packets/sec
tme-n5k-2(config-if)# show vpc statistics vpc 11
vPC Status: Up, vPC number: 11
Hardware: Port-Channel, address: 000d.ecdf.5fae (bia 000d.ecdf.5fae)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Input flow-control is off, output flow-control is off
Switchport monitor is off
Members in this channel: Eth1/1
Last clearing of "show interface" counters never
minute input rate 4968 bits/sec, 8 packets/sec
minute output rate 792 bits/sec, 1 packets/sec
Cisco Nexus 5000 Series Switch FCoE Configuration Example
Once the vPC is set up between the two Nexus 5000s, we can move on to configuring the FCoE topology. This cheat sheet presumes that basic configuration has been executed on the Nexus 5000 switch that will provide IP Address (mgmt0), switchname, password for admin, etc. and that the vPC configuration has been completed as outlined in the previous section. The following steps will walk through the basic FCoE configuration necessary to set up an FCoE topology in conjunction with the vPC topology.
Step 1 Enable FCoE on the Nexus 5000:
tme-n5k-1(config)# feature fcoe
FC license checked out successfully
fc_plugin extracted successfully
FC plugin loaded successfully
FCoE manager enabled successfully
FC enabled on all modules successfully
tme-n5k-2(config)# feature fcoe
FC license checked out successfully
fc_plugin extracted successfully
FC plugin loaded successfully
FCoE manager enabled successfully
FC enabled on all modules successfully
Note This can take a few moments to complete.
Step 2 Create a VSAN and map it to a VLAN that has been designated to carry FCoE traffic:
tme-n5k-1(config)# vsan database
tme-n5k-1(config-vsan-db)# vsan 100
tme-n5k-1(config-vsan-db)# exit
tme-n5k-1(config)# vlan 100
me-n5k-1(config-vlan)# fcoe vsan 100
tme-n5k-1(config-vlan)# show vlan fcoe
-------- -------- --------
tme-n5k-2(config)# vsan database
tme-n5k-2(config-vsan-db)# vsan 101
tme-n5k-2(config-vsan-db)# exit
tme-n5k-2(config)# vlan 101
tme-n5k-2(config-vlan)# fcoe vsan 101
tme-n5k-2(config-vlan)# show vlan fcoe
-------- -------- --------
Note VLAN and VSAN numbers are not required to be the same.
Step 3 Configure the VLANs that are allowed to transverse the vPC links:
tme-n5k-1(config)# int po 11
tme-n5k-1(config-if)# switchport trunk allowed vlan 1, 100
tme-n5k-1(config-if)# show int trunk
------------------------------------------------------------------
------------------------------------------------------------------
------------------------------------------------------------------
Port Vlans Allowed on Trunk
------------------------------------------------------------------
------------------------------------------------------------------
Port Vlans Err-disabled on Trunk
------------------------------------------------------------------
------------------------------------------------------------------
------------------------------------------------------------------
tme-n5k-2(config)# int po 11
tme-n5k-2(config-if)# switchport trunk allowed vlan 1, 101
tme-n5k-2(config-if)# show int trunk
------------------------------------------------------------------
------------------------------------------------------------------
------------------------------------------------------------------
Port Vlans Allowed on Trunk
------------------------------------------------------------------
------------------------------------------------------------------
Port Vlans Err-disabled on Trunk
------------------------------------------------------------------
------------------------------------------------------------------
------------------------------------------------------------------
Step 4 Create a virtual Fibre Channel interface (vfc) and add it to the VSAN that was created in the previous step:
tme-n5k-1(config)# int vfc 1
tme-n5k-1(config-if)# bind interface po11
Warning: VFC will not come up for pre-FIP CNA
tme-n5k-1(config-if)# no shut
tme-n5k-2(config)# int vfc 1
tme-n5k-2(config-if)# bind interface po11
Warning: VFC will not come up for pre-FIP CNA
tme-n5k-2(config-if)# no shut
tme-n5k-1(config)# vsan database
tme-n5k-1(config-vsan-db)# vsan 100 interface vfc 1
tme-n5k-1(config)# show vsan membership
vsan 4079(evfp_isolated_vsan) interfaces:
vsan 4094(isolated_vsan) interfaces:
tme-n5k-2(config)# vsan database
tme-n5k-2(config-vsan-db)# vsan 101 interface vfc 1
tme-n5k-2(config)# show vsan membership
vsan 4079(evfp_isolated_vsan) interfaces:
vsan 4094(isolated_vsan) interfaces:
Step 5 Verify that the vfc is up and operational:
tme-n5k-1(config-if)# show int brief
------------------------------------------------------------------
Ethernet VLAN Type Mode Status Reason Speed
------------------------------------------------------------------
Eth1/1 1 eth trunk up none 10G(D)
Eth1/2 1 eth access up none 10G(D)
Eth1/38 1 eth access down SFP not inserted 10G(D)
Eth1/39 1 eth trunk up none 10G(D)
Eth1/40 1 eth trunk up none 10G(D)
------------------------------------------------------------------
Port-channel VLAN Type Mode Status Reason Speed
------------------------------------------------------------------
Po1 1 eth trunk up none a-10G(D) none
Po11 1 eth trunk up none a-10G(D) none
------------------------------------------------------------------
Port VRF Status IP Address Speed MTU
------------------------------------------------------------------
mgmt0 -- up 172.25.182.166 1000 1500
------------------------------------------------------------------
Interface Vsan Admin Admin Status SFP Oper Oper Port
------------------------------------------------------------------
vfc1 100 F on up -- F auto --
tme-n5k-2(config-if)# show int brief
------------------------------------------------------------------
Ethernet VLAN Type Mode Status Reason Speed Port
------------------------------------------------------------------
Eth1/1 1 eth trunk up none 10G(D) 11
Eth1/2 1 eth access up none 10G(D) --
Eth1/38 1 eth access down SFP not inserted 10G(D) --
Eth1/39 1 eth trunk up none 10G(D) 1
Eth1/40 1 eth trunk up none 10G(D) 1
------------------------------------------------------------------
Port-channel VLAN Type Mode Status Reason Speed Protocol
------------------------------------------------------------------
Po1 1 eth trunk up none a-10G(D) none
Po11 1 eth trunk up none a-10G(D) none
------------------------------------------------------------------
Port VRF Status IP Address Speed MTU
------------------------------------------------------------------
mgmt0 -- up 172.25.182.167 1000 1500
------------------------------------------------------------------
Interface Vsan Admin Admin Status SFP Oper Oper
------------------------------------------------------------------
vfc1 101 F on up -- F auto --
Step 6 Verify that the virtual Fibre Channel interface has logged into the fabric:
tme-n5k-1# show flogi database
------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
-----------------------------------------------------------------------
vfc1 100 0x540000 21:00:00:c0:dd:11:2a:01 20:00:00:c0:dd:11:2a:01
Total number of flogi = 1.
tme-n5k-2# show flogi database
-----------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
------------------------------------------------------------------------
vfc1 101 0x540000 21:00:00:c0:dd:11:2a:01 20:00:00:c0:dd:11:2a:01
Total number of flogi = 1.
Step 7 Verify that the vPC is up and operational:
tme-n5k-1(config-if)# show vpc statistics vpc 11
vPC Status: Up, vPC number: 11
Hardware: Port-Channel, address: 000d.ecde.a908 (bia 000d.ecde.a908)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Input flow-control is off, output flow-control is off
Switchport monitor is off
Members in this channel: Eth1/1
Last clearing of "show interface" counters never
1 minute input rate 4968 bits/sec, 8 packets/sec
1 minute output rate 792 bits/sec, 1 packets/sec
tme-n5k-2(config-if)# show vpc statistics vpc 11
vPC Status: Up, vPC number: 11
Hardware: Port-Channel, address: 000d.ecdf.5fae (bia 000d.ecdf.5fae)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Input flow-control is off, output flow-control is off
Switchport monitor is off
Members in this channel: Eth1/1
Last clearing of "show interface" counters never
1 minute input rate 4968 bits/sec, 8 packets/sec
1 minute output rate 792 bits/sec, 1 packets/sec