Troubleshoot MDS 9000 Trunking/Port-Channel Links that Fail to Come Up

Introduction

This document describes why a Fibre Channel (FC) link between a Cisco Multilayer Director Switch (MDS) and another Cisco switch doesn't come Up when trunking or in a port-channel, however, the same link might come Up when it is not in a port-channel and trunking is disabled.

Background Information

This includes FC links between MDS switches and other MDS switches, Nexus switches, and Unified Computing System (UCS) Fabric Interconnects (FIs).

Tip: For more detailed information about port-channels and trunking, refer to the appropriate port-channel and trunking configuration guides.
Port-channel Guide: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/mds9000/sw/8_x/config/interfaces/cisco_mds9000_interfaces_config_guide_8x/configuring_portchannels.html
Trunking Guide: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/mds9000/sw/8_x/config/interfaces/cisco_mds9000_interfaces_config_guide_8x/configuring_trunking.html

All MDS switches can aggregate multiple physical links into a single virtual link via port-channels, as well as transport multiple Virtual Storage Area Networks (VSANs) over a link with the trunking feature when connected with another Cisco device.

In order to negotiate port-channels and trunking links, MDS switches use Exchange Peer Parameters (EPP) services to communicate across peer ports in an Inter-Switch Link (ISL). Part of the ISL negotiation process verifies that the peer is a Cisco device with the peer's Organizational Unique Identifier (OUI).

If a Cisco switch's OUI is not listed in the peer's OUI database as a Cisco OUI, the link fails to come up when added to a port-channel or if trunking is enabled on the interface.

Troubleshoot MDS Trunking/Port-Channel Links that Fail to Come Up

If the link is in a port-channel, check the port-channel interface haschannel mode active configured.

MDSswitch# show run interface port-channel 1
...
interface port-channel1
channel mode active      <<<<

If the link is connected to a N_Port Virtualization (NPV) switch, like a FI, check the NPIV and fport-channel-trunk features are enabled.

MDSswitch# show feature | include npiv
npiv 1 enabled
MDSswitch# show feature | include fport
fport-channel-trunk 1 enabled

If the link is trunking, check the trunking protocol is enabled globally.

MDSswitch# show trunk protocol
Trunk Protocol is enabled

Check trunking is enabled on the interface.

MDSswitch# show interface port-channel 1
port-channel1 is down (No operational members)
Hardware is Fibre Channel 
Port WWN is xx:xx:00:2a:6a:xx:xx:xx
Admin port mode is auto, trunk mode is on <<<<

If the link is connected at 8Gbps, ensure the fill pattern, or fill word, must be configured the same for both sides of the link. By default, the MDS, N5K/N6K, and UCS FI 6300 series devices use ARBFF. N9K and UCS FI 6400 series only support IDLE as the 8G fill pattern.

Use this command to confirm the current fill pattern on MDS/N5K/N6K interfaces fc x/y:

slot x show hardware internal fcmac port y port-info | i FILL

MDSswitch# slot 1 show hardware internal fcmac port 2 port-info | i FILL
FILL_WORD(raw) : ARBFF (0xbc94ffff)

Note: The output on Nexus switches can be 0x0 (interface not up at 8Gbps), 0xBC94FFFF (ARBFF), or 0xBC95B5B5 (IDLE).

When an N9K or UCS FI 6400 is connected to a device that uses ARBFF by default, you must configure the interface on the connected device to use IDLE.

MDSswitch# conf t
Enter configuration commands, one per line. End with CNTL/Z.
MDSswitch(config)# interface fc1/2MDSswitch(config-if)# switchport fill-pattern IDLE speed 8000

On UCS FI 6300 series, you see FC uplinks' fill pattern and configured in the UCS Manager (UCSM) at SAN > SAN Cloud > Fabric > Uplink FC Interfaces > FC Interface x/yin the General tab.

Verify

Symptoms

• Interfaces come up as single ports with trunking disabled, but not in a port-channel.
• A Single interface fails to come up with trunking enabled.
• When in a port-channel or added to a port-channel, interfaces become error-disabled immediately.

Log Messages

Note: It is possible for the switch OUI to be absent from the peer's OUI database on one or both sides of a link. Both sides must be investigated.

For MDS and Nexus switches:

The switch without the peer's OUI in its OUI database records "physical flogi rejected, waiting for the port mode" and "EPP_SYNC" errors.

MDSswitch# show flogi internal event-history errors | include reject previous 1 next 1
...
10) Event:E_DEBUG, length:124, at 727676 usecs after Tue May 14 17:44:47 2013
[102] fs_fc2_msg_flogi: ifindex[0x12ae000] pwwn[xx:xx:00:2a:6a:xx:xx:xx] physical flogi rejected, waiting for the port mode

11) Event:E_DEBUG, length:124, at 661821 usecs after Tue May 14 17:44:39 2013
[102] fs_fc2_msg_flogi: ifindex[0x12ae000] pwwn[xx:xx:00:2a:6a:xx:xx:xx] physical flogi rejected, waiting for the port mode

12) Event:E_DEBUG, length:73, at 772303 usecs after Tue May 14 17:43:11 2013
[102] fs_flogi_send_flogi_reject: mts_q == 0, ifindex 0x12ae000, port 0x0


MDSswitch# show port internal event-history errors | include EPP previous 1 next 1
...
19) Event:E_DEBUG, length:48, at 465145 usecs after Tue May 14 17:44:49 2013
[102] epp_ac_accept_sync: Error sending EPP_SYNC
... 
27) Event:E_DEBUG, length:48, at 280537 usecs after Tue May 14 17:44:43 2013
[102] epp_ac_accept_sync: Error sending EPP_SYNC

To confirm the interface of the errors, translate the index withshow interface snmp-ifindexand filter by the ifindex in the error message. The example uses the ifindex 12ae000 found in theshow flogi internal event-history errors.

MDSswitch# show interface snmp-ifindex | include 12ae000
fc6/47 0019587072 (00012ae000)

For UCS FIs: The FI without the peer's OUI in its OUI database records "PI_FSM_EV_PORT_CONFIG_FAILURE" and "port fcx/y attempting vf tagging on non-cisco switch" errors.

UCS(nxos)# show port internal event-history errors

1) Event:E_DEBUG, length:117, at 225850 usecs after Wed Aug 1 10:18:26 2018
[102] pi_fsm_port_attr_change_init: Ifindex (fc1/47)0x102e000, Err disabled event (PI_FSM_EV_PORT_CONFIG_FAILURE)0xd7

2) Event:E_DEBUG, length:100, at 222587 usecs after Wed Aug 1 10:18:26 2018
[102] pm_process_fport_vf_tagging_capability: port fc1/47 attempting vf tagging on non-cisco switch

OUI Database

Extract the OUI from each switch. The OUI is the third, fourth, and fifth octet of the World Wide Name (WWN).

peer-switch# show wwn switch
Switch WWN is xx:xx:00:2a:6a:xx:xx:xx

In this example, the switch OUI is0x002A6A.

Check the switch's OUI database for the peer switch's OUI. If the command has no output, the OUI is absent from the database.

MDSswitch# show wwn oui | include 0x002a6a
MDSswitch#

Note: UCS FIs don't support theshow wwn ouicommand. Refer to Table 1. and Table 2. in order to see when OUIs were added to the FI OUI database.

Work Around

Switch OUI databases are updated through switch software upgrades. Until a switch can be upgraded to a software version with an OUI database that includes the peer switch's OUI, use single links configured outside of port-channels with trunking disabled.

Thewwn oui <hex oui> command on MDS and Nexus switches can manually add Cisco OUIs to the switch's OUI database.

On MDS switches, this feature is available in NX-OS Version 7.3(0)D1(1)and later.

On Nexus 5K/6K switches, this feature was added in NX-OS Version 7.3(3)N1(1)and later.
On Nexus 9000 switches on NX-OS 7.x, this feature is available in NX-IS Version 7.3(1)N1(1) and later.

On Nexus 9000 switches on other versions of NX-OS, this feature is available in NX-OS Version9.3(3)and later.

Example: MDSswitch# configure terminal
         MDSswitch(config)# wwn oui 0x0000fc
         MDSswitch# show wwn oui
         OUI Vendor Default/Static
         ----------------------------------------------------
         0x0000fc Cisco Static

Nexus 9000 switches on ACI images can add an OUI to the library in run time from the APIC in15.0(1k) and later.

OUI Database Additions

Table 1.1. MDS OUI Database Additions

Table 1.2. Nexus 5000/5500/5600/6000 OUI Database Additions

Table 1.3. Nexus 9000 OUI Database Additions

Table 1.4. UCS FI OUI Database Additions

Caveats

Cisco bug ID CSCut41864 - Port process crashes when interface > 64 used for NPV-NPIV connections

Cisco bug ID CSCuz52695 - UCS FI FC Uplinks Error Disable Immediately Connected NPV to MDS9396S

Cisco bug ID CSCvp95480 - 6454 FI - Restrict allowed WWxN addresses during UCSM Configuration (avoid multicast address)

Multiple bugs have been filed to document OUIs that were absent from switches' OUI libraries. They include:

MDS/UCS Bugs

Cisco bug ID CSCtr01652- Add N7K OUIs to liboui's list of recognized Cisco OUIs. MDS.

MDS/N5K/N6K Bugs

Cisco bug ID CSCty04686- Add MAC OUI "002a6a", "8c604f", "00defb" for 5k/UCS-FI

MDS Bugs

Cisco bug ID CSCta97375 - GLDN:SABRE: Sabre oui is not recognized as cisco oui 547FEE.

Cisco bug ID CSCur44218 - Add MiniUCS FI OUIs to liboui's list of recognized Cisco OUIs

Cisco bug ID CSCuz11494 - FC port channel does not come up with peer UCS OUI 00:c8:8b

Cisco bug ID CSCuu03717 - Add OUIs 00:3a:9c, 00:08:31, d0:a5:a6 to default OUI list

Cisco bug ID CSCuu26045 - Add MiniUCS FI OUI 0x74a02f to MDS list of recognized Cisco OUIs

Cisco bug ID CSCvx67383 - MDS Port-Channel towards ACI LEAF N9K with switch OUI 0x3C13CC (3C:13:CC) does not come up or trunk

Cisco bug ID CSCvx49403 - Port-Channel with Cisco switch OUI 0xDC774C does not come up or trunk.

Cisco bug ID CSCvr60193 - Add OUI C4:F7:D5 to the default OUI list

Cisco bug ID CSCvp19788- MDS 9000 Add OUI 70:EA:1A to OUI library

Cisco bug ID CSCvt28351 - Add six OUI's to the default OUI list

Cisco bug ID CSCvw14504 - Add OUI 0x4CE176 to OUI Database - Port-channels and trunking not working with OUI 0x4CE176

Cisco bug ID CSCvx67383 - MDS Port-Channel towards ACI LEAF N9K with switch OUI 0x3C13CC (3C:13:CC) does not come up or trunk

N5K/N6K Bugs

Cisco bug ID CSCvx72821 - Port-Channel with Cisco switch OUI 0xDC774C or 0x4CE176 or 0x3C13CC does not come up or trunk.

Cisco bug ID CSCvp54881 - Unable to create San Port Channel between UCS & Nexus 56128p

N9K Bugs

Cisco bug ID CSCvp21786 - Nexus 9000:::Unable to bring up F mode san-port-channel or trunking between MDS and N9K

Cisco bug ID CSCvx68507 - Port-Channel with Cisco switch OUI 0xDC774C or 0x4CE176 or 0x3C13CC does not come up or trunk.