Field Notice: FN74209 - Cisco UCS Manager May Flap Virtual Fibre Channel Interfaces - Software Upgrade Recommended

Available Languages

Updated:November 6, 2024

Document ID:FN74209

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

High

Impact Rating:

High

First Published:

2024-Nov-06

Last Published:

2024-Nov-06

Revision:

1.0

Cisco Bug IDs:

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Products Affected

Affected Software Product	Affected Release	Affected Release Number	Comments
Unified Computing System (UCS) Infrastructure Software Bundle	4.3	4.3(4a), 4.3(4b), 4.3(4c), 4.3(4d)

Defect Information

Defect ID	Headline
CSCwk91747	VFC Flap during FI Reboot
CSCwm30262	VLAN change triggered vHBAs/vfc reprogramming

Problem Description

Virtual Fibre Channel (vFC) and virtual host bus adapter (vHBA) interfaces may flap with no warning in the following scenarios when they are running Cisco UCS Release 4.3(4a), 4.3(4b), 4.3(4c), or 4.3(4d):

One of the fabric interconnects is rebooted (impacts all servers that have vHBAs)
Fabric interconnect management processes restart (impacts all servers that have vHBAs)
There are any changes to the virtual network interface card (vNIC) or vHBA on the service profile that is associated with the server (impacts the specific server for which the service profile was modified)
Cisco Integrated Management Controller (IMC) loses connectivity due to Cisco IMC resetting or for any other reason (impacts the specific server that has these Cisco IMC connectivity issues)

For affected products, these scenarios will result in the vFC or vHBA interface going down and coming back up within a few seconds.

Background

During any reconfigure or reboot operation related to VLANs within the system, the Data Management Engine (DME) process will evaluate the virtual Ethernet (vEth) interfaces that are associated with the port channel from the virtual interface card (VIC) upstream. Due to the design of the system, Fibre Channel traffic from the host upstream or downstream is carried by Fibre Channel over Ethernet (FCOE) over some vEth links in this port channel.

During the evaluation of and expected change to ethernet that is carrying vEth interfaces in the VIC port channel, FCOE instances that are carrying vEth interfaces are mistakenly taken down as well. This causes the momentary vFC and vHBA outage.

Problem Symptom

Any of the four scenarios in the Problem Description section of this field notice can result in vFC or vHBA interface flaps. Depending on the specific configuration, this may lead to an All Paths Down (APD) scenario at the host level.

Symptoms:

vFC/vHBA link flaps
APD at the host level

The host-level log may appear as follows:

2024-09-17T15:08:23.233Z In(14) vobd[2097948]: [scsiCorrelator] 5516906010185us: [vob.scsi.scsipath.pathstate.deadver2] scsiPath vmhba1:C0:T2:L7 changed state from on (device ID: naa.60000970000xxxx02306533030303438)
2024-09-17T15:08:23.233Z In(14) vobd[2097948]: [scsiCorrelator] 5517504798424us: [esx.problem.storage.redundancy.degraded] Path redundancy to storage device naa.60000970000xxxx02306533030303438 degraded. Path vmhba1:C0:T2:L7 is down. Affected datastores: Unknown.
2024-09-17T15:08:23.233Z In(14) vobd[2097948]: [scsiCorrelator] 5516906010217us: [vob.scsi.scsipath.remove] Remove path: vmhba1:C0:T2:L7
2024-09-17T15:08:23.288Z In(14) vobd[2097948]: [scsiCorrelator] 5516906017712us: [vob.scsi.scsipath.pathstate.deadver2] scsiPath vmhba1:C0:T2:L40 changed state from on (device ID: naa.60000970000xxxx02306533030304330)
2024-09-17T15:08:23.289Z In(14) vobd[2097948]: [scsiCorrelator] 5517504853584us: [esx.problem.storage.redundancy.degraded] Path redundancy to storage device naa.60000970000xxxx02306533030304330 degraded. Path vmhba1:C0:T2:L40 is down. Affected datastores: Unknown.
2024-09-17T15:08:23.290Z In(14) vobd[2097948]: [scsiCorrelator] 5516906017741us: [vob.scsi.scsipath.remove] Remove path: vmhba1:C0:T2:L40
2024-09-17T15:08:23.315Z In(14) vobd[2097948]: [scsiCorrelator] 5516906019911us: [vob.scsi.scsipath.pathstate.deadver2] scsiPath vmhba1:C0:T4:L7 changed state from on (device ID: naa.60000970000xxxx02306533030303438)
2024-09-17T15:08:23.315Z In(14) vobd[2097948]: [scsiCorrelator] 5517504879848us: [esx.problem.storage.redundancy.degraded] Path redundancy to storage device naa.60000970000xxxx02306533030303438 degraded. Path vmhba1:C0:T4:L7 is down. Affected datastores: Unknown.
2024-09-17T15:08:23.315Z In(14) vobd[2097948]: [scsiCorrelator] 5516906019932us: [vob.scsi.scsipath.remove] Remove path: vmhba1:C0:T4:L7
2024-09-17T15:56:19.218Z In(14) vobd[2097948]: [scsiCorrelator] 5519781681843us: [vob.scsi.scsipath.pathstate.deadver2] scsiPath vmhba0:C0:T3:L7 changed state from on (device ID: naa.60000970000xxxx02306533030303438)
2024-09-17T15:56:19.219Z In(14) vobd[2097948]: [scsiCorrelator] 5520380783678us: [esx.problem.storage.connectivity.lost] Lost connectivity to storage device naa.60000970000xxxx02306533030303438. Path vmhba0:C0:T3:L7 is down. Affected datastores: Unknown.
2024-09-17T15:56:19.219Z In(14) vobd[2097948]: [scsiCorrelator] 5519781681910us: [vob.scsi.scsipath.pathstate.deadver2] scsiPath vmhba0:C0:T4:L7 changed state from on (device ID: naa.60000970000xxxx02306533030303438)
2024-09-17T15:56:19.219Z In(14) vobd[2097948]: [scsiCorrelator] 5520380783793us: [esx.problem.storage.connectivity.lost] Lost connectivity to storage device naa.60000970000xxxx02306533030303438. Path vmhba0:C0:T4:L7 is down. Affected datastores: Unknown.
2024-09-17T15:56:19.220Z In(14) vobd[2097948]: [scsiCorrelator] 5519781684093us: [vob.scsi.scsipath.remove] Remove path: vmhba0:C0:T3:L7
2024-09-17T15:56:19.293Z In(14) vobd[2097948]: [scsiCorrelator] 5519781691647us: [vob.scsi.scsipath.pathstate.deadver2] scsiPath vmhba0:C0:T3:L40 changed state from on (device ID: naa.60000970000xxxx02306533030304330)
2024-09-17T15:56:19.294Z In(14) vobd[2097948]: [scsiCorrelator] 5519781691663us: [vob.scsi.scsipath.remove] Remove path: vmhba0:C0:T3:L40
2024-09-17T15:56:19.407Z In(14) vobd[2097948]: [scsiCorrelator] 5519781702121us: [vob.scsi.scsipath.pathstate.deadver2] scsiPath vmhba0:C0:T4:L40 changed state from on (device ID: naa.60000970000xxxx02306533030304330) 2024-09-17T15:56:19.407Z In(14) vobd[2097948]: [scsiCorrelator] 5519781702135us: [vob.scsi.scsipath.remove] Remove path: vmhba0:C0:T4:L40

Within the Cisco UCS logs, impacted vFC and vHBA links would show as follows:

PortAG attempts to create the FC VIF (vFC)
PortAG identifies the FC VIF (vFC) as already existing
The VFC is added to the pinning poll
PortAG admin takes down the already existing vFC, causing the outage

[INFO][0xf2a38b40][Oct 1 18:54:03.469][app_sam_portAG:diffRun] Processing VIF 1853
[INFO][0xf2a38b40][Oct 1 18:54:03.469][app_sam_portAG:write] Writing PAYLOAD!
[INFO][0xf2a38b40][Oct 1 18:54:03.469][app_sam_portAG:write] write: Writing this string:"type":"REQUEST","api":"SCI_GetFcVifInventory","args":{"aInInterface":[{"portType":7,"chassis":0,"slot":0,"vifId":1853}]}}
[INFO][0xf2a38b40][Oct 1 18:54:03.469][app_sam_portAG:print] lStr = {"type":"REPLY","api":"SCI_GetFcVifInventory","retval":0,"args":{"aOutInv":{"interface":{"portType":7,"chassis":0,"slot":0,"vifId":1853},"operState":2,"stateReason":{"buf":""},"portVsan":102},"aOutErrorDetails":{"buf":""}}}
[INFO][0xf2a38b40][Oct 1 18:54:03.469][app_sam_portAG:diffRun] Would create Fc VIF 1853 <---- 1 ***************************
[INFO][0xf2a38b40][Oct 1 18:54:03.469][app_sam_portAG:diffRun] Processing VIF 1852
[INFO][0xf2a38b40][Oct 1 18:54:03.469][app_sam_portAG:diffRun] Eth VIF 1852 already exists <----- 2 ***************************
[INFO][0xf2a38b40][Oct 1 18:54:03.469][rest_coupler:http_get] encoded http_get URL is: http://127.0.0.1:8080/api_local/mo/sys/veth/inst/veth-%5BVethernet1852%5D.json
[INFO][0xf2a38b40][Oct 1 18:54:03.471][rest_coupler:performHt] RESTAPI: http get SUCCESS..
[INFO][0xf2a38b40][Oct 1 18:54:03.471][rest_coupler:SCI_getAl] trunkVlans is 162
[INFO][0xf2a38b40][Oct 1 18:54:03.471][app_sam_portAG:diffRun] Processing VIF 1350
[INFO][0xf2a38b40][Oct 1 18:54:03.471][app_sam_portAG:diffRun] lIsMgmtVif = true, isMgmtVif = false, ignoring VIF 1350...
[INFO][0xf2a38b40][Oct 1 18:54:03.471][app_sam_portAG:diffRun] Processing VIF 1850
[INFO][0xf2a38b40][Oct 1 18:54:03.471][app_sam_portAG:diffRun] Eth VIF 1850 already exists
[INFO][0xf2a38b40][Oct 1 18:54:03.471][rest_coupler:http_get] encoded http_get URL is: http://127.0.0.1:8080/api_local/mo/sys/veth/inst/veth-%5BVethernet1850%5D.json
[INFO][0xf2a38b40][Oct 1 18:54:03.472][rest_coupler:performHt] RESTAPI: http get SUCCESS..
[INFO][0xf2a38b40][Oct 1 18:54:03.472][rest_coupler:SCI_getAl] trunkVlans is 160
[INFO][0xf2a38b40][Oct 1 18:54:03.472][app_sam_portAG:trigger] In triggerVfcPinningPoll(); shallow association case, triggering vfc pinning poll
[INFO][0xf2a38b40][Oct 1 18:54:03.472][app_sam_portAG:trigger] In triggerVfcPinningPoll(); vfc 1853 added to pinning poll <----- 3 ***************************
[INFO][0xf2a38b40][Oct 1 18:54:03.472][app_sam_portAG:ethVifB] VifConfigurator::ethVifBind: Coupler has SCI_BindVifWithChannelNo
[INFO][0xf2a38b40][Oct 1 18:54:03.472][rest_coupler:REST_Chec] RESTAPI: Entering REST_CheckAndReBindVif
[INFO][0xf2a38b40][Oct 1 18:54:03.472][rest_coupler:http_get] encoded http_get URL is: http://127.0.0.1:8080/api_local/mo/sys/veth/inst/veth-%5BVethernet10045%5D.json
[INFO][0xf2a38b40][Oct 1 18:54:03.473][rest_coupler:performHt] RESTAPI: http get SUCCESS..
[INFO][0xf2a38b40][Oct 1 18:54:03.474][rest_coupler:http_post] encoded URL is: http://127.0.0.1:8080/api_local/mo/sys/veth/inst.json?rsp-subtree=sync
[INFO][0xf2a38b40][Oct 1 18:54:03.474][rest_coupler:http_post] post Data is: {"l1VethIf":{ "attributes": {"rn": "veth-[Vethernet10045]","boundIf":"po1385","boundChannelId":"1853","adminSt":"down","status":"created,modified",}}} <----- 4 ***************************
[INFO][0xf2a38b40][Oct 1 18:54:03.478][rest_coupler:execute] RESTAPI: http post SUCCESS.

Workaround/Solution

The only workaround for this issue is to avoid any of the four scenarios in the Problem Description section of this field notice until an affected device has been upgraded to a fixed release.

To fix the issue that is described in this field notice, upgrade to a fixed software release.

Affected Software Product	Affected Release	First Fixed Release
Cisco UCS Infrastructure Software Bundle	4.3(4a) 4.3(4b) 4.3(4c) 4.3(4d)	4.3(4e) 4.3(5a)

Download the latest release of UCS-B Series Infrastructure Firmware from the Cisco Software Center.

Revision History

Version	Description	Section	Date
1.0	Initial Release	—	2024-NOV-06

For More Information

For further assistance or for more information about this field notice, contact the Cisco Technical Assistance Center (TAC) using one of the following methods:

Receive Email Notification About New Field Notices

To receive email updates about Field Notices (reliability and safety issues), Security Advisories (network security issues), and end-of-life announcements for specific Cisco products, set up a profile in My Notifications.

Was this Document Helpful?

Feedback

Contact Cisco

Open a Support Case
(Requires a Cisco Service Contract)