Field Notice: FN74128 - MDS 9396S Fabric Switches Might Disruptively Reload or MDS 9700 32 Gbps FC Modules Might Log EOBC Failure - Software Upgrade Recommended

This issue has platform dependent symptoms:

• A Cisco MDS 9396S 16G Fabric Switch might reset unexpectedly. During the reset, ports on the affected device go down and disrupt traffic forwarding through the device.
• A Cisco MDS 9700 32 Gbps FC linecard might log a transient and nondisruptive Ethernet out-of-band Channel (EOBC) heartbeat failure syslog message.

Only a subset of devices are affected by this issue. It is not possible to determine in advance which devices will be affected or when.

This issue applies to the above products only when running one of the following Cisco MDS NX-OS releases:

• 8.4 train: Release 8.4(2d) and later
• 9.3 train: Any 9.3(x) release
• 9.4 train: Any 9.4(x) release earlier than Release 9.4(2)

Earlier releases were affected by a similar issue described in FN72346.

This issue is caused by a PCIe (Peripheral Component Interconnect Express) operation that might fail after an indeterminate runtime. The success or failure of the operation depends on which address is being accessed by the operation. This can lead to a variety of symptoms and prevents prediction of which devices will be affected.

The symptom is different for each product:

• For a Cisco MDS 9396S Fabric switches, the switch resets without any user intervention.
  This is disruptive to traffic. The reason is 'HA policy of Reset' and the service is 'f16_mac_usd'. This can be seen with the show system reset-reason command:

  Reason: Reset triggered due to HA policy of Reset
  Service: f16_mac_usd hap reset
  Version: 8.4(2d)

• For a Cisco MDS 9700 32 Gbps FC linecard, a transient nondisruptive EOBC heartbeat failure occurs. 'EOBC heartbeat failure' syslog messages from the module occur on both active and standby supervisors. These can be seen with the show logging logfile command, as shown in the following example:

  %MODULE-4-MOD_WARNING: Module 1 (Serial number: ...) reported warning 1/1-1/0 due to EOBC heartbeat failure on standby sup in device DEV_EOBC_MAC (device error 0xc0a0504f)
  %MODULE-4-MOD_WARNING: Module 1 (Serial number: ...) reported warning 1/1-1/0 due to EOBC heartbeat failure in device DEV_EOBC_MAC (device error 0xc0a0514d)
  

In both cases, a machine check error and 'MCSR=0x80200000' value are logged after the event. To determine whether a device has been affected by this issue, run the following commands:

• For Cisco MDS 9396S Fabric switches, use the show system internal kernel nvram-messages previous command.
• For Cisco MDS 9700 32 Gbps FC linecards, use the slot <slot> show system internal kernel messages command, where <slot> is the slot number of the module.

If the device is affected by this issue, the following output is displayed:

Machine check in kernel mode on Core ... comm [...] pid [...]
L2 Machine check on Core ...: MCSR=0x80200000 ...

Solution: To permanently resolve this issue, upgrade to a fixed version of Cisco MDS NX-OS. This issue is resolved in Cisco MDS NX-OS 9.4(2) and later releases.

Workaround:

1. For Cisco MDS 9396S Fabric switches there is no workaround.

2. Cisco MDS 9700 32 Gbps FC linecards do not require any workaround as the issue is transient and nondisruptive.

To determine the release of Cisco MDS NX-OS running on a device, enter the show version CLI command. The release is shown in the 'system' field, as shown in the following example:

switch# show version

...

Software
  BIOS:      version 5.2.25
  loader:    version N/A
  kickstart: version 9.2(1)
  system:    version 9.2(1)

To determine a device Product ID, enter the show inventory CLI command. The Product ID is shown in the 'PID' field.

• For MDS 9396S fabric switches:

  switch# show inventory
  NAME: "Chassis",  DESCR: "MDS 9396S 96X16G FC (2 RU) Chassis"
  PID: DS-C9396S-K9        ,  VID: V00 ,  SN: JPG191800A0

• For MDS 9700 32 Gbps FC modules:

  switch# show inventory
  NAME: "Slot 1" ,  DESCR: "4/8/16/32 Gbps Advanced FC Module"
  PID: DS-X9648-1536K9     ,  VID: V01 ,  SN: JAE210308AT

This issue is due to an incomplete fix for FN72346.