Field Notice: FN72346 - MDS 9396S Fabric Switches and MDS 9700 32 Gbps FC Modules Might Disruptively Reload - Software Upgrade Recommended

A Cisco MDS 9700 Series Multilayer Director 32G FC Line Card or a Cisco MDS 9396S 16G Multilayer Fabric Switch may randomly display one of the following symptoms:

• Briefly stops responding to keepalives from the supervisor to the device. The device recovers and data traffic is not affected.
• Stops responding to keepalives from the supervisor to the device long enough for the supervisor to reload the affected device.
• Reloads due to a service crash and boots successfully.
• Reloads due to a service crash and hangs.

During any reload, the ports on the device may go down, disrupting traffic-forwarding through those ports.

Only a subset of devices are affected by this issue. It is not possible to determine in advance which devices will be affected.

This issue applies only to these Cisco NX-OS Software releases for MDS:

• 8.1(x)
• 8.2(x)
• 8.3(x)
• 8.4(x) earlier than 8.4(2d)
• 9.2(x)

This issue is caused by a PCIe (Peripheral Component Interconnect Express) operation that might fail after a fixed period of runtime. The success or failure of the operation depends on what concurrently occurs on the CPU at that point. This can lead to a variety of symptoms and prevents prediction of which devices will be affected.

For MDS 9396S Fabric Switches

If the switch encounters this issue, it will always reload, which is disruptive to traffic. The reload will either:

• Fail, causing the switch to hang and become unresponsive to management and inband traffic. A few minutes later, fabric protocols such as FSPF and trunking protocols time out, and the switch becomes isolated from the fabric.
  
  
• Succeed, causing the switch to become operational again. After the reset, the command show system reset-reason may show various reasons for the reset. The reset may be caused by signal 6 killing critical processes such as 'f16_mac_usd'. If a crash happens, the show logging logfile command will display a syslog message of the process that crashed on the linecard:

  %SYSMGR-SLOT1-2-SERVICE_CRASHED: Service "f16_mac_usd" (PID 1234) hasn't caught signal 6 (core will be saved).

  The show cores command will show the core file with a matching process name, PID, and timestamp:

  Module  Instance  Process-name     PID       Date(Year-Month-Day Time)
  ------  --------  ---------------  --------  -------------------------
  1       1         f16_mac_usd      1234      2022-01-01 01:23:45

In both cases, the following lines are displayed after the reload by using the show system internal kernel nvram-messages previous command:

L2 Machine check on Core 2: MCSR=0x80200000 l2plbstats1=0xc0000
00001000  100c: 00000010 f04: 00000000 10b: 00000000

The first line will contain the strings "L2 Machine check" and "MCSR=0x80200000". The second line will contain the string "100c: 00000010".

For MDS 9700 32 Gbps FC Line Cards

The affected module will do one of the following:

• Continue to run without interruption to traffic. The output of the show logging logfile command will include the following lines:

  %MODULE-4-MOD_WARNING: Module 1 (Serial number: ...) reported warning 1/1-1/0 due to LC CPU not responding in device DEV_EOBC_MAC (device error 0xc0a0505c)
  
  %KERN-2-SYSTEM_MSG: [40437235.724980] LC [slot:1] CPU not responding... LC Status 0xf8  - kernel

  The output of the slot x show system internal kernel messages command (where x is the slot number of the relevant device) will include the following lines:
  

  [40435278.830804] L2 Machine check on Core 0: MCSR=0x80200000 l2plbstats1=0xc0000
  
  [40435279.604197] 110c: 00001000  100c: 00010010 f04: 00000000 10b: 00000000

  The first line will contain the strings "L2 Machine check" and "MCSR=0x80200000". The second line will contain the string "100c: 00000010". The previous logs might appear more than once for a given module, a few minutes apart, which indicates multiple events.
  
  
• Reload and interrupt traffic. The following symptoms may occur:
  
  
  • The module no longer responds and is reset by the supervisor. This line is logged in the output of the show logging logfile command:

    %MODULE-2-MOD_NOT_ALIVE: Module 1 not responding... resetting (Serial number: ...)

  • The module resets. This line is logged in the output of the show logging logfile command:

    %SYSMGR-SLOT1-2-SERVICE_CRASHED: Service "f32cmn_usd" (PID 1517) hasn't caught signal 6 (core will be saved).

    In this case, the show cores command will show a core file from the module with a timestamp just before the module reload:

    Module  Instance  Process-name     PID       Date(Year-Month-Day Time)
    ------  --------  ---------------  --------  -------------------------
    1       1         f32cmn_usd       1517      2021-08-07 04:59:31

    The core file might have a different process name, such as 'f32mac' or 'f32cmn_usd' or 'sm15_usd'.

Some platforms are affected by more than one Field Notice. Refer to the table below to determine the minimum version of Cisco MDS NX-OS to address each Field Notice.

1. Cisco MDS 9220i is not supported in the 8.4 train.

Refer to the table below to determine the minimum version of Cisco MDS NX-OS to address FN72223, FN72237, and FN72346 by platform.

1. Cisco MDS 9220i is not supported in the 8.4 train.

Solution: In order to permanently resolve this issue, upgrade to a fixed version of Cisco MDS NX-OS. See Field Notice FN74128 for information about fixed versions of Cisco MDS NX-OS.

For MDS 9396S Fabric switches, a nondisruptive upgrade to a fixed version might trigger the issue described in Field Notice 72237. Refer to Field Notice 72237 to determine the appropriate steps.

For MDS 9700 with 32 Gbps FC linecards, the switch can be upgraded nondisruptively with the install all command.

Workaround: None.

In order to determine the release of Cisco MDS NX-OS running on a device, enter the show version CLI command. The release is shown in the 'system' field.

switch# show version
...
Software
  BIOS:      version 5.2.25
  loader:    version N/A
  kickstart: version 9.2(1)
  system:    version 9.2(1)

In order to determine a device Product ID, enter the show inventory CLI command. The Product ID is shown in the 'PID' field.

For MDS 9396S fabric switches:

switch# show inventory
NAME: "Chassis",  DESCR: "MDS 9396S 96X16G FC (2 RU) Chassis"
PID: DS-C9396S-K9        ,  VID: V00 ,  SN: JPG191800A0

For MDS 9700 32 Gbps FC modules:

switch# show inventory
NAME: "Slot 1" ,  DESCR: "4/8/16/32 Gbps Advanced FC Module"
PID: DS-X9648-1536K9     ,  VID: V01 ,  SN: JAE210308AT         

Refer to your Cisco MDS switch supplier for their recommended version of Cisco MDS NX-OS. For the latest recommendations from Cisco, refer to the Recommended Releases for Cisco MDS 9000 Series Switches.