Field Notice: FN74142 - Cisco Catalyst 9600 Switches Supervisor Cards Can Experience Memory Failures - Hardware Upgrade Available

A limited number of Cisco Catalyst 9600 Supervisor Engine 1 cards may fail to boot up and encounter a boot loop. This issue may be seen after a software upgrade or other action that results in a power cycle of the supervisor card. The issue is specific to the supervisor card with the product identifier (PID) C9600-SUP-1. Supervisor cards with the PID C9600X-SUP-2 are not affected. 

The dual in-line memory module(s) (DIMM) used in the Catalyst 9600 Supervisor Engine 1 encountered an issue during manufacturing. This issue can result in a higher rate of failure.

In this case, a manufacturing deviation in specific modules impacts 16GB DIMMs. This deviation was contained to a specific date range, and the DIMMs that use these chips were manufactured during the middle to end of 2020.

Since the discovery of this deviation, additional limits have been imposed on the manufacturing process to help prevent future DIMMs from experiencing this process variation.

DIMMs in affected Catalyst 9600 Supervisor Engine 1 cards will exhibit persistent, correctable memory errors. If left untreated, affected Catalyst 9600 Supervisor Engine 1 cards will eventually encounter an uncorrectable memory event. This may result in an unexpected reboot of the switch or switches.

Various DIMM reliability, availability, and serviceability (RAS) features or even operating system features can mask the extent of these correctable errors. 

The following error messages may be seen:

Initializing Hardware...... 
Initializing Hardware...... 
***BOOT FAIL***: Rommon invalid or timeout. 
***BOOT FAIL***: Rommon invalid or timeout.

For reference, this is an example of a normal boot log message:

Initializing Hardware... 
Initializing Hardware...... 
System Bootstrap, Version 17.6.1r, RELEASE SOFTWARE (P) Compiled 18-03-2021 12:00:00.00 by rel
Current ROMMON image : Primary Rommon Image 
Last reset cause:PowerOn 
C9600-SUP-1 platform with 16777216 Kbytes of main memory

Cisco recommends replacing the affected DIMM C9K-MEM-16GB= to mitigate the potential for unexpected switch failure. For information about validating the serial number of a supervisor card, see the How to Identify Affected Products section of this field notice.

All serial numbers are the supervisor card serial number, not the DIMM serial number.

Replacement Instructions: First, validate the serial number(s) of the affected units as described in the How to Identify Affected Products section of this field notice. Then, see the Upgrade Program Information section for replacement instructions.

Returns of Affected Devices: Once the affected devices have been replaced, they need to be returned to Cisco. Instructions for how to raise a case for returning the device are provided in the Additional Information section. Note that this process is separate from our standard return process, which is typically included within the Return Merchandise Authorization (RMA).

DIMM Replacement

Customers have the option to replace the DIMMs directly. For video instructions for replacing a DIMM, see Replace a DIMM Module on Cat9600 Supervisor Engine 1.

Cisco is offering field services free of charge for DIMM replacement through a qualified Cisco third-party field engineer. After the new DIMMs have arrived on site and you are ready to schedule the replacement, email ciscodimmswap@parkplacetech.com to engage the field services team.

Important notes for replacement orders:

• After the replacement order is submitted, a pop-up message will confirm the information. It can take up to five business days to process the final order. After that, a notification with the details of the final order will be sent.
• Avoid placing more than one replacement order for the same serial number, as this could slow down the order processing.
• The replacement orders are proactive replacements and do not adhere to normal Service Level Agreements or service contracts. Furthermore, some countries have strict importation regulations. For these reasons, extended lead time may occur for these replacements.

Impacted supervisor cards can be identified based on their PIDs and serial numbers. To identify impacted products, use the Serial Number Validation Tool, as described in the Serial Number Validation section of this field notice.

Do not use the information in this section to submit replacement orders. For instructions on how to request a replacement, see the Upgrade Program Information section.

Return the Affected Device: Once the affected devices have been replaced, return them to Cisco. Use the following steps to return devices, either one-by-one or in bulk:

- Go to Support Case Manager (cisco.com) to raise a return-only RMA.

- To initiate an RMA, complete the following steps:

• Choose Return of Defective Asset.
• Choose Return Queries for the Request Type.
• Choose New Services Return – RMA Needed for the Sub-Request Type.
• In the Additional Request Details field, enter the following information and submit the return-only RMA request:
• TAC SR number: (not mandatory)
• Customer CCO ID:
• Contract Number: (not mandatory)
• Part ID(s):
• Serial Number:
• Company Name:
• Country: 
• Street Address: 
• State/Province: 
• Zip/Postal Code: 
• City: 
• On-site Contact Name: 
• On-site Contact Phone number: 
• On-site Contact E-mail: 
• Reason for RMA Creation: FN# FN74142
• Bill to ID: (not mandatory)