Field Notice: FN - 70078 - CRS-X Linecards Version V04 or Later Are Not Supported by Cisco IOS XR Software Releases Earlier Than 5.3.4 - Workaround Provided

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.0	27-Nov-17	Initial Release

Products Affected

Affected Product ID	Comments
CRS-MSC400G=
CRS-MSC400G
CRS-MSC-X
CRS-FP-X
CRS-MSC-X=
100GE-MSC400G-BUN=
10GE-MSC400G-BUN=
CRS-FP-X=	Part Alternate
CRS-LSP-X
CRS-LSP-X=
CRS-MSC-X-L=
CRS-FP-X-L=
CRS-MSC-X-L
CRS-FP-X-L
CRS-FP200G=
CRS-FP200G
CRS-MSC200G
CRS-MSC200G=
CRS-FP200G-L
CRS-FP200G-L=

Defect Information

Defect ID	Headline
CSCuw72333	Implementation of 500 Mhz sTCAM broadcom support for CRS-X LineCard
CSCuq89524	CPU support for CRS-X Line cards and Shelf controller card

Problem Description

CRS-X linecards, as mentioned in the Products Affected section, with Version V04 or later are not compatible with Cisco IOS^® XR Software Releases earlier than 5.3.4. These cards might fail to boot or might move into the IN-RESET state if they use non-compatible Cisco IOS versions.

Background

Cisco has upgraded the hardware of the CRS-X linecards to address the End of Life (EoL) issue of the existing ternary content-addressable memory (TCAM) module and the CPU. Version V04 is rolled out to support the replacement parts for TCAM and CPU.

These replaced components result in changes to the initializing sequence during bootup and require you to install the reload SMU (Software Maintenance Upgrade) if you run Cisco IOS-XR software releases earlier than 5.3.4. Dependent upon on the Cisco IOS-XR version, two SMUs might be required, one for the TCAM change and other for the CPU change.

Note: The SMU for CPU is required for versions earlier than 5.3.2 only.

Problem Symptom

When the customer installs a CRS-X linecard with version V04 or later with a non-compatible Cisco IOS XR software release, the card fails to boot to the XR-RUN state. Even if the card boots up, these types of errors will be observed before it moves into IN-RESET state.

Sample Error Logs

Missing TCAM SMU

Error messages in 5.3.x; the linecard goes to the XR FAILURE state after a few reload attempts.

LC/0/3/CPU0:Mar 20 15:32:41.690 : npu_driver[271]: %L2-NPU-3-ERROR_MSG : *** NPU0:
 FATAL error at {line:806}: failure in tcm, rc=19, No such  device
LC/0/3/CPU0:Mar 20 15:32:41.691 : npu_driver[271]: %L2-NPU-3-INIT_ERROR_MSG : NPU0:
 Init Failed. Caused by:No such device
LC/0/3/CPU0:Mar 20 15:32:41.691 : npu_driver[271]: %L2-NPU-3-ERR_EXIT : Exit on error:
 Failed asic power-on reset init: Caused by No such device  :
 pkg/bin/npu_driver : (PID=106555) :  -Traceback= 420956c 420a14c 421071a ca8e96e b6bb050
LC/0/3/CPU0:Mar 20 15:32:41.895 : npu_driver[271]: %L2-NPU-3-ERROR_MSG : *** NPU1:
 FATAL error at {line:806}: failure in tcm, rc=19, No such device
LC/0/3/CPU0:Mar 20 15:32:41.896 : npu_driver[271]: %L2-NPU-3-INIT_ERROR_MSG : NPU1:
 Init Failed. Caused by:No such device
LC/0/3/CPU0:Mar 20 15:32:42.561 : npu_driver[271]: %L2-NPU-6-INFO_MSG : NPU0:
 *** Total Valid NPUs:2 (max=2) *** 
LC/0/3/CPU0:Mar 20 15:32:42.561 : npu_driver[271]:
 *****************************************************
LC/0/3/CPU0:Mar 20 15:32:42.561 : npu_driver[271]:
 ***Failure detected! Not all ASICs are intialized***
LC/0/3/CPU0:Mar 20 15:32:42.561 : npu_driver[271]:
 ***Skipping Interface Creation***
LC/0/3/CPU0:Mar 20 15:32:42.561 : npu_driver[271]:
 ***LC will be reloaded for full recovery***
LC/0/3/CPU0:Mar 20 15:32:42.561 : npu_driver[271]:
 *****************************************************

Error messages in 5.1.x; the linecard reaches the XR RUN state, but there will not be any interfaces created.

LC/0/3/CPU0:Mar 21 13:15:47.032 : npu_driver[264]: npu_platform_stcam_dump_80_bit_reg
 reading 80bit i2cAddr 0x1, data 0x00000000 0x00000000 0x00000000
LC/0/3/CPU0:Mar 21 13:15:47.944 : npu_driver[264]: npu_platform_stcam_check_dev_status:
 ERROR: PAT sTCAM dev status check, timed out waiting for link up for slice 0
LC/0/3/CPU0:Mar 21 13:15:49.095 : npu_driver[264]: %L2-NPU-3-ERROR_MSG : *** NPU0: FATAL
 error at {line:657}: failure in tcm, rc=19, No such device
LC/0/3/CPU0:Mar 21 13:15:49.095 : npu_driver[264]: %L2-NPU-3-INIT_ERROR_MSG : NPU0:
 Init Failed. Caused by:No such device
LC/0/3/CPU0:Mar 21 13:15:49.095 : syslog_dev[80]: npu_driver[264]:
LC/0/3/CPU0:Mar 21 13:15:49.095 : syslog_dev[80]: npu_driver[264]: *** *** NPU0: FATAL
 error at {line:657}: failure in tcm, rc=19, No such device
LC/0/3/CPU0:Mar 21 13:15:49.095 : npu_driver[264]: %L2-NPU-3-ERR_EXIT : Exit on error:
 Failed asic power-on reset init: Caused by No such device  : pkg/bin/npu_driver :
 (PID=102458) :  -Traceback= 42081ba 4208c48 420a2ee a572148 827b050
LC/0/3/CPU0:Mar 21 13:15:49.096 : syslog_dev[80]: npu_driver[264]: npu_device_hotplug_cb:
{Err=0x13:No such device} npu_device_hw_init failed for NPU instance 0

Missing CPU SMU

Error messages when the linecard fails to boot properly and complains about pcie_attach_device failed:

LC/0/1/CPU0:Apr 13 06:32:38.212 : pciesvr[70]: pcie_hal_platform_asic_bringup_prologue:
Kyanite fpga sw version 0x15002f
LC/0/1/CPU0:Apr 13 06:32:38.273 : init[65540]: %OS-INIT-7-MBI_STARTED :
 total time 38.613seconds
LC/0/1/CPU0:Apr 13 06:32:39.291 : procmgr[74]: %PLATFORM-KENSHO_GL-3-PCIE_ATTACH_FAIL :
 pcie_attach_device failed!
LC/0/1/CPU0:Apr 13 06:32:39.294 : procmgr[74]: %PLATFORM-PROCMGR-3-INIT_ERROR :
 Procmgr initialization error. reason: 'Subsystem(3960)' detected the 'fatal' condition
 'Code(6)': No such device
LC/0/1/CPU0:Apr 13 06:32:40.314 : procmgr[74]: %PLATFORM-KENSHO_GL-3-PCIE_ATTACH_FAIL :
 pcie_attach_device failed!
LC/0/1/CPU0:Apr 13 06:32:40.317 : procmgr[74]: %PLATFORM-PROCMGR-3-INIT_ERROR :
 Procmgr initialization error. reason: 'Subsystem(3960)' detected the 'fatal' condition
 'Code(6)': No such device
LC/0/1/CPU0:Apr 13 06:32:41.334 : procmgr[74]: %PLATFORM-KENSHO_GL-3-PCIE_ATTACH_FAIL :
 pcie_attach_device failed!
LC/0/1/CPU0:Apr 13 06:32:41.337 : procmgr[74]: %PLATFORM-PROCMGR-3-INIT_ERROR :
 Procmgr initialization error. reason: 'Subsystem(3960)' detected the 'fatal' condition
 'Code(6)': No such device
LC/0/1/CPU0:Apr 13 06:32:42.366 : procmgr[74]: %PLATFORM-KENSHO_GL-3-PCIE_ATTACH_FAIL :
 pcie_attach_device failed!
LC/0/1/CPU0:Apr 13 06:32:42.369 : procmgr[74]: %PLATFORM-PROCMGR-3-INIT_ERROR :
 Procmgr initialization error. reason: 'Subsystem(3960)' detected the 'fatal' condition
 'Code(6)': No such device
LC/0/1/CPU0:Apr 13 06:32:43.387 : procmgr[74]: %PLATFORM-KENSHO_GL-3-PCIE_ATTACH_FAIL :
 pcie_attach_device failed!
LC/0/1/CPU0:Apr 13 06:32:43.390 : procmgr[74]: %PLATFORM-PROCMGR-3-INIT_ERROR :
 Procmgr initialization error. reason: 'Subsystem(3960)' detected the 'fatal' condition
 'Code(6)': No such device
LC/0/1/CPU0:Apr 13 06:32:43.400 : init[-1]: reboot: Failed to get RP isolation status.
 Treated as non-isolated mode, 'Subsystem(1786)' detected the 'warning' condition 'Code(15)'

Error messages when fabric-ping issues occur. Be aware that fabric-ping error can occur for other issues as well.

RP/0/RP1/CPU0:Apr 22 17:43:05.672 jst: online_diag_rp[347]: %DIAG-XR_DIAG-3-FABRIC_PING_FAILURE :
 (U) Fabric Ping Failure, 1 of 3 nodes failed(L): 0/3/CPU0
RP/0/RP1/CPU0:Apr 22 17:43:13.532 jst: online_diag_rp[347]: %DIAG-XR_DIAG-3-FABRIC_PING_FAILURE :
 (U) Fabric Ping Failure - data path (Level 3 L) on plane 0, 1 of 3 nodes (33%) failed: 0/3/CPU0
RP/0/RP1/CPU0:Apr 22 17:43:13.533 jst: online_diag_rp[347]: %DIAG-XR_DIAG-3-FABRIC_PING_FAILURE :
 (U) Fabric Ping Failure - data path (Level 3 L) on plane 1, 1 of 3 nodes (33%) failed: 0/3/CPU0
RP/0/RP1/CPU0:Apr 22 17:43:13.534 jst: online_diag_rp[347]: %DIAG-XR_DIAG-3-FABRIC_PING_FAILURE :
 (U) Fabric Ping Failure - data path (Level 3 L) on plane 2, 1 of 3 nodes (33%) failed: 0/3/CPU0
RP/0/RP1/CPU0:Apr 22 17:43:13.534 jst: online_diag_rp[347]: %DIAG-XR_DIAG-3-FABRIC_PING_FAILURE :
 (U) Fabric Ping Failure - data path (Level 3 L) on plane 3, 1 of 3 nodes (33%) failed: 0/3/CPU0
RP/0/RP1/CPU0:Apr 22 17:43:13.535 jst: online_diag_rp[347]: %DIAG-XR_DIAG-3-FABRIC_PING_FAILURE :
 (U) Fabric Ping Failure - data path (Level 3 L) on plane 4, 1 of 3 nodes (33%) failed: 0/3/CPU0
RP/0/RP1/CPU0:Apr 22 17:43:13.536 jst: online_diag_rp[347]: %DIAG-XR_DIAG-3-FABRIC_PING_FAILURE :
 (U) Fabric Ping Failure - data path (Level 3 L) on plane 5, 1 of 3 nodes (33%) failed: 0/3/CPU0
RP/0/RP1/CPU0:Apr 22 17:43:13.536 jst: online_diag_rp[347]: %DIAG-XR_DIAG-3-FABRIC_PING_FAILURE :
 (U) Fabric Ping Failure - data path (Level 3 L) on plane 6, 1 of 3 nodes (33%) failed: 0/3/CPU0
RP/0/RP1/CPU0:Apr 22 17:43:13.537 jst: online_diag_rp[347]: %DIAG-XR_DIAG-3-FABRIC_PING_FAILURE :
 (U) Fabric Ping Failure - data path (Level 3 L) on plane 7, 1 of 3 nodes (33%) failed: 0/3/CPU0
RP/0/RP1/CPU0:Apr 22 17:43:18.165 jst: online_diag_rp[347]: %DIAG-XR_DIAG-6-INFO : (U) FIM:
 single-node failure detected - 0/3/CPU0 consecutive ucast/mcast failures: 1/0
RP/0/RP1/CPU0:Apr 22 17:43:18.165 jst: online_diag_rp[347]: %DIAG-DIAG-3-TEST_FAIL : RP 0/RP1/CPU0:
 FabricUcastMcastTest{ID=13} has failed. Error code = 0x4 (DIAG_PARTIAL_FAILURE)
RP/0/RP1/CPU0:Apr 22 17:43:18.165 jst: online_diag_rp[347]: %DIAG-DIAG-3-MINOR : RP 0/RP1/CPU0:
 Diagnostics detected a Minor Error. Please use 'show diagnostic result location ' to see test
 results.
RP/0/RP1/CPU0:SB-CRSX(admin)#RP/0/RP1/CPU0:Apr 22 17:43:13.537 jst: online_diag_rp[347]:
 %DIAG-XR_DIAG-3-FABRIC_PING_FAILURE : (U) Fabric Ping Failure - data path (Level 3 L) on plane 7,
 1 of 3 nodes (33%) failed: 0/3/CPU0
RP/0/RP1/CPU0:Apr 22 17:43:18.165 jst: online_diag_rp[347]: %DIAG-DIAG-3-TEST_FAIL : RP 0/RP1/CPU0:
 FabricUcastMcastTest{ID=13} has failed. Error code = 0x4 (DIAG_PARTIAL_FAILURE)
RP/0/RP1/CPU0:Apr 22 17:43:18.165 jst: online_diag_rp[347]: %DIAG-DIAG-3-MINOR : RP 0/RP1/CPU0:
 Diagnostics detected a Minor Error. Please use 'show diagnostic result location ' to see test results.
RP/0/RP1/CPU0:SB-CRSX(admin)#show diagnostic resuRP/0/RP1/CPU0:SB-CRSX(admin)#show diagnostic result
 lRP/0/RP1/CPU0:Apr 22 17:44:20.579 jst: online_diag_rp[347]: %DIAG-XR_DIAG-3-FABRIC_PING_FAILURE :
 (U) Fabric Ping Failure, 1 of 3 nodes failed(L): 0/3/CPU0

Workaround/Solution

RMAs should not be created for failure of CRS-X Version V04 linecards, which is related to non-installation of SMUs as discussed in this section.

The CRS-X linecard with version V04 or later, which runs Cisco IOS-XR Software Releases earlier than 5.3.4 requires one of these workaround options in order to address a software incompatibility issue.

Option 1: Install the SMUs

In order to continue with the existing software versions earlier than 5.3.2, install both the reload SMUs for TCAM and CPU as stated in this section. For the 5.3.2 and 5.3.3 software versions, only the TCAM SMU is required. These SMUs are backward compatible and can be installed in any order. More details are provided in this section.

A. SMU for TCAM Upgrade - Cisco bug ID CSCuw72333

Release	Platform	SMU ID	ETA	Customer
5.1.1	CRS-PX	AA12789	Posted	All CRS-X customers
5.1.2	CRS-PX	AA12788	Posted	All CRS-X customers
5.1.3	CRS-PX	AA12787	Posted	All CRS-X customers
5.1.4	CRS-PX	AA12786	Posted	All CRS-X customers
5.3.1	CRS-PX	AA12785	Posted	All CRS-X customers
5.3.2	CRS-PX	AA12784	Posted	All CRS-X customers
5.3.3	CRS-PX	AA12783	Posted	All CRS-X customers

 

Release 5.1.1
If you run release 5.1.1, upgrade with this SMU:
Description: Implementation of 500 Mhz sTCAM broadcom support for CRS-X LineCard
Release: 5.1.1
Release Date: 19-MAY-2017
File Name: hfr-px-5.1.1.CSCuw72333.pie

Release 5.1.2
If you run release 5.1.2, upgrade with this SMU:
Description: Implementation of 500 Mhz sTCAM broadcom support for CRS-X LineCard
Release: 5.1.2
Release Date: 10-MAY-2017
File Name: hfr-px-5.1.2.CSCuw72333.pie

Release 5.1.3
If you run release 5.1.3, upgrade with this SMU:
Description: Implementation of 500 Mhz sTCAM broadcom support for CRS-X LineCard
Release: 5.1.3
Release Date: 09-MAY-2017
File Name: hfr-px-5.1.3.CSCuw72333.pie

Release 5.1.4
If you run release 5.1.4, upgrade with this SMU:
Description: Implementation of 500 Mhz sTCAM broadcom support for CRS-X LineCard
Release: 5.1.4
Release Date: 04-MAY-2017
File Name: hfr-px-5.1.4.CSCuw72333.pie

Release 5.3.1
If you run release 5.3.1, upgrade with this SMU:
Description: Implementation of 500 Mhz sTCAM broadcom support for CRS-X LineCard
Release: 5.3.1
Release Date: 15-MAY-2017
File Name: hfr-px-5.3.1.CSCuw72333.pie

Release 5.3.2
If you run release 5.3.2, upgrade with this SMU:
Description: Implementation of 500 Mhz sTCAM broadcom support for CRS-X LineCard
Release: 5.3.2
Release Date: 03-MAY-2017
File Name: hfr-px-5.3.2.CSCuw72333.pie

Release 5.3.3
If you run release 5.3.3, upgrade with this SMU:
Description: Implementation of 500 Mhz sTCAM broadcom support for CRS-X LineCard
Release: 5.3.3
Release Date: 26-APR-2017
File Name: hfr-px-5.3.3.CSCuw72333.pie

 

B. SMU for CPU Upgrade - Cisco bug ID CSCuq89524

Release	Platform	SMU ID	ETA	Customer
5.1.1	CRS-PX	AA09775	Posted	All CRS-X customers
5.1.2	CRS-PX	AA09774	Posted	All CRS-X customers
5.1.3	CRS-PX	AA09773	Posted	All CRS-X customers
5.1.4	CRS-PX	AA09835	Posted	All CRS-X customers
5.3.0	CRS-PX	AA09843	Posted	All CRS-X customers
5.3.1	CRS-PX	AA09895	Posted	All CRS-X customers

 

(SMU is incorporated in releases 5.3.2 and 5.3.3)

Release 5.1.1
If you run release 5.1.1, upgrade with this SMU:
Description: CPU support for CRS-X Line cards and Shelf controller card
Release: 5.1.1
Release Date: 29-MAY-2015
File Name: hfr-px-5.1.1.CSCuq89524.pie

Release 5.1.2
If you run release 5.1.2, upgrade with this SMU:
Description: CPU support for CRS-X Line cards and Shelf controller card
Release: 5.1.2
Release Date: 08-MAY-2015
File Name: hfr-px-5.1.2.CSCuq89524.pie

Release 5.1.3
If you run release 5.1.3, upgrade with this SMU:
Description: CPU support for CRS-X Line cards and Shelf controller card
Release: 5.1.3
Release Date: 18-MAY-2015
File Name: hfr-px-5.1.3.CSCuq89524.pie

Release 5.1.4
If you run release 5.1.4, upgrade with this SMU:
Description: CPU support for CRS-X Line cards and Shelf controller card
Release: 5.1.4
Release Date: 21-SEP-2015
File Name: hfr-px-5.1.4.CSCuq89524.pie

Release 5.3.0
If you run release 5.3.0, upgrade with this SMU:
Description: CPU support for CRS-X Line cards and Shelf controller card
Release: 5.3.0
Release Date: 31-MAY-2015
File Name: hfr-px-5.3.0.CSCuq89524.pie

Release 5.3.1
If you run release 5.3.1, upgrade with this SMU:
Description: CPU support for CRS-X Line cards and Shelf controller card
Release: 5.3.1
Release Date: 04-JUNE-2015
File Name: hfr-px-5.3.1.CSCuq89524.pie

Option 2: Upgrade

Cisco IOS XR software can be migrated to one of the latest Cisco IOS XR Software Releases: 5.3.4, 6.1.x, or later. These releases support both hardware changes and do not require any of the SMUs.

Open the CRS-X 16-Slot Single-Shelf System Download Software page for the latest software releases.

How To Identify Affected Products

Look for a special label on the packaging box which indicates this board is Version V04 and contains the revised TCAM Memory, CPU, and a link to the Field Notice. A representation of the label is shown here:

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email
• By telephone

Receive Email Notification For New Field Notices

Cisco Notification Service—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.