Field Notice: FN74223 - Some Cisco NCS 540 Series Medium Density Routers May Become Unresponsive After Upgrade to Cisco IOS XR Software Release 7.11.1 or Later - Workaround Provided

Some medium density Cisco Network Convergence System (NCS) 540 Series Routers may not boot when subjected to a field programmable device (FPD) upgrade during installation of Cisco IOS XR Software Release 7.11.1 or later.

The issue occurs because a key variable in the Trust Anchor Module (TAM) is corrupted during the upgrade attempt. Cisco is preparing an SMU to fix the issue.

Routers impacted by the issue may not boot after reload or power cycle. The PWR LED on the front panel of the router will be Flashing Green, and the FAN LED will be Flashing Amber, indicating a failure of the TAm to initialize.

The video NCS540-TAm Init Fail on the Cisco Video Portal provides detailed information about the problem symptom.

The software fix for this issue will be available by end of the first quarter of calendar year 2025. SMUs for specific Cisco IOS XR Software releases will be prepared based upon requests from customers.

Cisco recommends deferring the software upgrade on products mentioned as part of this field notice until the fix or an SMU is available.

For scenarios where software upgrades are on priority, administrators can disable fpd auto-upgrade and then manually upgrade all FPD devices except TAmFw and stdbyFPGA. For devices using this workaround, it is highly recommended to install the SMU once it is available and then re-enable the fpd auto-upgrade feature.

In the following example, fpd auto-upgrade disable is configured before the Cisco IOS XR Software is upgraded:

RP/0/RP0/CPU0:EG#conf
DDD MM DD HH:MM:SS UTC
 
RP/0/RP0/CPU0:EG(config)#
RP/0/RP0/CPU0:EG(config)#fpd auto-upgrade disable
RP/0/RP0/CPU0:EG(config)#commit
DDD MM DD HH:MM:SS UTC
RP/0/RP0/CPU0:DDD MM DD HH:MM:SS UTC: config[66319]: %MGBL-CONFIG-6-DB_COMMIT : Configuration committed by user 'root123'. Use 'show configuration commit 
RP/0/RP0/CPU0:EG(config)#end
RP/0/RP0/CPU0:DDD MM DD HH:MM:SS UTC: config[66319]: %MGBL-SYS-5-CONFIG_I : Configured from console by root123
RP/0/RP0/CPU0:EG#show hw-module fpd
DDD MM DD HH:MM:SS UTC
 
Auto-upgrade:Disabled
Attribute codes: B golden, P protect, S secure, A Anti Theft aware
                                                                         FPD Versions
                                                                        ==============
Location   Card type             HWver FPD device       ATR Status   Running Programd  Reload Loc
-------------------------------------------------------------------------------------------------
0/RP0/CPU0 N540X-16Z4G8Q2C-A     1.0   ADM_FW               CURRENT   14.03   14.03       NOT REQ
0/RP0/CPU0 N540X-16Z4G8Q2C-A     1.0   ADMConfig            CURRENT    1.05    1.05       NOT REQ
0/RP0/CPU0 N540X-16Z4G8Q2C-A     1.0   IoFpga               NEED UPGD  2.12    2.12         0/RP0
0/RP0/CPU0 N540X-16Z4G8Q2C-A     1.0   IoFpgaGolden     B   CURRENT            2.07         0/RP0
0/RP0/CPU0 N540X-16Z4G8Q2C-A     1.0   Primary-BIOS     SA  NEED UPGD  1.46    1.46         0/RP0
0/RP0/CPU0 N540X-16Z4G8Q2C-A     1.0   StdbyFpga        S   NEED UPGD  0.40    0.40         0/RP0
0/RP0/CPU0 N540X-16Z4G8Q2C-A     1.0   StdbyFpgaGolden  BS  CURRENT            0.40         0/RP0
0/RP0/CPU0 N540X-16Z4G8Q2C-A     1.0   TamFw            S   NEED UPGD  4.11    4.11         0/RP0
0/RP0/CPU0 N540X-16Z4G8Q2C-A     1.0   TamFwGolden      BS  CURRENT            4.11         0/RP0
RP/0/RP0/CPU0:EG#

FPD upgrades can be done manually on devices that have the ATR Status of NEED UPGD. Note that TamFw and StdbyFpga devices should not be subjected to FPD upgrades even when their ATR statuses are NEED UPGD. The ATR statuses of these two devices should be disregarded.

The following example shows a manual FPD upgrade of an IoFpga device:

RP/0/RP0/CPU0:EG#upgrade hw-module location 0/rP0/CPU0 fpd IoFpga
DDD MM DD HH:MM:SS UTC
upgrade command issued (use "show hw-module fpd" to check upgrade status)
RP/0/RP0/CPU0:EG#
RP/0/RP0/CPU0:EG#RP/0/RP0/CPU0:DDD MM DD HH:MM:SS UTC: fpd-serv[121]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Upgrade for the following FPDs has been committ 
RP/0/RP0/CPU0:DDD MM DD HH:MM:SS UTC: fpd-serv[121]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Location           FPD name             Force    
RP/0/RP0/CPU0:DDD MM DD HH:MM:SS UTC: fpd-serv[121]: %INFRA-FPD_Manager-1-UPGRADE_ALERT :===============================================
RP/0/RP0/CPU0:DDD MM DD HH:MM:SS UTC: fpd-serv[121]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP0/CPU0         IoFpga                FALSE  
RP/0/RP0/CPU0:DDD MM DD HH:MM:SS UTC: fpd_client[411]: %PLATFORM-FPD_CLIENT-1-UPGRADE_ALERT : Upgrade dependent FPD IoFpga@0/RP0/CPU0 
RP/0/RP0/CPU0:DDD MM DD HH:MM:SS UTC: fpd_client[411]: %PLATFORM-FPD_CLIENT-1-UPGRADE_COMPLETE : FPD upgrade complete for IoFpga@0/RP0/CPU0 [image upgraded 
RP/0/RP0/CPU0:DDD MM DD HH:MM:SS UTC: shelfmgr[169]: %PLATFORM-SHELFMGR-6-INFO_LOG : 0/RP0/CPU0 is operational 
RP/0/RP0/CPU0:DDD MM DD HH:MM:SS UTC: fpd-serv[121]: %PKT_INFRA-FM-3-FAULT_MAJOR : ALARM_MAJOR :FPD-NEED-UPGRADE :CLEAR :0/RP0/CPU0:  
RP/0/RP0/CPU0:EG#