Upgrading Field-Programmable Device

An FPD is a field programmable logic device which contains non-volatile, re-programmable memory to define its internal wiring and functionality. The contents of this non-volatile memory are called the FPD image or FPD firmware. Over the lifespan of an FPD, FPD firmware images may need upgrades for bug fixes or functionality improvements. These upgrades are performed in the field with minimum system impact.

Prerequisites for FPD Image Upgrades

You must install the FPD pie before you install the SMUs or Service Packs. If you install the SMU or Service Packs before the FPD pie, the FPDs on the line card may not upgrade. In such cases, you must remove the SMUs and Service Packs and reload the router.

Overview of FPD Image Upgrade Support

An FPD image is used to upgrade the software on an FPD.

FPD versions must be compatible with the Cisco IOS XR software that is running on the router; if an incompatibility exists between an FPD version and the Cisco IOS XR software, the device with the FPGA may not operate properly until the incompatibility is resolved.


Note

Downgrade of FPDs is not recommended.

Note

FPD auto upgrade is not supported on this router.

FPD upgrade service

The main tasks of the FPD upgrade service are:

  • Check FPD image version to decide if a specific firmware image needs an upgrade or not.

  • Automatic FPD Image Upgrade (if enabled).

  • Manual FPD Image Upgrade using the upgrade hw-module fpd command.

  • Invoke the appropriate device driver with a name of the new image to load.

An FPD image package is used to upgrade FPD images. The install activate command is used to place the FPD binary files into the expected location on the boot devices.

Supported Upgrade Methods

Method

Remarks

Manual Upgrade

Upgrade using CLI, force upgrade supported.


Important

FPD auto upgrade is not supported on this router.

Determining Upgrade Requirement

Use the show hw-module fpd command to determine if an FPD upgrade is required. Check for NEED UPGD in the Status column.

Use the show fpd package command to find out which FPGAs are supported with your current software release and minimum hardware requirements for each module.

Manual FPD upgrade

Manual FPD upgrade is performed using the upgrade hw-module fpd command. All cards or all of FPGA in a card can be upgraded. If reload is required to activate FPD , the upgrade should be complete. Interface module (IMs) and RSPs cannot be reloaded during the process of the FPD upgrade.

FPD upgrade is transaction-based:

  • Each fpd upgrade cli execution is one transaction

  • Only one transaction is allowed at any given time

  • One transaction may include one or many FPD upgrade(s)

The force option can be used to forcibly upgrade the FPD (regardless of whether it is required or not). It triggers all FPDs to be upgraded or downgraded. The force option can also be used to downgrade or upgrade the FPGAs even after the version check. However, the force option must be used cautiously and only to recover a component from a failed upgrade.


Note

In some cases, FPDs can have primary and backup images.

How to Upgrade FPD Images

  • Migrate the software to a later Cisco IOS XR software release.

  • Swap IMs or RSPs from a system running a different Cisco IOS XR software release.

  • Insert a new IM or RSP.

In the event that there is an FPD incompatibility with your card, you may receive an error message. If you upgrade to a newer version of the Cisco IOS XR software and there is an FPD incompatibility, you receive the following message:

LC/0/1/CPU0:Dec 23 16:33:47.945 : spa_192_jacket_v2[203]: %PLATFORM-UPGRADE_FPD-4-DOWN_REV : spa fpga2 instance 0 is down-rev (V0.6), upgrade to (V1.0). Use the "upgrade hw-module fpd" CLI in admin mode.

If the FPD image on the card is newer then what is required by the currently running Cisco IOS XR software image on the router, you receive the following error message:

LC/0/1/CPU0:Dec 23 16:33:47.955 : spa_192_jacket_v2[203]: %PLATFORM-UPGRADE_FPD-4-UP_REV : spa fpga instance 1 is severely up-rev (V2.1), downgrade to (V1.6). Use the "upgrade hw-module fpd" CLI in admin mode.

You should perform the FPD upgrade procedure if you receive such messages. Cards may not function properly if FPD incompatibilities are not resolved.


Note

The use of the force option when performing a FPD upgrade is not recommended except under explicit direction from Cisco engineering or TAC.

Before you begin

  • The FPD upgrade procedure is performed while the card is online. At the end of the procedure the card must be reloaded before the FPD upgrade is complete. To automatically reload the card, you can use the hw-module reload command during your next maintenance window. The upgrade procedure is not complete until the card is reloaded.

  • During the FPD upgrade, you must not do the following:

    • Reload, perform an online insertion and removal (OIR) of a line card (LC), or power down the chassis. Doing so may cause the node to enter an unusable state.

    • Press Ctrl-C if the console appears to hang without any output. Doing so may abort the upgrade.

  • If you are not sure whether a card requires an FPD upgrade, you can install the card and use the show hw-module fpd command to determine if the FPD image on the card is compatible with the currently running Cisco IOS XR software release.

Configuration Examples for FPD Image Upgrade

The following examples indicates the use of commands associated with the FPD image upgrade procedure.

show fpd package Command Output: Example

Use the show fpd package command in System Admin EXEC mode to find out which IMs and RSPs are supported with your current Cisco IOS XR software release, which FPD image package you need for each IM or RSP, and what the minimum hardware requirements are for each module. If multiple FPD images are available for your card, they are listed as Subtype fpga2, fpga3, and so on.


Note

The FPD name used in the FPD Description column of the output of the show fpd package command includes the last ten characters of DCO-PID. Depending on the slot and port numbers, the FPD name is appended with DCO_0, DCO_1, or DCO_2. For example, the FPD names for CFP2-WDM-D-1HL in port 0 and port 1 are -WDM-D-1HL_DCO_0 and WDM-D-1HL_DCO_1 respectively.

The following example shows a sample output from the show fpd package command: 

sysadmin_vm:0_RP0# show fpd package
Fri Oct 16 07:20:50.021 UTC

=============================== ================================================
                                        Field Programmable Device Package
                                ================================================
                                               Req     SW      Min Req   Min Req
Card Type            FPD Description           Reload  Ver     SW Ver  Board Ver
=================== ========================== ====== ======= ======== =========
--------------------------------------------------------------------------------
A900-IMA8CS1Z-CC     IMFPGA                     YES     1.96    1.96    0.0  
--------------------------------------------------------------------------------
A900-IMA8CS1Z-M      IMFPGA                     YES     1.96    1.96    0.0  
--------------------------------------------------------------------------------
A900-IMA8Z           IMFPGA                     YES    17.05   17.05    0.0  
--------------------------------------------------------------------------------
A900-IMA8Z-CC        IMFPGA                     YES    17.05   17.05    0.0  
--------------------------------------------------------------------------------
A900-IMA8Z-L         IMFPGA                     YES     1.45    1.45    0.0  
--------------------------------------------------------------------------------
A900-PWR1200-A       DCA-PriMCU(A)              NO      0.11    0.11    0.0  
                     DCA-SecMCU(A)              NO      1.04    1.04    0.0  
--------------------------------------------------------------------------------
A900-PWR1200-D       LIT-PriMCU(A)              NO      2.04    0.04    0.0  
                     LIT-SecMCU(A)              NO      1.23    1.23    0.0  
--------------------------------------------------------------------------------
A907-FAN-E           PSOC(A)                    NO      1.65    1.65    0.0  
                     PSOC(A)                    NO      1.66    1.66    0.4  
--------------------------------------------------------------------------------
N560-4-FAN-H         PSOC(A)                    NO    177.02  177.02    0.0  
--------------------------------------------------------------------------------
N560-4-FAN-H-CC      PSOC(A)                    NO    177.02  177.02    0.0  
--------------------------------------------------------------------------------
N560-4-PWR-FAN       PSOC(A)                    NO    177.08  177.08    0.0  
--------------------------------------------------------------------------------
N560-4-PWR-FAN-CC    PSOC(A)                    NO    177.08  177.08    0.0  
--------------------------------------------------------------------------------
N560-4-RSP4          ADM(A)                     NO      1.06    1.06    0.0  
                     IOFPGA(A)                  YES     0.64    0.64    0.0  
                     PRIMARY-BIOS(A)            YES     0.17    0.17    0.0  
                     SATA(A)                    YES     2.10    2.10    0.0  
--------------------------------------------------------------------------------
N560-4-RSP4-CC       ADM(A)                     NO      1.06    1.06    0.0  
                     IOFPGA(A)                  YES     0.64    0.64    0.0  
                     PRIMARY-BIOS(A)            YES     0.17    0.17    0.0  
                     SATA(A)                    YES     2.10    2.10    0.0  
--------------------------------------------------------------------------------
N560-4-RSP4E         ADM(A)                     NO      1.06    1.06    0.0  
                     IOFPGA(A)                  YES     0.64    0.64    0.0  
                     PRIMARY-BIOS(A)            YES     0.17    0.17    0.0  
                     SATA(A)                    YES     2.10    2.10    0.0  
--------------------------------------------------------------------------------
N560-4-RSP4E-CC      ADM(A)                     NO      1.06    1.06    0.0  
                     IOFPGA(A)                  YES     0.64    0.64    0.0  
                     PRIMARY-BIOS(A)            YES     0.17    0.17    0.0  
                     SATA(A)                    YES     2.10    2.10    0.0  
--------------------------------------------------------------------------------
N560-FAN-H           PSOC(A)                    NO      2.02    2.02    0.0  
--------------------------------------------------------------------------------
N560-IMA-8Q/4L       IMFPGA                     YES     1.08    1.08    0.0  
--------------------------------------------------------------------------------
N560-IMA1W           CFP2-D-DCO                 NO     38.273  38.273   0.0  
                     CFP2-DE-DCO                NO     38.273  38.273   0.0  
                     CFP2-DET-DCO               NO     38.273  38.273   0.0  
                     CFP2-DETS-DCO              NO     38.273  38.273   0.0  
                     CFP2-DS-DCO                NO     38.273  38.273   0.0  
                     CFP2-DS100-DCO             NO     38.273  38.273   0.0  
                     IMFPGA                     YES     1.24    1.24    0.0  
--------------------------------------------------------------------------------
N560-IMA2C-CC        IMFPGA                     YES     5.04    5.04    0.0  
--------------------------------------------------------------------------------
N560-PWR1200-D-E     QCS-PriMCU(A)              NO      1.82    1.82    0.0  
                     QCS-SecMCU(A)              NO      1.84    1.84    0.0  
--------------------------------------------------------------------------------
N560-RSP4            ADM(A)                     NO      1.06    1.06    0.0  
                     IOFPGA(A)                  YES     0.64    0.64    0.0  
                     PRIMARY-BIOS(A)            YES     0.17    0.17    0.0  
                     SATA(A)                    YES     2.10    2.10    0.0  
--------------------------------------------------------------------------------
N560-RSP4-E          ADM(A)                     NO      1.06    1.06    0.0  
                     IOFPGA(A)                  YES     0.64    0.64    0.0  
                     PRIMARY-BIOS(A)            YES     0.17    0.17    0.0  
                     SATA(A)                    YES     2.10    2.10    0.0  
--------------------------------------------------------------------------------
NCS560-IMA2C         IMFPGA                     YES     5.04    5.04    0.0  
--------------------------------------------------------------------------------
NCS560-IMA2C-DD      IMFPGA                     YES     1.24    1.24    0.0  
--------------------------------------------------------------------------------
NCS560-IMA2C-L       IMFPGA                     YES     1.24    1.24    0.0
upgrade hw-module fpd Command Output: Example

Use the upgrade hw-module fpd command to upgrade the FPD image. 

RP/0/RP0/CPU0:ios# upgrade hw-module location 0/RP0 fpd ADM
Wed Oct 28 07:46:49.805 UTC
upgrade command issued (use "show hw-module fpd" to check upgrade status)
RP/0/RP0/CPU0:ios#RP/0/RP0/CPU0:Oct 28 07:46:51.949 UTC: optics_driver[222]: %PKT_INFRA-FM-2-FAULT_CRITICAL : ALARM_CRITICAL :OPTICS RX POWER LANE-0 LOW ALARM :CLEAR : Optics0/11/0/1:  Optics0/11/0/1 
0/RP0/ADMIN0:Oct 28 07:46:54.154 UTC: fpdserv[4899]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Upgrade for the following FPDs has been committed:   
0/RP0/ADMIN0:Oct 28 07:46:54.154 UTC: fpdserv[4899]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Location           FPD name             Force     
0/RP0/ADMIN0:Oct 28 07:46:54.154 UTC: fpdserv[4899]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : ==================================================   
0/RP0/ADMIN0:Oct 28 07:46:54.154 UTC: fpdserv[4899]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP0              ADM                  FALSE  
0/RP0/ADMIN0:Oct 28 07:46:59.203 UTC: control_driver[3690]: %INFRA-FPD_Driver-6-UPGRADE_RESULT : Upgrade completes 20 percent for fpd ADM@location 0/RP0.  
RP/0/RP0/CPU0:ios#0/RP0/ADMIN0:Oct 28 07:47:09.204 UTC: control_driver[3690]: %INFRA-FPD_Driver-6-UPGRADE_RESULT : Upgrade completes 70 percent for fpd ADM@location 0/RP0.  
0/RP0/ADMIN0:Oct 28 07:47:10.854 UTC: control_driver[3690]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD ADM@0/RP0 image programming completed with UPGRADE DONE state Info: [image 1.05 to 1.06 version]   
0/RP0/ADMIN0:Oct 28 07:47:10.855 UTC: control_driver[3690]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD ADM @location 0/RP0 FPD upgraded and activated!   
0/RP0/ADMIN0:Oct 28 07:47:10.857 UTC: shelf_mgr[3705]: %INFRA-SHELF_MGR-6-CARD_SW_OPERATIONAL : Card: 0/RP0 software state going to Operational  
0/RP0/ADMIN0:Oct 28 07:47:10.857 UTC: shelf_mgr[3705]: %INFRA-SHELF_MGR-6-CARD_HW_OPERATIONAL : Card: 0/RP0 hardware state going to Operational  
RP/0/RP0/CPU0:Oct 28 07:47:17.931 UTC: fpd-serv[393]: %INFRA-FPD_Manager-1-UPGRADE_ALERT :  FPD Upgrade Completed(use "show hw-module fpd" to check upgrade status)  
0/RP0/ADMIN0:Oct 28 07:47:19.155 UTC: fpdserv[4899]: %INFRA-FPD_Manager-1-UPGRADE_ALERT :  FPD Upgrade Completed(use "show hw-module fpd" to check upgrade status) 
RP/0/RP0/CPU0:ios#
show platform Command Output: Example

Use the show platform command to verify that the IM is up and running. 

RP/0/RP1/CPU0:router# show platform 
Tue Oct 20 04:42:31.936 UTC
Node              Type                       State             Config state
--------------------------------------------------------------------------------
0/0/CPU0          A900-IMA8CS1Z-M            OPERATIONAL       NSHUT
0/1/CPU0          A900-IMA8CS1Z-M            OPERATIONAL       NSHUT
0/2/CPU0          A900-IMA8CS1Z-M            OPERATIONAL       NSHUT
0/4/CPU0          A900-IMA8Z-L               OPERATIONAL       NSHUT
0/5/CPU0          A900-IMA8Z-L               OPERATIONAL       NSHUT
0/7/CPU0          N560-IMA1W                 OPERATIONAL       NSHUT
0/9/CPU0          N560-IMA2C-DD              OPERATIONAL       NSHUT
0/10/CPU0         A900-IMA8Z                 OPERATIONAL       NSHUT
0/11/CPU0         A900-IMA8Z-L               OPERATIONAL       NSHUT
0/RP0/CPU0        N560-RSP4-E(Standby)       UNKNOWN           NSHUT
0/RP1/CPU0        N560-RSP4-E(Active)        IOS XR RUN        NSHUT
0/FT0/CPU0        N560-FAN-H                 OPERATIONAL       NSHUT
0/PM2/CPU0        A900-PWR1200-A             OPERATIONAL       NSHUT

Auto FPD Upgrade

Table 1. Feature History Table

Feature Name

Release Information

Feature Description

Auto FPD Upgrade

Release 7.3.2

This functionality enables automatic upgrade and reload for field-programmable devices (FPDs) whenever the Cisco IOS XR image has a newer FPD version. This functionality upgrades all route processor and interface module FPDs simultaneously while displaying upgrade triggers on the console.

Effective Cisco IOS XR Release 7.3.2, you can enable automatic upgrade of FPD by using the “fpd auto-upgrade enable” command.

To automatically upgrade all FPDs, use:

RP/0/RP0/CPU0:IOS(config)#fpd auto-upgrade enable

To reload the interface modules following the fpd auto-upgrade, use:

 RP/0/RP0/CPU0:IOS(config)#fpd auto-reload enable

Limitations and Usage Guidelines

Limitations

  • FPD auto-upgrade should be enabled only in the XR VM and not in the System Admin VM.

  • With auto-upgrade enabled, if any card is in RELOAD REQUIRED state, auto-upgrade is re-triggered during any SSO or FPD-serv process restart.

  • When an interface module (IM) or route processor (RP) is in RELOAD REQUIRED state and auto-upgrade is enabled, FPD upgrades are triggered again.

  • With auto-upgrade enabled, if Ian M is inserted, auto-upgrade is triggered. During this phase optics alarms are generated. If auto-reload is not enabled, you must reload the IMs manually to clear these alarms.

  • SATA allows you to upgrade or downgrade when an FPD version change is available. Therefore, when auto-upgrade is enabled, the system automatically downgrades if lower versions are available. This behavior is specific only to SATA FPDs.

  • You must disable auto-upgrade during XR ISSU; otherwise, the router goes into a state where redundancy cannot be achieved. In this case, standby RP must be reloaded to achieve redundancy.

  • FPD auto-reload is applicable for IMs only. IMs are automatically reloaded after the fpd auto-upgrade process is completed.

Usage Guidelines—Online Insertion of IMs

When an IM with a lower FPD version is inserted, one of the following scenarios apply:

  • If fpd auto-upgrade and auto-reload are enabled, and a new IM is inserted, the system upgrades the IMs FPDs automatically with the latest FPDs and reloads the IMs.

  • If fpd auto-upgrade and auto-reload are both disabled, no action is required.

  • If fpd auto-upgrade is enabled and auto-reload is disabled, the following alarms are displayed on the console: 

    RP/0/RP1/CPU0:Jun 1 10:05:46.095 UTC: optics_driver[231]: %PKT_INFRA-FM-3-FAULT_MAJOR : ALARM_MAJOR :OPTICS SUPPORTED_ERROR :DECLARE : Optics0/5/0/6: Optics0/5/0/6
RP/0/RP1/CPU0:Jun 1 10:05:46.096 UTC: optics_driver[231]: %PKT_INFRA-FM-2-FAULT_CRITICAL : ALARM_CRITICAL :OPTICS NOT SUPPORTED :DECLARE : Optics0/5/0/6: Optics0/5/0/6

    You must reload the IM manually to clear these alarms

Usage Guidelines—Online Insertion of RPs

When fpd auto-upgrade is enabled and a new RP is inserted, the system upgrades the RP FPDs automatically with the latest FPDs.


Note

RPs are not reloaded automatically. You must manually reload the RP or chassis for the latest FPD version to reflect.

Note

Reload impacts the network traffic.

To reload the RP:

RP/0/RP0/CPU0:IOS# admin
Mon Jun 28 17:00:39.340 UTC

sysadmin-vm:0_RP1# hw-module location 0/RP1 reload 
Mon Jun  28 17:00:52.178 UTC+00:00
Reload hardware module ? [no,yes] yes 
#result Card graceful reload request on 0/RP1 succeeded.
RP/0/RP0/CPU0:IOS#

RP/0/RP0/CPU0:ios# show hw-module fpd
Fri Jun  4 10:08:01.784 UTC

Auto-upgrade:Enabled
                                                                         FPD Versions
                                                                        ==============
Location   Card type             HWver FPD device       ATR Status    Running   Programd
-----------------------------------------------------------------------------------------
0/1        N560-IMA2C-DD         0.0   IMFPGA               CURRENT     1.27      1.27
0/3        N560-IMA2C            0.0   IMFPGA               CURRENT     5.01      5.01
0/5        A900-IMA8CS1Z-M       0.0   IMFPGA               CURRENT     1.98      1.98
0/RP0      N560-4-RSP4           0.0   ADM                  CURRENT     1.06      1.06
0/RP0      N560-4-RSP4           0.0   IOFPGA               CURRENT     0.64      0.64
0/RP0      N560-4-RSP4           0.0   PRIMARY-BIOS         CURRENT     0.18      0.18
0/RP0      N560-4-RSP4           0.0   SATA                 CURRENT     2.10      2.10
0/RP1      N560-4-RSP4           0.0   ADM                  CURRENT     1.06      1.06
0/RP1      N560-4-RSP4           0.0   IOFPGA               CURRENT     0.64      0.64
0/RP1      N560-4-RSP4           0.0   PRIMARY-BIOS         CURRENT     0.19      0.19
0/RP1      N560-4-RSP4           0.0   SATA                 CURRENT     2.10      2.10
0/FT0      N560-4-PWR-FAN        0.1   PSOC                 CURRENT   177.08    177.08
0/FT1      N560-4-FAN-H          0.1   PSOC                 CURRENT   177.02    177.02
0/FT2      N560-4-FAN-H          0.1   PSOC                 CURRENT   177.02    177.02
RP/0/RP0/CPU0:ios#
Table 2. Action Required on FPDs After Auto Upgrade

FPD

Action Required

IOFPGA

Manual reload required

ADM

Upgraded version available immediately

PRIMARY-BIOS

Manual reload required

SATA

Upgraded version available immediately

PSOC

Upgraded version available immediately

IMFPGA

Manual reload required, if auto-reload is not configured

Configuring Auto FPD During System Upgrade

In case of Software upgrade (without ISSU), configure the fpd auto-upgrade enable command. All the IM and RP FPDs are automatically upgraded in the currently installed image (V1). After the upgrade, the router automatically reloads and comes up with the new image (V2) with the upgraded FPDs already running. No additional reloads are required.


Note

System reloads are part of the SU process, therefore you can disable the FPD auto reload functionality by using the fpd auto-reload disable command.

  1. Enable FPD auto-upgrade
    RP/0/RP0/CPU0:IOS#conf    
RP/0/RP0/CPU0:IOS(config)#fpd auto-upgrade enable
RP/0/RP0/CPU0:IOS#commit
  2. Check for FPD Versions
    RP/0/RP0/CPU0:Router#show hw-module fpd 
Mon Jun 28 21:41:19.187 UTC

Auto-upgrade:Enabled
                                                                         FPD Versions
                                                                        ==============
Location   Card type             HWver FPD device       ATR Status    Running   Programd  
-----------------------------------------------------------------------------------------
0/2        NCS4200-1T16G-PS      0.0   IMFPGA               CURRENT     1.98      1.98    
0/3        A900-IMA8CS1Z-M       0.0   IMFPGA               CURRENT     1.95      1.95    
0/4        A900-IMA8Z            0.0   IMFPGA               CURRENT    17.05     17.05    
0/5        A900-IMA8Z-L          0.0   IMFPGA               CURRENT     1.48      1.48    
0/8        NCS4200-1T16G-PS      0.0   IMFPGA               CURRENT     1.98      1.98    
0/9        N560-IMA1W            66.32 CFP2-DS-DCO          CURRENT    38.27397  38.27397 
0/9        N560-IMA1W            0.0   IMFPGA               CURRENT     1.28      1.28    
0/15       NCS4200-1T16G-PS      0.0   IMFPGA               CURRENT     1.98      1.98    
0/RP0      N560-RSP4             0.0   ADM                  CURRENT     1.06      1.06    
0/RP0      N560-RSP4             0.0   IOFPGA               CURRENT     0.64      0.64    
0/RP0      N560-RSP4             0.0   PRIMARY-BIOS         CURRENT     0.19      0.19    
0/RP0      N560-RSP4             0.0   SATA                 CURRENT     1.30      1.30    
0/RP1      N560-RSP4             0.0   ADM                  CURRENT     1.05      1.05    
0/RP1      N560-RSP4             0.0   IOFPGA               CURRENT     0.64      0.64    
0/RP1      N560-RSP4             0.0   PRIMARY-BIOS         CURRENT     0.19      0.19    
0/RP1      N560-RSP4             0.0   SATA                 CURRENT     1.30      1.30    
0/FT0      N560-FAN-H            1.0   PSOC                 CURRENT     2.02      2.02    
RP/0/RP0/CPU0:Router#

  3. Check that Auto Upgrades are Triggered for FPDs with Newer Versions Available


    Note

    At this step, all RsP, IMs, and fan FPD upgrades are initiated and completed. All cards are upgraded before the router reloads. 

    RP/0/RP1/CPU0:UUT-RSP4# copy tftp://<ncs560-mini-x-7.3.2.iso> harddisk:/
RP/0/RP0/CPU0:IOS#install add source harddisk: ncs560-mini-x-7.3.2.iso ncs560-mcast-2.0.0.0-r732.x86_64.rpm ncs560-mgbl-2.0.0.0-r732.x86_64.rpm ncs560-mpls-1.0.0.0-r732.x86_64.rpm
RP/0/RP0/CPU0:IOS#install ncs560-mini-x-7.3.2 ncs560-mcast-2.0.0.0-r732.x86_64 ncs560-mgbl-2.0.0.0-r732.x86_64 ncs560-mpls-1.0.0.0-r732.x86_64
RP/0/RP0/CPU0:IOS#install commit

    RP/0/RP0/CPU0:ROUTER# install activate ncs560-mini-x-7.3.2.28I
Mon Jun 28 21:30:17.673 UTC
2021-06-28 21:30:20 Install operation 31 started by root123:
  install activate pkg ncs560-mini-x-7.3.2.28I 
2021-06-28 21:30:20 Package list:
2021-06-28 21:30:20     ncs560-mini-x-7.3.2.28I
RP/0/RP0/CPU0:Jun 28 21:32:41.204 UTC: sdr_instmgr[1213]: %PKT_INFRA-FM-6-FAULT_INFO : INSTALL-IN-PROGRESS :DECLARE :0/RP0/CPU0: INSTALL_IN_PROGRESS Alarm : being DECLARED for the system   
This install operation will reload the system, continue?
 [yes/no]:[yes] yes
2021-06-28 21:33:01 Install operation will continue in the background
	...
RP/0/RP0/CPU0:ROUTER#RP/0/RP0/CPU0:Jun 28 21:41:40.910 UTC: fpd-serv[168]: %PKT_INFRA-FM-3-FAULT_MAJOR : ALARM_MAJOR :FPD-NEED-UPGRADE :DECLARE :0/RP0:  
RP/0/RP0/CPU0:Jun 28 21:41:41.159 UTC: fpd-serv[168]: %PKT_INFRA-FM-3-FAULT_MAJOR : ALARM_MAJOR :FPD-NEED-UPGRADE :CLEAR :0/RP0:  
0/RP0/ADMIN0:Jun 28 21:41:42.565 UTC: control_driver[3205]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD SATA@0/RP0 image programming completed with UPGRADE DONE state Info: [SDD firmware upgraded from 1.30 to 2.10]   
0/RP0/ADMIN0:Jun 28 21:41:42.566 UTC: control_driver[3205]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD SATA @location 0/RP0 FPD upgraded and activated!   
0/RP0/ADMIN0:Jun 28 21:41:42.570 UTC: shelf_mgr[3220]: %INFRA-SHELF_MGR-6-CARD_SW_OPERATIONAL : Card: 0/RP0 software state going to Operational  
0/RP0/ADMIN0:Jun 28 21:41:42.570 UTC: shelf_mgr[3220]: %INFRA-SHELF_MGR-6-CARD_HW_OPERATIONAL : Card: 0/RP0 hardware state going to Operational  
RP/0/RP0/CPU0:Jun 28 21:41:42.486 UTC: fpd-serv[168]: %PKT_INFRA-FM-3-FAULT_MAJOR : ALARM_MAJOR :FPD-NEED-UPGRADE :DECLARE :0/RP1:  
0/RP1/ADMIN0:Jun 28 21:41:44.182 UTC: control_driver[3220]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD SATA@0/RP1 image programming completed with UPGRADE DONE state Info: [SDD firmware upgraded from 1.30 to 2.10]   
0/RP1/ADMIN0:Jun 28 21:41:44.182 UTC: control_driver[3220]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD SATA @location 0/RP1 FPD upgraded and activated!   
0/RP1/ADMIN0:Jun 28 21:41:48.905 UTC: control_driver[3220]: %INFRA-FPD_Driver-6-UPGRADE_RESULT : Upgrade completes 20 percent for fpd ADM@location 0/RP1.  
0/RP1/ADMIN0:Jun 28 21:41:48.905 UTC: control_driver[3220]: %INFRA-FPD_Driver-6-UPGRADE_RESULT : Upgrade completes 50 percent for fpd PRIMARY-BIOS@location 0/RP1.  
0/RP1/ADMIN0:Jun 28 21:42:10.160 UTC: control_driver[3220]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD PRIMARY-BIOS@0/RP1 image programming completed with UPGRADE DONE state Info: [  Upgrade Complete ]   
0/RP1/ADMIN0:Jun 28 21:42:10.161 UTC: control_driver[3220]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD PRIMARY-BIOS @location 0/RP1 upgrade completed.    
0/RP1/ADMIN0:Jun 28 21:42:11.060 UTC: control_driver[3220]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD ADM@0/RP1 image programming completed with UPGRADE DONE state Info: [image 1.05 to 1.06 version]   
0/RP1/ADMIN0:Jun 28 21:42:11.061 UTC: control_driver[3220]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD ADM @location 0/RP1 FPD upgraded and activated!   
0/RP0/ADMIN0:Jun 28 21:42:11.062 UTC: shelf_mgr[3220]: %INFRA-SHELF_MGR-6-CARD_SW_OPERATIONAL : Card: 0/RP1 software state going to Operational  
0/RP0/ADMIN0:Jun 28 21:42:11.062 UTC: shelf_mgr[3220]: %INFRA-SHELF_MGR-6-CARD_HW_OPERATIONAL : Card: 0/RP1 hardware state going to Operational  
RP/0/RP0/CPU0:Jun 28 21:45:14.615 UTC: fpd_imfpga[121]: %INFRA-FPD_Driver-6-UPGRADE_RESULT : Upgrade completes 50 percent for fpd IMFPGA@location 0/5. 
RP/0/RP0/CPU0:Jun 28 21:45:14.616 UTC: fpd_imfpga[121]: %INFRA-FPD_Driver-6-UPGRADE_RESULT : Upgrade completes 50 percent for fpd IMFPGA@location 0/3.
RP/0/RP0/CPU0:Jun 28 21:48:24.763 UTC: fpd_imfpga[121]: %INFRA-FPD_Driver-6-UPGRADE_RESULT : Upgrade completes 90 percent for fpd IMFPGA@location 0/5. 

RP/0/RP0/CPU0:ROUTER#RP/0/RP0/CPU0:Jun 28 21:48:43.929 UTC: fpd_imfpga[121]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD IMFPGA@0/5 image programming completed with UPGRADE DONE state Info: [DONE ]  
RP/0/RP0/CPU0:Jun 28 21:48:43.940 UTC: fpd_imfpga[121]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD IMFPGA @location 0/5 upgrade completed.   
0/RP0/ADMIN0:Jun 28 21:48:45.354 UTC: shelf_mgr[3220]: %INFRA-SHELF_MGR-6-CARD_HW_OPERATIONAL : Card: 0/5 hardware state going to Operational  
0/RP0/ADMIN0:Jun 28 21:48:45.354 UTC: shelf_mgr[3220]: %INFRA-SHELF_MGR-6-CARD_SW_OPERATIONAL : Card: 0/RP0 software state going to Operational  
0/RP0/ADMIN0:Jun 28 21:48:45.354 UTC: shelf_mgr[3220]: %INFRA-SHELF_MGR-6-CARD_HW_OPERATIONAL : Card: 0/RP0 hardware state going to Operational
P/0/RP0/CPU0:Jun 28 21:50:21.630 UTC: fpd_imfpga[121]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD IMFPGA@0/3 image programming completed with UPGRADE DONE state Info: [DONE ]  
RP/0/RP0/CPU0:Jun 28 21:50:21.642 UTC: fpd_imfpga[121]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD IMFPGA @location 0/3 upgrade completed.   
0/RP0/ADMIN0:Jun 28 21:50:23.056 UTC: shelf_mgr[3220]: %INFRA-SHELF_MGR-6-CARD_HW_OPERATIONAL : Card: 0/3 hardware state going to Operational  
0/RP0/ADMIN0:Jun 28 21:50:23.056 UTC: shelf_mgr[3220]: %INFRA-SHELF_MGR-6-CARD_SW_OPERATIONAL : Card: 0/RP0 software state going to Operational  
0/RP0/ADMIN0:Jun 28 21:50:23.057 UTC: shelf_mgr[3220]: %INFRA-SHELF_MGR-6-CARD_HW_OPERATIONAL : Card: 0/RP0 hardware state going to Operational

  4. Check the RP FPD Versions and FPD Status

    When the router is operational after the reload, all the RP, IMs, and fan FPDs are upgraded to the latest FPD versions. 

    sysadmin-vm:0_RP1# show hw-module fpd 
Mon Jun 28 21:51:20.187 UTC

Auto-upgrade:Enabled
                                                                         FPD Versions
                                                                        ==============
Location   Card type             HWver FPD device       ATR Status    Running   Programd  
-----------------------------------------------------------------------------------------
0/2        NCS4200-1T16G-PS      0.0   IMFPGA               CURRENT     1.98      1.98    
0/3        A900-IMA8CS1Z-M       0.0   IMFPGA               CURRENT     1.98      1.98    
0/4        A900-IMA8Z            0.0   IMFPGA               CURRENT    17.05     17.05    
0/5        A900-IMA8Z-L          0.0   IMFPGA               CURRENT     1.49      1.49    
0/8        NCS4200-1T16G-PS      0.0   IMFPGA               CURRENT     1.98      1.98    
0/9        N560-IMA1W            66.32 CFP2-DS-DCO          CURRENT    38.27397  38.27397 
0/9        N560-IMA1W            0.0   IMFPGA               CURRENT     1.28      1.28    
0/15       NCS4200-1T16G-PS      0.0   IMFPGA               CURRENT     1.98      1.98    
0/RP0      N560-RSP4             0.0   ADM                  CURRENT     1.06      1.06    
0/RP0      N560-RSP4             0.0   IOFPGA               CURRENT     0.64      0.64    
0/RP0      N560-RSP4             0.0   PRIMARY-BIOS         CURRENT     0.19      0.19    
0/RP0      N560-RSP4             0.0   SATA                 CURRENT     2.10      2.10    
0/RP1      N560-RSP4             0.0   ADM                  CURRENT     1.05      1.05    
0/RP1      N560-RSP4             0.0   IOFPGA               CURRENT     0.64      0.64    
0/RP1      N560-RSP4             0.0   PRIMARY-BIOS         CURRENT     0.19      0.96    
0/RP1      N560-RSP4             0.0   SATA                 CURRENT     2.10      2.10    
0/FT0      N560-FAN-H            1.0   PSOC                 CURRENT     2.02      2.02    
RP/0/RP0/CPU0:ROUTER#

Configuring Auto FPD During ISSU

ISSU occurs in two phases—in the System Admin VM and then in the XR VM.

In the System Admin VM mode, first execute the fpd auto-upgrade enable command. This configuration causes the FPDs on the route processor (RP) to automatically upgrade to the latest versions.

In the XR VM mode, you must disable the FPD auto upgrade. This is because IM FPGA upgrades are not automatically triggered as part of ISSU.


Note

Less than 50ms traffic loss is expected during ISSU.

Note

During both Admin and XR ISSU, always disable the auto reload functionality by using the fpd auto-reload disable command.

  1. Enable FPD auto-upgrade in XR VM
    RP/0/RP0/CPU0:IOS#conf    
RP/0/RP0/CPU0:IOS(config)#fpd auto-upgrade enable 
RP/0/RP0/CPU0:IOS#commit
  2. Disable FPD auto-reload in XR VM
    RP/0/RP0/CPU0:IOS#conf    
RP/0/RP0/CPU0:IOS(config)#fpd auto-reload disable 
RP/0/RP0/CPU0:IOS#commit
  3. Check for FPD Versions
    RP/0/RP0/CPU0:IOS#show hw-module fpd 
Sun Aug  1 19:36:13.869 UTC

Auto-upgrade:Enabled
                                                                         FPD Versions
                                                                        ==============
Location   Card type             HWver FPD device       ATR Status    Running   Programd  
-----------------------------------------------------------------------------------------
0/1        NCS4200-1T16G-PS      0.0   IMFPGA               CURRENT     1.95      1.95    
0/3        NCS4200-1T16G-PS      0.0   IMFPGA               CURRENT     1.95      1.95    
0/4        A900-IMA8Z            0.0   IMFPGA               CURRENT    17.05     17.05    
0/7        N560-IMA2C            0.0   IMFPGA               CURRENT     4.96      4.96    
0/9        A900-IMA8Z            0.0   IMFPGA               CURRENT    17.05     17.05    
0/RP0      N560-RSP4-E           0.0   ADM                  CURRENT     1.06      1.06    
0/RP0      N560-RSP4-E           0.0   IOFPGA               CURRENT     0.67      0.67    
0/RP0      N560-RSP4-E           0.0   PRIMARY-BIOS         CURRENT     0.16      0.16    
0/RP0      N560-RSP4-E           0.0   SATA                 CURRENT     1.30      1.30    
0/RP1      N560-RSP4-E           0.0   ADM                  CURRENT     1.06      1.06    
0/RP1      N560-RSP4-E           0.0   IOFPGA               CURRENT     0.67      0.67    
0/RP1      N560-RSP4-E           0.0   PRIMARY-BIOS         CURRENT     0.16      0.16    
0/RP1      N560-RSP4-E           0.0   SATA                 CURRENT     1.30      1.30    
0/FT0      N560-FAN-H            1.0   PSOC                 CURRENT     2.02      2.02    
RP/0/RP0/CPU0:IOS#

  4. Perform sysadmin ISSU Upgrade

    Check that FPD upgrades are triggered as part of ISSU:
    RP/0/RP0/CPU0:IOS#admin
sysadmin-vm:0_RP0# install extract ncs560-mini-x-7.3.2
Mon Aug  2  04:54:17.250 UTC+00:00
result Mon Aug  2 04:54:18 2021 Install operation 1 (install extract) started by user 'cafyauto' will continue asynchronously.
sysadmin-vm:0_RP0# 0/RP0/ADMIN0:Aug  2 04:58:47.990 UTC: inst_mgr[4350]: %INFRA-INSTMGR-2-OPERATION_SUCCESS : Install operation 1 completed successfully  
Mon Aug  2 04:58:47 2021 Install operation 1 completed successfully.
sysadmin-vm:0_RP0# install activate issu  host-7.3.2 ncs560-sysadmin-7.3.2
Mon Aug  2  05:26:31.442 UTC+00:00
op-initiated true
operation-id 2
result start-success
id 2
sysadmin-vm:0_RP0# 2021-08-02 05:26:31 Admin install operation 2 started
sysadmin-vm:0_RP0# 2021-08-02 05:26:31 install activate issu host-7.3.2 ncs560-sysadmin-7.3.2
sysadmin-vm:0_RP0# 2021-08-02 05:26:35 Install operation 2 started: ISSU prepare
install activate issu host-7.3.2 ncs560-sysadmin-7.3.2
sysadmin-vm:0_RP0# 0/RP0/ADMIN0:Aug  2 05:30:22.441 UTC: inst_mgr[4350]: %INFRA-INSTMGR-2-OPERATION_SUCCESS : Install operation 2 completed successfully  
2021-08-02 05:30:22 Install operation 2 completed successfully
sysadmin-vm:0_RP0# 2021-08-02 05:30:23 Admin install operation 3 started
sysadmin-vm:0_RP0# 2021-08-02 05:30:23 install activate issu host-7.3.2 ncs560-sysadmin-7.3.2
sysadmin-vm:0_RP0# 2021-08-02 05:30:25 Install operation 3 started: ISSU activate
install activate issu host-7.3.2 ncs560-sysadmin-7.3.2
sysadmin-vm:0_RP0#
sysadmin-vm:0_RP0# 2021-08-02 05:31:10 Admin VM of node 0/RP1 will now reload as part of ISSU operation 3
0/RP0/ADMIN0:Aug  2 05:31:17.314 UTC: shelf_mgr[3212]: %INFRA-SHELF_MGR-4-VM_RELOAD : Reloading VM on 0/RP1  
0/RP0/ADMIN0:Aug  2 05:32:12.842 UTC: shelf_mgr[3212]: %INFRA-SHELF_MGR-6-CARD_SW_OPERATIONAL : Card: 0/RP1 software state going to Operational  
RP/0/RP0/CPU0:Aug  2 05:33:05.763 UTC: fpd-serv[419]: %PKT_INFRA-FM-3-FAULT_MAJOR : ALARM_MAJOR :FPD-NEED-UPGRADE :DECLARE :0/RP1:  

sysadmin-vm:0_RP0# 2021-08-02 05:33:28 Admin VM of node 0/RP0 will now reload as part of ISSU operation 3
RP/0/RP0/CPU0:Aug  2 05:35:44.331 UTC: fpd-serv[419]: %PKT_INFRA-FM-3-FAULT_MAJOR : ALARM_MAJOR :FPD-NEED-UPGRADE :DECLARE :0/RP0:  
0/RP1/ADMIN0:Aug  2 05:36:01.213 UTC: inst_mgr[6351]: %INFRA-INSTMGR-2-OPERATION_SUCCESS : Install operation 3 completed successfully  
0/RP0/ADMIN0:Aug  2 05:36:29.242 UTC: control_driver[3920]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : Auto fpd triggered for FPD SATA upgrade @ location 0/RP0 please wait until all fpd upgrades are done  
0/RP0/ADMIN0:Aug  2 05:36:29.545 UTC: control_driver[3920]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD SATA@0/RP0 image programming completed with UPGRADE DONE state Info: [SDD firmware upgraded from 1.30 to 2.10]   
0/RP0/ADMIN0:Aug  2 05:36:29.545 UTC: control_driver[3920]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD SATA @location 0/RP0 FPD upgraded and activated!   
0/RP1/ADMIN0:Aug  2 05:36:29.557 UTC: shelf_mgr[4032]: %INFRA-SHELF_MGR-6-CARD_SW_OPERATIONAL : Card: 0/RP0 software state going to Operational  
0/RP1/ADMIN0:Aug  2 05:36:29.557 UTC: shelf_mgr[4032]: %INFRA-SHELF_MGR-6-CARD_HW_OPERATIONAL : Card: 0/RP0 hardware state going to Operational  
0/RP1/ADMIN0:Aug  2 05:36:29.667 UTC: control_driver[4015]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : Auto fpd triggered for FPD SATA upgrade @ location 0/RP1 please wait until all fpd upgrades are done  
0/RP1/ADMIN0:Aug  2 05:36:29.981 UTC: control_driver[4015]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD SATA@0/RP1 image programming completed with UPGRADE DONE state Info: [SDD firmware upgraded from 1.30 to 2.10]   
0/RP1/ADMIN0:Aug  2 05:36:29.981 UTC: control_driver[4015]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD SATA @location 0/RP1 FPD upgraded and activated!   
0/RP1/ADMIN0:Aug  2 05:36:29.987 UTC: shelf_mgr[4032]: %INFRA-SHELF_MGR-6-CARD_SW_OPERATIONAL : Card: 0/RP1 software state going to Operational  
0/RP1/ADMIN0:Aug  2 05:36:29.987 UTC: shelf_mgr[4032]: %INFRA-SHELF_MGR-6-CARD_HW_OPERATIONAL : Card: 0/RP1 hardware state going to Operational  
0/RP1/ADMIN0:Aug  2 05:36:30.059 UTC: control_driver[4015]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : Auto fpd triggered for FPD PRIMARY-BIOS upgrade @ location 0/RP1 please wait until all fpd upgrades are done  
0/RP0/ADMIN0:Aug  2 05:36:31.704 UTC: control_driver[3920]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : Auto fpd triggered for FPD PRIMARY-BIOS upgrade @ location 0/RP0 please wait until all fpd upgrades are done  
0/RP1/ADMIN0:Aug  2 05:36:35.977 UTC: control_driver[4015]: %INFRA-FPD_Driver-6-UPGRADE_RESULT : Upgrade completes 50 percent for fpd PRIMARY-BIOS@location 0/RP1.  
0/RP0/ADMIN0:Aug  2 05:36:36.706 UTC: control_driver[3920]: %INFRA-FPD_Driver-6-UPGRADE_RESULT : Upgrade completes 50 percent for fpd PRIMARY-BIOS@location 0/RP0.  
0/RP1/ADMIN0:Aug  2 05:36:55.985 UTC: control_driver[4015]: %INFRA-FPD_Driver-6-UPGRADE_RESULT : Upgrade completes 100 percent for fpd PRIMARY-BIOS@location 0/RP1.  
0/RP1/ADMIN0:Aug  2 05:36:56.012 UTC: control_driver[4015]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD PRIMARY-BIOS@0/RP1 image programming completed with UPGRADE DONE state Info: [  Upgrade Complete ]   
0/RP1/ADMIN0:Aug  2 05:36:56.012 UTC: control_driver[4015]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD PRIMARY-BIOS @location 0/RP1 upgrade completed.    
0/RP1/ADMIN0:Aug  2 05:36:56.018 UTC: shelf_mgr[4032]: %INFRA-SHELF_MGR-6-CARD_SW_OPERATIONAL : Card: 0/RP1 software state going to Operational  
0/RP1/ADMIN0:Aug  2 05:36:56.018 UTC: shelf_mgr[4032]: %INFRA-SHELF_MGR-6-CARD_HW_OPERATIONAL : Card: 0/RP1 hardware state going to Operational  
0/RP0/ADMIN0:Aug  2 05:36:56.582 UTC: control_driver[3920]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD PRIMARY-BIOS@0/RP0 image programming completed with UPGRADE DONE state Info: [  Upgrade Complete ]   
0/RP0/ADMIN0:Aug  2 05:36:56.869 UTC: control_driver[3920]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD PRIMARY-BIOS @location 0/RP0 upgrade completed.    
0/RP1/ADMIN0:Aug  2 05:36:56.871 UTC: shelf_mgr[4032]: %INFRA-SHELF_MGR-6-CARD_SW_OPERATIONAL : Card: 0/RP0 software state going to Operational  
0/RP1/ADMIN0:Aug  2 05:36:56.872 UTC: shelf_mgr[4032]: %INFRA-SHELF_MGR-6-CARD_HW_OPERATIONAL : Card: 0/RP0 hardware state going to Operational  
RP/0/RP0/CPU0:IOS#admin
Mon Aug  2 05:49:27.392 UTC

cafyauto connected from 192.0.0.4 using ssh on sysadmin-vm:0_RP1
sysadmin-vm:0_RP1# install commit
Mon Aug  2  05:49:40.694 UTC+00:00
result Mon Aug  2 05:49:41 2021 Install operation 4 (install commit) started by user 'cafyauto' will continue asynchronously.
sysadmin-vm:0_RP1# Mon Aug  2 05:49:44 2021 Install operation 4 completed successfully.
sysadmin-vm:0_RP1# 0/RP1/ADMIN0:Aug  2 05:49:44.733 UTC: inst_mgr[6351]: %INFRA-INSTMGR-2-OPERATION_SUCCESS : Install operation 4 completed successfully  
0/RP1/ADMIN0:Aug  2 05:49:46.936 UTC: inst_mgr[6351]: %PKT_INFRA-FM-6-FAULT_INFO : INSTALL-IN-PROGRESS :CLEAR :0/RP1: Calvados INSTALL_IN_PROGRESS Alarm : being CLEARED for the system    
sysadmin-vm:0_RP1#
  5. Check the FPD Versions
    sysadmin-vm:0_RP1# show hw-module fpd 
Mon Aug  2  05:49:31.591 UTC+00:00
                                                                     FPD Versions
                                                                    ==============
Location  Card type         HWver FPD device       ATR Status      Run     Programd
-----------------------------------------------------------------------------------
0/RP0     N560-RSP4-E       0.0   ADM                  CURRENT     1.06      1.06    
0/RP0     N560-RSP4-E       0.0   IOFPGA               CURRENT     0.67      0.64    
0/RP0     N560-RSP4-E       0.0   PRIMARY-BIOS         RLOAD REQ   0.16      0.19    
0/RP0     N560-RSP4-E       0.0   SATA                 CURRENT     2.10      2.10    
0/RP1     N560-RSP4-E       0.0   ADM                  CURRENT     1.06      1.06    
0/RP1     N560-RSP4-E       0.0   IOFPGA               CURRENT     0.67      0.64    
0/RP1     N560-RSP4-E       0.0   PRIMARY-BIOS         RLOAD REQ   0.16      0.19    
0/RP1     N560-RSP4-E       0.0   SATA                 CURRENT     2.10      2.10    
0/FT0     N560-FAN-H        1.0   PSOC                 CURRENT     2.02      2.02    
sysadmin-vm:0_RP1#
  6. Disable FPD auto-upgrade in XR VM
    RP/0/RP1/CPU0:PP-Router#conf
RP/0/RP1/CPU0:PP-Router(config)#fpd auto-upgrade disable 
RP/0/RP1/CPU0:PP-Router(config)#commit
  7. Perform ISSU Upgrade in XR VM
    RP/0/RP0/CPU0:IOS#install extract ncs560-mini-x-7.3.2
Mon Aug  2 05:55:19.880 UTC
2021-08-02 05:55:22 Install operation 100 started by cafyauto:
  install extract ncs560-mini-x-7.3.2 
2021-08-02 05:55:22 Package list:
2021-08-02 05:55:22     ncs560-mini-x-7.3.2
2021-08-02 05:55:23 Install operation will continue in the background
RP/0/RP0/CPU0:IOS#2021-08-02 05:59:42 Install operation 100 finished successfully
RP/0/RP0/CPU0:Aug  2 05:59:42.840 UTC: sdr_instmgr[1213]: %INSTALL-INSTMGR-2-OPERATION_SUCCESS : Install operation 100 finished successfully 
RP/0/RP0/CPU0:IOS#install activate issu  ncs560-xr-7.3.2
Mon Aug  2 06:02:06.097 UTC
2021-08-02 06:02:08 Install operation 101 started by cafyauto:
  install activate issu ncs560-xr-7.3.2 
2021-08-02 06:02:08 Package list:
2021-08-02 06:02:08     ncs560-xr-7.3.2
This install operation will start the issu, continue?
 [yes/no]:[yes] yes
2021-08-02 06:04:03 Install operation will continue in the background
RP/0/RP0/CPU0:IOS#RP/0/RP0/CPU0:Aug  2 06:04:11.315 UTC: issudir[342]: %PKT_INFRA-FM-6-FAULT_INFO : ISSU-IN-PROGRESS :DECLARE :0/RP0/CPU0: ISSU_IN_PROGRESS Alarm : being DECLARED for the system   
…
RP/0/RP0/CPU0:IOS# RP/0/RP1/CPU0:Aug  2 06:18:13.640 UTC: imfpga[345]: Change ISSU Node State from [Secondary] to [Primary]
RP/0/RP1/CPU0:Aug  2 06:18:13.659 UTC: tmgctrl[444]: ISSU Node Role Change Notification [Primary]
RP/0/RP1/CPU0:Aug  2 06:18:13.725 UTC: issudir[139]: %INSTALL-ISSU_INFRA-1-OPERATION_ISSU_RUN_DONE : ISSU Run has completed for install operation '101'. Issue the command "install activate issu cleanup" to continue the operation. 
RP/0/RP1/CPU0:Aug  2 06:18:13.729 UTC: issudir[139]: %PKT_INFRA-FM-6-FAULT_INFO : ISSU-IN-PROGRESS :DECLARE :0/RP1/CPU0:  
RP/0/RP1/CPU0:Aug  2 06:19:13.695 UTC: fpd-serv[216]: %PKT_INFRA-FM-3-FAULT_MAJOR : ALARM_MAJOR :FPD-NEED-UPGRADE :DECLARE :0/3:  
RP/0/RP1/CPU0:Aug  2 06:19:13.695 UTC: fpd-serv[216]: %PKT_INFRA-FM-3-FAULT_MAJOR : ALARM_MAJOR :FPD-NEED-UPGRADE :DECLARE :0/7:  
RP/0/RP1/CPU0:Aug  2 06:19:53.982 UTC: fpd-serv[216]: %PKT_INFRA-FM-3-FAULT_MAJOR : ALARM_MAJOR :FPD-NEED-UPGRADE :DECLARE :0/1:  
RP/0/RP1/CPU0:Aug  2 06:19:13.764 UTC: imfpga[343]: ISSU Phase Change Notification [Phase Cleanup]
RP/0/RP1/CPU0:Aug  2 06:19:19.442 UTC: tmgctrl[444]: ISSU Phase Change Notification [Phase Completed], Becoming Timing Primary now
RP/0/RP1/CPU0:Aug  2 06:19:19.452 UTC: sdr_instmgr[1213]: %PKT_INFRA-FM-6-FAULT_INFO : INSTALL-IN-PROGRESS :DECLARE :0/RP1/CPU0: INSTALL_IN_PROGRESS Alarm : being DECLARED for the system   
RP/0/RP1/CPU0:Aug  2 06:19:19.519 UTC: issudir[139]: %PKT_INFRA-FM-6-FAULT_INFO : ISSU-IN-PROGRESS :CLEAR :0/RP1/CPU0:  
RP/0/RP1/CPU0:Aug  2 06:19:19.538 UTC: tmgctrl[444]: ISSU Phase Change Notification [Phase Completed] Processing done
RP/0/RP1/CPU0:Aug  2 06:19:20.480 UTC: sdr_instmgr[1213]: %INSTALL-INSTMGR-2-OPERATION_SUCCESS : Install operation 101 finished successfully

RP/0/RP1/CPU0:IOS# install commit 
Mon Aug  2 06:44:07.572 UTC
2021-08-02 06:44:10 Install operation 103 started by root123:
  install commit 
2021-08-02  06:44:11 Install operation will continue in the background
RP/0/RP1/CPU0:ISO#0/RP1/ADMIN0:Aug  2 06:44:20.839 UTC: inst_mgr[6353]: %PKT_INFRA-FM-6-FAULT_INFO : INSTALL-IN-PROGRESS :CLEAR :0/RP1: Calvados INSTALL_IN_PROGRESS Alarm : being CLEARED for the system    
RP/0/RP1/CPU0:Aug  2 16:44:20.579 UTC: sdr_instmgr[1213]: %PKT_INFRA-FM-6-FAULT_INFO : INSTALL-IN-PROGRESS :CLEAR :0/RP1/CPU0: INSTALL_IN_PROGRESS Alarm : being CLEARED for the system   
2021-08-02 06:44:23 Install operation 103 finished successfully
RP/0/RP1/CPU0:Aug  2 06:44:23.201 UTC: sdr_instmgr[1213]: %INSTALL-INSTMGR-2-OPERATION_SUCCESS : Install operation 103 finished successfully
sysadmin-vm:0_RP1# show version
Mon Aug  2  07:24:58.414 UTC+00:00
Cisco IOS XR Admin Software, Version 7.3.2
Copyright (c) 2013-2021 by Cisco Systems, Inc.

Build Information:
 Built By     : ingunawa
 Built On     : Thu Jul 29 01:27:41 PDT 2021
 Build Host   : iox-ucs-003
 Workspace    : /auto/iox-ucs-003-san2/prod/7.3.2.DT_IMAGE/ncs560/ws
 Version      : 7.3.2
 Location     : /opt/cisco/calvados/packages/
 Label        : 7.3.2
System uptime is 1 hour, 53 minutes

sysadmin-vm:0_RP1# exit
Mon Aug  2  07:25:02.260 UTC+00:00
RP/0/RP1/CPU0:IOS#show version
Mon Aug  2 07:25:04.063 UTC
Cisco IOS XR Software, Version 7.3.2
Copyright (c) 2013-2021 by Cisco Systems, Inc.

Build Information:
 Built By     : ingunawa
 Built On     : Thu Jul 29 03:34:42 PDT 2021
 Built Host   : iox-ucs-003
 Workspace    : /auto/iox-ucs-003-san2/prod/7.3.2.DT_IMAGE/ncs560/ws
 Version      : 7.3.2
 Location     : /opt/cisco/XR/packages/
 Label        : 7.3.2

cisco NCS-560 () processor
System uptime is 1 hour 13 minutes

RP/0/RP1/CPU0:IOS #

  8. Check the IMFPGA FPD Versions


    Note

    IMFPGA FPDs are not upgraded as part of ISSU, as auto-upgrade is disabled. Based on your network design, you must enable and upgrade the IM FPDs as well as reload the IMs at a time when it has the least impact on traffic. 

    RP/0/RP1/CPU0:IOS#show hw-module fpd 
Mon Aug  2 06:38:52.253 UTC

Auto-upgrade:Disabled
                                                                         FPD Versions
                                                                        ==============
Location   Card type             HWver FPD device       ATR Status    Running   Programd  
-----------------------------------------------------------------------------------------
0/1        NCS4200-1T16G-PS      0.0   IMFPGA               NEED UPGD   1.95      1.95    
0/3        NCS4200-1T16G-PS      0.0   IMFPGA               NEED UPGD   1.95      1.95    
0/4        A900-IMA8Z            0.0   IMFPGA               CURRENT    17.05     17.05    
0/7        N560-IMA2C            0.0   IMFPGA               NEED UPGD   4.96      4.96    
0/9        A900-IMA8Z            0.0   IMFPGA               CURRENT    17.05     17.05    
0/RP0      N560-RSP4-E           0.0   ADM                  CURRENT     1.06      1.06    
0/RP0      N560-RSP4-E           0.0   IOFPGA               CURRENT     0.67      0.67    
0/RP0      N560-RSP4-E           0.0   PRIMARY-BIOS         RLOAD REQ   0.16      0.19    
0/RP0      N560-RSP4-E           0.0   SATA                 CURRENT     2.10      2.10    
0/RP1      N560-RSP4-E           0.0   ADM                  CURRENT     1.06      1.06    
0/RP1      N560-RSP4-E           0.0   IOFPGA               CURRENT     0.67      0.67    
0/RP1      N560-RSP4-E           0.0   PRIMARY-BIOS         RLOAD REQ   0.16      0.19    
0/RP1      N560-RSP4-E           0.0   SATA                 CURRENT     2.10      2.10    
0/FT0      N560-FAN-H            1.0   PSOC                 CURRENT     2.02      2.02    
RP/0/RP1/CPU0:IOS#
  9. Verify Standby is Node and NSR ready
    RP/0/RP1/CPU0:IOS#show redundancy
Mon Aug  2 06:39:44.953 UTC
Redundancy information for node 0/RP1/CPU0:
==========================================
Node 0/RP1/CPU0 is in ACTIVE role
Partner node (0/RP0/CPU0) is in STANDBY role
Standby node in 0/RP0/CPU0 is ready
Standby node in 0/RP0/CPU0 is NSR-ready

Reload and boot info
----------------------
RP reloaded Mon Aug  2 06:11:55 2021: 27 minutes ago
Active node booted Mon Aug  2 06:11:55 2021: 27 minutes ago
Standby node boot Mon Aug  2 06:20:02 2021: 19 minutes ago
Standby node last went not ready Mon Aug  2 06:22:46 2021: 16 minutes ago
Standby node last went ready Mon Aug  2 06:26:47 2021: 12 minutes ago
Standby node last went not NSR-ready Mon Aug  2 06:15:01 2021: 24 minutes ago
Standby node last went NSR-ready Mon Aug  2 06:24:52 2021: 14 minutes ago
There have been 0 switch-overs since reload

Active node reload "CARD_SHUTDOWN"
Standby node reload "CARD_SHUTDOWN "

RP/0/RP1/CPU0:IOS#

  10. IM FPD auto-upgrade: After ISSU

    1. Configure the fpd auto-upgrade enable command to auto upgrade FPDs for all IMs


      Note

      After ISSU is complete for RPs, configure FPD auto upgrade and FPD auto reload.


      Note

      FPD auto reload affects network traffic. Ensure you schedule the reload of IMs at a time when it has the least impact on traffic.


      Note

      Do not exit configure exclusive mode until all IM FPGA upgrades are automatically completed. Use a separate VTY session to monitor and check status. 

      RP/0/RP1/CPU0:IOS# configure exclusive
Mon Aug  2 06:45:11.297 UTC
RP/0/RP1/CPU0:IOS(config)# fpd auto-upgrade enable
RP/0/RP0/CPU0:IOS(config)# fpd auto-reload enable
RP/0/RP0/CPU0:IOS# commit

      Note

      During this step, all IMs that need FPD upgrades, are upgraded simultaneously. IMs are reloaded automatically for the versions to reflect.

    2. Exit configure exclusive mode when IMFPGA upgrades are completed on all IMs.
      RP/0/RP1/CPU0:ROUTER#RP/0/RP1/CPU0:Jun 29 06:44:28.808 UTC: fpd_imfpga[430]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : Auto fpd triggered for FPD IMFPGA upgrade @ location 0/5/CPU0 please wait until all fpd upgrades are done 
RP/0/RP1/CPU0:Jun 29 06:45:25.105 UTC: fpd-serv[125]: %PKT_INFRA-FM-3-FAULT_MAJOR : ALARM_MAJOR :FPD-NEED-UPGRADE :DECLARE :0/5:  
RP/0/RP1/CPU0:Jun 29 06:48:02.020 UTC: fpd_imfpga[430]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD IMFPGA@0/5 image programming completed with UPGRADE DONE state Info: [DONE ]  
RP/0/RP1/CPU0:Jun 29 06:48:02.032 UTC: fpd_imfpga[430]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD IMFPGA @location 0/5 upgrade completed.   
0/RP1/ADMIN0:Jun 29 06:48:02.473 UTC: shelf_mgr[3695]: %INFRA-SHELF_MGR-6-CARD_HW_OPERATIONAL : Card: 0/5 hardware state going to Operational  
0/RP1/ADMIN0:Jun 29 06:48:02.473 UTC: shelf_mgr[3695]: %INFRA-SHELF_MGR-6-CARD_SW_OPERATIONAL : Card: 0/RP1 software state going to Operational  
0/RP1/ADMIN0:Jun 29 06:48:02.474 UTC: shelf_mgr[3695]: %INFRA-SHELF_MGR-6-CARD_HW_OPERATIONAL : Card: 0/RP1 hardware state going to Operational  
RP/0/RP1/CPU0:Jun 29 06:48:08.741 UTC: fpd-serv[125]: %INFRA-FPD_SERVER-1-UPGRADE_ALERT : fpd auto-reload configured on this box.  Going to reload node 0/5/CPU0 to activate newly upgraded FPD(s).  
0/RP1/ADMIN0:Jun 29 06:48:19.182 UTC: shelf_mgr[3695]: %INFRA-SHELF_MGR-4-CARD_RELOAD : Reloading card 0/5  
0/RP1/ADMIN0:Jun 29 06:48:19.186 UTC: shelf_mgr[3695]: %INFRA-SHELF_MGR-6-HW_EVENT : Rcvd HW event HW_EVENT_RESET, event_reason_str '' for card 0/5  
0/RP1/ADMIN0:Jun 29 06:48:21.581 UTC: shelf_mgr[3695]: %INFRA-SHELF_MGR-6-HW_EVENT : Rcvd HW event HW_EVENT_POWERED_OFF, event_reason_str 'reset requested by user' for card 0/5  
RP/0/RP1/CPU0:Jun 29 06:48:58.746 UTC: fpd-serv[125]: %PKT_INFRA-FM-3-FAULT_MAJOR : ALARM_MAJOR :FPD-NEED-UPGRADE :CLEAR :0/5:  
0/RP1/ADMIN0:Jun 29 06:49:52.266 UTC: shelf_mgr[3695]: %INFRA-SHELF_MGR-6-HW_EVENT : Rcvd HW event HW_EVENT_POWERED_OFF, event_reason_str '' for card 0/5  
0/RP1/ADMIN0:Jun 29 06:49:57.586 UTC: shelf_mgr[3695]: %INFRA-SHELF_MGR-6-HW_EVENT : Rcvd HW event HW_EVENT_POWERED_ON, event_reason_str '' for card 0/5  
0/RP1/ADMIN0:Jun 29 06:50:23.999 UTC: shelf_mgr[3695]: %INFRA-SHELF_MGR-6-HW_EVENT : Rcvd HW event HW_EVENT_OK, event_reason_str '' for card 0/5  
0/RP1/ADMIN0:Jun 29 06:50:23.999 UTC: shelf_mgr[3695]: %INFRA-SHELF_MGR-6-CARD_HW_OPERATIONAL : Card: 0/5 hardware state going to Operational

  11. Break Reload Actions in Multiple Steps to Avoid Traffic Loss

    1. Reload Standby RP

      In the following example, RP1 is the active and RP0 is the standby route processor. After the standby RP reloads and boots up, the RP FPDs come to node ready and NSR ready state.


      Note

      It is expected behavior to see different versions of the RPs in System Admin VM and XR VM.

      		 
      RP/0/RP1/CPU0:IOS# admin
Mon Aug  2 07:34:26.615 UTC
Last login: Mon Aug  2 07:20:24 2021 from 192.0.4.4

cafyauto connected from 192.0.4.4 using ssh on sysadmin-vm:0_RP1
sysadmin-vm:0_RP1# hw-module location 0/RP0 reload 
Mon Aug  2  07:34:47.432 UTC+00:00
Reload hardware module ? [no,yes] yes
0/RP1/ADMIN0:Aug  2 07:34:48.728 UTC: shelf_mgr[4032]: %INFRA-SHELF_MGR-6-USER_ACTION : User cafyauto(192.0.4.4) requested CLI action 'graceful card reload' for location 0/RP0  
result Card graceful reload request on 0/RP0 succeeded.
0/RP1/ADMIN0:Aug  2 07:35:01.151 UTC: shelf_mgr[4032]: %INFRA-SHELF_MGR-4-CARD_RELOAD : Reloading card 0/RP0
RP/0/RP1/CPU0:Aug  2 07:35:01.263 UTC: fpd-serv[216]: %PKT_INFRA-FM-3-FAULT_MAJOR : ALARM_MAJOR :FPD-NEED-UPGRADE :CLEAR :0/RP0:
RP/0/RP1/CPU0:IOS#show hw-module fpd 

Auto-upgrade:Enabled
                                                                         FPD Versions
                                                                        ==============
Location   Card type             HWver FPD device       ATR Status    Running   Programd  
-----------------------------------------------------------------------------------------
0/1        NCS4200-1T16G-PS      0.0   IMFPGA               CURRENT     1.102     1.10    
0/3        NCS4200-1T16G-PS      0.0   IMFPGA               RLOAD REQ   1.95      1.10    
0/4        A900-IMA8Z            0.0   IMFPGA               CURRENT    17.05     17.05    
0/7        N560-IMA2C            0.0   IMFPGA               RLOAD REQ   4.96      5.01    
0/9        A900-IMA8Z            0.0   IMFPGA               CURRENT    17.05     17.05    
0/RP0      N560-RSP4-E           0.0   ADM                  CURRENT     1.06      1.06    
0/RP0      N560-RSP4-E           0.0   IOFPGA               CURRENT     0.67      0.67    
0/RP0      N560-RSP4-E           0.0   PRIMARY-BIOS         CURRENT     0.19      0.19    
0/RP0      N560-RSP4-E           0.0   SATA                 CURRENT     2.10      2.10    
0/RP1      N560-RSP4-E           0.0   ADM                  CURRENT     1.06      1.06    
0/RP1      N560-RSP4-E           0.0   IOFPGA               CURRENT     0.67      0.67    
0/RP1      N560-RSP4-E           0.0   PRIMARY-BIOS         RLOAD REQ   0.16      0.19    
0/RP1      N560-RSP4-E           0.0   SATA                 CURRENT     2.10      2.10    
0/FT0      N560-FAN-H            1.0   PSOC                 CURRENT     2.02      2.02    
RP/0/RP1/CPU0:IOS#

    2. Redundancy Switchover—Swap to the Updated RP as Active RP

      After the standby RP is reloaded and booted up, perform the redundancy switchover to this RP with upgraded FPD, as active RP.


      Note

      Follow Step 11a to upgrade and reload the new standby RP.

      		 
      RP/0/RP1/CPU0:IOS#redundancy switchover 
Mon Jun 28 17:16:44.914 UTC
Proceed with switchover 0/RP0/CPU0 -> 0/RP1/CPU0? [confirm] 

RP/0/RP1/CPU0:Jun 28 17:16:46.157 UTC: rmf_svr[234]: %HA-REDCON-4-FAILOVER_REQUESTED : failover has been requested by operator, waiting to initiate 
Initiating switch-over.

    3. Note

      You can either perform steps 11a and 11b, or you can perform step 11c.

      Reload the Router to Reflect the Latest FPD Versions


      Note

      Reloading the router impacts network traffic.

      		 
      RP/0/RP1/CPU0:IOS#admin
Wed Sep  8 06:10:48.761 UTC
Last login: Wed Sep  8 06:05:30 2021 from 192.0.4.6

root connected from 192.0.4.6 using ssh on sysadmin-vm:0_RP1
sysadmin-vm:0_RP1# hw-module location all reload 
Wed Sep  8  06:10:56.203 UTC+00:00
Reloading the module will be traffic impacting if not properly drained. Continue to Reload hardware module ? [no,yes] yes
result Card graceful reload request on all acknowledged.
sysadmin-vm:0_RP1#

Automatic FPD Upgrade for PSU

During the installation and Power Supply Unit (PSU) insertion process, the Field-Programmable Devices (FPD) associated with the PSUs are automatically upgraded.


Note

The PSUs are upgraded sequentially, hence the PSU FPD upgrades take longer. You can choose to exclude PSUs from the auto upgrade flow. This restricts the PSUs from being upgraded either upon insertion, or during system upgrade.

To exclude the PSU FPDs from auto upgrading, use the following CLI:

fpd auto-upgrade exclude pm 

RP/0/RSP0/CPU0:router# show running-config fed auto-upgrade
Wed Mar 30 20:52:55.079 UTC
fpd auto-upgrade enable
fpd auto-upgrade exclude pm

Note

When you upgrade from an earlier unsupported version to a version that supports Automatic FPD upgrade for PSU, the PSU upgrade might happen on bootup.