In Service Software Upgrade

This document contains the following topics:

Overview

In-Service Software Upgrade (ISSU) provides the ability to upgrade the IOS XR 64-bit version on the routers with minimal disruption on the control plane and forwarding plane. ISSU supports upgrading a Cisco IOS XR 64-bit image from a lower to a higher version. ISSU supports Zero Topology Loss (ZTL) and Minimum Packet Loss (MPL). Packet loss is for less than 50 milliseconds unless specified otherwise in the Release Notes.

SMUs containing software fixes also can be installed using ISSU. See the corresponding SMU Readme for more information on installing the fixes.

You can perform ISSU installation in a single-step or as multi-step processs. However, you must perform the pre-installation tasks before executing ISSU. During the pre-installation tasks and ISSU execution, V1 refers to the image currently running on the router and V2 refers to the upgraded image.

ISSU execution contains the following phases:

  • Prepare phase: The installable files are pre-checked and loaded on the router before activation. This phase is optional.

  • Activate phase: The new image (V2) is downloaded to all nodes in the router replacing the old image (V1). This phase can be either run in consequent phases like Load, Run, and Cleanup or by using a one-shot Activate phase.

  • Commit phase: In this phase, the ISSU installation is committed (complete with V2) on all nodes.

ISSU supports upgrading the System Admin VM and XR VM individually. Using ISSU, the System Admin VM and XR VM can also be upgraded sequentially. The upgrade sequence is System Admin ISSU followed by IOS XR ISSU.


Note

You cannot upgrade both VMs simultaneously through the ISSU processs.

Note

Committing the upgrade from XR VM commits both, the System Admin and XR software. However, committing the upgrade from System Admin VM commits only the System Admin software.


Note

When RP1 is the active RP and System Admin VM ISSU is triggered, there is an additional VM switch over compared to performing System Admin VM ISSU from RP0. This is an expected behavior.

Restrictions and Usage Guidelines


Caution

When performing an ISSU from Cisco IOS XR Release 7.1.2 to Cisco IOS XR Release 7.6.2 image, during the ISSU upgrade load phase, the process may be aborted due to multiple processes that don’t declare on time.

In the XR-7.1.2 image 40G optics isn’t supported in the 100G port, so during ISSU load phase V1 in 7.1.2 image (RP0/RP1) maintained the same 100G port but in V2 (RP0/RP1) in 7.6.2, image changed to 40G port because 40G optics was inserted. So you must remove the 40G optics while performing an ISSU operation from XR-7.1.2 to XR-7.6.2 image.


Note

When performing an ISSU from Cisco IOS XR Release 7.0.2 to Cisco IOS XR Release 7.3.2 with SR-TE enabled, ensure that you configure RSVP on the router before the upgrade. The RSVP configuration ensures that the ISSU operation doesn’t fail.

You can remove the RSVP configuration after the upgrade is complete.

Note

When performing an ISSU from Cisco IOS XR Release 7.0.2 to Cisco IOS XR Release 7.3.2 with SR-TE enabled, ensure that you configure RSVP on the router before the upgrade. The RSVP configuration ensures that the ISSU operation doesn’t fail.

You can remove the RSVP configuration after the upgrade is complete.

Note

Before performing the ISSU-SMU deactivation, ensure that you consider the dependencies on the relevant SMUs already present on the router. This consideration avoids conflicts between components during ISSU operation.

Pre-installation Tasks

Procedure

Step 1

Configure NTP on the XR VM. When NTP is configured, the System Admin VM automatically syncronizes with NTP running on the RP. If the NTP server is not available, configure clock on both, the XR VM and the System Admin VM in configuration mode. Ensure that your clock is set to the correct location and timezone.

Example:

RP/0/RP0/CPU0:ios# show ntp associations  

   address         ref clock         st  when  poll reach  delay  offset    disp
*~202.153.144.25   10.64.58.51       2   159  1024  377   14.80   0.001   0.019
* sys_peer, # selected, + candidate, - outlayer, x falseticker, ~ configured

RP/0/RP0/CPU0:ios# admin show ntp associations  

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 192.0.4.4       .INIT.          16 u    -  256    0    0.000    0.000   0.000
*192.0.0.4       202.153.144.25   3 u   62  256  377    0.218   -0.034   0.291

RP/0/RP0/CPU0:ios#

Step 2

Using the show redundancy summary command, ensure that the dual RP systems are synchronized and they are in active and standby roles respectively. The RP status should either be Final Band or Running.

Example:

RP/0/RP0/CPU0:ios# show redundancy summary  
    Active Node    Standby Node
    -----------    ------------
     0/RP0/CPU0      0/RP1/CPU0 (Node Ready, NSR:Ready)

RP/0/RP0/CPU0:ios# show platform vm  
Node name       Node type       Partner name    SW status       IP address
--------------- --------------- --------------- --------------- ---------------
0/RP1/CPU0      RP (STANDBY)    0/RP0/CPU0      FINAL Band      192.0.4.4
0/RP0/CPU0      RP (ACTIVE)     0/RP1/CPU0      FINAL Band      192.0.0.4

RP/0/RP0/CPU0:ios#

Step 3

Ensure that the firmware on the RP and interface modules (IMs) is upgraded to the latest version. You can upgrade the router cards in a single step by using the upgrade hw-module location all fpd all command. Use the show hw-module location fpd command to verify the firmware versions.

Example:

RP/0/RP0/CPU0:ios# show hw-module fpd  
Sun Apr  5 12:47:21.347 CEST
                                                               FPD Versions
                                                               =================
Location   Card type             HWver FPD device       ATR Status   Running Programd
-----------------------------------------------------------------------------------
0/1        A900-IMA8CS1Z-M       0.0   IMFPGA               CURRENT    1.95    1.95  
0/5        A900-IMA8Z            0.0   IMFPGA               CURRENT   17.05   17.05  
0/9        N560-IMA2C            0.0   IMFPGA               CURRENT    4.80    4.80  
0/10       A900-IMA8Z            0.0   IMFPGA               CURRENT   17.05   17.05  
0/RP0      N560-RSP4-E           0.0   ADM                  CURRENT    1.05    1.05  
0/RP0      N560-RSP4-E           0.0   IOFPGA               CURRENT    0.56    0.56  
0/RP0      N560-RSP4-E           0.0   PRIMARY-BIOS         CURRENT    0.16    0.16  
0/RP0      N560-RSP4-E           0.0   SATA                 CURRENT    1.30    1.30  
0/RP1      N560-RSP4-E           0.0   ADM                  CURRENT    1.05    1.05  
0/RP1      N560-RSP4-E           0.0   IOFPGA               CURRENT    0.56    0.56  
0/RP1      N560-RSP4-E           0.0   PRIMARY-BIOS         CURRENT    0.16    0.16  
0/RP1      N560-RSP4-E           0.0   SATA                 CURRENT    1.30    1.30  
0/FT0      N560-FAN-H            1.0   PSOC                 CURRENT    2.02    2.02 
 
RP/0/RP0/CPU0:ios#

Step 4

Check the disk storage space on both, System Admin VM and XR VM and ensure that sufficient disk space is available. Remove files such as, show-tech, cores, kernel dumps, manually created text, log, debug information and so on. This example shows verifying the disk storage space for System Admin VM and XR VM on both RPs. You must also verify the disk space on both RPs. If required, verify the disk storage on line cards using the show media location command.

Example:

RP/0/RP0/CPU0:ios# show media  location all  

Media Information for node0_RP1_CPU0.


----------------------------------------------
Partition                    Size     Used  Percent    Avail
rootfs:                      3.9G     2.3G      63%     1.4G
harddisk:                    5.6G     2.6G      49%     2.8G
log:                         459M     134M      32%     291M
config:                      459M      10M       3%     415M
disk0:                       2.0G      46M       3%     1.8G
---------------------------------------------------
rootfs: = root file system (read-only)
log: = system log files (read-only)
config: = configuration storage (read-only)

Media Information for node0_RP0_CPU0.


----------------------------------------------
Partition                    Size     Used  Percent    Avail
rootfs:                      3.9G     2.3G      63%     1.4G
harddisk:                    5.6G     3.6G      68%     1.8G
log:                         459M     167M      40%     259M
config:                      459M      11M       3%     414M
disk0:                       2.0G      81M       5%     1.8G
---------------------------------------------------
rootfs: = root file system (read-only)
log: = system log files (read-only)
config: = configuration storage (read-only)
RP/0/RP0/CPU0:ios#ad show media  location all 

************************************************************
Location : 0/RP1
************************************************************
---------------------------------------------------------------------
Partition                    Size     Used  Percent    Avail
---------------------------------------------------------------------
rootfs:                      2.4G     616M      28%     1.7G
harddisk:                    5.6G     2.8G      53%     2.5G
log:                         459M     119M      28%     306M
config:                      459M     4.1M       1%     421M
disk0:                      1011M     1.7M       1%     940M
install:                     5.6G     2.2G      41%     3.1G
install:/tmp                 5.6G     2.2G      41%     3.1G
install:/cache               5.6G     2.2G      41%     3.1G
rootfs:/install/tmp          5.6G     2.2G      41%     3.1G
---------------------------------------------------------------------
rootfs:  = root file system (read-only)
log:     = system log files (read-only)
config:  = configuration storage (read-only)
install: = install repository (read-only)

************************************************************
Location : 0/RP0
************************************************************
---------------------------------------------------------------------
Partition                    Size     Used  Percent    Avail
---------------------------------------------------------------------
rootfs:                      2.4G     616M      28%     1.7G
harddisk:                    5.6G     2.8G      53%     2.5G
log:                         459M     128M      31%     297M
config:                      459M     4.0M       1%     421M
disk0:                      1011M     1.9M       1%     940M
install:                     5.6G     2.2G      41%     3.1G
install:/tmp                 5.6G     2.2G      41%     3.1G
install:/cache               5.6G     2.2G      41%     3.1G
rootfs:/install/tmp          5.6G     2.2G      41%     3.1G
---------------------------------------------------------------------
rootfs:  = root file system (read-only)
log:     = system log files (read-only)
config:  = configuration storage (read-only)
install: = install repository (read-only)

RP/0/RP0/CPU0:ios#

Step 5

Populate the repository with RPMs and SMUs. Pick and install individual RPMs and SMUs, one by one, or make a tarball and install it, or break it down with multiple tarballs.

Example:

RP/0/RP0/CPU0:ios# install add source harddisk:/ 702_ncs560_V2.tar
 
Fri Apr  3 11:02:01.208 UTC
Apr 03 11:02:03 Install operation 7 started by sanity:
 install add source harddisk:/ 702_ncs560_V2.tar 
Apr 03 11:02:10 Install operation will continue in the background
RP/0/RP0/CPU0:tb2-ncs560-pe1#

RP/0/RP0/CPU0:ios# show install request 

Fri Apr  3 11:02:41.567 UTC
User sanity, Op Id 7
install add  
702_ncs560_V2.tar
The install add operation 7 is 60% complete
Add validate request sent to sysadmin

RP/0/RP0/CPU0:ios#

Step 6

Use the show install repository command to validate that packages, images, or SMUs are populated properly in the router's repository. There should be a one to one relationship between V1 and V2 images and SMUs. For example, if you install a SMU on V1, you also need the corresponding V2 version in the repository to execute ISSU.

Example:

RP/0/RP0/CPU0:ios# show install repository | in mini 

ncs560-mini-x-6.6.3		<— V1 iso image currently running

ncs560-mini-x-7.0.2		<— V2 iso image to upgrade

RP/0/RP0/CPU0:ios#

If you are using the install activate issu ncs560-mini-x-7.0.2 command in the sysadmin prompt, the system automatically extracts the sysadmin and host ISO files and continues with the sysadmin ISSU operation. You must extract the XR IOS file separately for the XR ISSU to take place.

Step 7

Depending on the version of the image, extract the ISO image in System Admin VM or XR VM.

Example:

For XR VM:

RP/0/RP0/CPU0:ios# install extract ncs560-mini-x-7.0.2 

Wed Apr  1  23:07:10.119 UTC+00:00
RP/0/RP0/CPU0:ios#

RP/0/RP0/CPU0:ios# show install repository | in xr 
ncs560-xr-6.6.3			
ncs560-xr-7.0.2		 extracted V2 image		
RP/0/RP0/CPU0:ios#

For System Admin VM:

sysadmin-vm:0_RP0# install extract ncs560-mini-x-7.0.2 

Wed Apr  1  23:07:10.119 UTC+00:00
result Wed Apr  1 23:07:11 2020 Install operation 1 (install extract) started by user 'sanity' will continue asynchronously.

sysadmin-vm:0_RP0#

sysadmin-vm:0_RP0# show install repository all | in sys|host 
ncs560-sysadmin-6.6.3	
host-6.6.3		
ncs560-sysadmin-7.0.2		 extracted V2 image
host-7.0.2		            extracted V2 image


sysadmin-vm:0_RP0#

ISSU - Single-step Installation

To perform ISSU on the router, ISSU must be performed for both, System Admin VM and XR VM in the following order—first, System Admin VM (under the admin prompt) and then XR VM (under the IOS-XR prompt).

This section shows how to perform ISSU on the router with the IOS XR 64-bit image in a single step. You can either upgrade the system or install a patch in a single step. The system upgrade is done using an ISO image file, while the patch installation is done using packages and SMUs.


Note

Depending on whether you are installing a System Admin package or a XR package, execute these commands in the System Admin EXEC mode or XR EXEC mode respectively

Before you begin


Note

Effective Cisco IOS XR Release 7.3.2, in the XR VM, use the fpd auto-upgrade enable command to automatically upgrade the FPD. For more information, see the Auto FPD Upgrade section in the Upgrading Field-Programmable Device chapter.

  • Copy the package to be installed either on the router's hard disk or on a network server to which the router has access.

  • Ensure that the dual route processor (RP) system with standby is in "is ready" state.

Procedure

Step 1

Use the install activate issu command to activate the ISSU installation in XR VM or System Admin VM in a single step.

Example:

For System Admin VM:

sysadmin-vm:0_RP0# install activate issu ncs560-sysadmin-release-version host-release-version

For XR VM:

RP/0/RP0/CPU0:ios# install activate issu ncs560-xr-release-version package-name-release-version

Step 2

Use the install commit command to commit the newly-active software.

Example:

For System Admin VM:

sysadmin-vm:0_RP0# install commit

For XR VM:

RP/0/RP0/CPU0:ios# install commit

Example

This example shows how to perform System Admin VM upgrade using ISSU and how to verify the installation using the show commands.

sysadmin-vm:0_RP0# show install repository all | in sys|host 
ncs560-sysadmin-6.6.3	
host-6.6.3		
ncs560-sysadmin-7.0.2
host-7.0.2


sysadmin-vm:0_RP0:# install activate issu ncs560-sysadmin-7.0.2 host-7.0.2  

This install operation will result in admin VMs reload
 Do you want to proceed [yes/no]: yes
 Proceeding with operation 
 result Wed Oct 31 21:12:21 2018 Install operation 2 (install prepare and activate issu)
 started by user 'root' will continue asynchronously.

sysadmin-vm:0_RP0:#

!# Monitoring the progress of the installation.
!# The installation may take up to 30 minutes.

sysadmin-vm:0_RP0:# show install request  
User root, Op Id 2 
 install prepare issu
 host-7.0.2
 This operation is 40% complete
 Waiting for agents to complete host prepare ..

sysadmin-vm:0_RP0:#


sysadmin-vm:0_RP0:# show install request  
User root, Op Id 2
 install activate issu
 ISSU stage Phase1
 ncs560-sysadmin-7.0.2
   Node 0/RP0 [RP] : 90% of current state is completed
   Node 0/RP1 [RP] : 90% of current state is completed
   
sysadmin-vm:0_RP0:MYISSU#

!# Message after successful completion. Admin VM will reload after this message. . There should be no packet drop.
0/RP0/ADMIN0:Oct 31 21:27:53.260 : inst_mgr[5019]: %INFRA-INSTMGR-2-OPERATION_SUCCESS : Install operation 2 completed successfully

!# Verifying the active package

sysadmin-vm:0_RP1# show install active summary  
Active Packages: 1 
        ncs560-sysadmin-7.0.2 version=7.0.2 [Boot image]

!# Verifies the image previously committed
sysadmin-vm:0_RP1# show install commit summary  
Committed Packages: 1 
        ncs560-sysadmin-6.6.3 version=6.6.3 [Boot image]

!# Commits the latest image

sysadmin-vm:0_RP1# install commit
  

result Wed Oct 31 21:32:58 2018 Install operation 3 (install commit) started by user 'root' will continue asynchronously.
sysadmin-vm:0_RP1# 

0/RP1/ADMIN0:Oct 31 21:33:02.061 : inst_mgr[6913]:%INFRA-INSTMGR-2-OPERATION_SUCCESS : Install operation 3 completed successfully
Wed Oct 31 21:33:02 2018 Install operation 3 completed successfully.

This example shows how to perform XR VM upgrade using ISSU and verify the installation using the show commands.

!# Verify the active packages
RP/0/RP0/CPU0:ios# show install active summary  
Active Packages: 1
        ncs560-xr-6.6.3 version=6.6.3 [Boot image]
        !# Performing ISSU Installation

RP/0/RP0/CPU0:ios# install activate issu ncs560-xr-7.0.2  

RP/0/RP0/CPU0:ios# install activate issu ncs560-xr-7.0.2
Oct 31 21:48:14 Install operation 10 started by root:
install activate issu ncs560-xr-7.0.2
Oct 31 21:48:14 Package list:
Oct 31 21:48:14 ncs560-xr-7.0.2

This install operation will start the issu, continue?
[yes/no]:[yes] yes Oct 31 21:49:13 Install operation will continue in the background
RP/0/RP0/CPU0:ios#

RP/0/RP1/CPU0:ios# show issu summary 
 Fri Apr  3 12:33:48.324 UTC
INSTALL Operation ID    : Operation 10 Started at Fri Apr  3 11:56:11 2020
ISSU Progress           : 100.0%
Total ISSU Time         : 00:17:28
ISSU Type               : IMAGE(V1-6.6.3/V2-7.0.2)

Phase           Start-Time      End-Time                  State                     
--------------------------------------------------------------------------------
Prepare         11:49:40        11:55:26                  Completed                 
Load            11:56:11        12:06:42                  Completed                 
Run             12:30:37        12:30:48                  Completed                 
Cleanup         12:32:46        12:33:46                  Completed                 
--------------------------------------------------------------------------------
Current Status          : ISSU Orchestration Successfully Completed

Setup Information       : Single Chassis
ISSU Ready/Not Ready    : 0 / 0

Node ISSU readiness per rack per slot
Key: Ready - 'Y', Not ready - 'N', Primary node - '*', Complete - '-'

Rack 0   RP0   RP1

What to do next


Note

Starting Cisco IOS XR Release 7.3.2, if you have enabled the FPD auto-upgrade feature, you do not need to use the upgrade hw-module location all fpd all command to upgrade FPDs.

Use the show hw-module fpd command to verify the FPD versions.

For more information, see the Auto FPD Upgrade section in the Upgrading Field-Programmable Device chapter.

ISSU - Multi-step Installation

To perform ISSU on the router, ISSU must be performed for both, System Admin VM and XR VM in the following order—first, System Admin VM (under the admin prompt) and then XR VM (under the IOS-XR prompt).

This section shows how to perform ISSU on the router with IOS XR 64-bit in multiple steps.


Note

Depending on whether you are installing a System Admin package or a XR package, execute these commands in the System Admin EXEC mode or XR EXEC mode respectively.


Note

You should update the System Admin VM first and then update the XR VM. IOS XR 64-bit ISSU fails if the System Admin VM is not updated first.

Before you begin


Note

Effective Cisco IOS XR Release 7.3.2, in the XR VM, use the fpd auto-upgrade enable command to automatically upgrade the FPD. For more information, see the Auto FPD Upgrade section in the Upgrading Field-Programmable Device chapter.

  • Copy the package to be installed either on the router's hard disk or on a network server to which the router has access.

  • Ensure that the dual route processor (RP) system with standby is in "is ready" state.

  • Before starting any operation that triggers reload or switchover, monitor your system install health by running the show install health command in System Admin mode. By running this command, you can verify that show commands such as show install repository display the correct output.

Procedure

Step 1

(Optional) Prepare the installable files by using the install prepare issu package_name command. During the prepare phase, pre-activation checks are performed and the components of the installable files are loaded on to the router setup.

Example:

For XR VM:


RP/0/RP0/CPU0:ios# install prepare issu ncs560-xr-release-version

Step 2

Use the install activate issu load command to start the Load phase.

Example:

For XR VM:

RP/0/RP0/CPU0:ios# install activate issu load ncs560-xr-release-version

This step downloads the new image (V2) to all nodes in the router. The new image is checked for compatibility to ensure that the router can be upgraded. At the start of the Load phase, the router configuration mode is locked, and you cannot perform any configuration on the router until ISSU completes the phase. At the end of this stage, all standby nodes run V2 and all active nodes (including all line cards) still run the original software images (V1).

Step 3

Use the install activate issu run command to start the Run phase.

Example:

For XR VM:

RP/0/RP0/CPU0:ios# install activate issu run

This phase starts the version switch from V1 to V2. All the packages that have been prepared are activated to make the package configurations active on the router.

Step 4

Use the install activate issu cleanup command to start the Cleanup phase.

Example:

For XR VM:

RP/0/RP0/CPU0:ios# install activate issu cleanup

This phase initiates shutdown of VMs with previous versions after running the activation. The Cleanup phase concludes the ISSU process and the new software runs on all nodes in the system.

Step 5

Use the install commit command to commit the newly-active software.

Example:

For XR VM:

RP/0/RP0/CPU0:ios# install commit

Recovering from a Failed ISSU Operation

While performing the ISSU operation in IOS XR, the operation can abort due to multiple reasons, such as aa software issue. In such cases, you must perform a cleanup operation to restore the system to the stable state.

Abort an ISSU operation manually after the Prepare phase

Procedure

Step 1

To abort and clean up an ISSU operation manually after the Prepare phase, use the following command in the XR VM: 

RP/0/RP0/CPU0:ios# install prepare clean

Step 2

To abort the ISSU operation manually after the Load phase, use the following command in the XR VM: 

RP/0/RP0/CPU0:ios# install activate issu abort

To restore the IOS XR version on the standby RP, use the following command in the XR VM: 

RP/0/RP0/CPU0:ios# install activate issu abort cleanup

Note

 

The install activate issu abort cleanup command must be run only from current active RP.

Note

 

You cannot abort the ISSU operation after the Run phase.

Installing Packages Using ISSU: Related Commands

Command

Purpose

show issu

Displays the state or status of the ISSU operation. Effective with Cisco IOS XR Release 7.0.2, this command is also supported for System Admin VM ISSU.

admin show issu

Displays the status of the ISSU operation.

show install request

Displays the progress of the ISSU installation.

admin show install request

Displays details about the progress of the install operation.

show install active

Displays the active packages on the system.

install prepare clean

Clears the existing prepared image. If there is a failure in the Prepare phase, run this command to clear the prepared image.

show issu milestone summary

Displays information about the ISSU milestones.