Perform ISSU on Catalyst 3850 and Catalyst 9000 Series Switches

Introduction

This document describes the steps involved to perform In-Service Software Upgrade (ISSU) on Catalyst 3850 and Catalyst 9000 series switches.

What is ISSU

In-Service Software Upgrade is a process that upgrades an image to another image on a device while the network continues to forward packets. ISSU helps network administrators avoid a network outage when they perform a software upgrade. The images are upgraded in install mode, wherein, each package is upgraded individually.

ISSU is supported on all Catalyst 3850 and Catalyst 9000 series that support Stackwise-Virtual and also on Catalyst 9400 / 9600 stand-alone chassis with dual supervisors.

StackWise Virtual (SVL) comprises two switches that are connected together to form one virtual switch. SVL supports In-Service Software Upgrades.

Supported Platforms and Release Support Matrix

Before you proceed with ISSU, check if the platform actually supports ISSU. Also, verify if ISSU is supported between current code and target code. Mode details on the supported platforms and ISSU compatibility matrix can be found here:

ISSU Support between Releases

Pre-Requisites before ISSU

Note: The example in this document is based on Cisco Catalyst 9500 switches configured as Stackwise-Virtual. The steps are applicable to Cat9400 / Cat9600 stand-alone chassis with dual supervisor engines & also for Catalyst 3850 / Catalyst 9000 devices configured as Stackwise-Virtual.
To avoid any packet loss during the process, ISSU requires redundancy in your nodes towards both the active and standby devices. Since a reload occurs and a switchover between them is required to ensure to have high availability in your network.

1. Check Current Code Version

C9500#show version | in IOS XE 
Cisco IOS XE Software, Version 16.09.02

2. Check the Boot Mode

ISSU is supported only if both the switches in Stackwise Virtual are booted in Install mode.

C9500#show ver | in INSTALL
*    1 50    C9500-40X          16.9.2            CAT9K_IOSXE           INSTALL
     2 50    C9500-40X          16.9.2            CAT9K_IOSXE           INSTALL

On Catalyst 9400, the above output is not available. Check if the switch booted from packages.conf file

C9400#show version | in System image
System image file is "flash:packages.conf"
 

If the chassis is booted in a Bundle mode, ISSU is not supported. You can see errors like this if you try to run ISSU when the switches are running in Bundle mode.

*Nov 13 14:55:57.338: %INSTALL-5-INSTALL_START_INFO: Chassis 1 R1/0: install_engine: Started install one-shot ISSU flash:cat9k_iosxe.16.09.02.SPA.bininstall_add_activate_commit: Adding ISSU
ERROR: install_add_activate_commit: One-Shot ISSU operation is not supported in bundle boot mode
FAILED: install_add_activate_commit  exit(1) Tue Nov 13 14:56:03 UTC 2018

3. Check if There is Sufficient Available Memory on Flash

C9500#dir flash: | in free
11353194496 bytes total (8565174272 bytes free)

C9500#dir stby-flash: | in free
11353980928 bytes total (8566865920 bytes free)

Ensure that you have at least 1GB of space in flash to expand a new image. Clean up old installation files in case of insufficient space and use install remove inactive command.

4. Check if Switches are in SSO Mode

C9500#show redundancy 
Redundant System Information :
------------------------------
       Available system uptime = 4 minutes
Switchovers system experienced = 0
              Standby failures = 0
        Last switchover reason = none

                 Hardware Mode = Duplex
    Configured Redundancy Mode = sso
     Operating Redundancy Mode = sso
              Maintenance Mode = Disabled
                Communications = Up

Current Processor Information :
-------------------------------
               Active Location = slot 1
        Current Software state = ACTIVE
       Uptime in current state = 30 minutes
                 Image Version = Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.9.2, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Mon 05-Nov-18 19:32 by mcpre
                          BOOT = flash:packages.conf;
                   CONFIG_FILE = 
        Configuration register = 0x102

Peer Processor Information :
----------------------------
              Standby Location = slot 2
        Current Software state = STANDBY HOT 
       Uptime in current state = 26 minutes
                 Image Version = Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.9.2, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Mon 05-Nov-18 19:32 by mcpre
                          BOOT = flash:packages.conf;
                   CONFIG_FILE = 
        Configuration register = 0x102
        

5. Check if Auto-Boot is Enabled

C9500#show boot system 
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no                       <<<<< Manual Boot should be set to "no"       
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0
---------------------------
Switch 2
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0 

If Auto-Boot is not enabled, this can be changed as shown:

C9500(config)#no boot manual

6. Check the Current ISSU and Install States

Note: This step is very important!

C9500#show issu state detail
--- Starting local lock acquisition on switch 1 ---
Finished local lock acquisition on switch 1

No ISSU operation is in progress      <<<<<<<<< If see anything else, abort ISSU before proceeding.
Check on how to manually abort ISSU.



 
C9500#show install summary
[ Switch 1 2 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
            C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type  St   Filename/Version    
--------------------------------------------------------------------------------
IMG   C    16.9.2.0.2433      <<<<<<<< State should be Activated & Committed for current version alone.
 If not clear install state before proceeding. Check on how to clear install state.                                                  

--------------------------------------------------------------------------------
Auto abort timer: inactive
--------------------------------------------------------------------------------

7. Copy the New Image to be Upgraded / Downgraded

It is sufficient to copy the image only to the Active chassis (in case of Stackwise) or the Active supervisor (in case of Cat9400 dual SUP).

C9500#copy tftp: bootflash:
Address or name of remote host []? X.X.X.X
Source filename []? cat9k_iosxe.16.09.02.SPA.bin
Destination filename [cat9k_iosxe.16.09.02.SPA.bin]? 
 

ISSU Workflows - Actual Upgrade

Once all the pre-requisites are verified, you can proceed with the actual upgrade and use one of the methods listed:

• One Step Work Flow (only one step and does not support roll back)
• Three Step Work Flow (involves 3 steps and supports roll-back incase of failure)

One Step Work Flow

This workflow involves only one step and helps in optimization.

Note: You cannot roll back as the upgrade is committed automatically. Proceed to 3 Step Work Flow if you want to roll back.

// This example has SW-2 as Active and Sw-1 as Standby before starting ISSU


C9500#install add file flash:cat9k_iosxe.16.09.02.SPA.bin activate issu commit
install_add_activate_commit: START Fri Feb  8 10:07:51 jst 2019

*Feb  8 10:07:52.456 jst: %INSTALL-5-INSTALL_START_INFO: Switch 2 R0/0: install_engine: Started install one-shot ISSU flash:cat9k_iosxe.16.09.02.SPA.bininstall_add_activate_commit: Adding ISSU

--- Starting initial file syncing ---
[2]: Copying flash:cat9k_iosxe.16.09.02.SPA.bin from switch 2 to switch 1
[1]: Finished copying to switch 1  
Info: Finished copying flash:cat9k_iosxe.16.09.02.SPA.bin to the selected switch(es)
Finished initial file syncing

--- Starting Add ---
Performing Add on all members
  [1] Add package(s) on switch 1
  [1] Finished Add on switch 1
  [2] Add package(s) on switch 2
  [2] Finished Add on switch 2
Checking status of Add on [1 2]
Add: Passed on [1 2]
Finished Add

install_add_activate_commit: Activating ISSU

Going to start Oneshot ISSU install process

STAGE 0: Initial System Level Sanity Check before starting ISSU
===================================================
--- Verifying install_issu supported ---
--- Verifying standby is in Standby Hot state ---
--- Verifying booted from the valid media ---
--- Verifying AutoBoot mode is enabled ---
Finished Initial System Level Sanity Check


STAGE 1: Installing software on Standby
===================================================
--- Starting install_remote ---
Performing install_remote on Chassis remote
[1] install_remote package(s) on switch 1
[1] Finished install_remote on switch 1
install_remote: Passed on [1]
Finished install_remote


STAGE 2: Restarting Standby
===================================================
--- Starting standby reload ---
Finished standby reload

--- Starting wait for Standby to reach terminal redundancy state ---     <<<<< Standby (Sw-1) reloads here!!!


<<<<<<<< After Standby (Sw-1) comes up >>>>>>>>>>>

*Feb  8 10:19:10.223 jst: %REDUNDANCY-3-IPC: IOS versions do not match.

*Feb  8 10:19:48.421 jst: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded

*Feb  8 10:19:49.422 jst: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)   <<<<< Standby (Sw-1) comes up on new code and joins as Hot Standby..


*Feb  8 10:21:02.975 jst: %PLATFORM-6-HASTATUS_DETAIL: RP switchover, received chassis event became active. Switch to primary (count 1)   <<<<<< Switchover happens after this log and the previous active (Sw-2) reloads.

<<<<<<<< After new Standby (Sw-2) comes up >>>>>>>>

*Feb  8 10:27:09.715 jst: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded

*Feb  8 10:27:10.717 jst: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO).  <<<< ISSU commit starts after this automatically..


*Feb  8 10:28:27.302 jst: %INSTALL-5-INSTALL_START_INFO: Switch 2 R0/0: install_engine: Started install commit
%IOSXEBOOT-4-ISSU_ONE_SHOT: (rp/0): ISSU finished successfully

*Feb  8 10:29:32.127 jst: %INSTALL-5-INSTALL_COMPLETED_INFO: Switch 2 R0/0: install_engine: Completed install commit ISSU

 Proceed to after ISSU is completed.

Three Step Work Flow

• This workflow involves three steps: add, activate, and commit. After activation, all switches are upgraded to the new software version except that the software is not committed automatically, but must be performed manually via the install commit command.
• The advantage of this approach is the system can be rolled back to a previous software version.
• The system automatically rolls back if the rollback timer is not stopped and uses the install auto-abort-timer stop or the install commit command. If the rollback timer is stopped, the new software version could be run on the device for any duration and then rolled back to the previous version.

Step 1. Install add.

 This command downloads the image into the bootflash and expands it on both the switches.

// This example has SW-1 as Active and Sw-2 as Standby before starting ISSU

C9500#install add file flash:cat9k-universalk9.SPA.16.09.03.BETA.E1.SSA.bin.bin    
install_add: START Fri Feb  8 09:22:00 jst 2019

*Feb  8 09:22:02.055 jst: %INSTALL-5-INSTALL_START_INFO: Switch 1 R0/0: install_engine: Started install add flash:cat9k-universalk9.SPA.16.09.03.BETA.E1.SSA.bin.bininstall_add: Adding PACKAGE

--- Starting initial file syncing ---
[1]: Copying flash:cat9k-universalk9.SPA.16.09.03.BETA.E1.SSA.bin.bin from switch 1 to switch 2
[2]: Finished copying to switch 2  
Info: Finished copying flash:cat9k-universalk9.SPA.16.09.03.BETA.E1.SSA.bin.bin to the selected switch(es)
Finished initial file syncing

--- Starting Add ---
Performing Add on all members
  [1] Add package(s) on switch 1
  [1] Finished Add on switch 1
  [2] Add package(s) on switch 2
  [2] Finished Add on switch 2
Checking status of Add on [1 2]
Add: Passed on [1 2]
Finished Add

SUCCESS: install_add  Fri Feb  8 09:26:26 jst 2019  <<<< Wait until install_add says SUCCESS. If fails, manually abort ISSU.

Step 2. Install activate.

• When you execute this command, this sequence of events occurs:
  
  (i) A rollback timer is started. If the rollback timer expires, the system rolls back to the same state before the start of the ISSU. The rollback timer can be stopped when you use the install auto-abort-timer stop command. ISSU can be rolled back when you use install abort issu command.
  
      (ii) The standby switch is provisioned with the new software and it reloads with the new software version. Next, the active switch is provisioned with the new software and it reloads. The standby switch with the new image now becomes the active switch and the old active switch becomes the standby.

At the end of this procedure, both the switches run with the new software image.

C9500#install activate issu
install_activate: START Fri Feb  8 09:28:27 jst 2019
install_activate: Activating ISSU

*Feb  8 09:28:28.905 jst: %INSTALL-5-INSTALL_START_INFO: Switch 1 R0/0: install_engine: Started install activate ISSU
Going to start Activate ISSU install process

STAGE 0: Initial System Level Sanity Check before starting ISSU===================================================
--- Verifying install_issu supported ---
--- Verifying standby is in Standby Hot state ---
--- Verifying booted from the valid media ---
--- Verifying AutoBoot mode is enabled ---
Finished Initial System Level Sanity Check
STAGE 1: Installing software on Standby
===================================================
--- Starting install_remote ---
Performing install_remote on Chassis remote

*Feb  8 09:28:31.880 jst: %INSTALL-5-INSTALL_AUTO_ABORT_TIMER_PROGRESS: Switch 1 R0/0: rollback_timer: Install auto abort timer can expire in 7200 seconds


[2] install_remote package(s) on switch 2
[2] Finished install_remote on switch 2
install_remote: Passed on [2]
Finished install_remote

STAGE 2: Restarting Standby
===================================================
--- Starting standby reload ---
Finished standby reload--- Starting wait for Standby to reach terminal redundancy state ---      <<<<<<<< Standby (Sw-2) reloads here!!!*Feb  8 09:35:16.489 jst: %REDUNDANCY-3-IPC: IOS versions do not match.
 
*Feb  8 09:36:00.238 jst: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
*Feb  8 09:36:01.240 jst: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)     <<<< At this point, Standby (Sw-2) comes up with new code and joins as Hot Standby
Finished wait for Standby to reach terminal redundancy state

STAGE 3: Installing software on Active
===================================================
--- Starting install_active ---

Performing install_active on Chassis 11] install_active package(s) on switch 1
[1] Finished install_active on switch 1
install_active: Passed on [1]
Finished install_active
Chassis 1 reloading, reason - Non participant detected

STAGE 4: Restarting Active (switchover to standby)      <<<<<<<< At this point, there is a switchover and then the previous Active (Sw-1) reloads.
===================================================
--- Starting active reload ---
 New software can load after reboot process is completed
SUCCESS: install_activate  Fri Feb  8 09:37:14 jst 2019

At the end of Activate state, check the ISSU states.

C9500#show issu state detail
--- Starting local lock acquisition on switch 2 ---
Finished local lock acquisition on switch 2

Operation type: Step-by-step ISSU
Install type  : Image installation using ISSUCurrent state : Activated state
Last operation: Switchover

Completed operations:

Operation                           Start time
-------------------------------------------------------
Activate location standby Chassis 2 2019-02-08:09:28:32
Activate location active Chassis 1  2019-02-08:09:36:03
Switchover                          2019-02-08:09:37:16

State transition: Added -> Standby activated -> Active switched-over 

Auto abort timer: automatic, remaining time before rollback: 01:43:55
Running image: flash:packages.conf
Operating mode: sso, terminal state reached    <<<<< Wait until SSO terminal state before proceeding to commit.

Step 3.  Install commit.

The commit command performs the necessary clean up, enables the new software as permanent (removes the older version of the software), and stops the rollback timer. Any reboot after the commit can boot with new software.

C9500#install commit 
install_commit: START Fri Feb  8 09:45:22 jst 2019
install_commit: Committing ISSU

*Feb  8 09:45:23.533 jst: %INSTALL-5-INSTALL_START_INFO: Switch 2 R0/0: install_engine: Started install commit

Going to start Commit ISSU install process

STAGE 0: Initial System Level Sanity Check before starting ISSU
===================================================
--- Verifying install_issu supported ---
--- Verifying standby is in Standby Hot state ---
--- Verifying booted from the valid media ---
--- Verifying AutoBoot mode is enabled ---

Finished Initial System Level Sanity Check

--- Starting install_commit_2 ---
Performing install_commit_2 on Chassis 2
[2] install_commit_2 package(s) on switch 2
[2] Finished install_commit_2 on switch 2
install_commit_2: Passed on [2]
Finished install_commit_2


STAGE 1: Dispatching the commit command to remote
===================================================
--- Starting install_commit_remote ---
Performing install_commit_remote on Chassis 1
Feb  8 09:48:33.364: %INSTALL-5-INSTALL_START_INFO: R0/0: install_engine: Started install commit

*Feb  8 09:48:33.352 jst: %INSTALL-5-INSTALL_START_INFO: Switch 1 R0/0: install_engine: Started install commit


Feb  8 09:51:27.505: %INSTALL-5-INSTALL_COMPLETED_INFO: R0/0: install_engine: Completed install commit ISSU
[1] install_commit_remote package(s) on switch 1
[1] Finished install_commit_remote on switch 1
install_commit_remote: Passed on [1]
Finished install_commit_remote

SUCCESS: install_commit  Fri Feb  8 09:51:27 jst 2019     <<<<< ISSU is completed here!!!!!

Post ISSU Checklist

Once ISSU is successfully completed,

• Check if both switches run on the new software.
• Check show issu state detail output to be clean and not showing any ISSU in progress.
• Check show install issu history output to ensure successful ISSU operation (Command available only with 16.10.1 release and later).
• It is recommended to give enough soak time in the new software before you eanble any new feature.

Action on ISSU Failure

• If ISSU fails, it is expected that auto-abort can recover the system back to initial state (older image). However, if this fails as well, manual recovery of the chassis is expected.
• During manual recovery, check if both active and standby run the older image (if not, recover the individual chassis).
• After you ensure both chassis run the old image, run install remove inactive to remove any unused image packages.
• Once both chassis run the old software, manually clean all the internal states of ISSU operation. (Refer here on how to clean the internal ISSU states).

Abort ISSU

• In the 3-Step Work Flow, during the activate ISSU process, if the abort-timer expires, system can auto-abort to older image.

Note: If the standby does not reach SSO during abort, manual abort is required. Also, if for any reason you wish to abort the ISSU in between, manual abort is required.

EXAMPLE : During install add, we notice these errors:

C9400#install add file flash:cat9k_iosxe.16.09.02.SPA.bin
install_add: START Tue Nov 13 20:47:53 UTC 2018

*Nov 13 20:47:54.787: %INSTALL-5-INSTALL_START_INFO: Chassis 1 R1/0: install_engine: Started install add flash:cat9k_iosxe.16.09.02.SPA.bininstall_add: Adding PACKAGE

--- Starting initial file syncing ---
[1]: Copying flash:cat9k_iosxe.16.09.02.SPA.bin from chassis 1 to chassis 2
[2]: Finished copying to chassis 2  
Info: Finished copying flash:cat9k_iosxe.16.09.02.SPA.bin to the selected chassis
Finished initial file syncing

--- Starting Add ---
Performing Add on all members
  [1] Add package(s) on chassis 1
  [1] Finished Add on chassis 1
  [2] Add package(s) on chassis 2
    cp: cannot stat '/tmp/packages.conf': No such file or directory
  [2] Finished Add on chassis 2
Checking status of Add on [1 2]
Add: Passed on [1]. Failed on [2]
Finished Add

FAILED: install_add  exit(1) Tue Nov 13 20:51:58 UTC 2018   <<<<<< install_add failed. If see any such error, first do a manual abort before proceeding further.  


C9400#install abort issu 
install_abort: START Tue Nov 13 20:57:40 UTC 2018
install_abort: Abort type ISSU subtype NONE smutype NONE

*Nov 13 20:57:41.759: %INSTALL-5-INSTALL_START_INFO: Chassis 1 R1/0: install_engine: Started install abort ISSU
NOTE: Going to start Abort ISSU install process

STAGE 0: Initial System Level Sanity Check before starting ISSU
===================================================
--- Verifying install_issu supported ---
--- Verifying booted from the valid media ---
--- Verifying AutoBoot mode is enabled ---
Finished Initial System Level Sanity Check

FAILED: ABORT operation is not allowed in ADDED state
ERROR: install_abort  exit(2 ) Tue Nov 13 20:57:49 UTC 2018

*Nov 13 20:57:49.756: %INSTALL-5-INSTALL_COMPLETED_INFO: Chassis 1 R1/0: install_engine: Completed install abort ISSU

Clean ISSU State

If ISSU upgrade/downgrade/abort/auto-abort is not successful, manual clean up of ISSU internal states is required.

C9400#sh issu state detail
--- Starting local lock acquisition on chassis 1 ---
Finished local lock acquisition on chassis 1

Operation type: One-shot ISSU
Install type  : Image installation using ISSU
Current state : Added state
Last operation: Activate location standby Chassis 2   <<<< Previous Add is still pending. This needs to be cleared before starting ISSU

Completed operations:

Operation                           Start time
-------------------------------------------------------
Activate location standby Chassis 2 2018-11-13:16:26:34   

State transition: Added 

Auto abort timer: inactive
Running image: flash:packages.conf
Operating mode: sso, terminal state not reached

Enable Service Internal before you run this command

C9400#clear install state   
clear_install_state: START Tue Nov 13 17:05:47 UTC 2018
--- Starting clear_install_state ---
Performing clear_install_state on all members
  [1] clear_install_state package(s) on chassis 1
  [1] Finished clear_install_state on chassis 1
Checking status of clear_install_state on [1]
clear_install_state: Passed on [1]
Finished clear_install_state

C9400#sh issu state detail 
--- Starting local lock acquisition on chassis 1 ---
Finished local lock acquisition on chassis 1

No ISSU operation is in progress