Upgrade Guide for Catalyst 9000 Switches

Introduction

This document describes upgrade methods for Catalyst 9000 (Cat9K) switches.

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on these software and hardware versions.

• C9200
• C9300
• C9400
• C9500
• C9600

Note: Consult the appropriate configuration guide for the commands that are used in order to enable these features on other Cisco platforms.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

This document covers new and old upgrade procedures for Catalyst 9K type switches that use either BUNDLE or INSTALL modes.. 

Install Modes Overview 

Bundle Mode

Bundle mode is a fancy way to say that switch runs the traditional way of Cisco IOS®. You boot a .bin file that contains everything you need to run the Cisco IOS. As per traditional Cisco IOS, you have a boot statement that points to the .bin file you want to load and that gets loaded at the time of boot.

Switch#show boot
BOOT variable = bootflash:cat9k_iosxe.16.11.01.SPA.bin;
Configuration Register is 0x102
MANUAL_BOOT variable = no
BAUD variable = 9600
ENABLE_BREAK variable does not exist
BOOTMODE variable does not exist
IPXE_TIMEOUT variable does not exist
CONFIG_FILE variable =

You must set this boot statement with the boot system command. The next time you reload, the switch boots into 16.12.01 rather than 16.11.01.

Switch#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
Switch(config)#no boot system
Switch(config)#boot system bootflash:cat9k_iosxe.16.12.01.SPA.bin <--- 
Switch(config)#end
Switch#wr
Switch#
Switch#show run | include boot system
boot system bootflash:cat9k_iosxe.16.12.01.SPA.bin
Switch#
Switch#show boot
BOOT variable = bootflash:cat9k_iosxe.16.12.01.SPA.bin; <--- 
Configuration Register is 0x102
MANUAL_BOOT variable = no
BAUD variable = 9600
ENABLE_BREAK variable does not exist
BOOTMODE variable does not exist
IPXE_TIMEOUT variable does not exist
CONFIG_FILE variable =

Install Mode

Install mode is the newer and recommended mode to run. This breaks the .bin file up into smaller .pkg files that must be loaded into memory independently of each other, and allows you to boot faster and utilize memory better. The .bin file that you download from software.cisco.com has all the .pkg files you need inside. 

Note: Think of the .bin file as a .zip file. It is able be extracted to obtain the .pkg files. There is also a configuration file that is used to indicate what .pkg files are needed. This is the process that is used when you boot in install mode.

1. The boot statement is looked at to determine what configuration file to load (usually packages.conf).
2. The configuration file is opened and used to sort out what .pkg files to load.
3. The .pkg files are loaded and Cisco IOS completes the boot process.

• The .conf file is just a text file that is used as a pointer, and you are able open it up and look at it from the CLI.
• This configuration file is set to load 16.11.01. Now you do not have to change the boot statement in order to upgrade a device, all you have to do is change the packages.conf file to point to the new .pkg files. 

Note: This is done automatically and you never have to edit this file directly. However, you can check what package files the .conf file is pointed to before you do a reload to ensure it is pointed to the right ones.

Switch#more packages.conf
#! /usr/binos/bin/packages_conf.sh
<...snip...>
# This is for CAT9k
boot  rp 0 0   rp_boot         cat9k-rpboot.16.11.01.SPA.pkg
iso   rp 0 0   rp_base         cat9k-rpbase.16.11.01.SPA.pkg
iso   rp 0 0   rp_daemons      cat9k-rpbase.16.11.01.SPA.pkg
iso   rp 0 0   rp_iosd         cat9k-rpbase.16.11.01.SPA.pkg
iso   rp 0 0   rp_security     cat9k-rpbase.16.11.01.SPA.pkg
iso   rp 0 0   rp_wlc          cat9k-wlc.16.11.01.SPA.pkg
iso   rp 0 0   rp_webui        cat9k-webui.16.11.01.SPA.pkg
iso   rp 0 0   srdriver        cat9k-srdriver.16.11.01.SPA.pkg
iso   rp 0 0   guestshell      cat9k-guestshell.16.11.01.SPA.pkg

boot  rp 1 0   rp_boot         cat9k-rpboot.16.11.01.SPA.pkg
iso   rp 1 0   rp_base         cat9k-rpbase.16.11.01.SPA.pkg
iso   rp 1 0   rp_daemons      cat9k-rpbase.16.11.01.SPA.pkg
iso   rp 1 0   rp_iosd         cat9k-rpbase.16.11.01.SPA.pkg
iso   rp 1 0   rp_security     cat9k-rpbase.16.11.01.SPA.pkg
iso   rp 1 0   rp_wlc          cat9k-wlc.16.11.01.SPA.pkg
iso   rp 1 0   rp_webui        cat9k-webui.16.11.01.SPA.pkg
iso   rp 1 0   srdriver        cat9k-srdriver.16.11.01.SPA.pkg
iso   rp 1 0   guestshell      cat9k-guestshell.16.11.01.SPA.pkg

<...snip...>
#
# -start- superpackage .pkginfo
#
# pkginfo: Name: rp_super
# pkginfo: BuildTime: 2019-03-28_09.46
# pkginfo: ReleaseDate: Thu-28-Mar-19-01:19
# pkginfo: .BuildArch: x86_64
# pkginfo: BootArchitecture: i686
# pkginfo: .BootArch: i686
# pkginfo: RouteProcessor: cat9k
# pkginfo: Platform: CAT9K
# pkginfo: User: mcpre
# pkginfo: PackageName: universalk9
# pkginfo: Build: 16.11.01
# pkginfo: .SupportedBoards: cat9k
# pkginfo: .InstallModel:
# pkginfo: .PackageRole: rp_super
# pkginfo: .RestartRole: rp_super
# pkginfo: .UnifiedPlatformList: passport,nyquist,starfleet
# pkginfo: CardTypes:
# pkginfo: .CardTypes:
# pkginfo: .BuildPath: /scratch/mcpre/release/BLD-V16_11_01_FC3/binos/linkfarm/stage-cat9k/hard/rp_super_universalk9.x86_64
# pkginfo: .Version: 16.11.1.0.312.1553791584..Gibraltar
# pkginfo: .InstallVersion: 1.0.0
# pkginfo: .InstallCapCommitSupport: yes
#
# -end- superpackage .pkginfo
#

When you perform an upgrade in install mode, the process is always the same. However, the commands can be different per platform:

1. Copy the new .bin file onto the switch.
2. Extract the .pkg files from the .bin file.
3. Update the .conf file and reload the device into the new version.
4. Stop the rollback timer to confirm the upgrade is complete.

Most of the time you are able do steps 2-4 with a single command. This also allows us to rollback to a previous version. 

Upgrade Methods

Autoupgrade

If your active supervisor runs in install mode, your best bet is to use auto upgrade. This allows the Active sup to upgrade the Standby sup regardless of what image is loaded onto the standby (bundle/install/etc). However, you have to boot the standby into some valid image in order for the process to start. 

Note: Auto upgrade does not recover a sup from rommon.

Enable this command so that the switch automatically upgrades a standby sup if it detects a mismatch

software auto-upgrade enable

Or run the upgrade manually 

install autoupgrade

Note: Auto upgrade works in a Stackwise Virtual setup as long as the active chassis is in Install mode.

Install Mode

This next section talks about use of the "install" commands. All 9ks support these commands. However, this is the only option for 9600, 9400, and 9200.

Process Overview

As discussed, the upgrade is a 4 part process. However, the commands for steps 2-4 are a little cryptic so here is what each command does:

Install Mode Commands

Restrictions

Your boot statement must always be packages.conf in install mode. If you try to change the boot statement to another .conf file while you are in the middle of an upgrade, it could cause the upgrade to fail. If you have already run the ADD command, you need to activate the package you added or clean it out to start over. Do not change your boot statement to an inactive package.

Summary

If you want to move to install, you need to first change your boot statement to packages.conf if it is not already then add the .bin file. This means you have to copy the new .bin file onto bootflash either from TFTP or USB. If you have issues with the copy of the file onto the switch because there is not enough space, see the Cleanup section of this doc to clear out files that are not needed. 

These commands complete the upgrade and you do not need to do any of the other steps. However, if you do not want to complete the upgrade in one step, it is possible to break it down into each individual step.

configure terminal
no boot system
boot system bootflash:packages.conf
end
write
install add file bootflash:<new .bin file> activate commit

Install Add

• When you first start your upgrade you only have your current version.

Switch#show install summary
[ R0 R1 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
            C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type  St   Filename/Version
--------------------------------------------------------------------------------
IMG   C    16.11.1.0.312

--------------------------------------------------------------------------------
Auto abort timer: inactive
--------------------------------------------------------------------------------

In order to get started, you must add the new version to the list of versions that you want to work with.

install add file bootflash:cat9k_iosxe.16.12.01.SPA.bin

Once it is added, you can see the .pkg file in flash and it lists 16.12.01 as an inactive version.

Switch#dir | include .pkg
359097 -rw- 11359240 Sep 13 2019 16:10:08 +00:00 cat9k-cc_srdriver.16.11.01.SPA.pkg
359098 -rw- 84354052 Sep 13 2019 16:10:08 +00:00 cat9k-espbase.16.11.01.SPA.pkg
359099 -rw- 1676292 Sep 13 2019 16:10:08 +00:00 cat9k-guestshell.16.11.01.SPA.pkg
359100 -rw- 466576384 Sep 13 2019 16:10:08 +00:00 cat9k-rpbase.16.11.01.SPA.pkg
359106 -rw- 38552418 Sep 13 2019 16:10:30 +00:00 cat9k-rpboot.16.11.01.SPA.pkg
359101 -rw- 29877252 Sep 13 2019 16:10:08 +00:00 cat9k-sipbase.16.11.01.SPA.pkg
359102 -rw- 57259008 Sep 13 2019 16:10:08 +00:00 cat9k-sipspa.16.11.01.SPA.pkg
359103 -rw- 19936260 Sep 13 2019 16:10:08 +00:00 cat9k-srdriver.16.11.01.SPA.pkg
359104 -rw- 12321792 Sep 13 2019 16:10:08 +00:00 cat9k-webui.16.11.01.SPA.pkg
359105 -rw- 9216 Sep 13 2019 16:10:08 +00:00 cat9k-wlc.16.11.01.SPA.pkg
456963 -rw- 14222344 Sep 13 2019 17:05:35 +00:00 cat9k-cc_srdriver.16.12.01.SPA.pkg
456964 -rw- 88892420 Sep 13 2019 17:05:35 +00:00 cat9k-espbase.16.12.01.SPA.pkg
473282 -rw- 1684484 Sep 13 2019 17:05:35 +00:00 cat9k-guestshell.16.12.01.SPA.pkg
473283 -rw- 535475200 Sep 13 2019 17:05:35 +00:00 cat9k-rpbase.16.12.01.SPA.pkg
473289 -rw- 43111714 Sep 13 2019 17:06:00 +00:00 cat9k-rpboot.16.12.01.SPA.pkg
473284 -rw- 31425540 Sep 13 2019 17:05:35 +00:00 cat9k-sipbase.16.12.01.SPA.pkg
473285 -rw- 60183552 Sep 13 2019 17:05:35 +00:00 cat9k-sipspa.16.12.01.SPA.pkg
473286 -rw- 22676484 Sep 13 2019 17:05:35 +00:00 cat9k-srdriver.16.12.01.SPA.pkg
473287 -rw- 12854272 Sep 13 2019 17:05:35 +00:00 cat9k-webui.16.12.01.SPA.pkg
473288 -rw- 9216 Sep 13 2019 17:05:35 +00:00 cat9k-wlc.16.12.01.SPA.pkg

Switch#show install summary
[ R0 R1 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
            C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type  St   Filename/Version
--------------------------------------------------------------------------------
IMG   I    16.12.1.0.544 <-- Installed but still Inactive (I)
IMG   C    16.11.1.0.312

--------------------------------------------------------------------------------
Auto abort timer: inactive
--------------------------------------------------------------------------------

Switch#show install inactive
[ R0 R1 ] Inactive Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
            C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type  St   Filename/Version
--------------------------------------------------------------------------------
IMG   I    16.12.1.0.544  <-- Installed but still Inactive (I)

Install Activate

• Once reloaded, you can see the version is active but not committed.

The next step is to activate the new version.

• This means reload the sup and load the new version.
• This edits the .conf file to point to the new .pkg files.

install activate

It asks you to confirm if you want to reload before the device reloads.

This operation may require a reload of the system. Do you want to proceed? [y/n]

Switch#show install sum
[ R0 R1 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
            C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type  St   Filename/Version
--------------------------------------------------------------------------------
IMG   U    16.12.1.0.544

--------------------------------------------------------------------------------
Auto abort timer: active on install_activate, time before rollback - 01:52:08   <--- when this hits zero, the device reloads back to original version.
--------------------------------------------------------------------------------

Switch#show install uncommitted
[ R0 R1 ] Uncommitted Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
            C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type  St   Filename/Version
--------------------------------------------------------------------------------
IMG   U    16.12.1.0.544

Install Commit

To stop the rollback timer and finish the upgrade process, you need to commit the new version.

install commit

This is the last step in the upgrade process and your new version now shows as the active committed version.

Switch#show install summary
[ R0 R1 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
            C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type  St   Filename/Version
--------------------------------------------------------------------------------
IMG   C    16.12.1.0.544  <--- Now Active and Committed

--------------------------------------------------------------------------------
Auto abort timer: inactive
--------------------------------------------------------------------------------

Install Abort

After you have run the activate command and before you commit, if you decide there is something wrong with the new version, you can abort the upgrade. This causes the device to reload again back to the previous version.

install abort

Prompt Level

it is possible to set the prompt-level to none at the end of a install command and it does not ask you yes/no for any questions (normally at the end of the commit command of the upgrade, it asks you if you want to reload). If you set the prompt-level to none, the switch reloads as soon as it is ready to do so.

install add file bootflash:cat9k_iosxe.16.12.01.SPA.bin activate commit prompt-level none

Auto Upgrade Timer

If an upgrade takes too long, the switch assumes something went wrong and it goes ahead and aborts the upgrade. The default timer is 120 minutes, but you can stop the timer manually if you need to.

install auto-abort-timer stop

You can set the rollback timer when you do the activate step.

Switch#install activate auto-abort-timer ?
  <30-1200>  Number of minutes the abort-timer is set for

Bundle Mode

This section describes the classic method of software upgrade with the use of a boot statement that point to the .bin file (versus a .pkg file).

Warning: If you want to upgrade a switch to bundle mode, ensure you have copied the .bin to EVERY switch in the stack!

After you have loaded the Cisco IOS file, all you need to do is change the boot statement and reload.

configure terminal
no boot system
boot system bootflash:<new filename>
end
write
reload

Legacy Method

Caution: This method applies to 9300 & 9500 switches only

This section talks about use of the request platform commands and is supported on the 9500 and 9300.

• This method is NOT recommended for 9300 or 9500. Please use one of the methods mentioned previously. 
• These commands are deprecated as of 16.10.1 and it is recommended that you use the install commands.
• The three most important commands are listed here (There are many more options but it is preferred to use the install commands instead of these).

Expand

This command takes whatever .bin file you specify and extract the .pkg files.

request platform software package expand switch all file flash:cat9k_iosxe.16.09.02.SPA.bin

Note: If the switch uses bundle mode. First run the expand command then change your boot statement to packages.conf and reload to get to install mode.

Install 

If you are already in install mode, use this command to move to another version:

request platform software package install switch all file flash:test auto-copy new

Clean

The next command does the same thing as install remove inactive:

request platform software package clean

Stackwise Virtual 

Dual-Sup

Stackwise virtual is very similar to dual sup setup. If you do a normal upgrade, both chassis reboot at the same time.

configure terminal
no boot system
boot sys bootflash:packages.conf
end
write
install add file bootflash:<new .bin file> activate commit

Note: To perform an ISSU there is one more consideration you have to take in stackwise virtual vs a dual sup setup on a single chassis.

• Because each chassis reloads once during the ISSU process, any devices that are connected to only one chassis go down for a whole reboot cycle.
• It is recommended to use a dual homed MEC for all connections to avoid this situation. See ISSU section for more details on ISSU.

Roll Back Install Mode

Once you complete an upgrade, it is possible rollback to the previous version if needed. If fact, you are able rollback to multiple versions if the files are still on flash.

Switch#show install rollback
ID      Label          Description
-----------------------------------------------
2   	No Label        No Description
1   	No Label        No Description



Switch#show install rollback id 1
Rollback id - 1 (Created on Fri Sep 13 13:14:40 2019)
Label: No Label
Description: No Description
Reload required: NO
State (St): I - Inactive, U - Activated & Uncommitted,
            C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type  St   Filename/Version
--------------------------------------------------------------------------------
IMG   C    16.11.1.0.312

The previous output shows I have two rollback points. The first rollback points to rollback to 16.11.01 (label your rollback points if you want to put a description or date).

Switch#install label 1 ?
  description  Add a description to specified install point
  label-name   Add a label name to specified install point

Switch#install label 1 label-name 16_11_01
install_add_label: START Fri Sep 13 16:43:48 UTC 2019
--- Starting install_add_label ---
Performing install_add_label on Active/Standby
  [R0] install_add_label package(s) on R0
  [R0] Finished install_add_label on R0
  [R1] install_add_label package(s) on R1
  [R1] Finished install_add_label on R1
Checking status of install_add_label on [R0 R1]
install_add_label: Passed on [R0 R1]
Finished install_add_label

SUCCESS: install_add_label  Fri Sep 13 16:43:57 UTC 2019
Switch#show install rollback
ID      Label          Description
-----------------------------------------------
2   	No Label        No Description
1   	16_11_01        No Description

If you now want to perform the rollback, you just need to run the rollback command:

install rollback to id 1

Quad-Sup

Not supported at the time of this writing. Scheduled for support on 17.2.1.

ISSU

The previous upgrade nethods talk about the upgrade of a single sup or the upgrade of both sups at the same time. However, with dual sups you can do an ISSU (In Service Software Upgrade) which allows one sup to be upgraded at a time so that there is little to no outage.

Note: For more information on 9300 xFSU see this document Understand Extended Fast Software Upgrade on Catalyst 9300 Series Switches

Caution: Once you run the command noted here, the process starts and reloads sup automatically. Do not run the command until you are ready for sups to start to reboot. Unlike the normal upgrade process, it does not ask for a confirmation from you before the reload happens. 

install add file bootflash:cat9k_iosxe.16.11.01.SPA.bin activate issu commit

• Once you run this command, the ISSU process extracts the files, reload the standby sup, wait for it to get back to SSO then failover reloads the active. 
• ISSU is only supported in Install mode. If active supervisor is in bundle, you cannot preform an ISSU process.
• 9200 and 9300 do not support ISSU. 9300 does support reloadfast.

Upgrade Options

ReloadFast

• ReloadFast is a feature that completes as much of the upgrade possible before the data plane is taken offline to minimize downtime . If you use reloadfast, all switches in the stack reload at the same time. However, they usually reboot much faster than a normal install.
• You can do reloadfast even on a single switch!
• This feature is supported on version 16.8.1 or later. It is only supported in INSTALL mode. 
• To prevent any loops that can happen while a fast reload occurs, it is only supported on switches with a single uplink to the STP root. A port-channel counts as a single uplink. 
• It is also not supported if the upgrade requires a microcode or FPGA upgrade. 
• C9300 always tries to do a reloadfast upgrade by default unless it cannot for one of the previously stated reasons. However, you can explicitly ask it to do so with the this command:

install add file flash:cat9k_iosxe.16.10.01.SPA.bin activate reloadfast commit

Cleanup

Once you are done with an upgrade, you have the option to clean up all the files you do not need by removal of all inactive files. It looks at the current loaded .pkg files and removes everything else. You can review all the files it plans to delete before it deletes them. 

install remove inactive

Note: This removes all .bin files so if you want to cleanup before you do an upgrade, ensure you do it before you copy on the new .bin file.

Troubleshoot

V-Mismatch

If you get a V-mismatch, this means the software that runs on the switch is different than the active. If you are in install mode, you have the option to resolve this with auto-upgrade. If you are in bundle mode, this is a little tricky to resolve.

Switch#show switch
Switch/Stack Mac Address : f8b7.e206.4900 - Local Mac Address
Mac persistency wait time: Indefinite
                                             H/W   Current
Switch#   Role    Mac Address     Priority Version  State
-------------------------------------------------------------------------------------
 1       Member   701f.5300.fa00     15     V01     V-Mismatch  <---Indicates a version mismatch
*2       Active   f8b7.e206.4900     14     V01     Ready

OPTION 1

• Remove the mismatched switch from the stack and upgrade it to the correct version then re-add it back to the stack.

OPTION 2

• You cannot change the boot statement on a mismatched switch. However, if you have access to console and can access the switch rommon, it is possible to manually boot into the correct version.

Caution: Ensure you have console access to the switch before you try this, to ensure you are able to recover the switch.

Steps for Option 2

1. Stack the switch and let it boot to V-Mismatchstate.
2. Delete old images from this switch that you do not want (flash: file system is accessible even if it is mismatched).
3. Copy the new .bin file you do want to flash.
4. Power cycle just this switch (it fails to boot since the image is deleted and this drops a switch into rommon).
5. (From console) manually boot the new image you copied to flash.
6. Once booted to right image, config is synced from Active.

Switch#dir ?
  /all             List all files
  /recursive       List files recursively
  all-filesystems  List files on all filesystems
  cns:             Directory or file name
  crashinfo-1:     Directory or file name
  crashinfo-2:     Directory or file name
  crashinfo:       Directory or file name
  flash-1:         Directory or file name   <----Flash for switch #1
  flash-2:         Directory or file name   <----Flash for switch #2
  flash:           Directory or file name

Use this command once you are in rommon

switch:  boot flash:cat9k_iosxe.16.12.01.SPA.bin <-- Image you copied that Active is running

OPTION 3 

• The first two options are usually able to recover the switch that is mismatched without the need to bring down the stack.
• However, if this is a new deployment or the you are okay to reload the whole stack, you have the option to copy the file from the mismatched switch to the rest of the switches and make the rest of the switches match the new switch rather than the other way around.
• This option could be useful if it is a new deployment and you do not have physical access, as you can accomplish this all from telnet.
• Once they are in sync, you can upgrade all of them to the right version. 

Recover from Rommon

If you get stuck in rommon with no valid image, you have the option to copy the file via USB onto the switch or via TFTP.

OPTION 1 

Boot right from a USB stick with the this command:

boot usbflash0:cat9k_iosxe.16.12.01.SPA.bin

OPTION 2

Copy from a TFTP server is a little more tricky. You either need to setup your laptop as a TFTP server and connect an ethernet cable to the management port of the device, or you need to connect the management port of the device to a network port of a device that is in operation. You then configure the device with your image as a tftp server:

tftp-server flash:cat9k_iosxe.16.12.01.SPA.bin

Cisco IOS device as the TFTP server is preferable so you do not have to worry about windows firewalls or computer account privilege issues . Once you have the TFTP server setup, you need to configure the this in rommon:

DEFAULT_GATEWAY=172.27.74.1
IP_ADDRESS=172.27.74.111
IP_SUBNET_MASK=255.255.255.0
TFTP_SERVER=172.19.64.31

Test your config by ping to the tftp server:

ping 172.19.64.31

Once connectivity is established, boot right to the file on TFTP:

boot tftp://172.19.64.31/cat9k_iosxe.16.12.01.SPA.bin

Note: More info on rommon recovery is found at the this link: recover_from_corrupt_or_missing_file_image_or_in_rommon_mode .

Bin File Missing from Member

Verify if the .bin file is seen on all member devices in the stack.

Step 1: 

Check the naming convention used on the stack with show file systems:

Switch#show file systems
File Systems:

       Size(b)       Free(b)      Type  Flags  Prefixes
             -             -    opaque     rw   system:
             -             -    opaque     rw   tmpsys:
*  11250098176    6275858432      disk     rw   bootflash: flash:         <- Name of Active Switch
   11250171904    4123000832      disk     rw   bootflash-2:              <- Name of Standby Switch
    1651314688    1344073728      disk     rw   crashinfo:
    1651507200    1180696576      disk     rw   crashinfo-2:
  236092686336  224026628096      disk     rw   disk0:
    8166649856    8053047296      disk     ro   webui:
             -             -    opaque     rw   null:
             -             -    opaque     ro   tar:
             -             -   network     rw   tftp:
      33554432      33494025     nvram     rw   nvram:
             -             -   network     rw   rcp:
             -             -   network     rw   http:
             -             -   network     rw   ftp:
             -             -   network     rw   scp:
             -             -   network     rw   sftp:
             -             -   network     rw   https:
             -             -    opaque     ro   cns:
      33554432      33507337     nvram     rw   stby-nvram:
             -             -     nvram     rw   stby-rcsf:
   11250098176    4122718208      disk     rw   stby-bootflash:
    1651314688    1180504064      disk     rw   stby-crashinfo:
             -             -    opaque     rw   revrcsf:

Note: If you are encountering an error of not enough file space, show file systems indicates size and free space available.

Step 2: 

Check each directory individually for the .bin file:

Switch#dir bootflash: | i .bin  
81126   -rw-       1265422219  Jun 19 2023 03:04:30 +00:00  cat9k_iosxe.17.11.01.SPA.bin              <- The Active has 17.11.01.SPA.bin
194733  -rw-       1027574083  Jun 13 2023 01:33:41 +00:00  cat9k_iosxe.17.06.05.SPA.bin
Switch#dir bootflash-2: | i .bin
210971  -rw-       1027574083  Jun 29 2023 22:19:23 +00:00  cat9k_iosxe.17.06.05.SPA.bin              <- The Standby does not have 17.11.01.SPA.bin

Note: It is expected for a device in install mode to only have the .bin file on the active until the install add command is committed. If an issue is seen proceed to Step 3.

Step 3: 

If manual intervention is required, there are 2 options to copy the .bin file from the Active to a Member/Standby.

 Option 1

Copy the .bin file directly to the Member/Standby:

Switch#copy bootflash:cat9k_iosxe.17.11.01.SPA.bin bootflash-2:

 Option 2

Copy the .bin to a USB (If direct copy is not possible)

Switch#copy bootflash:cat9k_iosxe.17.11.01.SPA.bin usbflash0:

Caution: The USB must be formatted for FAT32 and verify the Switch reads it by using show file systems.

Once it is copied to the USB, insert it into the Standby/Member:

Switch#copy usbflash0:cat9k_iosxe.17.11.01.SPA.bin bootflash-2:

Step 4: 

Verify the .bin file is on all devices:

Switch#dir flash: | i .bin
81126   -rw-       1265422219  Jun 19 2023 03:04:30 +00:00  cat9k_iosxe.17.11.01.SPA.bin           <- The Active has 17.11.01.SPA.bin
194733  -rw-       1027574083  Jun 13 2023 01:33:41 +00:00  cat9k_iosxe.17.06.05.SPA.bin
Switch#dir bootflash-2: | i .bin
210971  -rw-       1027574083  Jun 29 2023 22:19:23 +00:00  cat9k_iosxe.17.06.05.SPA.bin
210974  -rw-       1265422219  Jun 20 2023 03:32:23 +00:00  cat9k_iosxe.17.11.01.SPA.bin           <- The Standby now has 17.11.01.SPA.bin

Super Package Already Installed

If you get an error output when attempting to run the full install add file flash:name activate commit stating Super package already added, this means the package is already populated in the inactive state and is waiting to be activated and commited.

Switch#install add file flash:cat9k_iosxe.17.11.01.SPA.bin activate commit
install_add_activate_commit: START Wed Jul 26 12:36:58 UTC 2023

*Jul 26 12:36:58.491: %INSTALL-5-INSTALL_START_INFO: R0/0: install_engine: Started install one-shot bootflash:cat9k_iosxe.17.11.01.SPA.bininstall_add_activate_commit: Adding PACKAGE
install_add_activate_commit: Checking whether new add is allowed ....
FAILED: install_add_activate_commit : Super package already added. Add operation not allowed. 'install remove inactive' can be used to discard added packages

You can verify this state by issuing the show install summary command

Switch#show install summary
[ R0 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
            C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type  St   Filename/Version
--------------------------------------------------------------------------------
IMG   C    17.06.05.0.5797
IMG   I    17.11.01.0.1324                                                          <- Here it is seen as Inactive

--------------------------------------------------------------------------------
Auto abort timer: inactive
--------------------------------------------------------------------------------

There are 2 options that can be taken to resolve this issue

Option 1

Perform the install remove inactive command to clear the added packages

Switch#install remove inactive
install_remove: START Wed Jul 26 13:42:14 UTC 2023

Cleaning up unnecessary package files
No path specified, will use booted path bootflash:packages.conf
Cleaning bootflash:
  Scanning boot directory for packages ... done.
  Preparing packages list to delete ... 
    cat9k-cc_srdriver.16.09.03.SPA.pkg            
      File is in use, will not delete.
    cat9k-espbase.16.09.03.SPA.pkg
      File is in use, will not delete.
<...snip...>
  
The following files will be deleted:
[R0]:
/bootflash/cat9k-cc_srdriver.17.06.05.SPA.pkg    
/bootflash/cat9k-cc_srdriver.17.11.01.SPA.pkg
/bootflash/cat9k-espbase.17.06.05.SPA.pkg
/bootflash/cat9k-espbase.17.11.01.SPA.pkg
<...snip...>

Do you want to remove the above files? [y/n]y
[R0]:
Deleting file bootflash:cat9k-cc_srdriver.17.06.05.SPA.pkg ... done.     
Deleting file bootflash:cat9k-cc_srdriver.17.11.01.SPA.pkg ... done.
Deleting file bootflash:cat9k-espbase.17.06.05.SPA.pkg ... done.
Deleting file bootflash:cat9k-espbase.17.11.01.SPA.pkg ... done.
<...snip...>

SUCCESS: Files deleted.
--- Starting Post_Remove_Cleanup ---
Performing Post_Remove_Cleanup on Active/Standby
  [R0] Post_Remove_Cleanup package(s) on R0
  [R0] Finished Post_Remove_Cleanup on R0
Checking status of Post_Remove_Cleanup on [R0]
Post_Remove_Cleanup: Passed on [R0]
Finished Post_Remove_Cleanup

Option 2

Clear install state if install remove inactive does not resolve issue

Caution: This requires a reload of the device

Switch(config)#service internal                                        <- Service internal is required to be turned on for clear install state to work
Switch(config)#exit
Switch#clear install state
clear_install_state: START Wed Jul 26 12:25:09 UTC 2023

This command will remove all the provisioned SMUs, and rollback points. Use this command with caution.
A reload is required for this process. Press y to continue [y/n]y
--- Starting clear_install_state ---
Performing clear_install_state on Active/Standby
  [1] clear_install_state package(s) on R0
  [1] Finished clear_install_state on R0
Checking status of clear_install_state on [R0]
clear_install_state: Passed on [R0]
Finished clear_install_state

Send model notification for  before reload
Install will reload the system now!

Verify Both Option 1 and 2 can be verified by the show install summary command to ensure the Inactive Image is no longer there.

Switch#show install summary
[ R0 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
            C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type  St   Filename/Version
--------------------------------------------------------------------------------
IMG   C    17.06.05.0.5797                                                           <- 17.11.01 is no longer seen

--------------------------------------------------------------------------------
Auto abort timer: inactive
--------------------------------------------------------------------------------

Proceed with running the full install add file flash:name activate commit with no issue

Incorrect Boot Variable

This section covers issues seen when performing an upgrade and the wrong boot variable is set.

Scenario 1

• The Boot variable is set to bootflash:packages.conf
• For some reason, the switch reloads before the install add file flash:name activate commit occurs
• This corrupts ther packages.conf file or it is empty and the switch cannot find the boot parameters

Switch#show boot
BOOT variable = bootflash:packages.conf;
MANUAL_BOOT variable = no

When the switch reboots/reloads an error is seen

Preparing to autoboot. [Press Ctrl-C to interrupt]  0     
boot: attempting to boot from [bootflash:packages.conf]
ERROR: failed to boot from bootflash:packages.conf (Not Found)
Please wait while the system restarts.

Initializing Hardware......

• Break into ROMMON by either (CTRL-C) or pressing the Mode Button physically on the switch/supervisor
• Issue the set command
• Verify variable of BOOT=

switch:set
BAUD=9600
BOOT=bootflash:packages.conf;                                                      
BOOTLDR=packages.conf
BSI=0
<snip>

There are 2 options to boot into a desired IOS

Note: You can also boot into the previous ios bin file if desired

Option 1

Set new boot statement using new ios version

switch:BOOT=cat9k_iosxe.17.09.03.SPA.bin

Verify the BOOT statement is updated

switch:set
BAUD=9600
BOOT=bootflash:cat9k_iosxe.17.09.03.SPA.bin                                        <-Boot statement successfully updated
BOOTLDR=packages.conf
BSI=0
<snip>

Option 2

Boot immediately without changing variable until switch is loaded into ios

switch:boot bootflash:cat9k_iosxe.17.09.03.SPA.bin

Note: If unsure what .bin file exists on flash, issue dir flash: for list of files

Scenario 2

The command install add file flash:name activate commit is run but the boot variable set to .bin file instead of packages.conf

Switch#show boot

Current Boot Variables:
BOOT variable = flash:cat9k_iosxe_npe.17.03.03.SPA.bin;

Boot Variables on next reload:
BOOT variable = flash:cat9k_iosxe_npe.17.03.03.SPA.bin;
Manual Boot = no
Enable Break = yes
Boot Mode = DEVICE
iPXE Timeout = 0


Switch#install add file flash:cat9k_iosxe.17.09.03.SPA.bin activate commit
install_add_activate_commit: START Tue Sep 12 15:01:12 UTC 2023

install_add_activate_commit: Adding PACKAGE
install_add_activate_commit: Checking whether new add is allowed ....

This operation requires a reload of the system. Do you want to proceed? 
Please confirm you have changed boot config to flash:packages.conf [y/n]y

--- Starting initial file syncing ---
Info: Finished copying flash:cat9k_iosxe.17.09.03.SPA.bin to the selected switch(es)
Finished initial file syncing
<snip>

Upon reload an output can be seen with the switch loading into the 17.3.3.SPA.bin instead of packages.conf which contains 17.9.3.

Initializing Hardware......

System Bootstrap, Version 17.9.2r, RELEASE SOFTWARE (P)
Compiled Wed 11/23/2022 12:30:48.96 by rel

Current ROMMON image : Primary
Last reset cause     : CpuReset
C9500-12Q platform with 16777216 Kbytes of main memory

Preparing to autoboot. [Press Ctrl-C to interrupt]  0     
boot: attempting to boot from [flash:cat9k_iosxe_npe.17.03.03.SPA.bin]
boot: reading file cat9k_iosxe_npe.17.03.03.SPA.bin

Once the switch is booted up, verify that the packages.conf is correctly updated to be 17.9.3.

Switch#more flash:packages.conf
boot   rp 0 0   rp_boot cat9k-rpboot.17.09.03.SPA.pkg
boot   rp 1 0   rp_boot cat9k-rpboot.17.09.03.SPA.pkg

iso   rp 0 0   rp_base cat9k-rpbase.17.09.03.SPA.pkg
iso   rp 1 0   rp_base cat9k-rpbase.17.09.03.SPA.pkg                                                       <-All .pkg set to 17.9.3 correctly

iso   rp 0 0   rp_daemons cat9k-rpbase.17.09.03.SPA.pkg
iso   rp 1 0   rp_daemons cat9k-rpbase.17.09.03.SPA.pkg

iso   rp 0 0   rp_iosd cat9k-rpbase.17.09.03.SPA.pkg
iso   rp 1 0   rp_iosd cat9k-rpbase.17.09.03.SPA.pkg

iso   rp 0 0   rp_security cat9k-rpbase.17.09.03.SPA.pkg
iso   rp 1 0   rp_security cat9k-rpbase.17.09.03.SPA.pkg

iso   rp 0 0   rp_wlc cat9k-wlc.17.09.03.SPA.pkg
iso   rp 1 0   rp_wlc cat9k-wlc.17.09.03.SPA.pkg

iso   rp 0 0   rp_webui cat9k-webui.17.09.03.SPA.pkg
iso   rp 1 0   rp_webui cat9k-webui.17.09.03.SPA.pkg
<snip>

Change the boot variable to be flash:packages.conf

Switch(config)#no boot system
Switch(config)#boot system flash:packages.conf
Switch(config)#do wr                                                                  <- Configuration must be saved for boot variable to be updated properly before reloading
Switch#show boot
Current Boot Variables:
BOOT variable = flash:packages.conf;

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no
Enable Break = yes
Boot Mode = DEVICE
iPXE Timeout = 0

Reload the switch to get it to be in install mode and on the proper ios image of 17.9.3

Switch#reload
Initializing Hardware......

System Bootstrap, Version 17.9.2r, RELEASE SOFTWARE (P)
Compiled Wed 11/23/2022 12:30:48.96 by rel

Current ROMMON image : Primary
Last reset cause     : CpuReset
C9500-12Q platform with 16777216 Kbytes of main memory

Preparing to autoboot. [Press Ctrl-C to interrupt]  0     
boot: attempting to boot from [flash:packages.conf]                                           <-Boot variable correctly loading from packages.conf instead of .bin file
boot: reading file packages.conf
<snip>

Verify version after switch has loaded

Switch#show version
Cisco IOS XE Software, Version 17.09.03
Cisco IOS Software [Cupertino], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.9.3, RELEASE SOFTWARE (fc6)

Related Information

In-Service Software Upgrade (ISSU) on Catalyst 3850, Catalyst 9000 series switches

Cisco Bug IDs

Cisco bug ID CSCvr29736 - Doc bug: 9600 cannot do reloadfast
Cisco bug ID CSCvr29864 - Enhancement: 9300 add reloadfast to install command
Cisco bug ID CSCvr29886 - Enhancement: Be able to change boot statement of switch in v-mismatch

Cisco bug ID CSCvr30159 - Enhancement: Add check for manual boot when you run install add command

Cisco bug ID CSCvr30189 - Enhancement: Add an option to undo an "install add" without the need to delete the .bin files