Realice ISSU en switches Catalyst serie 3850 y 9000

Introducción

Este documento describe los pasos necesarios para realizar In-Service Software Upgrade (ISSU) en los switches Catalyst de las series 3850 y 9000.

Qué es ISSU

La actualización de software en funcionamiento es un proceso que actualiza una imagen a otra imagen en un dispositivo mientras la red continúa reenviando paquetes. ISSU ayuda a los administradores de red a evitar interrupciones de red cuando realizan una actualización de software. Las imágenes se actualizan en el modo de instalación, en el que cada paquete se actualiza individualmente.

ISSU es compatible con todos los switches Catalyst serie 3850 y 9000 que admiten Stackwise-Virtual y también con los chasis independientes Catalyst 9400 / 9600 con supervisores duales.

StackWise Virtual (SVL) consta de dos switches que se conectan entre sí para formar un switch virtual. SVL admite actualizaciones de software en servicio.

ISSU admite actualizaciones, reversiones y rollbacks.

Matriz de compatibilidad de versiones y plataformas admitidas

Antes de continuar con ISSU, verifique si la plataforma realmente soporta ISSU. Además, verifique si ISSU es compatible entre el código actual y el código de destino. Los detalles del modo de las plataformas soportadas y la matriz de compatibilidad ISSU se pueden encontrar aquí:

Soporte de ISSU entre Versiones

Requisitos previos antes de ISSU

Nota: El ejemplo de este documento se basa en los switches Cisco Catalyst 9500 configurados como Stackwise-Virtual. Los pasos son aplicables al chasis independiente Cat9400 / Cat9600 con motores supervisores duales y también para los dispositivos Catalyst 3850 / Catalyst 9000 configurados como Stackwise-Virtual.

1. Compruebe la versión del código actual

C9500#show version | in IOS XE 
Cisco IOS XE Software, Version 16.09.02

2. Compruebe el modo de arranque

ISSU se soporta solamente si ambos switches en Stackwise Virtual se inician en el modo Install.

C9500#show ver | in INSTALL
*    1 50    C9500-40X          16.9.2            CAT9K_IOSXE           INSTALL
     2 50    C9500-40X          16.9.2            CAT9K_IOSXE           INSTALL

On Catalyst 9400, the above output is not available. Check if the switch booted from packages.conf file

C9400#show version | in System image
System image file is "flash:packages.conf"
 

Si el chasis se inicia en el modo Bundle, ISSU no es compatible. Puede ver errores como este si intenta ejecutar ISSU cuando los switches se están ejecutando en el modo Bundle.

*Nov 13 14:55:57.338: %INSTALL-5-INSTALL_START_INFO: Chassis 1 R1/0: install_engine: Started install one-shot ISSU flash:cat9k_iosxe.16.09.02.SPA.bininstall_add_activate_commit: Adding ISSU
ERROR: install_add_activate_commit: One-Shot ISSU operation is not supported in bundle boot mode
FAILED: install_add_activate_commit  exit(1) Tue Nov 13 14:56:03 UTC 2018

3. Compruebe si hay suficiente memoria disponible en la memoria flash

C9500#dir flash: | in free
11353194496 bytes total (8565174272 bytes free)

C9500#dir stby-flash: | in free
11353980928 bytes total (8566865920 bytes free)

Asegúrese de tener al menos 1 GB de espacio en la memoria flash para ampliar una nueva imagen. Limpie los archivos de instalación antiguos en caso de espacio insuficiente y utilice el comando install remove inactive.

4. Compruebe si los switches están en modo SSO

C9500#show redundancy 
Redundant System Information :
------------------------------
       Available system uptime = 4 minutes
Switchovers system experienced = 0
              Standby failures = 0
        Last switchover reason = none

                 Hardware Mode = Duplex
    Configured Redundancy Mode = sso
     Operating Redundancy Mode = sso
              Maintenance Mode = Disabled
                Communications = Up

Current Processor Information :
-------------------------------
               Active Location = slot 1
        Current Software state = ACTIVE
       Uptime in current state = 30 minutes
                 Image Version = Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.9.2, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Mon 05-Nov-18 19:32 by mcpre
                          BOOT = flash:packages.conf;
                   CONFIG_FILE = 
        Configuration register = 0x102

Peer Processor Information :
----------------------------
              Standby Location = slot 2
        Current Software state = STANDBY HOT 
       Uptime in current state = 26 minutes
                 Image Version = Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.9.2, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Mon 05-Nov-18 19:32 by mcpre
                          BOOT = flash:packages.conf;
                   CONFIG_FILE = 
        Configuration register = 0x102
        

5. Compruebe si el arranque automático está activado

C9500#show boot system 
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no                       <<<<< Manual Boot should be set to "no"       
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0
---------------------------
Switch 2
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0 

Si el inicio automático no está habilitado, se puede cambiar como se muestra:

C9500(config)#no boot manual

6. Compruebe el ISSU actual y los estados de instalación

C9500#show issu state detail
--- Starting local lock acquisition on switch 1 ---
Finished local lock acquisition on switch 1

No ISSU operation is in progress      <<<<<<<<< If see anything else, abort ISSU before proceeding.
Check on how to manually abort ISSU.



 
C9500#show install summary
[ Switch 1 2 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
            C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type  St   Filename/Version    
--------------------------------------------------------------------------------
IMG   C    16.9.2.0.2433      <<<<<<<< State should be Activated & Committed for current version alone.
 If not clear install state before proceeding. Check on how to clear install state.                                                  

--------------------------------------------------------------------------------
Auto abort timer: inactive
--------------------------------------------------------------------------------

7. Copie la nueva imagen para actualizarla o bajarla de categoría

Basta con copiar la imagen solo en el chasis activo (en el caso de Stackwise) o en el supervisor activo (en el caso de Cat9400 con SUP dual).

C9500#copy tftp: bootflash:
Address or name of remote host []? X.X.X.X
Source filename []? cat9k_iosxe.16.09.02.SPA.bin
Destination filename [cat9k_iosxe.16.09.02.SPA.bin]? 
 

Flujos de trabajo de ISSU - Actualización real

Una vez verificados todos los requisitos previos, puede continuar con la actualización real y utilizar uno de los métodos enumerados:

• Flujo de trabajo de un paso (solo un paso y no admite rollback)
• Flujo de trabajo de tres pasos (incluye 3 pasos y admite la reversión en caso de fallo)

Flujo de trabajo de un paso

Este flujo de trabajo solo implica un paso y ayuda en la optimización.

// This example has SW-2 as Active and Sw-1 as Standby before starting ISSU


C9500#install add file flash:cat9k_iosxe.16.09.02.SPA.bin activate issu commit
install_add_activate_commit: START Fri Feb  8 10:07:51 jst 2019

*Feb  8 10:07:52.456 jst: %INSTALL-5-INSTALL_START_INFO: Switch 2 R0/0: install_engine: Started install one-shot ISSU flash:cat9k_iosxe.16.09.02.SPA.bininstall_add_activate_commit: Adding ISSU

--- Starting initial file syncing ---
[2]: Copying flash:cat9k_iosxe.16.09.02.SPA.bin from switch 2 to switch 1
[1]: Finished copying to switch 1  
Info: Finished copying flash:cat9k_iosxe.16.09.02.SPA.bin to the selected switch(es)
Finished initial file syncing

--- Starting Add ---
Performing Add on all members
  [1] Add package(s) on switch 1
  [1] Finished Add on switch 1
  [2] Add package(s) on switch 2
  [2] Finished Add on switch 2
Checking status of Add on [1 2]
Add: Passed on [1 2]
Finished Add

install_add_activate_commit: Activating ISSU

Going to start Oneshot ISSU install process

STAGE 0: Initial System Level Sanity Check before starting ISSU
===================================================
--- Verifying install_issu supported ---
--- Verifying standby is in Standby Hot state ---
--- Verifying booted from the valid media ---
--- Verifying AutoBoot mode is enabled ---
Finished Initial System Level Sanity Check


STAGE 1: Installing software on Standby
===================================================
--- Starting install_remote ---
Performing install_remote on Chassis remote
[1] install_remote package(s) on switch 1
[1] Finished install_remote on switch 1
install_remote: Passed on [1]
Finished install_remote


STAGE 2: Restarting Standby
===================================================
--- Starting standby reload ---
Finished standby reload

--- Starting wait for Standby to reach terminal redundancy state ---     <<<<< Standby (Sw-1) reloads here!!!


<<<<<<<< After Standby (Sw-1) comes up >>>>>>>>>>>

*Feb  8 10:19:10.223 jst: %REDUNDANCY-3-IPC: IOS versions do not match.

*Feb  8 10:19:48.421 jst: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded

*Feb  8 10:19:49.422 jst: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)   <<<<< Standby (Sw-1) comes up on new code and joins as Hot Standby..


*Feb  8 10:21:02.975 jst: %PLATFORM-6-HASTATUS_DETAIL: RP switchover, received chassis event became active. Switch to primary (count 1)   <<<<<< Switchover happens after this log and the previous active (Sw-2) reloads.

<<<<<<<< After new Standby (Sw-2) comes up >>>>>>>>

*Feb  8 10:27:09.715 jst: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded

*Feb  8 10:27:10.717 jst: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO).  <<<< ISSU commit starts after this automatically..


*Feb  8 10:28:27.302 jst: %INSTALL-5-INSTALL_START_INFO: Switch 2 R0/0: install_engine: Started install commit
%IOSXEBOOT-4-ISSU_ONE_SHOT: (rp/0): ISSU finished successfully

*Feb  8 10:29:32.127 jst: %INSTALL-5-INSTALL_COMPLETED_INFO: Switch 2 R0/0: install_engine: Completed install commit ISSU

 Continúe con después de completar ISSU.

Flujo de trabajo en tres pasos

• Este flujo de trabajo consta de tres pasos: agregar, activar y confirmar. Después de la activación, todos los switches se actualizan a la nueva versión de software, excepto que el software no se confirma automáticamente, sino que se debe realizar manualmente mediante el comando install commit.
• La ventaja de este enfoque es que el sistema se puede revertir a una versión de software anterior.
• El sistema retrocede automáticamente si el temporizador de reversión no se detiene y utiliza el comando install auto-abort-timer stop o el comando install commit. Si se detiene el temporizador de reversión, la nueva versión de software se puede ejecutar en el dispositivo durante cualquier tiempo y, a continuación, se puede revertir a la versión anterior.

Paso 1. Instale add.

 Este comando descarga la imagen en la memoria flash de inicialización y la expande en ambos switches.

// This example has SW-1 as Active and Sw-2 as Standby before starting ISSU

C9500#install add file flash:cat9k-universalk9.SPA.16.09.03.BETA.E1.SSA.bin.bin    
install_add: START Fri Feb  8 09:22:00 jst 2019

*Feb  8 09:22:02.055 jst: %INSTALL-5-INSTALL_START_INFO: Switch 1 R0/0: install_engine: Started install add flash:cat9k-universalk9.SPA.16.09.03.BETA.E1.SSA.bin.bininstall_add: Adding PACKAGE

--- Starting initial file syncing ---
[1]: Copying flash:cat9k-universalk9.SPA.16.09.03.BETA.E1.SSA.bin.bin from switch 1 to switch 2
[2]: Finished copying to switch 2  
Info: Finished copying flash:cat9k-universalk9.SPA.16.09.03.BETA.E1.SSA.bin.bin to the selected switch(es)
Finished initial file syncing

--- Starting Add ---
Performing Add on all members
  [1] Add package(s) on switch 1
  [1] Finished Add on switch 1
  [2] Add package(s) on switch 2
  [2] Finished Add on switch 2
Checking status of Add on [1 2]
Add: Passed on [1 2]
Finished Add

SUCCESS: install_add  Fri Feb  8 09:26:26 jst 2019  <<<< Wait until install_add says SUCCESS. If fails, manually abort ISSU.

Paso 2. Instalar activar.

• Al ejecutar este comando, se produce esta secuencia de eventos:
  
  (i) Se inicia un temporizador de reversión. Si el temporizador de reversión caduca, el sistema vuelve al mismo estado antes del inicio de ISSU. El temporizador de reversión se puede detener cuando se utiliza el comando install auto-abort-timer stop. ISSU se puede revertir cuando utiliza el comando install abort issu.
  
      (ii) El switch en espera se suministra con el nuevo software y se recarga con la nueva versión del software. A continuación, el switch activo se aprovisiona con el nuevo software y se recarga. El switch en espera con la nueva imagen se convierte ahora en el switch activo y el switch activo antiguo se convierte en el switch en espera.

Al final de este procedimiento, ambos switches se ejecutan con la nueva imagen de software.

C9500#install activate issu
install_activate: START Fri Feb  8 09:28:27 jst 2019
install_activate: Activating ISSU

*Feb  8 09:28:28.905 jst: %INSTALL-5-INSTALL_START_INFO: Switch 1 R0/0: install_engine: Started install activate ISSU
Going to start Activate ISSU install process

STAGE 0: Initial System Level Sanity Check before starting ISSU===================================================
--- Verifying install_issu supported ---
--- Verifying standby is in Standby Hot state ---
--- Verifying booted from the valid media ---
--- Verifying AutoBoot mode is enabled ---
Finished Initial System Level Sanity Check
STAGE 1: Installing software on Standby
===================================================
--- Starting install_remote ---
Performing install_remote on Chassis remote

*Feb  8 09:28:31.880 jst: %INSTALL-5-INSTALL_AUTO_ABORT_TIMER_PROGRESS: Switch 1 R0/0: rollback_timer: Install auto abort timer can expire in 7200 seconds


[2] install_remote package(s) on switch 2
[2] Finished install_remote on switch 2
install_remote: Passed on [2]
Finished install_remote

STAGE 2: Restarting Standby
===================================================
--- Starting standby reload ---
Finished standby reload--- Starting wait for Standby to reach terminal redundancy state ---      <<<<<<<< Standby (Sw-2) reloads here!!!*Feb  8 09:35:16.489 jst: %REDUNDANCY-3-IPC: IOS versions do not match.
 
*Feb  8 09:36:00.238 jst: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
*Feb  8 09:36:01.240 jst: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)     <<<< At this point, Standby (Sw-2) comes up with new code and joins as Hot Standby
Finished wait for Standby to reach terminal redundancy state

STAGE 3: Installing software on Active
===================================================
--- Starting install_active ---

Performing install_active on Chassis 11] install_active package(s) on switch 1
[1] Finished install_active on switch 1
install_active: Passed on [1]
Finished install_active
Chassis 1 reloading, reason - Non participant detected

STAGE 4: Restarting Active (switchover to standby)      <<<<<<<< At this point, there is a switchover and then the previous Active (Sw-1) reloads.
===================================================
--- Starting active reload ---
 New software can load after reboot process is completed
SUCCESS: install_activate  Fri Feb  8 09:37:14 jst 2019

Al final del estado Activate, verifique los estados ISSU.

C9500#show issu state detail
--- Starting local lock acquisition on switch 2 ---
Finished local lock acquisition on switch 2

Operation type: Step-by-step ISSU
Install type  : Image installation using ISSUCurrent state : Activated state
Last operation: Switchover

Completed operations:

Operation                           Start time
-------------------------------------------------------
Activate location standby Chassis 2 2019-02-08:09:28:32
Activate location active Chassis 1  2019-02-08:09:36:03
Switchover                          2019-02-08:09:37:16

State transition: Added -> Standby activated -> Active switched-over 

Auto abort timer: automatic, remaining time before rollback: 01:43:55
Running image: flash:packages.conf
Operating mode: sso, terminal state reached    <<<<< Wait until SSO terminal state before proceeding to commit.

Paso 3.  Instalar commit.

El comando commit realiza la limpieza necesaria, habilita el nuevo software como permanente (elimina la versión anterior del software) y detiene el temporizador de reversión. Cualquier reinicio después de la confirmación puede arrancar con nuevo software.

C9500#install commit 
install_commit: START Fri Feb  8 09:45:22 jst 2019
install_commit: Committing ISSU

*Feb  8 09:45:23.533 jst: %INSTALL-5-INSTALL_START_INFO: Switch 2 R0/0: install_engine: Started install commit

Going to start Commit ISSU install process

STAGE 0: Initial System Level Sanity Check before starting ISSU
===================================================
--- Verifying install_issu supported ---
--- Verifying standby is in Standby Hot state ---
--- Verifying booted from the valid media ---
--- Verifying AutoBoot mode is enabled ---

Finished Initial System Level Sanity Check

--- Starting install_commit_2 ---
Performing install_commit_2 on Chassis 2
[2] install_commit_2 package(s) on switch 2
[2] Finished install_commit_2 on switch 2
install_commit_2: Passed on [2]
Finished install_commit_2


STAGE 1: Dispatching the commit command to remote
===================================================
--- Starting install_commit_remote ---
Performing install_commit_remote on Chassis 1
Feb  8 09:48:33.364: %INSTALL-5-INSTALL_START_INFO: R0/0: install_engine: Started install commit

*Feb  8 09:48:33.352 jst: %INSTALL-5-INSTALL_START_INFO: Switch 1 R0/0: install_engine: Started install commit


Feb  8 09:51:27.505: %INSTALL-5-INSTALL_COMPLETED_INFO: R0/0: install_engine: Completed install commit ISSU
[1] install_commit_remote package(s) on switch 1
[1] Finished install_commit_remote on switch 1
install_commit_remote: Passed on [1]
Finished install_commit_remote

SUCCESS: install_commit  Fri Feb  8 09:51:27 jst 2019     <<<<< ISSU is completed here!!!!!

Lista de comprobación posterior a ISSU

Una vez que ISSU se haya completado correctamente,

• Verifique si ambos switches se ejecutan en el nuevo software.
• Marque show issu state detail output para estar limpio y no mostrar ningún ISSU en progreso.
• Marque show install issu history output para garantizar una operación ISSU exitosa (comando disponible solamente con la versión 16.10.1 y posteriores).
• Se recomienda dejar suficiente tiempo en remojo en el nuevo software antes de habilitar cualquier función nueva.

Acción sobre falla de ISSU

• Si ISSU falla, se espera que la anulación automática pueda recuperar el sistema al estado inicial (imagen anterior). Sin embargo, si esto también falla, se espera una recuperación manual del chasis.
• Durante la recuperación manual, verifique si tanto el activo como el en espera ejecutan la imagen anterior (si no, recupere el chasis individual).
• Después de asegurarse de que ambos chasis ejecuten la imagen antigua, ejecute install remove inactive para eliminar cualquier paquete de imagen no utilizado.
• Una vez que ambos chasis ejecuten el software antiguo, limpie manualmente todos los estados internos del funcionamiento de ISSU. (Consulte aquí sobre cómo limpiar los estados ISSU internos).

Anular ISSU

• En el flujo de trabajo de 3 pasos, durante el proceso de activación de ISSU, si caduca el temporizador de anulación, el sistema puede abortar automáticamente a una imagen anterior.

EXAMPLE : During install add, we notice these errors:

C9400#install add file flash:cat9k_iosxe.16.09.02.SPA.bin
install_add: START Tue Nov 13 20:47:53 UTC 2018

*Nov 13 20:47:54.787: %INSTALL-5-INSTALL_START_INFO: Chassis 1 R1/0: install_engine: Started install add flash:cat9k_iosxe.16.09.02.SPA.bininstall_add: Adding PACKAGE

--- Starting initial file syncing ---
[1]: Copying flash:cat9k_iosxe.16.09.02.SPA.bin from chassis 1 to chassis 2
[2]: Finished copying to chassis 2  
Info: Finished copying flash:cat9k_iosxe.16.09.02.SPA.bin to the selected chassis
Finished initial file syncing

--- Starting Add ---
Performing Add on all members
  [1] Add package(s) on chassis 1
  [1] Finished Add on chassis 1
  [2] Add package(s) on chassis 2
    cp: cannot stat '/tmp/packages.conf': No such file or directory
  [2] Finished Add on chassis 2
Checking status of Add on [1 2]
Add: Passed on [1]. Failed on [2]
Finished Add

FAILED: install_add  exit(1) Tue Nov 13 20:51:58 UTC 2018   <<<<<< install_add failed. If see any such error, first do a manual abort before proceeding further.  


C9400#install abort issu 
install_abort: START Tue Nov 13 20:57:40 UTC 2018
install_abort: Abort type ISSU subtype NONE smutype NONE

*Nov 13 20:57:41.759: %INSTALL-5-INSTALL_START_INFO: Chassis 1 R1/0: install_engine: Started install abort ISSU
NOTE: Going to start Abort ISSU install process

STAGE 0: Initial System Level Sanity Check before starting ISSU
===================================================
--- Verifying install_issu supported ---
--- Verifying booted from the valid media ---
--- Verifying AutoBoot mode is enabled ---
Finished Initial System Level Sanity Check

FAILED: ABORT operation is not allowed in ADDED state
ERROR: install_abort  exit(2 ) Tue Nov 13 20:57:49 UTC 2018

*Nov 13 20:57:49.756: %INSTALL-5-INSTALL_COMPLETED_INFO: Chassis 1 R1/0: install_engine: Completed install abort ISSU

Estado de ISSU limpio

Si el upgrade/downgrade/abort/auto-abort de ISSU no es exitoso, se requiere una limpieza manual de los estados internos de ISSU.

C9400#sh issu state detail
--- Starting local lock acquisition on chassis 1 ---
Finished local lock acquisition on chassis 1

Operation type: One-shot ISSU
Install type  : Image installation using ISSU
Current state : Added state
Last operation: Activate location standby Chassis 2   <<<< Previous Add is still pending. This needs to be cleared before starting ISSU

Completed operations:

Operation                           Start time
-------------------------------------------------------
Activate location standby Chassis 2 2018-11-13:16:26:34   

State transition: Added 

Auto abort timer: inactive
Running image: flash:packages.conf
Operating mode: sso, terminal state not reached

Enable Service Internal before you run this command

C9400#clear install state   
clear_install_state: START Tue Nov 13 17:05:47 UTC 2018
--- Starting clear_install_state ---
Performing clear_install_state on all members
  [1] clear_install_state package(s) on chassis 1
  [1] Finished clear_install_state on chassis 1
Checking status of clear_install_state on [1]
clear_install_state: Passed on [1]
Finished clear_install_state

C9400#sh issu state detail 
--- Starting local lock acquisition on chassis 1 ---
Finished local lock acquisition on chassis 1

No ISSU operation is in progress