Upgrade FTD HA via CLI Managed by FMC

Introduction

This document describes a detailed procedure to upgrade Cisco Firepower Threat Defense (FTD) devices via the Command Line Interface (CLI).

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Cisco Secure Firewall Management Center (FMC)
• Cisco Secure Firewall Threat Defense (FTD)

Components Used

The information in this document is based on these software and hardware versions:

- Cisco Secure Firewall Management Center v7.2.8

- Cisco Firepower Threat Defense for VMWare v7.2.2

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

Specific requirements for this document include:

• Cisco Secure Firewall Threat Defense running version 7.2 or higher
• Cisco Secure Firewall Management Center running version 7.2 or higher

Configure

Upgrading a pair of FTD devices via CLI requires the upgrade package file to be present on the device. It is essential to have no pending deployments as a prerequisite for a successful upgrade via CLI.

Preparing for Upgrade

Warning: Check the upgrade order, Standby / Active to avoid any traffic outages.

1. Begin with the device configured as Standby.

2. Access the CLI in expert mode by entering expert followed by sudo su in the clish mode. Confirm the device password to elevate privileges and enter expert mode.

Copyright 2004-2022, Cisco and/or its affiliates. All rights reserved. 
Cisco is a registered trademark of Cisco Systems, Inc. 
All other trademarks are property of their respective owners.

Cisco Firepower Extensible Operating System (FX-OS) v2.12.0 (build 1104)
Cisco Firepower Threat Defense for VMware v7.2.2 (build 54)

> expert
admin@firepower:~$ sudo su

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

Password: 
root@firepower:/home/admin# 
root@firepower:/home/admin# cd
root@firepower:~# 
root@firepower:~# 

Check Failover Status

Verify the failover status to ensure that the steps are applied to the Secondary FTD, which can be displayed as Secondary and Standby Ready.

firepower#
firepower# sh failover state

               State          Last Failure Reason      Date/Time
This host  -   Secondary
               Standby Ready  None              
Other host -   Primary
               Active         None

====Configuration State===
        Sync Done - STANDBY
====Communication State===
        Mac set

firepower#
firepower#

Upload the Upgrade Package

Upload the upgrade package to both devices through the FMC by navigating to Settings > Updates > Product Updates > Upload local software update package. Choose the previously downloaded package from software.cisco.com and select Upload.

Once you have uploaded Firepower package on the FMC, continue with Upgrade button.

Upgrade Button

On the upgrade wizard you need to select the FTD HA devices, then select the devices, and click Add to Selection.

Add to Selection

Then, you can copy the upgrade Package on the devices, a message appears to continue Upgrade Packages.

Copy Upgrade Package Button

On the Notification task, you can find the job copying the files to device When the task is finished, it is completed and successful.

Task Copying Files to Devices

You can verify the package is uploaded to the devices on this path:

root@firepower:/ngfw/var/sf/updates# 
root@firepower:/ngfw/var/sf/updates# ls -l
total 2181772
-rw-r--r-- 1 root root 1110405120 Jul 18 01:08 Cisco_FTD_Upgrade-7.2.2-54.sh.REL.tar
-rw-r--r-- 1 root root        815 Jul 18 01:23 Cisco_FTD_Upgrade-7.2.2-54.sh.REL.tar.METADATA
-rw-r--r-- 1 root root 1123706880 Jul 18 02:36 Cisco_FTD_Upgrade-7.2.7-500.sh.REL.tar
-rw-r--r-- 1 root root        854 Jul 18 02:37 Cisco_FTD_Upgrade-7.2.7-500.sh.REL.tar.METADATA
root@firepower:/ngfw/var/sf/updates# 

Readiness Check

Execute the readiness check from the CLI on the secondary device using the command:

root@firepower:/ngfw/var/sf/updates# install_update.pl --detach --readiness-check /ngfw/var/sf/updates/<FTD_Upgrade_Package.sh.REL.tar>

Here is an example:

root@firepower:/ngfw/var/sf/updates# install_update.pl --detach --readiness-check /ngfw/var/sf/updates/Cisco_FTD_Upgrade-7.2.7-500.sh.REL.tar
ARGV[0] = --detach
ARGV[1] = --readiness-check
ARGV[2] = /ngfw/var/sf/updates/Cisco_FTD_Upgrade-7.2.7-500.sh.REL.tar
bundle_filepath: /ngfw/var/sf/updates/Cisco_FTD_Upgrade-7.2.7-500.sh.REL.tar
install_update.pl begins. bundle_filepath: /var/sf/updates/Cisco_FTD_Upgrade-7.2.7-500.sh.REL.tar
[Readiness-Info]filename : /var/sf/updates/Cisco_FTD_Upgrade-7.2.7-500.sh.REL.tar  at /usr/local/sf/lib/perl/5.24.4/SF/Update.pm line 1374.
This was not run through the SF::System APIs at /usr/local/sf/lib/perl/5.24.4/SF/System/Wrappers.pm line 127.
Makeself GetUpdate Info params FILEPATH : /var/tmp/upgrade-patch/Cisco_FTD_Upgrade_Readiness-7.2.7-500.sh at /usr/local/sf/lib/perl/5.24.4/SF/Update/Makeself.pm line 33.
FILEPATH directory name /var/tmp/upgrade-patch at /usr/local/sf/lib/perl/5.24.4/SF/Update/Makeself.pm line 47.
Inside GetInfo FILEPATH :/var/tmp/upgrade-patch/Cisco_FTD_Upgrade_Readiness-7.2.7-500.sh at /usr/local/sf/lib/perl/5.24.4/SF/Update/Makeself.pm line 272.
root@firepower:/ngfw/var/sf/updates#

Monitor the readiness check process at this path:

root@firepower:/ngfw/var/log/sf/Cisco_FTD_Upgrade-7.2.7/upgrade_readiness

root@firepower:/ngfw/var/log/sf/Cisco_FTD_Upgrade-7.2.7/upgrade_readiness# cat upgrade_readiness_status.log
TIMESTAMP:Thu Jul 18 02:43:05 UTC 2024 PERCENT: 0%  MESSAGE:Running script 000_start/000_00_run_cli_kick_start.sh...
TIMESTAMP:Thu Jul 18 02:43:05 UTC 2024 PERCENT: 5%  MESSAGE:Running script 000_start/000_check_platform_support.sh...
TIMESTAMP:Thu Jul 18 02:43:06 UTC 2024 PERCENT:10%  MESSAGE:Running script 000_start/100_start_messages.sh...
TIMESTAMP:Thu Jul 18 02:43:06 UTC 2024 PERCENT:14%  MESSAGE:Running script 000_start/101_run_pruning.pl...
TIMESTAMP:Thu Jul 18 02:43:41 UTC 2024 PERCENT:19%  MESSAGE:Running script 000_start/105_check_model_number.sh...
TIMESTAMP:Thu Jul 18 02:43:42 UTC 2024 PERCENT:24%  MESSAGE:Running script 000_start/106_check_HA_state.pl...
TIMESTAMP:Thu Jul 18 02:43:42 UTC 2024 PERCENT:29%  MESSAGE:Running script 000_start/107_version_check.sh...
TIMESTAMP:Thu Jul 18 02:43:42 UTC 2024 PERCENT:33%  MESSAGE:Running script 000_start/108_clean_user_stale_entries.pl...
TIMESTAMP:Thu Jul 18 02:43:43 UTC 2024 PERCENT:38%  MESSAGE:Running script 000_start/110_DB_integrity_check.sh...
TIMESTAMP:Thu Jul 18 02:43:47 UTC 2024 PERCENT:43%  MESSAGE:Running script 000_start/113_EO_integrity_check.pl...
TIMESTAMP:Thu Jul 18 02:43:50 UTC 2024 PERCENT:48%  MESSAGE:Running script 000_start/250_check_system_files.sh...
TIMESTAMP:Thu Jul 18 02:43:50 UTC 2024 PERCENT:52%  MESSAGE:Running script 000_start/410_check_disk_space.sh...
TIMESTAMP:Thu Jul 18 02:43:55 UTC 2024 PERCENT:57%  MESSAGE:Running script 200_pre/001_check_reg.pl...
TIMESTAMP:Thu Jul 18 02:43:55 UTC 2024 PERCENT:62%  MESSAGE:Running script 200_pre/002_check_mounts.sh...
TIMESTAMP:Thu Jul 18 02:43:56 UTC 2024 PERCENT:67%  MESSAGE:Running script 200_pre/004_check_deploy_package.pl...
TIMESTAMP:Thu Jul 18 02:43:56 UTC 2024 PERCENT:71%  MESSAGE:Running script 200_pre/005_check_manager.pl...
TIMESTAMP:Thu Jul 18 02:43:56 UTC 2024 PERCENT:76%  MESSAGE:Running script 200_pre/006_check_snort.sh...
TIMESTAMP:Thu Jul 18 02:43:57 UTC 2024 PERCENT:81%  MESSAGE:Running script 200_pre/007_check_sru_install.sh...
TIMESTAMP:Thu Jul 18 02:43:57 UTC 2024 PERCENT:86%  MESSAGE:Running script 200_pre/009_check_snort_preproc.sh...
TIMESTAMP:Thu Jul 18 02:43:58 UTC 2024 PERCENT:90%  MESSAGE:Running script 200_pre/011_check_self.sh...
TIMESTAMP:Thu Jul 18 02:43:58 UTC 2024 PERCENT:95%  MESSAGE:Running script 200_pre/015_verify_rpm.sh...
TIMESTAMP:Thu Jul 18 02:44:00 UTC 2024 PERCENT:100%  MESSAGE:Readiness Check completed successfully.
root@firepower:/ngfw/var/log/sf/Cisco_FTD_Upgrade-7.2.7/upgrade_readiness# 

If readiness check fails, contact Cisco TAC.

Upgrade Installation

Proceed with the upgrade installation on the Secondary FTD. Navigate to the folder containing the upgrade file and execute the installation command:

root@firepower:/ngfw/var/sf/updates# install_update.pl --detach <FTD_Upgrade_Package.sh.REL.tar>

Once the upgrade has been executed, there is going to be an output like the next example:

root@firepower:/ngfw/var/sf/updates# install_update.pl --detach Cisco_FTD_Upgrade-7.2.7-500.sh.REL.tar
ARGV[0] = Cisco_FTD_Upgrade-7.2.7-500.sh.REL.tar
bundle_filepath: Cisco_FTD_Upgrade-7.2.7-500.sh.REL.tar
updated absolute bundle_filepath: /ngfw/var/sf/updates/Cisco_FTD_Upgrade-7.2.7-500.sh.REL.tar
install_update.pl begins. bundle_filepath: /var/sf/updates/Cisco_FTD_Upgrade-7.2.7-500.sh.REL.tar
Makeself GetUpdate Info params FILEPATH : /var/tmp/upgrade-patch/Cisco_FTD_Upgrade-7.2.7-500.sh at /usr/local/sf/lib/perl/5.24.4/SF/Update/Makeself.pm line 33.
FILEPATH directory name /var/tmp/upgrade-patch at /usr/local/sf/lib/perl/5.24.4/SF/Update/Makeself.pm line 47.
Inside GetInfo FILEPATH :/var/tmp/upgrade-patch/Cisco_FTD_Upgrade-7.2.7-500.sh at /usr/local/sf/lib/perl/5.24.4/SF/Update/Makeself.pm line 272.
Use of uninitialized value in string at /usr/local/sf/lib/perl/5.24.4/SF/Update/StatusProc.pm line 196.
Use of uninitialized value in string at /usr/local/sf/lib/perl/5.24.4/SF/Update/StatusProc.pm line 196.
Use of uninitialized value in string at /usr/local/sf/lib/perl/5.24.4/SF/Update/StatusProc.pm line 196.
Use of uninitialized value $in_container in string eq at /usr/local/sf/lib/perl/5.24.4/SF/Update/StatusProc.pm line 174.
Verifying archive integrity... All good.
Uncompressing Cisco FTD Upgrade / Sat Apr 27 04:09:29 UTC 2024.................................................................................................................................................................................................................................................................................................................................
Entering is_fmc_managed
Device is FMC Managed
[240718 02:48:13:868] Found original ftd upgrade file /var/sf/updates/Cisco_FTD_Upgrade-7.2.7-500.sh.REL.tar
[240718 02:48:16:990] MAIN_UPGRADE_SCRIPT_START
[240718 02:48:17:006] #####################################
[240718 02:48:17:007] # UPGRADE  STARTING 
[240718 02:48:17:008] #####################################
compare 7.2.2 and 6.2.3 and 
compare, newer installed 7.2.2 > 6.2.3
Entering create_upgrade_status_links...
Create upgrade_status.json and upgrade_status.log link in /ngfw/var/sf/sync/updates_status_logs
Running [ln -f /ngfw/var/log/sf/Cisco_FTD_Upgrade-7.2.7/upgrade_status.json /ngfw/var/sf/sync/updates_status_logs/upgrade_status.json] ...success
Link to JSON upgrade status file /ngfw/var/log/sf/Cisco_FTD_Upgrade-7.2.7/upgrade_status.json created in folder /ngfw/var/sf/sync/updates_status_logs
Running [ln -f /ngfw/var/log/sf/Cisco_FTD_Upgrade-7.2.7/upgrade_status.log /ngfw/var/sf/sync/updates_status_logs/current_update_status.log] ...success
Link to log upgrade status file /ngfw/var/log/sf/Cisco_FTD_Upgrade-7.2.7/upgrade_status.log created in folder /ngfw/var/sf/sync/updates_status_logs
[240718 02:48:17:229] BEGIN 000_start/000_00_run_cli_kick_start.sh
[240718 02:48:18:421] END 000_start/000_00_run_cli_kick_start.sh
[240718 02:48:18:525] BEGIN 000_start/000_00_run_troubleshoot.sh

On the FMC, there is a task running the upgrade on the Secondary device:

Task Running on FMC

Monitor the upgrade status using this path:

root@firepower:/ngfw/var/log/sf/Cisco_FTD_Upgrade-X.X.X# tail -f upgrade_status.log

Here is an example of the output:

root@firepower:/ngfw/var/log/sf/Cisco_FTD_Upgrade-7.2.7# tail -f upgrade_status.log
TIMESTAMP:Thu Jul 18 02:50:25 UTC 2024 PERCENT: 7%  MESSAGE:Running script 200_pre/202_disable_syncd.sh... TIMEREMAIN:13 mins
TIMESTAMP:Thu Jul 18 02:50:26 UTC 2024 PERCENT: 7%  MESSAGE:Running script 200_pre/400_restrict_rpc.sh... TIMEREMAIN:13 mins
TIMESTAMP:Thu Jul 18 02:50:26 UTC 2024 PERCENT: 7%  MESSAGE:Running script 200_pre/500_stop_system.sh... TIMEREMAIN:13 mins
TIMESTAMP:Thu Jul 18 02:50:53 UTC 2024 PERCENT:14%  MESSAGE:Running script 200_pre/501_recovery.sh... TIMEREMAIN:12 mins
TIMESTAMP:Thu Jul 18 02:50:53 UTC 2024 PERCENT:14%  MESSAGE:Running script 200_pre/505_revert_prep.sh... TIMEREMAIN:12 mins
TIMESTAMP:Thu Jul 18 02:51:46 UTC 2024 PERCENT:14%  MESSAGE:Running script 200_pre/999_enable_sync.sh... TIMEREMAIN:12 mins
TIMESTAMP:Thu Jul 18 02:51:46 UTC 2024 PERCENT:14%  MESSAGE:Running script 300_os/001_verify_bundle.sh... TIMEREMAIN:12 mins
TIMESTAMP:Thu Jul 18 02:51:47 UTC 2024 PERCENT:14%  MESSAGE:Running script 300_os/002_set_auto_neg.pl... TIMEREMAIN:12 mins
TIMESTAMP:Thu Jul 18 02:51:47 UTC 2024 PERCENT:14%  MESSAGE:Running script 300_os/060_fix_fstab.sh... TIMEREMAIN:12 mins
TIMESTAMP:Thu Jul 18 02:51:47 UTC 2024 PERCENT:14%  MESSAGE:Running script 300_os/100_install_Fire_Linux_OS_aquila.sh (in background: 200_pre/600_ftd_onbox_data_export.sh)... TIMEREMAIN:12 mins

When the Upgrade on the Secondary device has finished, you see this message:

240718 13:40:58:872] Attempting to remove upgrade lock 
[240718 13:40:58:873] Success, removed upgrade lock
Upgrade lock /ngfw/tmp/upgrade.lock removed successfully.
[240718 13:40:58:882] 
[240718 13:40:58:883] #######################################################
[240718 13:40:58:885] # UPGRADE  COMPLETE  #
[240718 13:40:58:887] #######################################################
Entering create_upgrade_status_links...
Create upgrade_status.json and upgrade_status.log link in /ngfw/Volume/root/ngfw/var/sf/sync/updates_status_logs
Running [ln -f /ngfw/Volume/root/ngfw/var/log/sf/Cisco_FTD_Upgrade-7.2.7/upgrade_status.json /ngfw/Volume/root/ngfw/var/sf/sync/updates_status_logs/upgrade_status.json] ...success
Link to JSON upgrade status file /ngfw/Volume/root/ngfw/var/log/sf/Cisco_FTD_Upgrade-7.2.7/upgrade_status.json created in folder /ngfw/Volume/root/ngfw/var/sf/sync/updates_status_logs
Running [ln -f /ngfw/Volume/root/ngfw/var/log/sf/Cisco_FTD_Upgrade-7.2.7/upgrade_status.log /ngfw/Volume/root/ngfw/var/sf/sync/updates_status_logs/current_update_status.log] ...success
Link to log upgrade status file /ngfw/Volume/root/ngfw/var/log/sf/Cisco_FTD_Upgrade-7.2.7/upgrade_status.log created in folder /ngfw/Volume/root/ngfw/var/sf/sync/updates_status_logs
Process 10677 exited.I am going away.
RC: 0
Update package reports success: almost finished...
Scheduling a reboot to occur in 5 seconds...
Process 12153 exited.I am going away.
root@firepower:/ngfw/var/sf/updates# 
Broadcast message from root@firepower (Thu Jul 18 13:41:05 2024):

The system is going down for reboot NOW!

After the upgrade from the Standby device is completed, the device is going to be rebooted. Once the devices comes up, check the failover status to ensure everything remains as initially configured.

On the Active FTD you can find:

firepower# show failover state

               State          Last Failure Reason      Date/Time
This host  -   Primary
               Active         None
Other host -   Secondary
               Standby Ready  Comm Failure             13:24:46 UTC Jul 18 2024

====Configuration State===
        Sync Done
====Communication State===
        Mac set

firepower# 

On Standby FTD, you find this:

firepower# 
firepower# sh failover state

               State          Last Failure Reason      Date/Time
This host  -   Secondary
               Standby Ready  None
Other host -   Primary
               Active         None

====Configuration State===
        Sync Skipped - STANDBY
====Communication State===
        Mac set

firepower# 

There is going to be a message showing that the versions are not the same. 

firepower# 
************WARNING****WARNING****WARNING********************************
   Mate version 9.18(4)201 is not identical with ours 9.18(2)200
************WARNING****WARNING****WARNING********************************

Perform the failover manually via CLI using the command failover active on Standby Device. Now the Standby device become Active.

Warning: At this point there is going a brief traffic interruption when failover happens.

firepower# 
firepower# failover active  

        Switching to Active
firepower# 
firepower# 
firepower# sh fail
firepower# sh failover state

               State          Last Failure Reason      Date/Time
This host  -   Secondary
               Active         None
Other host -   Primary
               Standby Ready  None

====Configuration State===
        Sync Skipped
====Communication State===
        Mac set

firepower# 

Once the failover has been completed, you can proceed upgrading the other device. Use the same steps described at the beginning of the document for the device that was previously Active and now is Standby.

Now both devices are upgraded. You can see with the command show version on Lina side. For the Primary Device:

firepower# 
firepower# show failover state

               State          Last Failure Reason      Date/Time
This host  -   Primary
               Standby Ready  None
Other host -   Secondary
               Active         None

====Configuration State===
        Sync Skipped - STANDBY
====Communication State===
        Mac set

firepower# 

For the Secondary Device:

firepower# 
firepower# sh failover state

               State          Last Failure Reason      Date/Time
This host  -   Secondary
               Active         None
Other host -   Primary
               Standby Ready  Comm Failure             14:03:06 UTC Jul 18 2024

====Configuration State===
        Sync Skipped
====Communication State===
        Mac set

firepower# 

At this point, you can switch over the devices from FMC like it was at the beginning.

Verify

After successfully upgrading both devices, verify the status within the FMC and on both FTDs using the command show version.

firepower# show version
-------------------[ firepower ]--------------------
Model                     : Cisco Firepower Threat Defense for VMware (75) Version 7.2.7 (Build 500)
UUID                      : 0edf9f22-78e6-11ea-8ed0-e0e5abf334e2
LSP version               : lsp-rel-20240306-2015
VDB version               : 353
----------------------------------------------------

On the FMC, you can see the version update and are ready to switch over as you had it at the beginning.

Switched Peers from FMC