Troubleshoot "Remote FMC Is Not Updated Successfully"

Introduction

This document describes how to troubleshoot "Remote FMC Is Not Updated Successfully. Complete the update on remote FMC before updating this peer."

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Firepower Management Center (FMC)
• Basic knowledge of the FMC CLI.

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

Error Message 

The error "Remote FMC is not updated successfully. Complete the update on remote FMC before updating this peer" displays on the FMC GUI when you try to upgrade the devices managed by the FMC High Availability (HA) pair. This error does not allow the upgrade of the managed devices to start. Here is how the error alert looks like from the GUI:

The error can also be verified from the CLI of the FMC with the expert mode command cat /var/log/httpd/httpd_error_log.1 | grep -i 'Remote FMC'.

> expert
root@FMC:~$ cat /var/log/httpd/httpd_error_log.1 | grep -i 'Remote FMC'

[Mon Jan 30 07:20:10.062741 2022] [cgi:error] [pid 5906] [client 192.168.1.10:45267] AH01215: (Remote FMC is not updated successfully. Complete the update on remote FMC before updating this peer.) in /usr/local/sf/htdocs/admin/update.cgi:331 at /usr/local/sf/lib/perl/5.10.1/SF.pm line 120.: /usr/local/sf/htdocs/admin/update.cgi, referer:  
[Mon Jan 30 07:22:43.370986 2022] [cgi:error] [pid 15376] [clien 192.168.1.10:45267] AH01215: (Remote FMC is not updated successfully. Complete the update on remote FMC before updating this peer.) in /usr/local/sf/htdocs/admin/update.cgi:331 at /usr/local/sf/lib/perl/5.10.1/SF.pm line 120.: /usr/local/sf/htdocs/admin/update.cgi, referer:  

Error Causes

This error is known to occur when there is a mismatch in the software patch version, Vulnerability Database (VDB) version, Intrusion Rules (SRU) version or Geolocation Database (GeoDB) version between the two FMCs in HA. The mismatch occurs when any of these listed version updates are stuck or fail to install. You cannot see this mismatch when you check the versions from the FMC UI under section Help > About, but it is recommended to check this page on both the FMCs to verify.

Note: The deployments to the managed devices can be successful with this but the software upgrades fail to start with this error.

Identify the Issue

Check the Versions on FMCs in HA from GUI

From the FMC GUI, go to Help > About to confirm the versions of Software Patch, VDB, SRU and GeoDB on both FMC in HA are all the same. These images show an example of a version match of two FMCs in HA from GUI:

.              

Verify Installation Status of VDB, SRU, GeoDB versions on FMCs in HA from CLI

From expert mode on FMC CLI, you need to verify if the VDB, SRU and GeoDB updates were installed completely without any failures on both FMCs in HA.

Note: In these sections, it is explained how to check the status.log of each image version folder. These image version folders must match the folder on the peer FMC. For example, if the VDB version folder that is installed on FMC is "vdb-4.5.0-338", then you have to check under the same folder for both the FMCs. Here, use the command cat /var/log/sf/vdb-4.5.0-338/status.log on both FMC to check the update status of VDB. The same applies for SRU and GeoDB updates as well.

Check VDB Installation Status

From the expert mode on FMC CLI, use this command cat /var/log/sf/<vdb-image-folder>/status.log to verify if the VDB update has been successful. Here is an example of a successful VDB installation:

root@FMC:~$ cat /var/log/sf/vdb-4.5.0-338/status.log
state:running
ui:The install has begun
ui:[ 0%] Running script pre/000_start.sh...
ui:[ 4%] Running script pre/010_check_versions.sh...
ui:[ 8%] Running script pre/011_check_versions.pl...
ui:[12%] Running script pre/020_check_space.sh...
ui:[15%] Running script pre/500_stop_rna.pl...
ui:[19%] Running script pre/999_finish.sh...
ui:[23%] Running script installer/000_start.sh...
ui:[27%] Running script installer/100_install_files.pl...
ui:[31%] Running script installer/200_install_fingerprints.sh...
ui:[35%] Running script installer/300_install_vdb.sh...
ui:[38%] Running script installer/400_install_rdps.pl...
ui:[42%] Running script installer/420_delete_obsolete_ids.pl...
ui:[46%] Running script installer/450_resave_detectors.pl...
ui:[50%] Running script installer/525_export_compliance_policies.pl...
ui:[54%] Running script installer/600_fix_dbcheck.sh...
ui:[58%] Running script installer/605_install_dbcheck_upgrade_script.sh...
ui:[62%] Running script installer/610_install_missing_upgrade_script.sh...
ui:[65%] Running script installer/615_purge_vdb_149_log.sh...
ui:[69%] Running script installer/900_update_version.sh...
ui:[73%] Running script installer/901_update_db_version.pl...
ui:[77%] Running script installer/950_reapply_to_sensor.pl...
ui:[81%] Running script installer/975_export_data.pl...
ui:[85%] Running script installer/999_finish.sh...
ui:[88%] Running script post/000_start.sh...
ui:[92%] Running script post/500_start_rna.pl...
ui:[96%] Running script post/999_finish.sh...
ui:[100%] The install completed successfully.
ui:The install has completed.
state:finished

Check SRU Installation Status

From the expert mode on FMC CLI, use the command cat /var/log/sf/<sru-image-folder>/status.log to verify if the SRU update has been successful. Here is an example of a successful SRU installation:

root@FMC:~$ cat /var/log/sf/sru-2021-05-03-001-vrt/status.log
state:running
ui:The force install has begun.
ui:[ 0%] Running script pre/000_start.sh...
ui:[ 5%] Running script pre/010_check_versions.sh...
ui:[11%] Running script pre/020_check_space.sh...
ui:[16%] Running script pre/999_finish.sh...
ui:[21%] Running script installer/000_start.sh...
ui:[26%] Running script installer/050_sru_log_start.pl...
ui:[32%] Running script installer/100_install_files.pl...
ui:[37%] Running script installer/510_install_policy.pl...
ui:[42%] Running script installer/520_install_rules.pl...
ui:[47%] Running script installer/521_rule_docs.sh...
ui:[53%] Running script installer/530_install_module_rules.pl...
ui:[58%] Running script installer/540_install_decoder_rules.pl...
ui:[63%] Running script installer/602_log_package.pl...
ui:[68%] Running script installer/900_update_version.sh...
ui:[74%] Running script installer/999_finish.sh...
ui:[79%] Running script post/000_start.sh...
ui:[84%] Running script post/500_copy_contents.sh...
ui:[89%] Running script post/900_iru_log_finish.pl...
ui:[95%] Running script post/999_finish.sh...
ui:[100%] The force install completed successfully.
ui:The force install has completed.
state:finished

Check GeoDB Installation Status

From the expert mode on FMC CLI, use the command cat /var/log/sf/<geodb-image-folder>/status.log to verify if the GeoDB update has been successful. Here is an example of a successful GeoDB installation:

root@FMC:~$ cat /var/log/sf/geodb-2022-08-02-100/status.log
state:running
ui:The install has begun.
ui:[ 0%] Running script installer/200_prechecks.pl...
ui:[33%] Running script installer/500_install_country_map.pl...
ui:[67%] Running script installer/601_fix_country.pl...
ui:[100%] The install completed successfully.
ui:The install has completed.
state:finished

If the installation failed or is stuck for any reason, you can see what step this failed or is stuck from this status.log. Here is an example of a GeoDB install fail on the FMC:

root@FMC:~$ cat /var/log/sf/geodb-2022-07-17-100/status.log
state:running
ui:The install has begun.
ui:[ 0%] Running script installer/200_prechecks.pl...
ui:[33%] Running script installer/500_install_country_map.pl...
ui:[67%] Running script installer/601_fix_country.pl...
ui:[67%] Fatal error: Error running script installer/601_fix_country.pl

Verify Installation Status of Software Version and Patch on FMCs in HA from CLI 

From expert mode on FMC CLI, use the command cat /etc/sf/patch_history to verify if both FMC have the same version and patch installed. Run this command to identify any mismatch on both FMCs. Here is an example of a patch mismatch from the CLI: 

root@FMC:~$ cat /etc/sf/patch_history
6.2.3-83
6.6.0-90
6.6.4-59
6.6.5-81
Hotfix_DE-8__413769962     <<<<<<<<<<<  Here the FMC seems to have a Hotfix installation image that is not present from the other FMC

-------------------------------------------------------------------

root@FMC:~$ cat /etc/sf/patch_history
6.2.3-83
6.6.0-90
6.6.4-59
6.6.5-81        

To further check if the installation of the hotfix in the FMC was successful, you need to check the status.log for this image folder:

root@FMC:~$ cat /var/log/sf/Cisco_Firepower_Mgmt_Center_Hotfix_DE-6.6.5.2/status.log

ui:[98%] Upgrade complete
ui:[99%] Running script 999_finish/999_z_must_remain_last_finalize_boot.sh...
ui:[99%] Running script 999_finish/999_zz_install_bundle.sh...
ui:[100%] The system will now restart services.
ui:System will now restart services.
ui:[100%] Installation completed successfully.
ui:Upgrade has completed.
state:finished

This example verifies that the patch image was not present in the one of the FMC in HA, whereas the other one had the patch successfully installed. 

Troubleshoot

To resolve the error, you have to perform a manual force install of the updates from the CLI of the FMC where the issue is identified. 

Disclaimer: Root access to the FMC devices is required in order to execute the commands under this section. Please use caution when running commands from the root of the FMC. 

VDB, SRU and GeoDB Update Issue

After you identify the VDB, SRU or GeoDB update issues, perform a manual force install from the CLI command install_update.pl /var/sf/updates/<image-file> --force. Here is an example of the manual force install for a GeoDB update:

> expert
root@FMC:~$ sudo su
<Enter the root password>
root@FMC:# install_update.pl /var/sf/updates/Cisco_Firepower_GEODB_FMC_Update-2022-08-02-100.sh.REL.tar --force

Note: Use the absolute path of the image file with the install_update.pl command as shown in the example. Do not untar any tar.gz files prior to force installation from CLI.

Hotfix Install Issue

For the hotfix/patch installation, you need to download the patch file and install it to the FMC where the patch file was not present either via GUI or CLI.

From FMC GUI: 

Go to System > Updates > Product Updates and upload the patch version to be installed. Then click on the Install option and choose the device to which you need to install the patch and proceed with installation. 

From FMC CLI:

To install the software/patch from the FMC CLI, upload the hotfix upgrade file to the path /var/log/sf/ on the FMC CLI and execute the command install_update.pl /var/log/sf/<image-file>. This command runs the the upgrade logs on the same screen to allow us to monitor the progress. Here is an example of the patch installation from the CLI:

> expert
root@FMC:~$ sudo su
<Enter the root password>
root@FMC:# install_update.pl /var/log/sf/Cisco_Firepower_Mgmt_Center_Hotfix_DE-6.6.5.2 

If there is a short timeout on the SSH session, use the command install_update.pl--detach /var/log/sf/<image-file> to run the installation in the background. This allows the upgrade to run even after the SSH session is closed.

Verify

VDB, SRU or GeoDB Update

After the manual force installation is complete, you can verify the status of the installation from the CLI with cat /var/log/sf/<image-version-folder>/status.log command for VDB, SRU and GeoDB update. Here is an example of the status.log output of a successful GeoDB installation:

root@FMC:/Volume/home/admin# cat /var/log/sf/geodb-2022-08-02-100/status.log
state:running
ui:The force install has begun.
ui:[ 0%] Running script installer/200_prechecks.pl...
ui:[33%] Running script installer/500_install_country_map.pl...
ui:[67%] Running script installer/601_fix_country.pl...
ui:[100%] The force install completed successfully.
ui:The force install has completed.
state:finished

Hotfix or Patch Update

After the manual installation of the update, execute the command cat /var/log/sf/<patch-image-folder>/status.log from CLI to verify the status of this installation. Here is an example of the status.log output of a successful installation:

root@FMC:/var/log/sf/Cisco_Firepower_Mgmt_Center_Hotfix_DE-6.6.5.2# tail -f status.log
ui:[98%] Upgrade complete
ui:[99%] Running script 999_finish/999_z_must_remain_last_finalize_boot.sh...
ui:[99%] Running script 999_finish/999_zz_install_bundle.sh...
ui:[100%] The system will now restart services.
ui:System will now restart services.
ui:[100%] Installation completed successfully.
ui:Upgrade has completed.
state:finished

Note: If the error is still present after you attempt the steps provided in this document, please open a service request with Cisco TAC.