Troubleshoot for FMC - HA

Available Languages

Download Options

PDF (131.0 KB)
View with Adobe Reader on a variety of devices
ePub (146.8 KB)
View in various apps on iPhone, iPad, Android, Sony Reader, or Windows Phone
Mobi (Kindle) (134.9 KB)
View on Kindle device or Kindle app on multiple devices

Updated:October 23, 2024

Document ID:222495

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Background information

Before you Begin

Troubleshoot commands

FMC troubleshooting commands.

FTD troubleshooting commands

Verification

FMC - HA validation

Communication from FTD to FMC-HA validation

Introduction

This document describes how to troubleshoot common synchronization issues in a High Availability (HA) set for the Firepower Management Center (FMC).

Prerequisites

Requirements

Cisco recommends that you have knowledge of the following topics:

FMC - HA configuration requirements
Basic knowledge of Linux shell.

Components Used

FMCv for VMware on version 7.2.8.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background information

The initial setup of the laboratory used for this document follows the requirements for the basic FMC-HA initial configuration.

Two FMCs with the same capacity or hardware version.
Two FMCs running the same software version, Intrusion Rule Update, Vulnerability Database, and Lightweight Security Package.
Two FMCs with the corresponding licenses.

Before you Begin

Ensure administrator has access to both FMCs.
Ensure administrator has access to the FTD devices managed by the FMC.

Troubleshoot commands

FMC troubleshooting commands.

To validate the connectivity between FMC devices, the user can run these commands.

> expert
admin@firepower:~$ sudo su

root@firepower:/Volume/home/admin# ping xx.xx.18.102
PING xx.xx.18.102 (xx.xx.18.102) 56(84) bytes of data.
64 bytes from xx.xx.18.102: icmp_seq=1 ttl=64 time=0.533 ms
64 bytes from xx.xx.18.102: icmp_seq=2 ttl=64 time=0.563 ms
64 bytes from xx.xx.18.102: icmp_seq=3 ttl=64 time=0.431 ms
^C
--- xx.xx.18.102 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 59ms
rtt min/avg/max/mdev = 0.431/0.509/0.563/0.056 ms

root@firepower:/Volume/home/admin# netstat -an | grep 8305
tcp 0 0 xx.xx.18.101:8305 0.0.0.0:* LISTEN 
tcp 0 0 xx.xx.18.101:8305 xx.xx.18.253:48759 ESTABLISHED
tcp 0 0 xx.xx.18.101:8305 xx.xx.18.254:53875 ESTABLISHED
tcp 0 0 xx.xx.18.101:8305 xx.xx.18.254:49205 ESTABLISHED
tcp 0 0 xx.xx.18.101:60871 xx.xx.18.253:8305 ESTABLISHE

ping <peer-ip-address> This command can be used to check the reachability between both devices.

netstat -an | grep 8305 This command displays the devices connected to port 8305.

Note: The port 8305 is the default port configured on the Firepower devices to establish the communication channel with the FMC.

To validate the FMC-HA setup configuration, the user can also run the script troubleshoot_HADC.pl. This is particularly useful in these scenarios:

When the FMC-HA integration health status is degraded.
If the access to the FMC Graphic User Interface (GUI) of one of the devices has been missing, but the FMC-CLI access is still working and accessible.

> expert
admin@firepower:~$ sudo su

root@firepower:/Volume/home/admin# troubleshoot_HADC.pl
**************** Troubleshooting Utility **************

1 Show HA Info Of FMC
2 Execute Sybase DBPing
3 Show Arbiter Status
4 Check Peer Connectivity
5 Print Messages of AQ Task
6 Show FMC HA Operations History (ASC order)
7 Dump To File: FMC HA Operations History (ASC order)
8 Last Successful Periodic Sync Time (When it completed)
9 Print HA Status Messages
10 Compare active and standby device list
11 Check manager status of standby missing devices
12 Check critical PM processes details
13 Get Remote Stale Sync AQ Info
14 Help
0 Exit

**************************************************************
Enter choice:

FTD troubleshooting commands

Troubleshooting the connectivity from the FTD to the FMC-HA allows the user to validate connectivity of devices that need to be registered on both FMCs or when HA is degraded, and displays the warning "Degraded – Synchronization incomplete (This Management Center has fewer devices registered)".

From the FTD clish level, the user can run these commands to validate the communication with the FMC.

> ping system xx.xx.18.102
PING xx.xx.18.102 (xx.xx.18.102) 56(84) bytes of data.
64 bytes from xx.xx.18.102: icmp_seq=1 ttl=64 time=0.595 ms
64 bytes from xx.xx.18.102: icmp_seq=2 ttl=64 time=0.683 ms
64 bytes from xx.xx.18.102: icmp_seq=3 ttl=64 time=0.642 ms
64 bytes from xx.xx.18.102: icmp_seq=4 ttl=64 time=24.4 ms
64 bytes from xx.xx.18.102: icmp_seq=5 ttl=64 time=11.4 ms
^C
--- xx.xx.18.102 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 128ms
rtt min/avg/max/mdev = 0.595/7.545/24.373/9.395 ms

> show managers
Type : Manager
Host : xx.xx..18.101
Display name : xx.xx..18.101
Version : 7.2.8 (Build 25)
Identifier : fc3e3572-xxxx-xxxx-xxxx-39e0098c166c
Registration : Completed
Management type : Configuration and analytics

Type : Manager
Host : xx.xx..18.102
Display name : xx.xx..18.102
Version : 7.2.8 (Build 25)
Identifier : bb333216-xxxx-xxxx-xxxx-c68c0c388b44
Registration : Completed
Management type : Configuration and analytics


> sftunnel-status
SFTUNNEL Start Time: Mon Oct 14 21:29:16 2024

Both IPv4 and IPv6 connectivity is supported
Broadcast count = 5
Reserved SSL connections: 0
Management Interfaces: 2
eth0 (control events) xx.xx..18.254,
tap_nlp (control events) 169.254.1.2,fd00:0:0:1::2

***********************

**RUN STATUS****xx.xx..18.102*************
Key File = /var/sf/peers/bb333216-xxxx-xxxx-xxxx-c68c0c388b44/sftunnel-key.pem
Cert File = /var/sf/peers/bb333216-xxxx-xxxx-xxxx-c68c0c388b44/sftunnel-cert.pem
CA Cert = /var/sf/peers/bb333216-xxxx-xxxx-xxxx-c68c0c388b44/cacert.pem
Cipher used = TLS_AES_256_GCM_SHA384 (strength:256 bits)
ChannelA Connected: Yes, Interface eth0
Cipher used = TLS_AES_256_GCM_SHA384 (strength:256 bits)
ChannelB Connected: Yes, Interface eth0
Registration: Completed.
IPv4 Connection to peer 'xx.xx..18.102' Start Time: Tue Oct 15 00:38:43 2024 UTC
IPv4 Last outbound connection to peer 'xx.xx..18.102' via Primary ip/host 'xx.xx..18.102'

PEER INFO:
sw_version 7.2.8
sw_build 25
Using light registration
Management Interfaces: 1
eth0 (control events) xx.xx..18.102,
Peer channel Channel-A is valid type (CONTROL), using 'eth0', connected to 'xx.xx..18.102' via 'xx.xx..18.254'
Peer channel Channel-B is valid type (EVENT), using 'eth0', connected to 'xx.xx..18.102' via 'xx.xx..18.254'

***********************

**RUN STATUS****xx.xx..18.101*************
Key File = /var/sf/peers/fc3e3572-xxxx-xxxx-xxxx-39e0098c166c/sftunnel-key.pem
Cert File = /var/sf/peers/fc3e3572-xxxx-xxxx-xxxx-39e0098c166c/sftunnel-cert.pem
CA Cert = /var/sf/peers/fc3e3572-xxxx-xxxx-xxxx-39e0098c166c/cacert.pem
Cipher used = TLS_AES_256_GCM_SHA384 (strength:256 bits)
ChannelA Connected: Yes, Interface eth0
Cipher used = TLS_AES_256_GCM_SHA384 (strength:256 bits)
ChannelB Connected: Yes, Interface eth0
Registration: Completed.
IPv4 Connection to peer 'xx.xx..18.101' Start Time: Mon Oct 14 21:29:15 2024 UTC
IPv4 Last outbound connection to peer 'xx.xx..18.101' via Primary ip/host 'xx.xx..18.101'

PEER INFO:

sw_version 7.2.8
sw_build 25
Using light registration
Management Interfaces: 1
eth0 (control events) xx.xx..18.101,
Peer channel Channel-A is valid type (CONTROL), using 'eth0', connected to 'xx.xx..18.101' via 'xx.xx..18.254'
Peer channel Channel-B is valid type (EVENT), using 'eth0', connected to 'xx.xx..18.101' via 'xx.xx..18.254'

***********************
**RPC STATUS****xx.xx..18.102*************
'uuid' => 'bb333216-xxxx-xxxx-xxxx-c68c0c388b44',
'uuid_gw' => '',
'last_changed' => 'Wed Oct 9 07:00:11 2024',
'active' => 1,
'name' => 'xx.xx..18.102',
'ip' => 'xx.xx..18.102',
'ipv6' => 'IPv6 is not configured for management'

**RPC STATUS****xx.xx..18.101*************
'uuid_gw' => '',
'uuid' => 'fc3e3572-xxxx-xxxx-xxxx-39e0098c166c',
'last_changed' => 'Mon Jun 10 18:59:54 2024',
'active' => 1,
'ip' => 'xx.xx..18.101',
'ipv6' => 'IPv6 is not configured for management',
'name' => 'xx.xx..18.101'

Check routes:
No peers to check

ping system <fmc-IP> To generate an ICMP, follow from the FTD management interface.

show managers This command lists the information of the managers where the device is registered.

sftunnel-status This command validates the communication channel established between the devices. This channel receives the name of sftunnel.

The commands to check the connectivity at the root level on the FTD are the same as the FMC. In the case of the FTD, it does not include a script that allows to validation of communication with the FMC, but it is possible to check the information generated during the registration process in the /ngfw/var/log/action.log.

Verification

For the next topology, the communication between the FMC-HA peers and the FTD01 can be validated using the commands previously described.

FMC-HA Topology

FMC - HA validation

For this validation, the basic guidelines to set an FMC-HA can also be validated using the command show version.

FMC Active

> show version
-------------------[ firepower ]--------------------
Model                     : Secure Firewall Management Center for VMware (66) Version 7.2.8 (Build 25)
UUID                      : fc3e3572-xxxx-xxxx-xxxx-39e0098c166c
Rules update version      : 2023-11-29-001-vrt
LSP version               : lsp-rel-20231129-1200
VDB version               : 395
----------------------------------------------------
> expert
admin@firepower:~$ sudo su

root@firepower:/Volume/home/admin# ping xx.xx.18.102
PING xx.xx.18.102 (xx.xx.18.102) 56(84) bytes of data.
64 bytes from xx.xx.18.102: icmp_seq=1 ttl=64 time=0.385 ms
64 bytes from xx.xx.18.102: icmp_seq=2 ttl=64 time=0.433 ms
64 bytes from xx.xx.18.102: icmp_seq=3 ttl=64 time=0.606 ms
64 bytes from xx.xx.18.102: icmp_seq=4 ttl=64 time=0.480 ms
64 bytes from xx.xx.18.102: icmp_seq=5 ttl=64 time=0.524 ms
^C
--- xx.xx.18.102 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 84ms
rtt min/avg/max/mdev = 0.385/0.485/0.606/0.079 ms

root@firepower:/Volume/home/admin# netstat -an | grep 8305
tcp        0      0 xx.xx.18.101:8305       xx.xx.18.254:53875      ESTABLISHED
tcp        0      0 xx.xx.18.101:8305       xx.xx.18.102:38239      ESTABLISHED ------ communication establish with FMC-Standby
tcp        0      0 xx.xx.18.101:8305       xx.xx.18.254:49205      ESTABLISHED
tcp        0      0 xx.xx.18.101:8305       xx.xx.18.253:34865      ESTABLISHED
tcp        0      0 xx.xx.18.101:60871      xx.xx.18.253:8305       ESTABLISHED
tcp        0      0 xx.xx.18.101:8305       xx.xx.18.102:42253      ESTABLISHED ------ communication establish with FMC-Standby

root@firepower:/Volume/home/admin# troubleshoot_HADC.pl
****************  Troubleshooting Utility  **************
 1   Show HA Info Of FMC
 2   Execute Sybase DBPing
 3   Show Arbiter Status
 4   Check Peer Connectivity
 5   Print Messages of AQ Task
 6   Show FMC HA Operations History (ASC order)
 7   Dump To File: FMC HA Operations History (ASC order)
 8   Last Successful Periodic Sync Time (When it completed)
 9   Print HA Status Messages
 10  Compare active and standby device list
 11  Check manager status of standby missing devices
 12  Check critical PM processes details
 13  Get Remote Stale Sync AQ Info
 14  Help
 0   Exit
**************************************************************
Enter choice: 1

HA Enabled: Yes
This FMC Role In HA: Active - Primary
Status out put: vmsDbEngine (system,gui) - Running 5093
In vmsDbEngineStatus(): vmsDbEngine process is running at /usr/local/sf/lib/perl/5.24.4/SF/Synchronize/HADC.pm line 3748.
Sybase Process: Running (vmsDbEngine, theSybase PM Process is Running)
Sybase Database Connectivity: Accepting DB Connections.
Sybase Database Name: csm_primary
Sybase Role: Active

****************  Troubleshooting Utility  **************
 1   Show HA Info Of FMC
 2   Execute Sybase DBPing
 3   Show Arbiter Status
 4   Check Peer Connectivity
 5   Print Messages of AQ Task
 6   Show FMC HA Operations History (ASC order)
 7   Dump To File: FMC HA Operations History (ASC order)
 8   Last Successful Periodic Sync Time (When it completed)
 9   Print HA Status Messages
 10  Compare active and standby device list
 11  Check manager status of standby missing devices
 12  Check critical PM processes details
 13  Get Remote Stale Sync AQ Info
 14  Help
 0   Exit
**************************************************************
Enter choice: 4
Peer UUID [Enter 'Return' For HA Peer(no UUID required)]:

Peer Is Connected
$VAR1 = {
          'vip' => '',
          'model_id' => 'E',
          'ip' => 'xx.xx.18.102',
          'persistent' => 0,
          'sw_version' => '7.2.8',
          'last_changed' => 1728457211,
          'active' => 1,
          'uuid' => 'bb333216-xxxx-xxxx-xxxx-c68c0c388b44',
          'upgrade_version' => '',
          'reg_state' => 0,
          'model_number' => '66',
          'primary_mgr' => 0,
          'name' => 'xx.xx.18.102',
          'uuid_gw' => '',
          'ipv6' => undef,
          'vip_local' => '',
          'priority' => 0,
          'reg_key' => '',
          'vnet' => undef,
          'role' => 0,
          'mgmt_mac_address' => '00:50:56:B3:D1:07'
        };

**************** Troubleshooting Utility **************
1 Show HA Info Of FMC
2 Execute Sybase DBPing
3 Show Arbiter Status
4 Check Peer Connectivity
5 Print Messages of AQ Task
6 Show FMC HA Operations History (ASC order)
7 Dump To File: FMC HA Operations History (ASC order)
8 Last Successful Periodic Sync Time (When it completed)
9 Print HA Status Messages
10 Compare active and standby device list
11 Check manager status of standby missing devices
12 Check critical PM processes details
13 Get Remote Stale Sync AQ Info
14 Help
0 Exit
**************************************************************
Enter choice: 8

---------------------------- Last periodic sync time details ------------------------------------

Last successful sync completed at: Wed Oct 16 16:44:23 2024 UTC
Current time: Wed Oct 16 16:46:34 2024 UTC
Last successful sync completed '2 minutes 11 seconds' ago.

---------------------------- Last periodic sync time details end ------------------------------------

**************** Troubleshooting Utility **************
1 Show HA Info Of FMC
2 Execute Sybase DBPing
3 Show Arbiter Status
4 Check Peer Connectivity
5 Print Messages of AQ Task
6 Show FMC HA Operations History (ASC order)
7 Dump To File: FMC HA Operations History (ASC order)
8 Last Successful Periodic Sync Time (When it completed)
9 Print HA Status Messages
10 Compare active and standby device list
11 Check manager status of standby missing devices
12 Check critical PM processes details
13 Get Remote Stale Sync AQ Info
14 Help
0 Exit
**************************************************************
Enter choice: 9

SYNC_ACTIVE: 1 at /usr/local/sf/lib/perl/5.24.4/SF/Synchronize.pm line 494, <STDIN> line 5.
Sybase state :  at /usr/local/sf/lib/perl/5.24.4/SF/Synchronize.pm line 701.
Sybase state :  at /usr/local/sf/lib/perl/5.24.4/SF/Synchronize.pm line 801.
Sync status : at /usr/local/sf/lib/perl/5.24.4/SF/Synchronize.pm line 802.
---------------------------- FMC HA status messages start ------------------------------------

Status: Healthy

---------------------------- FMC HA status messages end ------------------------------------
**************** Troubleshooting Utility **************
1 Show HA Info Of FMC
2 Execute Sybase DBPing
3 Show Arbiter Status
4 Check Peer Connectivity
5 Print Messages of AQ Task
6 Show FMC HA Operations History (ASC order)
7 Dump To File: FMC HA Operations History (ASC order)
8 Last Successful Periodic Sync Time (When it completed)
9 Print HA Status Messages
10 Compare active and standby device list
11 Check manager status of standby missing devices
12 Check critical PM processes details
13 Get Remote Stale Sync AQ Info
14 Help
0 Exit
**************************************************************
Enter choice: 10

Fetching standby missing device information...
Devices are in sync.

**************** Troubleshooting Utility **************
1 Show HA Info Of FMC
2 Execute Sybase DBPing
3 Show Arbiter Status
4 Check Peer Connectivity
5 Print Messages of AQ Task
6 Show FMC HA Operations History (ASC order)
7 Dump To File: FMC HA Operations History (ASC order)
8 Last Successful Periodic Sync Time (When it completed)
9 Print HA Status Messages
10 Compare active and standby device list
11 Check manager status of standby missing devices
12 Check critical PM processes details
13 Get Remote Stale Sync AQ Info
14 Help
0 Exit
**************************************************************
Enter choice: 0
Thank you

FMC Standby

> show version
-------------------[ firepower ]--------------------
Model                     : Secure Firewall Management Center for VMware (66) Version 7.2.8 (Build 25)
UUID                      : bb333216-xxxx-xxxx-xxxx-c68c0c388b44
Rules update version      : 2023-11-29-001-vrt
LSP version               : lsp-rel-20231129-1200
VDB version               : 395
----------------------------------------------------
> expert
admin@firepower:~$ sudo su

root@firepower:/Volume/home/admin# ping xx.xx.18.101
PING xx.xx.18.101 (xx.xx.18.101) 56(84) bytes of data.
64 bytes from xx.xx.18.101: icmp_seq=1 ttl=64 time=0.402 ms
64 bytes from xx.xx.18.101: icmp_seq=2 ttl=64 time=0.482 ms
64 bytes from xx.xx.18.101: icmp_seq=3 ttl=64 time=0.452 ms
64 bytes from xx.xx.18.101: icmp_seq=4 ttl=64 time=0.490 ms
64 bytes from xx.xx.18.101: icmp_seq=5 ttl=64 time=0.519 ms
^C
--- xx.xx.18.101 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 123ms
rtt min/avg/max/mdev = 0.402/0.469/0.519/0.039 ms

root@firepower:/Volume/home/admin# netstat -an | grep 8305
tcp        0      0 xx.xx.18.102:8305       xx.xx.18.254:50373      ESTABLISHED
tcp        0      0 xx.xx.18.102:8305       xx.xx.18.253:42083      ESTABLISHED
tcp        0      0 xx.xx.18.102:59439      xx.xx.18.254:8305       ESTABLISHED
tcp        0      0 xx.xx.18.102:36751      xx.xx.18.253:8305       ESTABLISHED
tcp        0      0 xx.xx.18.102:38239      xx.xx.18.101:8305       ESTABLISHED ------ communication establish with FMC-Active
tcp        0      0 xx.xx.18.102:42253      xx.xx.18.101:8305       ESTABLISHED ------ communication establish with FMC-Active

root@firepower:/Volume/home/admin#

root@firepower:/Volume/home/admin# troubleshoot_HADC.pl
**************** Troubleshooting Utility **************
1 Show HA Info Of FMC
2 Execute Sybase DBPing
3 Show Arbiter Status
4 Check Peer Connectivity
5 Print Messages of AQ Task
6 Show FMC HA Operations History (ASC order)
7 Dump To File: FMC HA Operations History (ASC order)
8 Last Successful Periodic Sync Time (When it completed)
9 Print HA Status Messages
10 Compare active and standby device list
11 Check manager status of standby missing devices
12 Check critical PM processes details
13 Get Remote Stale Sync AQ Info
14 Help
0 Exit
**************************************************************
Enter choice: 1

HA Enabled: Yes
This FMC Role In HA: Standby - Secondary
Status out put: vmsDbEngine (system,gui) - Running 29652
In vmsDbEngineStatus(): vmsDbEngine process is running at /usr/local/sf/lib/perl/5.24.4/SF/Synchronize/HADC.pm line 3748.
Sybase Process: Running (vmsDbEngine, theSybase PM Process is Running)
Sybase Database Connectivity: Accepting DB Connections.
Sybase Database Name: csm_secondary
Sybase Role: Standby

**************** Troubleshooting Utility **************
1 Show HA Info Of FMC
2 Execute Sybase DBPing
3 Show Arbiter Status
4 Check Peer Connectivity
5 Print Messages of AQ Task
6 Show FMC HA Operations History (ASC order)
7 Dump To File: FMC HA Operations History (ASC order)
8 Last Successful Periodic Sync Time (When it completed)
9 Print HA Status Messages
10 Compare active and standby device list
11 Check manager status of standby missing devices
12 Check critical PM processes details
13 Get Remote Stale Sync AQ Info
14 Help
0 Exit
**************************************************************
Enter choice: 4
Peer UUID [Enter 'Return' For HA Peer(no UUID required)]:

Peer Is Connected
$VAR1 = {
          'vnet' => undef,
          'upgrade_version' => '',
          'uuid_gw' => '',
          'name' => 'xx.xx.18.101',
          'primary_mgr' => 1,
          'sw_version' => '7.2.8',
          'persistent' => 0,
          'model_number' => '66',
          'last_changed' => 1718045994,
          'reg_key' => '',
          'active' => 1,
          'reg_state' => 0,
          'model_id' => 'E',
          'vip' => '',
          'mgmt_mac_address' => '00:50:56:B3:E1:57',
          'vip_local' => '',
          'ip' => 'xx.xx.18.101',
          'priority' => 0,
          'uuid' => 'fc3e3572-xxxx-xxxx-xxxx-39e0098c166c',
          'role' => 0,
          'ipv6' => undef
        };

**************** Troubleshooting Utility **************
1 Show HA Info Of FMC
2 Execute Sybase DBPing
3 Show Arbiter Status
4 Check Peer Connectivity
5 Print Messages of AQ Task
6 Show FMC HA Operations History (ASC order)
7 Dump To File: FMC HA Operations History (ASC order)
8 Last Successful Periodic Sync Time (When it completed)
9 Print HA Status Messages
10 Compare active and standby device list
11 Check manager status of standby missing devices
12 Check critical PM processes details
13 Get Remote Stale Sync AQ Info
14 Help
0 Exit
**************************************************************
Enter choice: 8 

---------------------------- Last periodic sync time details ------------------------------------

Last successful sync completed at: Wed Oct 16 16:46:06 2024 UTC
Current time: Wed Oct 16 16:47:35 2024 UTC
Last successful sync completed '1 minute 29 seconds' ago.

---------------------------- Last periodic sync time details end ------------------------------------

**************** Troubleshooting Utility **************
1 Show HA Info Of FMC
2 Execute Sybase DBPing
3 Show Arbiter Status
4 Check Peer Connectivity
5 Print Messages of AQ Task
6 Show FMC HA Operations History (ASC order)
7 Dump To File: FMC HA Operations History (ASC order)
8 Last Successful Periodic Sync Time (When it completed)
9 Print HA Status Messages
10 Compare active and standby device list
11 Check manager status of standby missing devices
12 Check critical PM processes details
13 Get Remote Stale Sync AQ Info
14 Help
0 Exit
**************************************************************
Enter choice: 9

SYNC_ACTIVE: 1 at /usr/local/sf/lib/perl/5.24.4/SF/Synchronize.pm line 494, <STDIN> line 5.
Found running Synchronization task: Initializing at /usr/local/sf/lib/perl/5.24.4/SF/Transaction/HADC.pm line 241.
Sybase state :  at /usr/local/sf/lib/perl/5.24.4/SF/Synchronize.pm line 701.
Sybase state :  at /usr/local/sf/lib/perl/5.24.4/SF/Synchronize.pm line 801.
Sync status :Synchronization Task In-progress  at /usr/local/sf/lib/perl/5.24.4/SF/Synchronize.pm line 802.
Found running Synchronization task: Initializing at /usr/local/sf/lib/perl/5.24.4/SF/Transaction/HADC.pm line 241.

---------------------------- FMC HA status messages start ------------------------------------

Status: Synchronization Task In-progress

---------------------------- FMC HA status messages end ------------------------------------

**************** Troubleshooting Utility **************
1 Show HA Info Of FMC
2 Execute Sybase DBPing
3 Show Arbiter Status
4 Check Peer Connectivity
5 Print Messages of AQ Task
6 Show FMC HA Operations History (ASC order)
7 Dump To File: FMC HA Operations History (ASC order)
8 Last Successful Periodic Sync Time (When it completed)
9 Print HA Status Messages
10 Compare active and standby device list
11 Check manager status of standby missing devices
12 Check critical PM processes details
13 Get Remote Stale Sync AQ Info
14 Help
0 Exit
**************************************************************
Enter choice: 10

Fetching standby missing device information...
Devices are in sync.

**************** Troubleshooting Utility **************
1 Show HA Info Of FMC
2 Execute Sybase DBPing
3 Show Arbiter Status
4 Check Peer Connectivity
5 Print Messages of AQ Task
6 Show FMC HA Operations History (ASC order)
7 Dump To File: FMC HA Operations History (ASC order)
8 Last Successful Periodic Sync Time (When it completed)
9 Print HA Status Messages
10 Compare active and standby device list
11 Check manager status of standby missing devices
12 Check critical PM processes details
13 Get Remote Stale Sync AQ Info
14 Help
0 Exit
**************************************************************
Enter choice: 0
Thank you

Communication from FTD to FMC-HA validation

> show version
-------------------[ firepower ]--------------------
Model                     : Cisco Firepower Threat Defense for VMware (75) Version 7.2.4 (Build 165)
UUID                      : 7064913a-xxxx-xxxx-xxxx-803aefd05d2c
LSP version               : lsp-rel-20231129-1200
VDB version               : 395
----------------------------------------------------

> ping system xx.xx.18.101 ------ ping to FMC-Active
PING xx.xx.18.101 (xx.xx.18.101) 56(84) bytes of data.
64 bytes from xx.xx.18.101: icmp_seq=1 ttl=64 time=14.1 ms
64 bytes from xx.xx.18.101: icmp_seq=2 ttl=64 time=27.8 ms
64 bytes from xx.xx.18.101: icmp_seq=3 ttl=64 time=26.1 ms
64 bytes from xx.xx.18.101: icmp_seq=6 ttl=64 time=55.7 ms
64 bytes from xx.xx.18.101: icmp_seq=7 ttl=64 time=39.9 ms
64 bytes from xx.xx.18.101: icmp_seq=8 ttl=64 time=38.9 ms
^C
--- xx.xx.18.101 ping statistics ---
8 packets transmitted, 6 received, 25% packet loss, time 76ms
rtt min/avg/max/mdev = 14.081/33.733/55.658/13.069 ms

> ping system xx.xx.18.102 ------ ping to FMC-Active
PING xx.xx.18.102 (xx.xx.18.102) 56(84) bytes of data.
64 bytes from xx.xx.18.102: icmp_seq=1 ttl=64 time=23.9 ms
64 bytes from xx.xx.18.102: icmp_seq=2 ttl=64 time=23.10 ms
64 bytes from xx.xx.18.102: icmp_seq=3 ttl=64 time=0.425 ms
64 bytes from xx.xx.18.102: icmp_seq=4 ttl=64 time=6.88 ms
64 bytes from xx.xx.18.102: icmp_seq=5 ttl=64 time=10.5 ms
^C
--- xx.xx.18.102 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 70ms
rtt min/avg/max/mdev = 0.425/13.131/23.969/9.380 ms

> show managers
Type                      : Manager
Host                      : xx.xx.18.101
Display name              : xx.xx.18.101
Version                   : 7.2.8 (Build 25)
Identifier                : fc3e3572-xxxx-xxxx-xxxx-39e0098c166c
Registration              : Completed
Management type           : Configuration and analytics

Type                      : Manager
Host                      : xx.xx.18.102
Display name              : xx.xx.18.102
Version                   : 7.2.8 (Build 25)
Identifier                : bb333216-xxxx-xxxx-xxxx-c68c0c388b44
Registration              : Completed
Management type           : Configuration and analytics

> sftunnel-status
SFTUNNEL Start Time: Mon Oct 14 21:29:16 2024
        Both IPv4 and IPv6 connectivity is supported
        Broadcast count = 17
        Reserved SSL connections: 0
        Management Interfaces: 2
        eth0 (control events) xx.xx.18.254,
        tap_nlp (control events) 169.254.1.2,fd00:0:0:1::2
***********************

**RUN STATUS****xx.xx.18.102************* --------------------------------------- sftunnel information of the FMC-Standby
        Key File    = /var/sf/peers/bb333216-xxxx-xxxx-xxxx-c68c0c388b44/sftunnel-key.pem
        Cert File   = /var/sf/peers/bb333216-xxxx-xxxx-xxxx-c68c0c388b44/sftunnel-cert.pem
        CA Cert     = /var/sf/peers/bb333216-xxxx-xxxx-xxxx-c68c0c388b44/cacert.pem
        Cipher used = TLS_AES_256_GCM_SHA384 (strength:256 bits)
        ChannelA Connected: Yes, Interface eth0
        Cipher used = TLS_AES_256_GCM_SHA384 (strength:256 bits)
        ChannelB Connected: Yes, Interface eth0
        Registration: Completed.
        IPv4 Connection to peer 'xx.xx.18.102' Start Time: Wed Oct 16 15:06:23 2024 UTC

        IPv4 Last outbound connection to peer 'xx.xx.18.102' via Primary ip/host 'xx.xx.18.102'

PEER INFO:
        sw_version 7.2.8
        sw_build 25
        Using light registration
        Management Interfaces: 1
        eth0 (control events) xx.xx.18.102,
        Peer channel Channel-A is valid  type (CONTROL), using 'eth0', connected to 'xx.xx.18.102' via 'xx.xx.18.254'
        Peer channel Channel-B is valid  type (EVENT), using 'eth0', connected to 'xx.xx.18.102' via 'xx.xx.18.254'

***********************

**RUN STATUS****xx.xx.18.101************* -------------------------------------- sftunnel information of the FMC-Active
        Key File    = /var/sf/peers/fc3e3572-xxxx-xxxx-xxxx-39e0098c166c/sftunnel-key.pem
        Cert File   = /var/sf/peers/fc3e3572-xxxx-xxxx-xxxx-39e0098c166c/sftunnel-cert.pem
        CA Cert     = /var/sf/peers/fc3e3572-xxxx-xxxx-xxxx-39e0098c166c/cacert.pem
        Cipher used = TLS_AES_256_GCM_SHA384 (strength:256 bits)
        ChannelA Connected: Yes, Interface eth0
        Cipher used = TLS_AES_256_GCM_SHA384 (strength:256 bits)
        ChannelB Connected: Yes, Interface eth0
        Registration: Completed.
        IPv4 Connection to peer 'xx.xx.18.101' Start Time: Mon Oct 14 21:29:15 2024 UTC

PEER INFO:
        sw_version 7.2.8
        sw_build 25
        Using light registration
        Management Interfaces: 1
        eth0 (control events) xx.xx.18.101,
        Peer channel Channel-A is valid  type (CONTROL), using 'eth0', connected to 'xx.xx.18.101' via 'xx.xx.18.254'
        Peer channel Channel-B is valid  type (EVENT), using 'eth0', connected to 'xx.xx.18.101' via 'xx.xx.18.254'

***********************

**RPC STATUS****xx.xx.18.101*************
  'name' => 'xx.xx.18.101',
  'last_changed' => 'Mon Jun 10 18:59:54 2024',
  'uuid_gw' => '',
  'ip' => 'xx.xx.18.101',
  'ipv6' => 'IPv6 is not configured for management',
  'active' => 1,
  'uuid' => 'fc3e3572-xxxx-xxxx-xxxx-39e0098c166c'

**RPC STATUS****xx.xx.18.102*************
  'name' => 'xx.xx.18.102',
  'last_changed' => 'Wed Oct  9 07:00:11 2024',
  'uuid_gw' => '',
  'ip' => 'xx.xx.18.102',
  'ipv6' => 'IPv6 is not configured for management',
  'active' => 1,
  'uuid' => 'bb333216-xxxx-xxxx-xxxx-c68c0c388b44'

Check routes:
No peers to check

Note: If the sftunnel information of one of the FMC is missing, this can indicate that the communication with the manager is compromised

FTD root level troubleshoot

> expert
admin@firepower:~$ sudo su

root@firepower:/home/admin# netstat -an | grep 8305
tcp        0      0 xx.xx.18.254:8305       xx.xx.18.102:59439      ESTABLISHED ------ communication establish with FMC-Standby
tcp        0      0 xx.xx.18.254:49205      xx.xx.18.101:8305       ESTABLISHED ------ communication establish with FMC-Active
tcp        0      0 xx.xx.18.254:50373      xx.xx.18.102:8305       ESTABLISHED ------ communication establish with FMC-Standby
tcp        0      0 xx.xx.18.254:53875      xx.xx.18.101:8305       ESTABLISHED ------ communication establish with FMC-Active

root@firepower:/home/admin# cat /ngfw/var/log/action_queue.log | less
Oct 16 15:06:50 firepower ActionQueueScrape.pl[4166]: Waiting for light registration to complete on device:  try 10 times... at /ngfw/usr/local/sf/lib/p
erl/5.24.4/SF/PeerManager/RegistrationCL.pm line 1805.
Oct 16 15:06:50 firepower ActionQueueScrape.pl[4166]: Found Registered peer with name xx.xx.18.102 (bb333216-xxxx-xxxx-xxxx-c68c0c388b44) at /ngfw/usr/local/sf/lib/perl/5.24.4/SF/PeerManager/RegistrationCL.pm line 1821.
Oct 16 15:06:50 firepower ActionQueueScrape.pl[4166]: Found peer with name xx.xx.18.102 - update DB at /ngfw/usr/local/sf/lib/perl/5.24.4/SF/PeerManager/RegistrationCL.pm line 1827.
Oct 16 15:06:50 firepower ActionQueueScrape.pl[4166]: Found Registered peer with name xx.xx.18.101 (fc3e3572-xxxx-xxxx-xxxx-39e0098c166c) at /ngfw/usr/local/sf/lib/perl/5.24.4/SF/PeerManager/RegistrationCL.pm line 1821.

Revision History

Revision	Publish Date	Comments
1.0	23-Oct-2024	Initial Release

Contributed by Cisco Engineers

Paulina Bibiana Paydon
Technical Consulting Engineer

Was this Document Helpful?

Feedback

Contact Cisco

Open a Support Case
(Requires a Cisco Service Contract)

This Document Applies to These Products

Secure Firewall Management Center

Troubleshoot for FMC - HA

Available Languages

Download Options

Bias-Free Language

Contents

Introduction

Prerequisites

Requirements

Components Used

Background information

Before you Begin

Troubleshoot commands

FMC troubleshooting commands.

FTD troubleshooting commands

Verification

FMC - HA validation

Communication from FTD to FMC-HA validation

Revision History

Contributed by Cisco Engineers

Was this Document Helpful?

Contact Cisco

This Document Applies to These Products