Disable Service Module Monitoring on ASA to Avoid Unwanted Failover Events (SFR/CX/IPS/CSC).

Introduction

This document describes how to disable monitoring on modules SourceFire (SFR), Context Aware (CX), Intrusion Prevention System (IPS), Content Security and Control (CSC) on an Adaptive Security Appliance (ASA) failover environment. 

Contributed by Cesar Lopez, Cisco TAC Engineer.

Prerequisites

Requirements

Cisco recommends that you have knowledge of the following topics:

• Configuration of Adaptive Security Appliance.
• Knowledge of ASA Failover for High Availability.

From version 9.3(1), this feature is configurable. Before the mentioned version, the module will always be monitored. A workaround can be used for previous versions described in this document.

Components Used

This document is based on these software and hardware versions:

• Cisco ASA version 9.3(1) and later.
• ASA 5500-X series with FirePOWER services, ASA CX Context-Aware Security or IPS module.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command

Background Information

By default, the ASA monitors an installed service module. If a failure is detected in the active unit module, the appliance failover is triggered. 

It can be helpful to disable this monitor when there is a scheduled service module reload or continuous module failures of the same without willing to have an ASA failover event.

Note: The ASA needs to be diverting traffic to the module in order to be monitored by the failover process.

Configure

Network Diagram

This document uses this setup:

Configurations

This configuration is used in lab devices to demonstrate the monitor feature mentioned in this document. Only the relevant configuration is included. Some of the lines of this output are omitted. 

ASA Version 9.3(3)
!
hostname ASA-FPWR
!
interface GigabitEthernet0/0
 nameif outside
 security-level 0
 ip address 10.88.247.5 255.255.255.224 standby 10.88.247.6
!
interface GigabitEthernet0/1
 nameif inside
 security-level 100
 ip address 192.168.10.111 255.255.255.0 standby 192.168.10.112
!
...
!
interface GigabitEthernet0/6
 description LAN Failover Interface
!
interface GigabitEthernet0/7
 description STATE Failover Interface
!
...

failover
failover lan unit primary
failover lan interface folink GigabitEthernet0/6
failover link statelink GigabitEthernet0/7
failover interface ip folink 1.1.1.1 255.255.255.0 standby 1.1.1.2
failover interface ip statelink 2.2.2.1 255.255.255.0 standby 2.2.2.2
!

...

!
class-map SFR
 match any
class-map inspection_default
 match default-inspection-traffic
!
!
policy-map type inspect dns migrated_dns_map_1
 parameters
 message-length maximum client auto
 message-length maximum 512
policy-map global_policy
 class inspection_default
 inspect dns migrated_dns_map_1
 inspect ftp
 inspect h323 h225
 inspect h323 ras
 inspect ip-options
 inspect netbios
 inspect rsh
 inspect rtsp
 inspect skinny
 inspect esmtp
 inspect sqlnet
 inspect sunrpc
 inspect tftp
 inspect sip
 inspect xdmcp
 class SFR
 sfr fail-open
!
service-policy global_policy global
prompt hostname context priority state
no call-home reporting anonymous
Cryptochecksum:b268e0095f175a26aa94d120e9041c29
: end

Check the current monitored components.

When the ASAs are in failover mode, the service module installed is monitored by default, just as the appliance interfaces. This command can be used, in order to see which current components are monitored:

ASA-FPWR/pri/act# show run all monitor-interface
monitor-interface outside
monitor-interface inside
monitor-interface service-module

Check the ASA units service module status. 

The show failover output shows the current status of each unit module:

ASA-FPWR/pri/act# show failover
Failover On
Failover unit Primary
Failover LAN Interface: folink GigabitEthernet0/6 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 2 of 316 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.3(3), Mate 9.3(3)
Last Failover at: 14:30:44 UTC Aug 6 2015
 This host: Primary - Active
 Active time: 85 (sec)
 slot 0: ASA5545 hw/sw rev (1.0/9.3(3)) status (Up Sys)
 Interface outside (10.88.247.5): Normal (Monitored)
 Interface inside (192.168.10.111): Normal (Monitored)
 slot 1: SFR5545 hw/sw rev (N/A/5.3.1-152) status (Up/Up)
 ASA FirePOWER, 5.3.1-152, Up
 Other host: Secondary - Standby Ready
 Active time: 396 (sec)
 slot 0: ASA5545 hw/sw rev (1.0/9.3(3)) status (Up Sys)
 Interface outside (10.88.247.6): Normal (Monitored)
 Interface inside (192.168.10.112): Normal (Monitored)
 slot 1: SFR5545 hw/sw rev (N/A/5.3.1-155) status (Up/Up)
 ASA FirePOWER, 5.3.1-155, Up

If the service module of an active unit goes down, a failover event occurs. The active unit becomes standby, and the previous standby unit takes the active role. In some scenarios, this causes some features that are not supported by a stateful failover, to reconverge. 

Verify the service module fail mode policy:

If a fail-openpolicy is used to send traffic to the module, traffic continues going through the ASA without being sent to the service module. This can be a more transparent way to overcome an expected module down status. 

Warning: If a fail-close policy has been applied, then, all traffic matching the class-map used to divert traffic to the module is dropped by the ASA. 

In order to know the policy status used, run the command show service-policy [sfr|cx|ips|csc] .

ASA-FPWR/pri/act# show service-policy sfr

Global policy:
 Service-policy: global_policy
 Class-map: SFR
 SFR: card status Up, mode fail-open
 packet input 0, packet output 0, drop 0, reset-drop 0

 The same can be seen by checking the Modular Policy Framework (MPF) configuration:

ASA-FPWR/pri/act# show run policy-map
!
policy-map type inspect dns migrated_dns_map_1
 parameters
 message-length maximum client auto
 message-length maximum 512
policy-map global_policy
 class inspection_default
 inspect dns migrated_dns_map_1
 inspect ftp
 inspect h323 h225
 inspect h323 ras
 inspect ip-options
 inspect netbios
 inspect rsh
 inspect rtsp
 inspect skinny
 inspect esmtp
 inspect sqlnet
 inspect sunrpc
 inspect tftp
 inspect sip
 inspect xdmcp
 class SFR
 sfr fail-open
!
ASA-FPWR/pri/act#

Disable service module monitoring.

This command, makes the failover process stop the monitoring of  the service module. Any planned reload or troubleshoot can be done to the module without a failover, in case of the module going "Down" or "Unresponsive".

no monitor-interface service-module

Verify

Verify that the service module monitoring is disabled.

Under the running configuration, the monitor-interface command is negated.  

ASA-FPWR/pri/act(config)# show run all monitor-interface
monitor-interface outside
monitor-interface inside
no monitor-interface service-module

To test reload the module hosted by the active unit.

For demonstration purposes, the FirePOWER module on this unit is reloaded to confirm if the Active failover unit stays on this role.

Output from the FirePOWER module in  ASA Primary/Active unit.

Sourcefire ASA5545 v5.3.1 (build 152)

Last login: Thu Aug 6 14:40:46 on ttyS1
>
> system reboot
This command will reboot the system. Continue?
Please enter 'YES' or 'NO': YES

Broadcast message from root (Thu Aug 6 14:40:59 2015):

The system is going down for reboot NOW!

Escape Sequence detected
Console session with module sfr terminated.

Output from the ASA Primary/Active unit while the module reloads. 

The unit stays on the Active role. 

ASA-FPWR/pri/act# show failover
Failover On
Failover unit Primary
Failover LAN Interface: folink GigabitEthernet0/6 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 2 of 316 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.3(3), Mate 9.3(3)
Last Failover at: 14:30:44 UTC Aug 6 2015
 This host: Primary - Active
 Active time: 616 (sec)
 slot 0: ASA5545 hw/sw rev (1.0/9.3(3)) status (Up Sys)
 Interface outside (10.88.247.5): Normal (Monitored)
 Interface inside (192.168.10.111): Normal (Monitored)
 slot 1: SFR5545 hw/sw rev (N/A/5.3.1-152) status (Unresponsive/Down)
 ASA FirePOWER, 5.3.1-152, Not Applicable
 Other host: Secondary - Standby Ready
 Active time: 396 (sec)
 slot 0: ASA5545 hw/sw rev (1.0/9.3(3)) status (Up Sys)
 Interface outside (10.88.247.6): Normal (Monitored)
 Interface inside (192.168.10.112): Normal (Monitored)
 slot 1: SFR5545 hw/sw rev (N/A/5.3.1-155) status (Up/Up)
 ASA FirePOWER, 5.3.1-155, Up

Output from the ASA Secondary/Standby unit while the module reloads:

The standby unit doesn't detect this status as a failure and doen't take the active role.

ASA-FPWR/sec/stby# show failover
Failover On
Failover unit Secondary
Failover LAN Interface: folink GigabitEthernet0/6 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 2 of 316 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.3(3), Mate 9.3(3)
Last Failover at: 14:30:59 UTC Aug 6 2015
 This host: Secondary - Standby Ready
 Active time: 396 (sec)
 slot 0: ASA5545 hw/sw rev (1.0/9.3(3)) status (Up Sys)
 Interface outside (10.88.247.6): Normal (Monitored)
 Interface inside (192.168.10.112): Normal (Monitored)
 slot 1: SFR5545 hw/sw rev (N/A/5.3.1-155) status (Up/Up)
 ASA FirePOWER, 5.3.1-155, Up
 Other host: Primary - Active
 Active time: 670 (sec)
 slot 0: ASA5545 hw/sw rev (1.0/9.3(3)) status (Up Sys)
 Interface outside (10.88.247.5): Normal (Monitored)
 Interface inside (192.168.10.111): Normal (Monitored)
 slot 1: SFR5545 hw/sw rev (N/A/5.3.1-152) status (Unresponsive/Down)
 ASA FirePOWER, 5.3.1-152, Not Applicable

Enable service module monitoring.

To enable module monitoring, run this command:

monitor-interface service-module

Verify that the service module is enabled.

Service module command is not negated anymore.  

ASA-FPWR/pri/act(config)# show run all monitor-interface
monitor-interface outside
monitor-interface inside
monitor-interface service-module

Troubleshoot

Issue 1. ASAs keep failing over,  and  this message "Service card in other unit has failed"  is showed.

If one or many failover events are detected, the show failover history can be used to know the possible reason. 

ASA-FPWR/sec/act# show failover history
==========================================================================
From State To State Reason
==========================================================================
14:38:58 UTC Aug 5 2015
Bulk Sync Standby Ready Detected an Active mate

14:39:05 UTC Aug 5 2015
Standby Ready Bulk Sync No Error

14:39:17 UTC Aug 5 2015
Bulk Sync Standby Ready No Error

14:48:12 UTC Aug 6 2015
Standby Ready Just Active Service card in other unit has failed

14:48:12 UTC Aug 6 2015
Just Active Active Drain Service card in other unit has failed

14:48:12 UTC Aug 6 2015
Active Drain Active Applying Config Service card in other unit has failed

14:48:12 UTC Aug 6 2015
Active Applying Config Active Config Applied Service card in other unit has failed

14:48:12 UTC Aug 6 2015
Active Config Applied Active Service card in other unit has failed

The now standby unit shows this message:

14:47:56 UTC Aug 6 2015
Standby Ready Failed Detect service card failure

If  "Service card in other unit has failed" message is seen, the failover happened because the active unit detected its own module as unresponsive. 

If the module stays in "Unresponsive" status, the affected ASA stays in Failed mode.

ASA-FPWR/sec/stby# Waiting for the earlier webvpn instance to terminate...
Previous instance shut down. Starting a new one.

 Switching to Active

ASA-FPWR/sec/act#
ASA-FPWR/sec/act# show failover
Failover On
Failover unit Secondary
Failover LAN Interface: folink GigabitEthernet0/6 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 2 of 316 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.3(3), Mate 9.3(3)
Last Failover at: 14:24:23 UTC Aug 6 2015
 This host: Secondary - Active
 Active time: 38 (sec)
 slot 0: ASA5545 hw/sw rev (1.0/9.3(3)) status (Up Sys)
 Interface outside (10.88.247.5): Normal (Waiting)
 Interface inside (192.168.10.111): Normal (Waiting)
 slot 1: SFR5545 hw/sw rev (N/A/5.3.1-155) status (Up/Up)
 ASA FirePOWER, 5.3.1-155, Up
 Other host: Primary - Failed
 Active time: 182 (sec)
 slot 0: ASA5545 hw/sw rev (1.0/9.3(3)) status (Up Sys)
 Interface outside (10.88.247.6): Normal (Waiting)
 Interface inside (192.168.10.112): Normal (Waiting)
 slot 1: SFR5545 hw/sw rev (N/A/5.3.1-152) status (Unresponsive/Down)
 ASA FirePOWER, 5.3.1-152, Not Applicable

Solution

Service module monitoring can be disabled while further steps to  troubleshoot the issue can be done in order to recover the module.

no monitor-interface service-module

Issue 2. My ASA doesn't support 9.3(1) or I cannot upgrade it. How can I avoid failover events? 

Legacy ASA5500 series don't support 9.3(1) version and, even if they don't support software modules, some of them have hardware modules such as CSC or the IPS. 

Even with the new ASA5500-X series, there are some appliances with versions below the one that supports disable monitoring. 

Solution

The ASA only monitors the module if there is a policy configured to pass traffic to it.So, in order to avoid a failover, the module policy can be removed.

Identify the class map and policy used.

In this case, this configuration is used to remove traffic diversion of a FirePOWER module.

class-map SFR
 match any
class-map inspection_default
 match default-inspection-traffic
!
!
policy-map type inspect dns migrated_dns_map_1
 parameters
 message-length maximum client auto
 message-length maximum 512
policy-map global_policy
 class inspection_default
 inspect dns migrated_dns_map_1
 inspect ftp
 inspect h323 h225
 inspect h323 ras
 inspect ip-options
 inspect netbios
 inspect rsh
 inspect rtsp
 inspect skinny
 inspect esmtp
 inspect sqlnet
 inspect sunrpc
 inspect tftp
 inspect sip
 inspect xdmcp
 class SFR
 sfr fail-open
!

 The command show service-policy [csc|cxsc|ips|sfr] can be used to detect the class map and current status.

ASA-FPWR/pri/act# show service-policy sfr

Global policy:
 Service-policy: global_policy
 Class-map: SFR
 SFR: card status Up, mode fail-open
 packet input 0, packet output 0, drop 0, reset-drop 

Disable traffic redirection to the module.

After the policy is removed, no further traffic is sent from the ASA to the module. 

ASA-FPWR/pri/act# conf t
ASA-FPWR/pri/act(config)# policy-map global_policy
ASA-FPWR/pri/act(config-pmap)# class SFR
ASA-FPWR/pri/act(config-pmap-c)# no sfr fail-open
ASA-FPWR/pri/act(config-pmap-c)# end
ASA-FPWR/pri/act#

Verify that the ASA redirection to the module is  disabled.

 The same show command can be used to verify that the traffic is no longer going to the module. The output must be empty.

 ASA-FPWR/pri/act# show service-policy sfr
ASA-FPWR/pri/act#

 Even if the module is unresponsive, the active unit remains in the same role. 

ASA-FPWR/pri/act# show module sfr

Mod Card Type Model Serial No.
---- -------------------------------------------- ------------------ -----------
 sfr FirePOWER Services Software Module ASA5545 FCH18457CNM

Mod MAC Address Range Hw Version Fw Version Sw Version
---- --------------------------------- ------------ ------------ ---------------
 sfr 74a0.2fa4.6c7a to 74a0.2fa4.6c7a N/A N/A 5.3.1-152

Mod SSM Application Name Status SSM Application Version
---- ------------------------------ ---------------- --------------------------
 sfr ASA FirePOWER Not Applicable 5.3.1-152

Mod Status Data Plane Status Compatibility
---- ------------------ --------------------- -------------
 sfr Unresponsive Not Applicable

ASA-FPWR/pri/act# show failover
Failover On
Failover unit Primary
Failover LAN Interface: folink GigabitEthernet0/6 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 2 of 316 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.3(3), Mate 9.3(3)
Last Failover at: 14:51:20 UTC Aug 6 2015
 This host: Primary - Active
 Active time: 428 (sec)
 slot 0: ASA5545 hw/sw rev (1.0/9.3(3)) status (Up Sys)
 Interface outside (10.88.247.5): Normal (Monitored)
 Interface inside (192.168.10.111): Normal (Monitored)
 slot 1: SFR5545 hw/sw rev (N/A/5.3.1-152) status (Unresponsive/Down)
 ASA FirePOWER, 5.3.1-152, Not Applicable
 Other host: Secondary - Standby Ready
 Active time: 204 (sec)
 slot 0: ASA5545 hw/sw rev (1.0/9.3(3)) status (Up Sys)
 Interface outside (10.88.247.6): Normal (Monitored)
 Interface inside (192.168.10.112): Normal (Monitored)
 slot 1: SFR5545 hw/sw rev (N/A/5.3.1-155) status (Up/Up)
 ASA FirePOWER, 5.3.1-155, Up

Enable traffic redirect to the module.

Once the traffic needs to be sent back to the module, the fail-open or fail-close policy can be added back. 

ASA-FPWR/pri/act(config)# policy-map global_policy
ASA-FPWR/pri/act(config-pmap)# class SFR
ASA-FPWR/pri/act(config-pmap-c)# sfr fail-open
ASA-FPWR/pri/act(config-pmap-c)# end
ASA-FPWR/pri/act#