Configure Catalyst 9800 WLC in HA SSO in Cisco IOS XE 16.12

Introduction

This document describes how to configure Catalyst 9800 Wireless Controllers in High Availability (HA) SSO in Cisco IOS® XE 16.10 to 16.12.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Catalyst 9800 Wireless LAN Controller (WLC)
• Stateful Switch Over (SSO)
• High Availability (HA)
• Redundancy Port (RP)

Components Used

The information in this document is based on these software and hardware versions:

• Virtual 9800-CL WLC  v16.10 to 16.12
• 9800-40 WLC  v16.10 to 16.12

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background information

In Cisco IOS XE 17.x and later for the Catalyst 9800. This document article focuses on HA SSO in versions 16.x.

The concept of a redundancy management interface adds a number of differences that are covered by the dedicated guide, High Availability SSO Deployment Guide for Cisco Catalyst 9800 Series Wireless Controllers, Cisco IOS XE Amsterdam 17. 

The redundancy explained in this article is 1:1, which means that one of the units operates in Active state while the other operates in Hot Standby state. If the active box is detected as unreachable, the Hot Standby unit becomes active and all APs and clients maintain their service through the new active unit.

Once both units are synchronized, the standby 9800 WLC mimics its configuration with the primary unit. Any configuration change performed on the active unit is replicated on the standby unit via the Redundancy Port (RP).

Configuration changes are no longer allowed to be performed on the standby 9800 WLC.

Besides the synchronization of the configuration between boxes, they also synchronize:

• APs in UP state (not APs in download state or APs in DTLS handshake)
• Clients in RUN state (If there is a client in Web Authentication required state and a switchover occurs, that client has to restart its association process),
• RRM configuration 
• (other settings)

Restrictions

Before you enable HA between two 9800 WLCs, perform these validations:

• Both devices must be of same PID. In case of 9800-CL, ensure the host environment (ESXi or KVM or ENCS) is the same for both instances.
• Both devices must run the same version of software.
• Both devices must run in the same Installation Mode (Either Bundle or Install). It is recommended that you run  Install mode for WLC. 
• Both devices must have redundant IPs in the same subnet. IP addresses used for redundancy must be unroutable without a gateway present in the subnet.
• Both devices must have a unique wireless management interface.
• Wireless management interface of both devices must belong to the same VLAN/subnet. 
• In the case of 9800-CL,
  • Verify same CPU, memory, and storage resources are allocated to both instances.
  • Verify VM snapshot is disabled for both instances.
  • Both devices must use the same interface number (For example: GigabitEthernet3) for HA

Configure HA on Hardware-based  9800 WLC

Network Diagram

This document is based on this topology:

Configurations

Redundancy SSO is enabled by default, however, communication configuration between the units is still required.

Before you configure any step, ensure both boxes run the same version.

Step 1. Connect 9800 WLCs to the network and ensure they are reachable by each other.

Wireless management interface from both boxes must belong to the same VLAN and subnet.

Step 2. Connect RP as shown in the network topology. There are two options to connect 9800 WLCs RP:

• RP - RJ45 10/100/1000 Redundancy Ethernet Port
• RP - 10-GE SFP port

Note: In Cisco IOS XE 16.10 to 16.12, it is recommended to connect the 9800s Redundancy Port to the same switch where the management connection of the 9800 is connected (see network topology). This is because there is no verification for the gateway reachability in these versions. A back-to-back connection works but cause higher chances of 9800 controllers to become active-active when connected that way. Note that the RP port does not use vlan tagging.

 Step 3. Assign the redundancy IP address to both 9800 WLCs

GUI:

Navigate to Administration > Device > Redundancy. Uncheck Clear Redundancy Config and enter the desired IP address.

Both units are to have unique IP addresses and both are to belong to the same subnet.

That subnet is not be routable anywhere in the network. 

9800 WLC-1

9800 WLC-2

 CLI:

16.10

9800 WLC-1# chassis ha-interface local-ip <!--IP address-->  remote-ip <!--IP address-->

9800 WLC-2# chassis ha-interface local-ip <!--IP address-->  remote-ip <!--IP address-->    

16.11

9800 WLC-1# chassis redundancy ha-interface local-ip <!--IP address--> remote-ip <!--IP address-->

9800 WLC-2# chassis redundancy ha-interface local-ip <!--IP address--> remote-ip <!--IP address-->

To specify which unit must be the active 9800 WLC, set the Chassis priority by either GUI or CLI. The unit with the higher priority is selected as primary.

GUI:

CLI:

16.10
# chassis 1 priority <1-15>
16.11
# chassis 1 priority <1-2>

 If a specific unit is not chosen to be active, the units elect Active based on lowest MAC address

Verify the current configuration with this command:

# show chassis ha-status local 

                      My state = ACTIVE
                    Peer state = DISABLED
        Last switchover reason = none
          Last switchover time = none
                 Image Version = ...

Chassis-HA   Local-IP         Remote-IP        MASK             HA-Interface
-----------------------------------------------------------------------------
This Boot:                                                                      

Next Boot: 203.0.113.2    <!--IP address-->   <!--IP address-->                    


Chassis-HA   Chassis#    Priority        IFMac Address         Peer Timeout
----------------------------------------------------------------------------
This Boot:    1                                                    0

Next Boot:    1                                                    0

Step 4. Save configurations on both 9800 WLCs

GUI:

CLI:

# write

Step 5. Restaart both unit at the same time

GUI:

Navigate to Administration > Management > Backup & Restore > Reload

CLI:

# reload

Configure HA on Virtual 9800 WLC

Network Diagram

This document is based on this topology:

Configurations

Set Up the Virtual Redundancy Network

Step 1. Open your vCenter client and navigate to Host > Configuration > Networking > Add Networking.

 Step 2. Select Virtual Machine and click Next.

Step 3. Select Create a vSphere standard switch and click Next.

Step 4. Optionally, customize the Network Label parameter. After that click Next.

 Step 5. Finish the wizard.

 Step 6. Link an interface from both virtual 9800 WLCs (one of every virtual 9800 WLC) to the Redundancy Network.

Right click the virtual 9800 WLC and click Edit Settings...

Select one of the available network adapters and assign it to the Redundancy Network, and click OK.

Do the same for both machines.

Set HA Configuration

Before a further configuration is performed, ensure that the wireless management interface from both units belong to the same VLAN and subnet and are reachable with each other.

Verify that both boxes run the same version.

Step 1. Assign the redundancy IP address to both 9800 WLCs

GUI:

Navigate to Administration > Device > Redundancy. Uncheck Clear Redundancy Config and enter the desired IP address.

Verify that both have have unique IP address and both units belong to the same subnet.

9800 WLC-1

9800 WLC-2

Note: Notice that GigabitEthernet3 was selected for HA interface. It is because the third interface of the virtual machine is the one associated to the Redundancy Network. This interface is used to enable communication between the two boxes before Cisco IOS boots up, transport of HA control messages (such as role selection, keep-alives, and so on) and provide the transport for Interprocess Communication (IPC) between the two boxes.

16.10 CLI:

9800 WLC-1# chassis ha-interface gigabitEthernet 3 local-ip <!--IP address--> remote-ip <!--IP address-->

9800 WLC-2# chassis ha-interface gigabitEthernet 3 local-ip <!--IP address--> remote-ip <!--IP address-->

16.12 CLI:

9800WLC1# chassis redundancy ha-interface g3 local-ip <!--IP address--> remote-ip <!--IP address-->
9800WLC2# chassis redundancy ha-interface g3 local-ip <!--IP address--> remote-ip <!--IP address-->

Note: Once the GigabitEthernet 3 interface is selected to be used as HA and the unit is restarted (even if HA is not built between two 9800 WLCs), you no longer see this interface listed in the output of show ip interface brief or any other command that shows the 9800 WLC's interfaces, this is because that interface is now marked for HA only.

Step 2. (Optional) To manually specify which box must be the active 9800 WLC, set the Active Chassis Priorityeither by GUI or CLI.

The chassis with the higher priority is selected as primary.

GUI:

CLI:

# chassis 1 priority <1-15>

 If a specific unit to be active is not specified, the boxes choose which one is the primary active 9800 WLC.

Step 3. Save configurations on both 9800 WLCs

GUI:

CLI:

# write

Step 4. Restart both boxes at the same time.

GUI:

Navigate to Administration > Management > Backup & Restore > Reload

CLI:

# reload

Enable Console Access to Standby 9800 WLC

Once HA is enabled and one of the boxes is assigned as active and the other one as standby hot, by default it is not allowed to reach exec mode (enable) on the standby box.

To enable it, log in via the SSH/console to the active 9800 WLC and enter these commands: 

# config t
# redundancy
# main-cpu
# standby console enable
# end

Force Switchover

To force a switchover between boxes, either manually reboot the active 9800 WLC or run this command:

# redundancy force-switchover

Break HA

Clear HA Configuration in both 9800 WLC

In order to break HA on both boxes you can perform these steps.

Step 1. Clear HA configuration of the current active 9800 WLC and force a redundancy switchover (It reboots the current active 9800 WLC, it boots back up with HA configuration cleared) 

16.10:
Active-9800 WLC# chassis clear
WARNING: Clearing the chassis HA configuration results in the chassis coming up in Stand Alone mode after reboot.The HA configuration remains the same on other chassis. Do you wish to continue? [y/n]? [yes]: y

Active-9800 WLC# redundancy force-switchover

16.11
Active-9800 WLC# clear chassis redundancy

Step 2. Once the standby 9800 WLC becomes active, log in and clear the redundancy configuration.

new-Acitve-9800 WLC# chassis clear
WARNING: Clearing the chassis HA configuration will result in the chassis coming up in Stand Alone mode after reboot.The HA configuration will remain the same on other chassis. Do you wish to continue? [y/n]? [yes]: y

Step 3. Update the IP address of the new active 9800 WLC. Optionally, update its hostname.

new-Acitve-9800 WLC# config t
new-Acitve-9800 WLC# hostname <new-hostname>
new-Acitve-9800 WLC# interface <wireless-mgmt-int-id>
new-Acitve-9800 WLC# ip address <a.b.c.d> <a.b.c.d>
new-Acitve-9800 WLC# exit

Step 4. Save the configuration and reload the new Active 9800 WLC

new-Acitve-9800 WLC# write
new-Acitve-9800 WLC# reload

After that, the second box restarts and comes back with new IP address configuration (to avoid IP address duplication with the former HA 9800 WLC) and with HA configuration cleared. The original active 9800 WLC keeps its original IP address.

Peer Timeout Configuration

Active and standby chassis send keep-alive messages to each other to ensure that both are still available.

Peer timeout is used to determine if peer chassis is lost if it does not receive any keep-alive message from the peer chassis in the configured peer timeout.

Default timeout is 500ms but is configurable through CLI. The configured timeout value syncs to standby 9800 WLC.

Use this command to customize this timer:

# chassis timer peer-timeout <500-16000 msec>

Use this command to clear the configured timer (if needed):

# chassis timer peer-timeout default

Upgrade

Standard (not AP or ISSU) upgrade can be done from the web UI. Once the WLCs are in a HA pair, they run the same version in the same (preferably INSTALL) mode.

The web UI upgrade page takes care of the software distribution to both controllers in the pair and installs and reboots both units at the same time.

This causes identical downtimes on both units. For other techniques which cause less downtime, please consult Patching and Installation Guide .

Verify

After both 9800 WLC units have rebooted and are synced to each other, you can access them via console and verity their current state with these commands:

9800 WLC-1# show chassis 

Chassis/Stack Mac Address : <!--IP address--> - Local Mac Address
Mac persistency wait time: Indefinite
Local Redundancy Port Type: Twisted Pair
 H/W Current
Chassis# Role Mac Address Priority Version State IP
-------------------------------------------------------------------------------------
*1 Active 00a3.8e23.a240 1 V02 Ready <!--IP address-->
 2 Standby 00a3.8e23.a280 1 V02 Ready <!--IP address-->

9800 WLC-1-stby# show chassis 

Chassis/Stack Mac Address : <!--IP address--> - Local Mac Address
Mac persistency wait time: Indefinite
Local Redundancy Port Type: Twisted Pair
                                             H/W   Current
Chassis#   Role    Mac Address     Priority Version  State                 IP
-------------------------------------------------------------------------------------
 1       Active   <!--address-->     1      V02     Ready           <!--IP address-->
*2       Standby  <!--address-->     1      V02     Ready           <!--IP address-->

Note: Standby shows IPs too, track Cisco bug ID CSCvm64484 for the fix.

The * (asterisk) points the chassis from which you run the command.

9800 WLC-1# show redundancy 

Redundant System Information :
------------------------------
       Available system uptime = 1 hour, 35 minutes
Switchovers system experienced = 0
              Standby failures = 0
        Last switchover reason = none

                 Hardware Mode = Duplex
    Configured Redundancy Mode = sso
     Operating Redundancy Mode = sso
              Maintenance Mode = Disabled
                Communications = Up

Current Processor Information :
-------------------------------
               Active Location = slot 1
        Current Software state = ACTIVE
       Uptime in current state = 1 hour, 35 minutes
                 Image Version = Cisco IOS Software [Fuji], WLC9500 Software (WLC9500_IOSXE), Experimental Version 16.10.20180920:011848 [v1610_throttle-/nobackup/mcpre/BLD-BLD_V1610_THROTTLE_LATEST_20180920_010739 154]
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Thu 20-Sep-18 03:07 by mcpre
                          BOOT = bootflash:packages.conf,12;
                   CONFIG_FILE = 
        Configuration register = 0x2102

Peer Processor Information :
----------------------------
              Standby Location = slot 2
        Current Software state = STANDBY HOT 
       Uptime in current state = 1 hour, 33 minutes
                 Image Version = Cisco IOS Software [Fuji], WLC9500 Software (WLC9500_IOSXE), Experimental Version 16.10.20180920:011848 [v1610_throttle-/nobackup/mcpre/BLD-BLD_V1610_THROTTLE_LATEST_20180920_010739 154]
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Thu 20-Sep-18 03:07 by mcpre
                          BOOT = bootflash:packages.conf,12;
                   CONFIG_FILE = 
        Configuration register = 0x2102

You can verify the current data from both active and standby 9800 WLC, to ensure it is the same on both.

Examples:

9800 WLC-1# show ap summary chassis active r0              
Number of APs: 2

AP Name                            Slots    AP Model  Ethernet MAC    Radio MAC       Location              Country     IP Address                                 State         
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
<!--AP Name-->                       2      3702I      <!--MAC-->    <!--MAC-->   CALO                  MX          <!--IP address-->                            Registered    
<!--AP Name-->                       2      3702I      <!--MAC-->    <!--MAC-->   abcde123456789012345  MX          <!--IP address-->                            Registered    

9800 WLC-1# show ap summary chassis standby r0             
Number of APs: 2

AP Name                            Slots    AP Model  Ethernet MAC    Radio MAC       Location              Country     IP Address                                 State         
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
<!--AP Name-->                       2      3702I     <!--MAC--> <!--MAC-->    CALO                    MX           <!--IP address-->                             Registered    
<!--AP Name-->                       2      3702I     <!--MAC--> <!--MAC-->    abcde123456789012345    MX           <!--IP address-->                             Registered    

Troubleshoot

Example of console output of a successful HA pair synchronization between two hardware 9800 WLCs:

9800 WLC-1

9800 WLC-1# chassis ha-interface local-ip <!--IP address--> remote-ip <!--IP address-->       

9800 WLC-1# show chassis 
Chassis/Stack Mac Address : <!--MAC address-->  - Local Mac Address
Mac persistency wait time: Indefinite
Local Redundancy Port Type: Twisted Pair
                                             H/W   Current
Chassis#   Role    Mac Address     Priority Version  State                 IP
-------------------------------------------------------------------------------------
*1       Active   <!--MAC address-->    1      V02     Ready                



9800 WLC-1# wr
Building configuration...
[OK]

9800 WLC-1# reload 
Reload command is issued on Active unit, this will reload the whole stack
Proceed with reload? [confirm]

MMM DD HH:MM:SS.XXX: %SYS-5-RELOAD: Reload requested by console. Reload Reason: Reload Command.
Chassis 1 reloading, reason - Reload command
.
.
.
*MMM DD HH:MM:SS.XXX: %IOSXE_REDUNDANCY-6-PEER: Active detected chassis 2 as standby.
*MMM DD HH:MM:SS.XXX: %STACKMGR-6-STANDBY_ELECTED: Chassis 1 R0/0: stack_mgr: Chassis 2 has been elected STANDBY.
*MMM DD HH:MM:SS.XXX: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process bt_logger
*MMM DD HH:MM:SS.XXX: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process mmap_schema
*MMM DD HH:MM:SS.XXX: %CMRP-5-PRERELEASE_HARDWARE: Chassis 2 R0/0: cmand: 0 is pre-release hardware
*MMM DD HH:MM:SS.XXX: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_FOUND(4))
*MMM DD HH:MM:SS.XXX: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_REDUNDANCY_STATE_CHANGE(5))
*MMM DD HH:MM:SS.XXX: %IOSXE_PEM-6-INSPEM_FM: PEM/FM Chassis 2 slot P0 inserted
*MMM DD HH:MM:SS.XXX: %IOSXE_PEM-6-INSPEM_FM: PEM/FM Chassis 2 slot P2 inserted
*MMM DD HH:MM:SS.XXX: % Redundancy mode change to SSO
*MMM DD HH:MM:SS.XXX: %VOICE_HA-7-STATUS: NONE->SSO; SSO mode will not take effect until after a platform reload.
*MMM DD HH:MM:SS.XXX: Syncing vlan database
*MMM DD HH:MM:SS.XXX: Vlan Database sync done from bootflash:vlan.dat to stby-bootflash:vlan.dat (616 bytes)
MMM DD HH:MM:SS.XXX: %PKI-6-AUTHORITATIVE_CLOCK: The system clock has been set.
MMM DD HH:MM:SS.XXX: %PKI-6-CS_ENABLED: Certificate server now enabled.
MMM DD HH:MM:SS.XXX: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
MMM DD HH:MM:SS.XXX: %VOICE_HA-7-STATUS: VOICE HA bulk sync done.
MMM DD HH:MM:SS.XXX: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)

9800 WLC-2

9800 WLC-2# chassis ha-interface local-ip <!--IP address--> remote-ip <!--IP address--> 

9800 WLC-2# show chassis 
Chassis/Stack Mac Address : <!--MAC address--> - Local Mac Address
Mac persistency wait time: Indefinite
Local Redundancy Port Type: Twisted Pair
                                             H/W   Current
Chassis#   Role    Mac Address     Priority Version  State                 IP
-------------------------------------------------------------------------------------
*1       Active   <!--MAC address-->     1      V02     Ready                


9800 WLC-2# wr
Building configuration...
[OK]

9800 WLC-2# reload 
Reload command is issued on Active unit, this will reload the whole stack
Proceed with reload? [confirm]

MMM DD HH:MM:SS.XXX: %SYS-5-RELOAD: Reload requested by console. Reload Reason: Reload Command.
Chassis 1 reloading, reason - Reload command
.
.
.
Press RETURN to get started!


*MMM DD HH:MM:SS.XXX: %IOSXE_PLATFORM-3-WDC_NOT_FOUND: WDC returned length: 0Adding registry invocations for the WLC platform

*MMM DD HH:MM:SS.XXX: %REDUNDANCY-3-PEER_MONITOR: PEER_FOUND event on standby
*MMM DD HH:MM:SS.XXX: %SMART_LIC-6-AGENT_READY: Smart Agent for Licensing is initialized
*MMM DD HH:MM:SS.XXX: %SMART_LIC-6-AGENT_ENABLED: Smart Agent for Licensing is enabled 
*MMM DD HH:MM:SS.XXX: %SMART_LIC-6-EXPORT_CONTROLLED: Usage of export controlled features is not allowed
*MMM DD HH:MM:SS.XXX: %SMART_LIC-6-HA_ROLE_CHANGED: Smart Agent HA role changed to Standby.
*MMM DD HH:MM:SS.XXX: dev_pluggable_optics_selftest attribute table internally inconsistent @ 0x1ED

*MMM DD HH:MM:SS.XXX: mcp_pm_subsys_init : Init done sucessfullySID Manager, starting initialization ...

*MMM DD HH:MM:SS.XXX: Notifications initializedSID Manager, completed initialization ...

*MMM DD HH:MM:SS.XXX: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*MMM DD HH:MM:SS.XXX: %SMART_LIC-6-AGENT_READY: Smart Agent for Licensing is initialized
*MMM DD HH:MM:SS.XXX: %SMART_LIC-6-AGENT_ENABLED: Smart Agent for Licensing is enabled 
*MMM DD HH:MM:SS.XXX: %SMART_LIC-6-EXPORT_CONTROLLED: Usage of export controlled features is not allowed
*MMM DD HH:MM:SS.XXX: %CRYPTO-4-AUDITWARN: Encryption audit check could not be performed
*MMM DD HH:MM:SS.XXX: %VOICE_HA-7-STATUS: CUBE HA-supported platform detected.
*MMM DD HH:MM:SS.XXX: %IOSXE_VMAN-3-MSGINITFAIL: Failed to initialize required Virt-manager resource: Initalize MQIPC
*MMM DD HH:MM:SS.XXX: mcp_pm_init_done : Called
*MMM DD HH:MM:SS.XXX: %ONEP_BASE-6-SS_ENABLED: ONEP: Service set Base was enabled by Default
*MMM DD HH:MM:SS.XXX: cwan_pseudo_oir_insert_one: [0/0] ctrlr[16506] already analyzed
*MMM DD HH:MM:SS.XXX: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named TP-self-signed-1598997203 has been generated or imported by crypto config
*MMM DD HH:MM:SS.XXX: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named ca has been generated or imported by crypto config
*MMM DD HH:MM:SS.XXX: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named ewlc-tp1 has been generated or imported by crypto config
*MMM DD HH:MM:SS.XXX: %AAA-5-USER_RESET: User admin failed attempts reset by console
*MMM DD HH:MM:SS.XXX: %CRYPTO_ENGINE-5-KEY_REPLACE: A key named TP-self-signed-1598997203 has been replaced by crypto config with new key data 
*MMM DD HH:MM:SS.XXX: %SSH-5-DISABLED: SSH 1.99 has been disabled
*MMM DD HH:MM:SS.XXX: %CRYPTO_ENGINE-5-KEY_REPLACE: A key named ca has been replaced by crypto config with new key data 
*MMM DD HH:MM:SS.XXX: %CRYPTO_ENGINE-5-KEY_REPLACE: A key named ewlc-tp1 has been replaced by crypto config with new key data 
MMM DD HH:MM:SS.XXX: %SPA_OIR-6-OFFLINECARD: SPA (BUILT-IN-4X10G/1G) offline in subslot 0/0
MMM DD HH:MM:SS.XXX: %IOSXE_OIR-6-INSCARD: Card (fp) inserted in slot F0
MMM DD HH:MM:SS.XXX: %IOSXE_OIR-6-ONLINECARD: Card (fp) online in slot F0
MMM DD HH:MM:SS.XXX: %IOSXE_OIR-6-INSCARD: Card (cc) inserted in slot 0
MMM DD HH:MM:SS.XXX: %IOSXE_OIR-6-ONLINECARD: Card (cc) online in slot 0
MMM DD HH:MM:SS.XXX: %IOSXE_OIR-6-INSSPA: SPA inserted in subslot 0/0
MMM DD HH:MM:SS.XXX: %IOSXE_OIR-3-SPA_INTF_ID_ALLOC_FAILED: Failed to allocate interface identifiers forSPA(BUILT-IN-4X10G/1G) in slot/bay: 0/0
MMM DD HH:MM:SS.XXX: %SYS-5-RESTART: System restarted --
Cisco IOS Software [Fuji], WLC9500 Software (WLC9500_IOSXE), Experimental Version 16.10.20180920:011848 [v1610_throttle-/nobackup/mcpre/BLD-BLD_V1610_THROTTLE_LATEST_20180920_010739 154]
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Thu 20-Sep-18 03:07 by mcpre
MMM DD HH:MM:SS.XXX: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to down
MMM DD HH:MM:SS.XXX: %SSH-5-ENABLED: SSH 1.99 has been enabled
MMM DD HH:MM:SS.XXX: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named TP-self-signed-1598997203.server has been generated or imported by crypto-engine
MMM DD HH:MM:SS.XXX: %SYS-6-BOOTTIME: Time taken to reboot after reload =  328 seconds
MMM DD HH:MM:SS.XXX: %SMART_LIC-3-HOT_STANDBY_OUT_OF_SYNC: Smart Licensing agent on hot standby is out of sync with active Smart Licensing agent
MMM DD HH:MM:SS.XXX: %SPA_OIR-6-ONLINECARD: SPA (BUILT-IN-4X10G/1G) online in subslot 0/0
MMM DD HH:MM:SS.XXX: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/2, link down due to local fault
MMM DD HH:MM:SS.XXX: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/3, link down due to local fault
MMM DD HH:MM:SS.XXX: BUILT-IN-4X10G/1G[0/0] : Unsupported rate(0) for the XCVR inserted inport 0 xcvr_type=0 admin_state=UNSHUT
MMM DD HH:MM:SS.XXX: BUILT-IN-4X10G/1G[0/0] : Unsupported rate(0) for the XCVR inserted inport 1 xcvr_type=0 admin_state=UNSHUT
MMM DD HH:MM:SS.XXX: %PKI-3-KEY_CMP_MISMATCH: Key in the certificate and stored key does not match for Trustpoint-TP-self-signed-1598997203.
MMM DD HH:MM:SS.XXX: %PKI-4-NOAUTOSAVE: Configuration was modified.  Issue "write memory" to save new certificate
MMM DD HH:MM:SS.XXX: %TRANSCEIVER-3-INIT_FAILURE: SIP0/0: Detected for transceiver module in TenGigabitEthernet0/0/0, module disabled 
MMM DD HH:MM:SS.XXX: %LINK-3-UPDOWN: SIP0/0: Interface TenGigabitEthernet0/0/0, changed state to up
MMM DD HH:MM:SS.XXX: %PLATFORM-6-RF_PROG_SUCCESS: RF state STANDBY HOT

Run this command on any unit and, by default, every 5 seconds you see an update of HA synchronization:

# show redundancy history monitor [ interval <5-3600 seconds > ]

Sep 21 15:24:24.727 RF_EVENT_CLIENT_PROGRESSION(503) RP Platform RF(1340) op=8 rc=0
Sep 21 15:24:24.727 RF_EVENT_CLIENT_PROGRESSION(503) RP Platform RF(1340) op=8 rc=11
Sep 21 15:24:24.740 RF_EVENT_CLIENT_PROGRESSION(503) CWAN Interface Events(1504) op=8 rc=0
Sep 21 15:24:24.741 RF_EVENT_CLIENT_PROGRESSION(503) CWAN Interface Events(1504) op=8 rc=0
Sep 21 15:24:24.741 RF_EVENT_CLIENT_PROGRESSION(503) CWAN Interface Events(1504) op=8 rc=11
Sep 21 15:24:24.741 RF_EVENT_CLIENT_PROGRESSION(503) NAT HA(401) op=8 rc=0
Sep 21 15:24:24.741 RF_EVENT_CLIENT_PROGRESSION(503) NAT HA(401) op=8 rc=0
Sep 21 15:24:24.741 RF_EVENT_CLIENT_PROGRESSION(503) NAT HA(401) op=8 rc=11
Sep 21 15:24:24.741 RF_EVENT_CLIENT_PROGRESSION(503) NAT64 HA(404) op=8 rc=0
Sep 21 15:24:24.741 RF_EVENT_CLIENT_PROGRESSION(503) NAT64 HA(404) op=8 rc=0
Sep 21 15:24:24.741 RF_EVENT_CLIENT_PROGRESSION(503) NAT64 HA(404) op=8 rc=11
Sep 21 15:24:24.743 RF_EVENT_CLIENT_PROGRESSION(503) DHCPv6 Relay(148) op=8 rc=0
Sep 21 15:24:24.743 RF_EVENT_CLIENT_PROGRESSION(503) DHCPv6 Relay(148) op=8 rc=0
Sep 21 15:24:24.743 RF_EVENT_CLIENT_PROGRESSION(503) DHCPv6 Relay(148) op=8 rc=11
Sep 21 15:24:24.782 RF_EVENT_CLIENT_PROGRESSION(503) DHCPv6 Server(149) op=8 rc=0
Sep 21 15:24:24.782 RF_EVENT_CLIENT_PROGRESSION(503) DHCPv6 Server(149) op=8 rc=0
Sep 21 15:24:24.782 RF_EVENT_CLIENT_PROGRESSION(503) DHCPv6 Server(149) op=8 rc=11
Sep 21 15:24:24.783 RF_EVENT_CLIENT_PROGRESSION(503) Last Slave(65000) op=8 rc=0
Sep 21 15:24:25.783 RF_PROG_STANDBY_HOT(105) Last Slave(65000) op=8 rc=0
Sep 21 15:24:25.783 RF_EVENT_CLIENT_PROGRESSION(503) Last Slave(65000) op=8 rc=0



End = e Freeze = f 

Enter Command: e
 

To have a more detailed view of the HA sync process, run this command:

# show logging process stack_mgr internal

Related Information

• Cisco Technical Support & Downloads