Field Notice: FN74222 - Full or Partial Cisco 9800 Series Wireless Controller Configuration Loss after High-Availability Stateful Switchover Failover - Software Upgrade Recommended

After a High-Availability Stateful Switchover (HA SSO) failover or reboot, a partial or complete configuration loss may be seen on Cisco C9800 Series Wireless Controllers (WLCs).

Observable symptoms include APs (Access Points) not being able to join the WLC and the WLC being unreachable or unable to be configured.

Due to this software defect, when the Replication Manager (repm) process increases CPU utilization, if the WLC reloads or does a failover for any reason, the configuration may be lost.

Observable symptoms vary and may appear as follows:

• The WLC becomes unreachable.
• The WLC is not able to apply configuration changes. Doing so may result in the following error:

  Country <XX> is unsupported. Please configure supported country code.

• APs can no longer join the WLC.
• APs stop broadcasting some or all Service Set Identifiers (SSIDs).
• Total or partial configuration is missing.
• There is no telemetry.
• The WLC reverts to the Day 0 configuration in the CLI and the Day 0 wizard in the GUI.
• The repm process indicates high CPU utilization over the time (e.g., greater than 60% and increasing) before the failover or reboot. Use the following command to monitor the repm process:

  Device# show processes cpu platform | include repm
  
  Pid PPid 5Sec 1Min 5Min Status Size Name 
  -------------------------------------------------------------------------------- 
  17795 17785 83% 89% 89% S 910372 repm

• The running configurations for Standby and Active WLCs show differences. View and compare the two running wireless configurations using the following instructions:
  • Active: Use the show running-config command.
  • Standby: Use SSH to access the Redundancy Management Interface (RMI) and use the show running-config command.

This is a software-only issue. No hardware replacement is needed.

Preventive actions

 To prevent this issue, use the following steps before upgrading:

1. Back up your configuration.
2. Monitor the repm process. If the repm CPU utilization is not high (<60%), proceed to the upgrade.

Upgrade to one of the following fixed Wireless IOS XE software releases:

• 17.9.6
• 17.9.5 SMU (Software Maintenance Upgrade) C9800-universalk9_wlc.17.09.05.CSCwj96199.SPA.smu.bin
• 17.12.3 SMU C9800-universalk9_wlc.17.12.03.CSCwj96199.SPA.smu.bin
• 17.12.4
• 17.15.1

If repm CPU utilization is high (>60%) and increasing over time, take the following actions to mitigate the issue:

1. Delete the persistent binary config file from both Active and Standby using the following commands:

   delete /force /recursive bootflash:.dbpersist/persistent-config.tar.gz
   delete /force /recursive bootflash:.dbpersist/persistent-config.meta
   delete /force /recursive bootflash-2:.dbpersist/persistent-config.tar.gz
   delete /force /recursive bootflash-2:.dbpersist/persistent-config.meta

   Note: If remote access to the CLI is not available, access the CLI using the console of each controller and delete these files.
2. Reload the whole stack (Active and Standby together) for the workaround to take effect. The commands will free up space and lower CPU utilization of the repm process.
   Note: This step will cause the controllers to go down for a few minutes until the Active controller reloads.
3. After applying the workaround, upgrade to a fixed software release.

Corrective actions

If a device is already affected and the configuration is lost, restore the configuration and upgrade to a fixed Wireless IOS XE software release.

To determine if a specific WLC is affected by this issue, confirm the software version the WLC is running and whether HA SSO is enabled.

To determine which software version a WLC is running, use the show version | include 9800|IOSXE  command.

To verify that an HA SSO pair is running on the device, use the show redundancy command.

If the output of the show redundancy command returns Configured Redundancy Mode = sso, and the WLC is running an affected software version, the WLC is susceptible to this issue.