Field Notice: FN70330 - Cisco IOS Access Point Stranded Due to Flash Corruption Issue - Workaround Provided

Some Cisco IOS^® Access Points (APs) report no flash access or file system corruption after an AP reboot. This is most commonly seen when a Wireless LAN Controller (WLC) is upgraded.

The corruption can cause these conditions on the AP: 

• The AP is unable to save the configuration.
• The AP is unable to perform an upgrade.
• The AP loses configuration.
• The AP is stuck in a booting loop.
• The AP is in bootloader status.

Due to a number of software issues, the flash file system on some Cisco IOS APs might become corrupt over time during normal operation.

Problem #1: No Flash Access

These symptoms are indicative of problem #1, no flash access.

Command: show file system

The size and free space on flash is shown as "-" instead of the actual value.

Problem Output

==============================
       Size(b)       Free(b)      Type  Flags  Prefixes				
*            -             -     flash     rw   flash:	
==============================

Normal Output

================================
       Size(b)       Free(b)      Type    Flags  Prefixes				
*     40900608       25282560     flash     rw     flash:
===============================

Command: show flash

The AP cannot display its files.

Directory of flash:/
 %Error opening flash:/ (Invalid argument)

Command: show logging

The AP might display these messages.

*Mar  1 00:01:04.159: %LWAPP-3-CLIENTERRORLOG: Save LWAPP Config: error saving config file

.....

*Mar  1 00:01:04.159: %LWAPP-3-CLIENTERRORLOG: Load nvram:/lwapp_ap.cfg config
failed, trying backup...
*Mar  1 00:01:04.159: %LWAPP-3-CLIENTERRORLOG: Load nvram:/lwapp_ap.cfg.bak
config failed...

Command: fsck flash:

The AP cannot verify its file system.

AP# fsck flash:

Fsck operation may take a while. Continue? [confirm]

%Error fscking flash: (Device or resource busy)

Problem #2: Corrupted Files

This symptom is indicative of problem #2, corrupted files.

The AP is stuck in this loop. This output is seen on the AP console:

Loading "flash:/ap1g1-k9w8-mx.v153_80mr_esc.201603111020/ap1g1-k9w8-mx.v153_80mr_esc.201603111020"
...###############################################################################################
##################################################################################################
##################################################################################################
##################################################################################################
##################################################################################################
####################bad mzip file, unknown zip method?

Solution

Caution: Before an upgrade, run the Wireless LAN (WLAN) Poller tool in order to identify affected Cisco IOS APs on the network and fix them.

Refer to the document Understanding Various AP-IOS Flash Corruption Issues in order to understand how to recover APs with flash issues and for details about how to install and run the WLAN Poller tool.

Upgrade the WLC to one of these AireOS releases:

• Release 8.3MR4Esc (Release 8.3.143.7 or later, available from the Technical Assistance Center (TAC))
• Release 8.3.150.0
• Release 8.5.140.0 or later
• Release 8.8.111.0 or later
• Release 8.9 or later

Notes:

• If you upgrade to a fixed version before you run the WLAN Poller tool, there is a risk that your Cisco IOS APs will be stranded during the upgrade.
• If the WLAN Poller tool needs to recover affected APs, it will cause those APs to reload once the image is downloaded and installed. Ensure that you run the WLAN Poller tool in a maintenance window.
• If it is not possible to upgrade the WLC to a fixed code version at this time, and you see that a significant number of APs enter into this state, you should run the WLAN Poller tool every 30 days or more frequently until you can upgrade.

These are the only Cisco IOS AP models affected: 1600, 1700, 2600, 2700, 3600, and 3700 Series APs.