Field Notice: FN74109 - Access Point Image Corruption During CAPWAP Upgrade May Result in Boot Failure - Software Upgrade Recommended

When some access points (APs) download software in CAPWAP (Control and Provisioning of Wireless Access Points) from a Cisco Catalyst 9800 Series Wireless Controller, the image may be corrupted. The AP may attempt to boot the corrupt image, resulting in the AP entering a boot loop. This field notice explains which deployments are and are not susceptible to this problem and how to recover APs that are in a boot loop.

For more information on the issues described in this field notice, see these short videos:

• How to tell whether you might be susceptible to this issue and how to upgrade safely

• How to recover APs with corrupted images 

Identify Affected Deployments

This issue is more likely to occur in a network path between a Cisco Catalyst 9800 Series Wireless Controller and an AP that has both high packet loss and a maximum transmission unit (MTU) that is lower than 1485. For more details on how to identify an at-risk network path, see https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/221869-safely-upgrade-access-points-avoiding-i.html.

Downloads from all Cisco Catalyst 9800 Series Wireless Controllers are susceptible to this vulnerability. Downloads from Cisco AireOS Controller are not susceptible.

The following Cisco AP models that are registered to Cisco Catalyst 9800 Series Wireless LAN Controllers (WLC) are affected: 

• Wave2 11ac APs (2800/3800/4800/1560/IW6330/ESW6300)
• Catalyst 9100 Series WiFi6 APs (9105/9115/9117/9120/9124/9130/WP-WIFI6)
• Catalyst 9100 Series WiFI6E APs (9136/9162/9164/9166)
• Cisco 6300 Series Embedded Services APs
• Catalyst IW6300 Series Heavy Duty APs

The following Cisco AP models are not affected:

• 1800/1540 series Wave 2 11ac APs
• Wave1 11ac APs (1700/2700/3700/1570/IW3700)

To prevent the APs from entering boot loops, upgrade APs to the following fixed versions of Cisco AireOS or Wireless IOS XE:

• 8.10.185.0 and later
• 17.3.7 and later
• 17.6.6 and later
• 17.9.3 and later
• 17.11.1 and later

In some deployments, AP image downloads over CAPWAP are subject to corruption. Although affected APs recognize that the downloaded image is corrupted, they still attempt to boot from that image, resulting in a boot loop condition.

Newer versions of u-boot (boot loader) have an Alt-boot enhancement. Using this Alt-boot enhancement, an AP can recognize that it is trying to boot from a corrupted image after five failures in a row. The AP will then attempt to boot from its backup partition. In such cases, it is possible to recover the AP without physical access to its console. However, APs that are in a boot loop with an older u-boot must be recovered through the console.

See Recover from a boot loop caused by image corruption on Wave 2 and 11ax Access Points (CSCvx32806) for more information about how to determine whether the u-boot of an AP has the Alt-boot enhancement.

• Safely Upgrade Access Points, Avoiding Image Corruption That Causes Boot Loop

If an AP has already downloaded the corrupt image and is now in a boot loop, the console will display a message as shown in the following example:

verify signature failed for /bootpart/part1/ramfs_data_cisco.cpio.lzma 

The following example shows another console message for an AP that is in a boot loop due to downloading a corrupt image:

verify signature failed for /bootpart/part2/ramfs_data_cisco.squashfs 

Note: To determine which boot partition is corrupt, see whether the console message includes part1 or part2.

If the AP was configured to log to an external syslog server, it will log the following error prior to the image download attempt, as shown in the following example:

Image signature verification failure: -3 

APs that are in a boot loop state will show an output of IEEE PD in the uplink switchport, as shown below:

Switch# show power inline

Available:195.0(w) Used:152.1(w) Remaining:42.9(w)Interface Admin  Oper  Power Device  Class Max (Watts)
--------- ------ ----- ----- ------- ----- ----------- 
Gi0/19    auto   on    15.4  Ieee PD 4     30.0

Solution for APs That Are Already in a Boot Loop 

To recover APs that are already in a boot loop condition as a result of image corruption following CAPWAP download, see Recover from a Boot Loop Caused by Image Corruption on Wave 2 and 11 ax Access Points (CSCvx32806).

APs that have Alt-boot enhancements can be recovered without physical access to the console. APs that are in a boot loop with an older u-boot must be recovered through the console.

Preparing for an Upgrade 

If you are planning to upgrade your access points, and if all of the following apply:

• Some or all of the APs are susceptible models.
• Your network path from Cisco Catalyst 9800 Series Wireless Controller to the APs is at risk.
• Your APs are now running software that lacks the fixes for booting bad images.

See Safely Upgrade Access Points, Avoiding Image Corruption That Causes Boot Loop for instructions on how to avoid AP image corruption during the upgrade process.

Additional Information 

• Safely Upgrade Access Points, Avoiding Image Corruption That Causes Boot Loop
• Recover from a boot loop caused by image corruption on Wave 2 and 11ax Access Points (CSCvx32806)

APs can enter a boot loop if they use an affected upgrade method, boot a corrupt image, or use an affected image path.

Affected Upgrade Methods

AP images that are upgraded through CAPWAP may be subject to image corruption. 

The following scenarios involve CAPWAP downloads:

• WLC image upgrade with predownload
• WLC image upgrade without predownload
• Efficient Image Upgrade for FlexConnect deployments during WLC image upgrade
• APs moving between WLCs that are running different versions

Note: Non-CAPWAP upgrade methods, such as TFTP or SFTP through the archive download-sw AP command or the HTTPS out-of-band upgrade method introduced in Cisco Wireless IOS XE 17.11, are not susceptible to image corruption.

Affected Versions: the Boot a Bad Image Syndrome

This problem, where the AP attempts to boot an image that it knows is corrupt, is addressed by the following Cisco bug IDs: CSCvx32806, CSCwc72021, and CSCwd90081, which are fixed in the the following Cisco AireOS and Cisco Wireless IOS XE releases:

• 8.10.185.0 and later
• 17.3.7 and later
• 17.6.6 and later
• 17.9.3 and later
• 17.11.1 and later

Note: Although an AP that has upgraded to a fixed software version may still download a corrupt image, it will not attempt to boot that image, but instead will continue to attempt the download until it succeeds.

Affected Network Paths

AP image corruption is more likely to occur over CAPWAP tunnels over a WAN that has the following path characteristics:

• A CAPWAP MTU that is less than 1485 bytes: The lower the MTU, the higher the risk. Low CAPWAP MTU may also be a symptom of packet loss.
• High packet loss.

AP image corruption is not observed when a LAN path between the Cisco Catalyst 9800 Series Wireless Controller and the AP has a full 1500 byte IP path MTU, low latency, and very low packet loss.

Note: Customers with affected AP models that are running software with the Boot a Bad Image Syndrome and that have an affected WAN path between the APs and the Cisco Catalyst 9800 Series Wireless Controller should take care when upgrading to ensure that their APs do not become stuck in a boot loop.

Products Considered Not Vulnerable

The following Cisco devices are not considered vulnerable:

• AireOS WLCs 
• Aironet 1800 APs
• Aironet 1540 Series Outdoor APs
• Wave 1 11ac APs (1700/2700/3700/1570/IW3700)
• Embedded Wireless Controllers (EWC)

The following Cisco products are affected

• Catalyst 9800 Series WLCs 
• Cisco 802.11ac Wave 2 APs (2800/3800/4800/1560/IW6330/ESW6300)
• Catalyst 9100 Series WiFi6 APs (9105/9115/9117/9120/9124/9130/WP-WIFI6)
• Catalyst 9100 Series WiFI6E APs (9136/9162/9164/9166)
• Cisco 6300 Series Embedded Services Access Points
• Catalyst IW6300 Series Heavy Duty Access Points