Troubleshoot Catalyst 9130 AP Join Issue

Introduction

This document describes a specific use case where the Access point is stuck at Code upgrade during the controller joining process.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Basic knowledge of Cisco WLC 9800
• Basic knowledge of Cisco Wave2 and/or 11AX APs
• Good understating for the AP join process with the catalyst 9800 WLC.

Components Used

The information in this document is based on these software and hardware versions:

• Catalyst 9800-L WLC, Cisco IOS® XE Cupertino 17.9.3
• Catalyst C9130AXI-E Access Point

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Topology

This troubleshooting flow is applicable for APs connected in local mode or APs connected in flex connect mode in a branch site.

Troubleshoot Topology

Use Case

This document discusses a specific use case in which the Access Point completes its discovery and join phase but becomes stuck at the Image data check phase. This issue arises due to a code upgrade push between the wireless LAN controller and the Access Point, resulting in an error message stating "No space left on device."

A. Access Point Completed Discovery Phase:

Feb 13 11:11:21 kernel: [*02/13/2024 11:11:21.4662] CAPWAP State: Discovery Feb 13 11:11:21 kernel: [*02/13/2024 11:11:21.4669] IP DNS query for CISCO-CAPWAP-CONTROLLER.cisco.com Feb 13 11:11:21 kernel: [*02/13/2024 11:11:21.5444] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0) Feb 13 11:11:21 kernel: [*02/13/2024 11:11:21.5454] Discovery Request sent to ff02::18c, discovery type UNKNOWN(0) Feb 13 11:11:21 kernel: [*02/13/2024 11:11:21.5456] Discovery Response from <WLC IP address> Feb 13 11:11:21 kernel: [*02/13/2024 11:11:21.5466] Discovery Response from <MAC of the controlloer> Feb 13 11:11:21 LED: State received is Dis_join_progress Feb 13 11:11:21 LED: LED state Changed from LED_GREEN to LED_CYCLIC_GRO_LED 

B. DTLS Set Up Initiated and Join Process Completed Logs:

Feb 13 11:11:31 kernel: [*02/13/2024 11:11:31.0002] CAPWAP State: DTLS Setup Feb 13 11:11:31 kernel: 
[*02/13/2024 11:11:31.6405] Feb 13 11:11:31 kernel: [*02/13/2024 11:11:31.6405] CAPWAP State: Join Feb 13 11:11:31 
kernel: [*02/13/2024 11:11:31.6422] Sending Join request to 10.228.104.4 through port 5248 Feb 13 11:11:31 kernel: 
[*02/13/2024 11:11:31.6454] Join Response from 10.228.104.4 Feb 13 11:11:31 kernel: [*02/13/2024 11:11:31.6454] 
AC accepted join request with result code: 0 Feb 13 11:11:31 kernel: [*02/13/2024 11:11:31.6489] Received wlcType 0,
 timer 30 Feb 13 11:11:31 kernel: [*02/13/2024 11:11:31.6489] TLV ID 2216 not found Feb 13 11:11:31 kernel: 
[*02/13/2024 11:11:31.6489] TLV-DEC-ERR-1: No proc for 2216  

C. Upon completion of the join process, the Access Point enters the Image data phase. During this phase, you can observe an image transfer from the controller to the AP, which fails, accompanied by the error message "No space left on device."

Feb 13 11:11:31 kernel: [*02/13/2024 11:11:31.6709] CAPWAP State: Image Data
Feb 13 11:11:31 kernel: [*02/13/2024 11:11:31.6712] AP image version 8.10.112.0 backup 0.0.0.0, Controller 17.9.3.50
Feb 13 11:11:31 kernel: [*02/13/2024 11:11:31.6712] Version does not match.
Feb 13 11:11:31 kernel: [*02/13/2024 11:11:31.7111] do PRECHECK, part1 is active part
Feb 13 11:11:31 upgrade: /tmp space: OK available 80268, required 40000 
Feb 13 11:11:31 kernel: [*02/13/2024 11:11:31.7322] upgrade.sh: /tmp space: OK available 80268, required 40000 
Feb 13 11:11:31 kernel: [*02/13/2024 11:11:31.7326] wtpImgFileReadRequest: request ap1g6a, local /tmp/part.tar
Feb 13 11:11:31 kernel: [*02/13/2024 11:11:31.7337] Image Data Request sent to 10.228.104.4, fileName [ap1g6a], slaveStatus 0
Feb 13 11:11:31 kernel: [*02/13/2024 11:11:31.7351] Image Data Response from 10.228.104.4
Feb 13 11:11:31 kernel: [*02/13/2024 11:11:31.7351] AC accepted join request with result code: 0
Feb 13 11:11:31 LED: State received is Dis_join_completed
Feb 13 11:11:31 LED: State received is Sofware_upgrade_progress
Feb 13 11:11:31 LED: LED state Changed from LED_CYCLIC_GRO_LED to LED_BLINKING_BLUE
Feb 13 11:11:45 kernel: [*02/13/2024 11:11:31.7393] <..................................................
Feb 13 11:11:53 kernel: [*02/13/2024 11:11:45.3443] ..................................................
Feb 13 11:12:07 kernel: [*02/13/2024 11:11:53.9200] ..................................................
Feb 13 11:12:16 kernel: [*02/13/2024 11:12:07.5228] ..........................Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
Feb 13 11:12:25 kernel: [*02/13/2024 11:12:18.7413] ........................
Feb 13 11:12:43 kernel: [*02/13/2024 11:12:25.5137] ..................................................
Feb 13 11:13:06 kernel: [*02/13/2024 11:12:43.6235] ...................................Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
Feb 13 11:13:07 kernel: [*02/13/2024 11:13:07.0982] ...Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
Feb 13 11:13:07 kernel: [*02/13/2024 11:13:07.5458] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
Feb 13 11:13:07 kernel: [*02/13/2024 11:13:07.5965] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
Feb 13 11:13:09 kernel: [*02/13/2024 11:13:07.6044] ............
Feb 13 11:13:21 kernel: [*02/13/2024 11:13:09.9353] .............> 84387840 bytes, 62742 msgs, 1195 last
Feb 13 11:13:21 kernel: [*02/13/2024 11:13:21.0817] Last block stored, IsPre 0, WriteTaskId 0
Feb 13 11:13:21 kernel: [*02/13/2024 11:13:21.0847] wtpProcessImageDataRequest(10): fileName ap1g6a, pre 0
Feb 13 11:13:21 upgrade: Start doing upgrade arg1=PREDOWNLOAD arg2= arg3= ...
Feb 13 11:13:21 kernel: [*02/13/2024 11:13:21.1278] do PREDOWNLOAD, part1 is active part
Feb 13 11:13:21 kernel: [*02/13/2024 11:13:21.1419] upgrade.sh: Start doing upgrade arg1=PREDOWNLOAD arg2= arg3= ...
Feb 13 11:13:21 upgrade: Using image /tmp/part.tar on axel-qca ...
Feb 13 11:13:21 kernel: [*02/13/2024 11:13:21.1945] upgrade.sh: Using image /tmp/part.tar on axel-qca ...
Feb 13 11:13:21 kernel: [*02/13/2024 11:13:21.1947] sh: write error: No space left on device
Feb 13 11:13:21 kernel: [*02/13/2024 11:13:21.1997] tar: write error: No space left on device
Feb 13 11:13:21 upgrade: ERROR: Image type mismatch. Expected:ap1g6a Got:
Feb 13 11:13:21 kernel: [*02/13/2024 11:13:21.2247] upgrade.sh: ERROR: Image type mismatch. Expected:ap1g6a Got:
Feb 13 11:13:21 upgrade: Cleanup for do_upgrade...
Feb 13 11:13:21 upgrade: /tmp/upgrade_in_progress cleaned
Feb 13 11:13:21 kernel: [*02/13/2024 11:13:21.2411] upgrade.sh: Cleanup for do_upgrade...
Feb 13 11:13:21 upgrade: Cleanup tmp files ...
Feb 13 11:13:21 kernel: [*02/13/2024 11:13:21.2581] upgrade.sh: /tmp/upgrade_in_progress cleaned
FebFeb 13 11:13:21 kernel: [*02/13/2024 11:13:21.3072] capwap-upgrade script returned failure when calling PREDOWNLOAD.
Feb 13 11:13:21 kernel: [*02/13/2024 11:13:21.3073] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
Feb 13 11:13:21 kernel: [*02/13/2024 11:13:21.3074] CAPWAP SM handler: Failed to process message type 15 state 10.
Feb 13 11:13:21 kernel: [*02/13/2024 11:13:21.3074] Failed to handle capwap control message from controller - status 2
Feb 13 11:13:21 kernel: [*02/13/2024 11:13:21.3074] Failed to process encrypted capwap packet 0x55aaaa2000 from 10.228.104.4
Feb 13 11:13:21 kernel: [*02/13/2024 11:13:21.3074] Failed to send capwap message 0 to the state machine. Packet already freed.
Feb 13 11:13:22 LED: State received is Sofware_upgrade_fail
Feb 13 11:13:22 LED: LED state Changed from LED_BLINKING_BLUE to LED_GREEN
Feb 13 11:13:25 kernel: [*02/13/2024 11:13:25.6121] Invalid event 56 & state 10 combination.
Feb 13 11:13:25 kernel: [*02/13/2024 11:13:25.6121] Failed to handle timer message.
Feb 13 11:13:28 kernel: [*02/13/2024 11:13:28.4629] Re-Tx Count=1, Max Re-Tx Value=5, SendSeqNum=5, NumofPendingMsgs=1
Feb 13 11:13:28 kernel: [*02/13/2024 11:13:28.4629] 
Feb 13 11:13:31 kernel: [*02/13/2024 11:13:31.3139] Re-Tx Count=2, Max Re-Tx Value=5, SendSeqNum=5, NumofPendingMsgs=1
Feb 13 11:13:31 kernel: [*02/13/2024 11:13:31.3139] 
Feb 13 11:13:34 kernel: [*02/13/2024 11:13:34.1648] Re-Tx Count=3, Max Re-Tx Value=5, SendSeqNum=5, NumofPendingMsgs=1
Feb 13 11:13:34 kernel: [*02/13/2024 11:13:34.1648] 
Feb 13 11:13:37 kernel: [*02/13/2024 11:13:37.0157] Re-Tx Count=4, Max Re-Tx Value=5, SendSeqNum=5, NumofPendingMsgs=1
Feb 13 11:13:37 kernel: [*02/13/2024 11:13:37.0157] 
Feb 13 11:13:39 kernel: [*02/13/2024 11:13:39.8666] Re-Tx Count=5, Max Re-Tx Value=5, SendSeqNum=5, NumofPendingMsgs=1
Feb 13 11:13:39 kernel: [*02/13/2024 11:13:39.8666] 
Feb 13 11:13:42 kernel: [*02/13/2024 11:13:42.7175] Max retransmission count exceeded, going back to DISCOVER mode.
Feb 13 11:13:42 kernel: [*02/13/2024 11:13:42.7175] Dropping msg CAPWAP_ECHO_REQUEST, type = 1, len = 0, eleLen = 8, sendSeqNum = 5
Feb 13 11:13:42 kernel: [*02/13/2024 11:13:42.7185] 
Feb 13 11:13:42 kernel: [*02/13/2024 11:13:42.7185] CAPWAP State: DTLS Teardown
Feb 13 11:13:42 kernel: [*02/13/2024 11:13:42.7303] Aborting image download(0x0): Dtls cleanup, ap1g6a
Feb 13 11:13:42 kernel: [*02/13/2024 11:13:42.7952] do ABORT, part1 is active part
Feb 13 11:13:42 upgrade: Cleanup tmp files ...
Feb 13 11:13:42 kernel: [*02/13/2024 11:13:42.8145] upgrade.sh: Cleanup tmp files ...   

Solution

There are multiple solutions to handle this:

Option 1. First, upgrade to an intermediate image that has the fix before moving to 17.7+. 1. Upgrade to 17.3.5 2. Proceed to upgrade to 17.7+.

Option 2. Manual DE Workaround: mount -o remount,size=100M /tmp/ This requires dev access.

Either of these options works and you might need help with TAC when working with Option 2.