Troubleshoot COS APs
Available Languages
Contents
Introduction
This document describes some of the troubleshooting tools available for APs running the COS operating system (Cheetah OS, Click OS, simply Cisco AP OS).
Prerequisites
Requirements
There are no specific requirements for this document.
Components Used
This document focuses on COS APs like APs models of the series 2800, 3800, 1560 and 4800, as well as new 11ax APs Catalyst 91xx.
This document focuses on many features available in AireOS 8.8 and later. And also Cisco IOS® XE 16.12.2s and later.
There can be comments about availability of certain features in prior releases.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Capture Packet Traces (Sniffer Traces)
Wired PCAP on AP Port
It is possible to take a pcap on the AP ethernet port. You can either display the result live on the CLI (with only summarized packet details) or save it as a full pcap in the AP flash.
The wired pcap captures everything on the ethernet side (both Rx/Tx) and the tap point inside the AP is imediately before the packet is put on wire.
However, it only captures AP CPU-plane traffic, which means traffic to and from the AP (AP DHCP, AP capwap control tunnel, ...) and does not show client traffic.
Note that the size is very limited (Max size limit of 5MB), so it can be required to configure filters to capture only the traffic you are interested in.
Ensure to stop the traffic capture with "no debug traffic wired ip capture" or simply "undebug all" before you try to copy it (otherwise the copy does not end as packets are still written).
Procedure
Step 1. Start the pcap; select the traffic type with “debug traffic wired ip capture”:
AP70DB.98E1.3DEC#debug traffic wired ip capture
% Writing packets to "/tmp/pcap/AP70DB.98E1.3DEC_capture.pcap0"
AP70DB.98E1.3DEC#reading from file /dev/click_wired_log, link-type EN10MB (Ethernet)
Step 2. Wait for the traffic to flow and then Stop the capture with the command "no debug traffic wired ip capture" or simply "undebug all":
AP70DB.98E1.3DEC#no debug traffic wired ip capture
Step 3. Copy the file to tftp/scp server:
AP70DB.98E1.3DEC#copy pcap AP70DB.98E1.3DEC_capture.pcap0 tftp 192.168.1.100
###################################################################################################################################################################### 100.0%
AP70DB.98E1.3DEC#
Step 4. Now you can open the file in wireshark. The file is pcap0. Change to pcap so that it automatically associates with wireshark.
Command Options
The debug traffic wired command has several options that can help you to capture specific traffic:
APC4F7.D54C.E77C#debug traffic wired
<0-3> wired debug interface number
filter filter packets with tcpdump filter string
ip Enable wired ip traffic dump
tcp Enable wired tcp traffic dump
udp Enable wired udp traffic dum
You can add "verbose" at the end of the debug command to see the hex dump of the packet. Be aware that this can overwhelm your CLI session very quickly if your filter is not narrow enough.
Wired PCAP through the use of Filter
The filter format corresponds with tcpdump capture filter format.
|
Filter Example |
Description |
|
|
Host |
“host 192.168.2.5” |
This filters the packet capture to only gather packets which go to or come from the host 192.168.2.5. |
|
“src host 192.168.2.5” |
This filters the packet capture to only gather packets that come from 192.168.2.5. |
|
|
“dst host 192.168.2.5“ |
This filters the packet capture to only gather packets which go to 192.168.2.5. |
|
|
Port |
“port 443” |
This filters the packet capture to only gather packets with a source or destination of port 443. |
|
“src port 1055” |
This captures traffic which is sourced from port 1055. |
|
|
“dst port 443” |
This captures traffic destined for port 443. |
Here is an example where the output displays on the console but also filtered to only see CAPWAP data packets:
APC4F7.D54C.E77C#debug traffic wired filter "port 5246"
APC4F7.D54C.E77C#reading from file /dev/click_wired_log, link-type EN10MB (Ethernet)
12:20:50.483125 IP APC4F7-D54C-E77C.lan.5264 > 192.168.1.15.5246: UDP, length 81
12:20:50.484361 IP 192.168.1.15.5246 > APC4F7-D54C-E77C.lan.5264: UDP, length 97
APC4F7.D54C.E77C#no debug traffic wired filter "port 5246"
APC4F7.D54C.E77C#Killed
APC4F7.D54C.E77C#
Example of output on File:
APC4F7.D54C.E77C#debug traffic wired filter "port 5246" capture
% Writing packets to "/tmp/pcap/APC4F7.D54C.E77C_capture.pcap0"
APC4F7.D54C.E77C#reading from file /dev/click_wired_log, link-type EN10MB (Ethernet)
APC4F7.D54C.E77C#no debug traffic wired filter "port 5246" capture
APC4F7.D54C.E77C#copy pcap APC4F7.D54C.E77C_capture.pcap0 tftp 192.168.1.100
###################################################################################################################################################################### 100.0%
APC4F7.D54C.E77C#
To open the capture on wireshark:
Radio Capture
It is possible to enable the capture of packets on the control-plane of the radio. Due to performance impact, it is not possible to capture on the radio dataplane.
This means that the client association flow (probes, authentication, association, eap, arp, dhcp packets as well as ipv6 control packets, icmp and ndp) is visible but not the data the client passes after the move to the connected state.
Procedure
Step 1. Add the tracked client mac address. Several mac addresses can be added. It is also possible to run the command for all clients but this is not recommended.
config ap client-trace address add < client-mac> --- Per client debugging. Allows multiple macs.
config ap client-trace all-clients <enable | disable> -- All clients debugging. Not recommended.
Step 2. Set a filter to only log specific protocols or all supported protocols:
config ap client-trace filter <all|arp|assoc|auth|dhcp|eap|icmp|ipv6|ndp|probe> <enable|disable>
Step 3. Chose to display the output on console (asynchronously):
configure ap client-trace output console-log enable
Step 4. Start the trace.
config ap client-trace start
Example:
AP0CD0.F894.46E4#show dot11 clients
Total dot11 clients: 1
Client MAC Slot ID WLAN ID AID WLAN Name RSSI Maxrate WGB
A8:DB:03:08:4C:4A 0 1 1 testewlcwlan -41 MCS92SS No
AP0CD0.F894.46E4#config ap client-trace address add A8:DB:03:08:4C:4A
AP0CD0.F894.46E4#config ap client-trace filter
all Trace ALL filters
arp Trace arp Packets
assoc Trace assoc Packets
auth Trace auth Packets
dhcp Trace dhcp Packets
eap Trace eap Packets
icmp Trace icmp Packets
ipv6 Trace IPv6 Packets
ndp Trace ndp Packets
probe Trace probe Packets
AP0CD0.F894.46E4#config ap client-trace filter all enable
AP0CD0.F894.46E4#configure ap client-trace output console-log enable
AP0CD0.F894.46E4#configure ap client-trace start
AP0CD0.F894.46E4#term mon
To stop the capture:
configure ap client-trace stop
configure ap client-trace clear
configure ap client-trace address clear
Verify
Verify Client Trace:
AP70DB.98E1.3DEC#show ap client-trace status
Client Trace Status : Started
Client Trace ALL Clients : disable
Client Trace Address : a8:db:03:08:4c:4a
Remote/Dump Client Trace Address : a8:db:03:08:4c:4a
Client Trace Filter : probe
Client Trace Filter : auth
Client Trace Filter : assoc
Client Trace Filter : eap
Client Trace Filter : dhcp
Client Trace Filter : dhcpv6
Client Trace Filter : icmp
Client Trace Filter : icmpv6
Client Trace Filter : ndp
Client Trace Filter : arp
Client Trace Output : eventbuf
Client Trace Output : console-log
Client Trace Output : dump
Client Trace Output : remote
Remote trace IP : 192.168.1.100
Remote trace dest port : 5688
NOTE - Only VIP packets are seen on remote if VIP is enabled
Dump packet length : 10
Client Trace Inline Monitor : disable
Client Trace Inline Monitor pkt-attach : disable
Example of a successful client connection:
The letters between brackets help you understand where that frame was seen (E for Ethernet, W for Wireless, C for the Click module when it is internal to the AP) and in which direction (Upload or Download).
Here is a small table of the meaning of those letters:
U - uplink packet(from client)
D - downlink packet(to click)
W - module wireless driver
E - module Ethernet driver
C - module Click
Note: As from 17.12.1 there is an addicional letter A added to help verifying if a packet was sent or not over the air:
[D:W] part to let aptrace know about mgmt/data pkts handed off to driver successfully by kclick
[D:A] part to let aptrace know about corresponding mgmt/data pkts tx completion status when the pkt is sent over the air
Other Options
View Log asyncronously:
The logs can then be consulted with the command: "show ap client-trace events mac xx:xx:xx:xx:xx:xx" (or replace the mac with "all")
AP0CD0.F894.46E4#show ap client-trace events mac a8:db:03:08:4c:4a
[*04/06/2020 10:11:54.287675] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr1v1> [U:W] DOT11_AUTHENTICATION : (.)
[*04/06/2020 10:11:54.288144] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr1v0> [D:W] DOT11_AUTHENTICATION : (.)
[*04/06/2020 10:11:54.289870] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr1v0> [U:W] DOT11_ASSOC_REQUEST : (.)
[*04/06/2020 10:11:54.317341] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr1v0> [D:W] DOT11_ASSOC_RESPONSE : (.)
[*04/06/2020 10:11:54.341370] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr1v0> [D:W] EAPOL_KEY.M1 : DescType 0x02 KeyInfo 0x008b
[*04/06/2020 10:11:54.374500] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr1v0> [U:W] EAPOL_KEY.M2 : DescType 0x02 KeyInfo 0x010b
[*04/06/2020 10:11:54.377237] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr1v0> [D:W] EAPOL_KEY.M3 : DescType 0x02 KeyInfo 0x13cb
[*04/06/2020 10:11:54.390255] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr1v0> [U:W] EAPOL_KEY.M4 : DescType 0x02 KeyInfo 0x030b
[*04/06/2020 10:11:54.396855] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr1v0> [U:W] DOT11_ACTION : (.)
[*04/06/2020 10:11:54.416650] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr1v0> [D:W] DOT11_ACTION : (.)
[*04/06/2020 10:11:54.469089] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr1v0> [U:W] DOT11_ACTION : (.)
[*04/06/2020 10:11:54.469157] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr1v0> [D:W] DOT11_ACTION : (.)
[*04/06/2020 10:11:57.921877] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr1v0> [U:W] DOT11_ACTION : (.)
[*04/06/2020 10:11:57.921942] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr1v0> [D:W] DOT11_ACTION : (.)
[*04/06/2020 10:15:36.123119] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr1v0> [D:W] DOT11_DEAUTHENTICATION : (.)
[*04/06/2020 10:15:36.127731] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr1v0> [D:W] DOT11_DISASSOC : (.)
[*04/06/2020 10:17:24.128751] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr0v0> [U:W] DOT11_AUTHENTICATION : (.)
[*04/06/2020 10:17:24.128870] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr0v1> [U:W] DOT11_AUTHENTICATION : (.)
[*04/06/2020 10:17:24.129303] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr0v0> [D:W] DOT11_AUTHENTICATION : (.)
[*04/06/2020 10:17:24.133026] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr0v0> [U:W] DOT11_ASSOC_REQUEST : (.)
[*04/06/2020 10:17:24.136095] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr0v0> [D:W] DOT11_ASSOC_RESPONSE : (.)
[*04/06/2020 10:17:24.138732] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr0v0> [D:W] EAPOL_KEY.M1 : DescType 0x02 KeyInfo 0x008b
[*04/06/2020 10:17:24.257295] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr0v0> [U:W] EAPOL_KEY.M2 : DescType 0x02 KeyInfo 0x010b
[*04/06/2020 10:17:24.258105] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr0v0> [D:W] EAPOL_KEY.M3 : DescType 0x02 KeyInfo 0x13cb
[*04/06/2020 10:17:24.278937] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr0v0> [U:W] EAPOL_KEY.M4 : DescType 0x02 KeyInfo 0x030b
[*04/06/2020 10:17:24.287459] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr0v0> [U:W] DOT11_ACTION : (.)
[*04/06/2020 10:17:24.301344] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr0v0> [D:W] DOT11_ACTION : (.)
[*04/06/2020 10:17:24.327482] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr0v0> [U:W] DOT11_ACTION : (.)
[*04/06/2020 10:17:24.327517] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr0v0> [D:W] DOT11_ACTION : (.)
[*04/06/2020 10:17:24.430136] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr0v0> [U:W] DOT11_ACTION : (.)
[*04/06/2020 10:17:24.430202] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr0v0> [D:W] DOT11_ACTION : (.)
[*04/06/2020 10:19:08.075326] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr0v0> [U:W] DOT11_PROBE_REQUEST : (.)
[*04/06/2020 10:19:08.075392] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr0v0> [D:W] DOT11_PROBE_RESPONSE : (.)
[*04/06/2020 10:19:08.075437] [AP0CD0.F894.46E4] [a8:db:03:08:4c:4a] <apr0v1> [U:W] DOT11_PROBE_REQUEST : (.)
Dump the packets in hex format
You can dump the packets in hex format in the CLI:
configure ap client-trace output dump address add xx:xx:xx:xx:xx:xx
configure ap client-trace output dump enable x -> Enter the packet dump length value
Then, you can clean the hex dump and save in txt format and import into wireshark:
Because the output can be very large and to consider that the output only mentions what frame type is seen and not any of the inner detail, it can be more efficient to redirect the packet capture to a laptop that run a a capture application (such as wireshark).
Enable the remote capture feature to send the packets to external device with wireshark:
config ap client-trace output remote enable
The command means the AP forwards every frame captured by the client-trace filter towards the laptop at 192.168.68.68 and uses PEEKREMOTE encapsulation (just like APs in sniffer mode) on port 5000.
One limitation is that the target laptop has to be in the same subnet as the AP where you run this command on. You can change the port number to accomodate any security policies in place in your network.
Once you received all the packets on the laptop that runs Wireshark, you can right click on the udp 5000 header and chose decode as and pick PEEKREMOTE as illustrated in this figure:
List of bugs and enhancements around this feature :
Cisco bug ID CSCvm09020
DNS not seen by client trace anymore on 8.8
Cisco bug ID CSCvm09015 client trace shows many ICMP_other with null sequence number
Cisco bug ID CSCvm02676 AP COS client-trace does not capture webauth packets
Cisco Bug ID CSCvm02613 AP COS client-trace remote output does not work
Cisco Bug ID CSCvm00855 lient-trace SEQ numbers inconsistent
Control the AP Client trace from the 9800 WLC
You can configure several APs to do a radio client trace and trigger it from the
Step 1. Configure an AP trace profile that defines which traffic to capture
config term
wireless profile ap trace <TRACE-NAME>
filter all
no filter probe
output console-log
Step 2. Add the AP trace profile to an AP join profile that is used by the APs that you target.
ap profile < ap join profile name>
trace <TRACE-NAME>Ensure that this ap join profile is applied to a site tag that is used by your target APs
Step 4 Trigger start/stop
ap trace client start ap <ap name> client all/<mac>
ap trace client stop ap <ap name> client all/<mac>
ap trace client start site <site tag> client all/<mac>
ap trace client stop site <site tag> client all/<mac>
Verification commands :
show wireless profile ap trace summary
show wireless profile ap trace detailed PROF_NAME detail
sh ap trace client summary
show ap trace unsupported-ap summaryClient debug bundle on the AP
Rather than collecting a radio debug/capture, it may be easier to use the client debug bundle feature if you are debugging one or more specific clients.
Step 1. Identify which client you want to troubleshoot.
9164#show dot11 clients
Total dot11 clients: 6
Client MAC Slot ID WLAN ID AID WLAN Name RSSI Maxrate is_wgb_wired is_
mld_sta
52:1E:34:C9:D6:F3 1 2 35 MySSID -62 M7 No
No
80:A9:97:2C:DC:6E 1 2 34 MySSID -47 MCS112SS No
No
E8:D8:D1:1F:71:F3 0 2 35 MySSID -62 M7 No
No
6A:E4:06:E7:AB:E1 1 2 33 MySSID -44 MCS112SS No
No
00:1D:63:70:AC:23 0 2 33 MySSID -56 M7 No
No
68:72:C3:FD:17:F5 0 2 34 MySSID -53 M15 No
No
Step 2. Start the debug for one or more client MAC address(es)
9164#debug client-bundle start debug 80:A9:97:2C:DC:6E
WORD <mac-addr2>
<cr>By default, nothing will be printed on screen. You can enable terminal monitor and see the debugs being printed live, but be aware that this wil make the terminal very hard to use. It's not required to print the debugs on the terminal in order to collect the bundle.
Step 3. You have to stop the debug bundle before you can upload the output of it :
debug client-bundle start debug 80:A9:97:2C:DC:6EStep 4. Upload the bundle to an FTP or SCP server (as reminder the WLC can act as SCP server)
9164#debug client-bundle upload tftp 192.168.129.29 80:a9:97:2c:dc:6e
2024-09-04 11:58:48 Creating client bundle, please wait...
2024-09-04 11:59:01 Client bundle file 9164-_client_bundle.17.15.1.6.20240904.115848.tgz created.
2024-09-04 11:59:01 TFTP uploading...
Successful file transfer:
9164_client_bundle.17.15.1.6.20240904.115848.tgz
9164#The TGZ bundle contains 4 files :
- 2 containing show commands pertaining to the radios and client
- 1 about the actual debugging (which is displayed on terminal if you do term mon)
- 1 containing syslogs
APs Catalyst 91xx in Sniffer Mode
The new Catalyst 9115, 9117, 9120 and 9130 can be configured in sniffer mode. The procedure is simular to previous AP models.
Troubleshooting Tips
Path MTU
Although Path MTU discovery finds the optimal MTU for the AP, it is possible to override this settings manually.
On AireOS 8.10.130 WLC, the command config ap pmtu disable <ap/all> sets a static MTU for one or all APs rather than to rely on the dynamic discovery mechanism.
To enable debugs at boot time
You can run config boot debug capwap to enable capwap,DTLS and DHCP debugs at the next boot time, even before the OS has booted and the prompt is shown.
You also have "config boot debug memory xxxx" for several memory debugs.
You can see if boot debugs are enabled or not at next reboot with "show boot".
They can be disabled with the addition of the disable keyword at the end such as "config boot debug capwap disable".
Power save mechanism
The power save of a given client can be troubleshot by running
debug client trace <mac address>
Clients QoS
To verify that QoS tags are applied, you can run "debug capwap client qos".
It displays the UP value of packets for wireless clients.
It is not mac filterable as of 8.8 ; enhancement request Cisco bug IDCSCvm08899 ).
labAP#debug capwap client qos [*08/20/2018 09:43:36.3171] chatter: set_qos_up :: SetQosPriority: bridged packet dst: 00:AE:FA:78:36:89 UP: 0 [*08/20/2018 09:43:45.0051] chatter: set_qos_up :: SetQosPriority: bridged packet dst: 00:AE:FA:78:36:89 UP: 0 [*08/20/2018 09:43:45.5463] chatter: set_qos_up :: SetQosPriority: bridged packet dst: 00:AE:FA:78:36:89 UP: 0 [*08/20/2018 09:43:46.5687] chatter: set_qos_up :: SetQosPriority: bridged packet dst: AC:81:12:C7:CD:35 UP: 5 [*08/20/2018 09:43:47.0982] chatter: set_qos_up :: SetQosPriority: bridged packet dst: AC:81:12:C7:CD:35 UP: 5
You can also verify the Qos UP to DSCP table on the AP as well as total amount of packets marked, shaped and dropped by Qos:
LabAP#show dot11 qos Qos Policy Maps (UPSTREAM) no policymap Qos Stats (UPSTREAM) total packets: 0 dropped packets: 0 marked packets: 0 shaped packets: 0 policed packets: 0 copied packets: 0 DSCP TO DOT1P (UPSTREAM) Default dscp2dot1p Table Value: [0]->0 [1]->2 [2]->10 [3]->18 [4]->26 [5]->34 [6]->46 [7]->48 Active dscp2dot1p Table Value: [0]->0 [1]->2 [2]->10 [3]->18 [4]->26 [5]->34 [6]->46 [7]->48 Qos Policy Maps (DOWNSTREAM) no policymap Qos Stats (DOWNSTREAM) total packets: 0 dropped packets: 0 marked packets: 0 shaped packets: 0 policed packets: 0 copied packets: 0 DSCP TO DOT1P (DOWNSTREAM) Default dscp2dot1p Table Value: [0]->0 [1]->-1 [2]->1 [3]->-1 [4]->1 [5]->-1 [6]->1 [7]->-1 [8]->-1 [9]->-1 [10]->2 [11]->-1 [12]->2 [13]->-1 [14]->2 [15]->-1 [16]->-1 [17]->-1 [18]->3 [19]->-1 [20]->3 [21]->-1 [22]->3 [23]->-1 [24]->-1 [25]->-1 [26]->4 [27]->-1 [28]->-1 [29]->-1 [30]->-1 [31]->-1 [32]->-1 [33]->-1 [34]->5 [35]->-1 [36]->-1 [37]->-1 [38]->-1 [39]->-1 [40]->-1 [41]->-1 [42]->-1 [43]->-1 [44]->-1 [45]->-1 [46]->6 [47]->-1 [48]->7 [49]->-1 [50]->-1 [51]->-1 [52]->-1 [53]->-1 [54]->-1 [55]->-1 [56]->7 [57]->-1 [58]->-1 [59]->-1 [60]->-1 [61]->-1 [62]->-1 [63]->-1 Active dscp2dot1p Table Value: [0]->0 [1]->-1 [2]->1 [3]->-1 [4]->1 [5]->-1 [6]->1 [7]->-1 [8]->-1 [9]->-1 [10]->2 [11]->-1 [12]->2 [13]->-1 [14]->2 [15]->-1 [16]->-1 [17]->-1 [18]->3 [19]->-1 [20]->3 [21]->-1 [22]->3 [23]->-1 [24]->-1 [25]->-1 [26]->4 [27]->-1 [28]->-1 [29]->-1 [30]->-1 [31]->-1 [32]->-1 [33]->-1 [34]->5 [35]->-1 [36]->-1 [37]->-1 [38]->-1 [39]->-1 [40]->-1 [41]->-1 [42]->-1 [43]->-1 [44]->-1 [45]->-1 [46]->6 [47]->-1 [48]->7 [49]->-1 [50]->-1 [51]->-1 [52]->-1 [53]->-1 [54]->-1 [55]->-1 [56]->7 [57]->-1 [58]->-1 [59]->-1 [60]->-1 [61]->-1 [62]->-1 [63]->-1 LabAP#
When Qos policies are defined on the WLC and downloaded on the Flexconnect AP, you can verify them with :
AP780C-F085-49E6#show policy-map
2 policymaps
Policy Map BWLimitAAAClients type:qos client:default
Class BWLimitAAAClients_AVC_UI_CLASS
drop
Class BWLimitAAAClients_ADV_UI_CLASS
set dscp af41 (34)
Class class-default
police rate 5000000 bps (625000Bytes/s)
conform-action
exceed-action
Policy Map platinum-up type:qos client:default
Class cm-dscp-set1-for-up-4
set dscp af41 (34)
Class cm-dscp-set2-for-up-4
set dscp af41 (34)
Class cm-dscp-for-up-5
set dscp af41 (34)
Class cm-dscp-for-up-6
set dscp ef (46)
Class cm-dscp-for-up-7
set dscp ef (46)
Class class-default
no actions
In case of Qos rate-limiting :
AP780C-F085-49E6#show rate-limit client
Config:
mac vap rt_rate_out rt_rate_in rt_burst_out rt_burst_in nrt_rate_out nrt_rate_in nrt_burst_out nrt_burst_in
A8:DB:03:6F:7A:46 2 0 0 0 0 0 0 0 0
Statistics:
name up down
Unshaped 0 0
Client RT pass 0 0
Client NRT pass 0 0
Client RT drops 0 0
Client NRT drops 0 38621
9 54922 0
Off-Channel scan
Debugging the off-channel scan of the AP can be useful when troubleshooting rogue detection (to validate if and when the AP goes on a specific channel to scan), but can also be useful in video troubleshoot where a sensitive real-time stream gets constant interruptions if the "off channel scan defer" feature is not used.
debug rrm off-channel defer
debug rrm off-chanel dbg (starting 17.8.1) debug rrm off-channel schedule
debug rrm off-channel voice (starting 17.8.1)
debug rrm schedule (starting 17.8.1, debug NDP packet tx) show trace dot_11 channel enable
[*06/11/2020 09:45:38.9530] wcp/rrm_userspace_0/rrm_schedule :: RRMSchedule process_int_duration_timer_local rrm_is_msmt_schedule_local passive scan on channel 5 bandwidth 20MHz
[*06/11/2020 09:45:39.0550] noise measurement channel 5 noise 89
[*06/11/2020 09:45:43.5490] wcp/rrm_userspace_1/rrm_schedule :: RRMSchedule process_int_duration_timer_local rrm_is_msmt_schedule_local passive scan on channel 140 bandwidth 20MHz
[*06/11/2020 09:45:43.6570] noise measurement channel 140 noise 97
Client Connectivity
It is possible to list clients that have been deauthenticated by the access point with the last event timestamp:
LabAP#show dot11 clients deauth
timestamp mac vap reason_code
Mon Aug 20 09:50:59 2018 AC:BC:32:A4:2C:D3 9 4
Mon Aug 20 09:52:14 2018 00:AE:FA:78:36:89 9 4
Mon Aug 20 10:31:54 2018 00:AE:FA:78:36:89 9 4In the previous output, the reason code is the deauthentication reason code as detailed in this link :
The vap refers to the identifier of the WLAN inside the AP (which is different from the WLAN ID on the WLC !!!).
You can cross-relate it with other outputs detailed subsequently which always mentions the vap of associated clients.
You can see the list of VAP ids with "show controllers Dot11Radio 0/1 wlan".
When clients are still associated, you can get details on their connection with:
LabAP#show dot11 clients
Total dot11 clients: 1
Client MAC Slot ID WLAN ID AID WLAN Name RSSI Maxrate WGB
00:AE:FA:78:36:89 1 10 1 TestSSID -25 MCS82SS No
A lot more details can be obtained about the client entry with:
LabAP#show client summ
Radio Driver client Summary:
==============================
wifi0
[*08/20/2018 11:54:59.5340]
[*08/20/2018 11:54:59.5340] Total STA List Count 0
[*08/20/2018 11:54:59.5340] | NO| MAC|STATE|
[*08/20/2018 11:54:59.5340] -----------------------------
wifi1
[*08/20/2018 11:54:59.5357]
[*08/20/2018 11:54:59.5357] Total STA List Count 1
[*08/20/2018 11:54:59.5357] | NO| MAC|STATE|
[*08/20/2018 11:54:59.5357] -----------------------------
[*08/20/2018 11:54:59.5357] | 1| 0:ffffffae:fffffffa:78:36:ffffff89| 8|
Radio Driver Client AID List:
==============================
wifi0
[*08/20/2018 11:54:59.5415]
[*08/20/2018 11:54:59.5415] Total STA-ID List Count 0
[*08/20/2018 11:54:59.5415] | NO| MAC|STA-ID|
[*08/20/2018 11:54:59.5415] ------------------------------
wifi1
[*08/20/2018 11:54:59.5431]
[*08/20/2018 11:54:59.5431] Total STA-ID List Count 1
[*08/20/2018 11:54:59.5431] | NO| MAC|STA-ID|
[*08/20/2018 11:54:59.5432] ------------------------------
[*08/20/2018 11:54:59.5432] | 1| 0:ffffffae:fffffffa:78:36:ffffff89| 6|
WCP client Summary:
=====================
mac radio vap aid state encr Maxrate is_wgb_wired wgb_mac_addr
00:AE:FA:78:36:89 1 9 1 FWD AES_CCM128 MCS82SS false 00:00:00:00:00:00
NSS client Summary:
=====================
Current Count: 3
| MAC | OPAQUE |PRI POL|VLAN|BR|TN|QCF|BSS|RADID|MYMAC|
|F8:0B:CB:E4:7F:41|00000000| 3| 0| 1| 1| 0| 2| 3| 1|
|F8:0B:CB:E4:7F:40|00000000| 3| 0| 1| 1| 0| 2| 3| 1|
|00:AE:FA:78:36:89|00000003| 1| 0| 1| 1| 0| 9| 1| 0|
Datapath IPv4 client Summary:
===============================
id vap port node tunnel mac seen_ip hashed_ip sniff_ago confirm_ago
00:AE:FA:78:36:89 9 apr1v9 192.0.2.13 - 00:AE:FA:78:36:89 192.168.68.209 10.228.153.45 5.990000 5.980000
Datapath IPv6 client Summary:
===============================
client mac seen_ip6 age scope port
1 00:AE:FA:78:36:89 fe80::2ae:faff:fe78:3689 61 link-local apr1v9
Wired client Summary:
=======================
mac port state local_client detect_ago associated_ago tx_pkts tx_bytes rx_pkts rx_bytesYou can force the disconnection of a specific client with :
test dot11 client deauthenticate Traffic counters can be obtained per-client with:
LabAP#show client statistics wireless 00:AE:FA:78:36:89 Client MAC address: 00:AE:FA:78:36:89 Tx Packets : 621 Tx Management Packets : 6 Tx Control Packets : 153 Tx Data Packets : 462 Tx Data Bytes : 145899 Tx Unicast Data Packets : 600 Rx Packets : 2910 Rx Management Packets : 13 Rx Control Packets : 943 Rx Data Packets : 1954 Rx Data Bytes : 145699 LabAP#
More on the radio level, a lot of information can be obtained in the "show controllers". When you add the client mac address, the supported data rates, current data reates, PHY capabilities as well as amount of retries and txfails, are displayed:
LabAP#show controllers dot11Radio 0 client 00:AE:FA:78:36:89
mac radio vap aid state encr Maxrate is_wgb_wired wgb_mac_addr
00:AE:FA:78:36:89 0 9 1 FWD AES_CCM128 M15 false 00:00:00:00:00:00
Configured rates for client 00:AE:FA:78:36:89
Legacy Rates(Mbps): 11
HT Rates(MCS):M0 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 M13 M14 M15
VHT Rates: 1SS:M0-7 2SS:M0-7
HT:yes VHT:yes HE:no 40MHz:no 80MHz:no 80+80MHz:no 160MHz:no
11w:no MFP:no 11h:no encrypt_polocy: 4
_wmm_enabled:yes qos_capable:yes WME(11e):no WMM_MIXED_MODE:no
short_preamble:yes short_slot_time:no short_hdr:yes SM_dyn:yes
short_GI_20M:yes short_GI_40M:no short_GI_80M:yes LDPC:yes AMSDU:yes AMSDU_long:no
su_mimo_capable:yes mu_mimo_capable:no is_wgb_wired:no is_wgb:no
Additional info for client 00:AE:FA:78:36:89
RSSI: -90
PS : Legacy (Sleeping)
Tx Rate: 0 Kbps
Rx Rate: 117000 Kbps
VHT_TXMAP: 0
CCX Ver: 4
Statistics for client 00:AE:FA:78:36:89
mac intf TxData TxMgmt TxUC TxBytes TxFail TxDcrd TxCumRetries RxData RxMgmt RxBytes RxErr TxRt RxRt idle_counter stats_ago expiration
00:AE:FA:78:36:89 apr0v9 8 1 6 1038 1 0 0 31 1 1599 0 0 117000 290 13.952000 0
Per TID packet statistics for client 00:AE:FA:78:36:89
Priority Rx Pkts Tx Pkts Rx(last 5 s) Tx (last 5 s) QID Tx Drops Tx Cur Qlimit
0 899 460 1 1 144 0 0 1024
1 0 0 0 0 145 0 0 1024
2 0 0 0 0 146 0 0 1024
3 59 0 0 0 147 0 0 1024
4 0 0 0 0 148 0 0 1024
5 0 0 0 0 149 0 0 1024
6 0 0 0 0 150 0 0 1024
7 0 0 0 0 151 0 0 1024
Legacy Rate Statistics:
(Mbps : Rx, Tx, Tx-Retries)
11 Mbps : 2, 0, 0
6 Mbps : 0, 9, 0
HT/VHT Rate Statistics:
(Rate/SS/Width : Rx, Rx-Ampdu, Tx, Tx-Ampdu, Tx-Retries)
0/1/20 : 4, 4, 0, 0, 0
6/2/20 : 4, 4, 0, 0, 0
7/2/20 : 5, 5, 0, 0, 0
webauth done:
falseIn order to constantly keep track of a client data rate and/or RSSI value, you can run "debug dot11 client rate address <mac> " and this logs this information every second:
LabAP#debug dot11 client rate address 00:AE:FA:78:36:89 [*08/20/2018 14:17:28.0928] MAC Tx-Pkts Rx-Pkts Tx-Rate Rx-Rate RSSI SNR Tx-Retries [*08/20/2018 14:17:28.0928] 00:AE:FA:78:36:89 0 0 12 a8.2-2s -45 53 0 [*08/20/2018 14:17:29.0931] 00:AE:FA:78:36:89 7 18 12 a8.2-2s -45 53 0 [*08/20/2018 14:17:30.0934] 00:AE:FA:78:36:89 3 18 12 a8.2-2s -45 53 0 [*08/20/2018 14:17:31.0937] 00:AE:FA:78:36:89 2 20 12 a8.2-2s -45 53 0 [*08/20/2018 14:17:32.0939] 00:AE:FA:78:36:89 2 20 12 a8.2-2s -45 53 0 [*08/20/2018 14:17:33.0942] 00:AE:FA:78:36:89 2 21 12 a8.2-2s -46 52 0 [*08/20/2018 14:17:34.0988] 00:AE:FA:78:36:89 1 4 12 a8.2-2s -46 52 0 [*08/20/2018 14:17:35.0990] 00:AE:FA:78:36:89 9 23 12 a8.2-2s -46 52 0 [*08/20/2018 14:17:36.0993] 00:AE:FA:78:36:89 3 7 12 a8.2-2s -46 52 0 [*08/20/2018 14:17:37.0996] 00:AE:FA:78:36:89 2 6 12 a8.2-2s -46 52 0 [*08/20/2018 14:17:38.0999] 00:AE:FA:78:36:89 2 14 12 a8.2-2s -46 52 0 [*08/20/2018 14:17:39.1002] 00:AE:FA:78:36:89 2 10 12 a8.2-2s -46 52 0 [*08/20/2018 14:17:40.1004] 00:AE:FA:78:36:89 1 6 12 a8.2-2s -46 52 0 [*08/20/2018 14:17:41.1007] 00:AE:FA:78:36:89 9 20 12 a8.2-2s -46 52 0 [*08/20/2018 14:17:42.1010] 00:AE:FA:78:36:89 0 0 12 a8.2-2s -46 52 0 [*08/20/2018 14:17:43.1013] 00:AE:FA:78:36:89 2 8 12 a8.2-2s -46 52 0 [*08/20/2018 14:17:44.1015] 00:AE:FA:78:36:89 0 0 12 a8.2-2s -46 52 0 [*08/20/2018 14:17:45.1018] 00:AE:FA:78:36:89 0 0 12 a8.2-2s -46 52 0 [*08/20/2018 14:17:46.1021] 00:AE:FA:78:36:89 0 0 12 a8.2-2s -46 52 0 [*08/20/2018 14:17:47.1024] 00:AE:FA:78:36:89 0 0 12 a8.2-2s -46 52 0 [*08/20/2018 14:17:48.1026] 00:AE:FA:78:36:89 7 15 12 a8.2-2s -46 52 0 [*08/20/2018 14:17:49.1029] 00:AE:FA:78:36:89 0 6 12 a8.2-2s -46 52 0 [*08/20/2018 14:17:50.1032] 00:AE:FA:78:36:89 0 0 12 a8.2-2s -46 52 0 [*08/20/2018 14:17:51.1035] 00:AE:FA:78:36:89 1 7 12 a8.2-2s -46 52 0 [*08/20/2018 14:17:52.1037] 00:AE:FA:78:36:89 0 17 12 a8.2-2s -46 52 0 [*08/20/2018 14:17:53.1040] 00:AE:FA:78:36:89 1 19 12 a8.2-2s -46 52 0 [*08/20/2018 14:17:54.1043] 00:AE:FA:78:36:89 2 17 12 a8.2-2s -46 52 0 [*08/20/2018 14:17:55.1046] 00:AE:FA:78:36:89 2 22 12 a8.2-2s -45 53 0 [*08/20/2018 14:17:56.1048] 00:AE:FA:78:36:89 1 18 12 a8.2-2s -45 53 0 [*08/20/2018 14:17:57.1053] 00:AE:FA:78:36:89 2 18 12 a8.2-2s -45 53 0 [*08/20/2018 14:17:58.1055] 00:AE:FA:78:36:89 12 37 12 a8.2-2s -45 53 0
In this output, the Tx and Rx packet counters are packets transmitted in the second interval since it last printed, same thing for the Tx Retries. However the RSSI, SNR and data rate are the values from the last packet of that interval (and not an average for all packets in that interval).
Flexconnect scenarios
You can verify what ACLs are currently applied to a client in a pre-auth (CWA for example) or post-auth scenario:
AP#show client access-lists pre-auth all f48c.507a.b9ad Pre-Auth URL ACLs for Client: F4:8C:50:7A:B9:AD IPv4 ACL: IPv6 ACL: ACTION URL-LIST Resolved IPs for Client: F4:8C:50:7A:B9:AD HIT-COUNT URL ACTION IP-LIST REDIRECT rule 0: allow true and ip proto 17 and src port 53 rule 1: allow true and ip proto 17 and dst port 53 rule 2: allow true and src 10.48.39.161mask 255.255.255.255 rule 3: allow true and dst 10.48.39.161mask 255.255.255.255 rule 4: deny true No IPv6 ACL found AP#show client access-lists post-auth all f48c.507a.b9ad Post-Auth URL ACLs for Client: F4:8C:50:7A:B9:AD IPv4 ACL: IPv6 ACL: ACTION URL-LIST Resolved IPs for Client: F4:8C:50:7A:B9:AD HIT-COUNT URL ACTION IP-LIST post-auth rule 0: deny true and dst 192.0.0.0mask 255.0.0.0 rule 1: deny true and src 192.0.0.0mask 255.0.0.0 rule 2: allow true No IPv6 ACL found
You can see the hit counters on the Flexconnect ACLs by enabling debug flexconnect access-list counter client <client MAC>
Subsequent executions of the show client access-list pre-auth/post-auth all <MAC> then add hit counters for each ACL entry. This works for all types of flex ACLs as of Cisco IOS® XE 17.13. In earlier versions, the same commands exist but only VLAN ACLs have their hit counters updated.
You can reset the ACL hit counters with clear counters access-list client <mac>
AP Filesystem
COS APs do not allow to list all the content of the file system as on unix platforms.
The command "show filesystems" gives a detail of the space usage and distribution on the current partition:
2802#show filesystems Filesystem Size Used Available Use% Mounted on /dev/ubivol/storage 57.5M 364.0K 54.1M 1% /storage 2802#
The command "show flash" lists the main files on the AP flash. You can also append the syslog or core keyword to list those specific folders.
ap_2802#show flash Directory of /storage/ total 84 -rw-r--r-- 1 root root 0 May 21 2018 1111 -rw-r--r-- 1 root root 6 Apr 15 11:09 BOOT_COUNT -rw-r--r-- 1 root root 6 Apr 15 11:09 BOOT_COUNT.reserve -rw-r--r-- 1 root root 29 Apr 15 11:09 RELOADED_AT_UTC drwxr-xr-x 2 root root 160 Mar 27 13:53 ap-images drwxr-xr-x 4 5 root 2016 Apr 15 11:10 application -rw-r--r-- 1 root root 6383 Apr 26 09:32 base_capwap_cfg_info -rw-r--r-- 1 root root 20 Apr 26 10:31 bigacl -rw-r--r-- 1 root root 1230 Mar 27 13:53 bootloader.log -rw-r--r-- 1 root root 5 Apr 26 09:29 bootloader_verify.shadow -rw-r--r-- 1 root root 18 Jun 30 2017 config -rw-r--r-- 1 root root 8116 Apr 26 09:32 config.flex -rw-r--r-- 1 root root 21 Apr 26 09:32 config.flex.mgroup -rw-r--r-- 1 root root 0 Apr 15 11:09 config.local -rw-r--r-- 1 root root 0 Jul 26 2018 config.mesh.dhcp -rw-r--r-- 1 root root 180 Apr 15 11:10 config.mobexp -rw-r--r-- 1 root root 0 Jun 5 2018 config.oeap -rw-r--r-- 1 root root 2253 Apr 26 09:43 config.wireless drwxr-xr-x 2 root root 160 Jun 30 2017 cores drwxr-xr-x 2 root root 320 Jun 30 2017 dropbear drwxr-xr-x 2 root root 160 Jun 30 2017 images -rw-r--r-- 1 root root 222 Jan 2 2000 last_good_uplink_config drwxr-xr-x 2 root root 160 Jun 30 2017 lists -rw-r--r-- 1 root root 215 Apr 16 11:01 part1_info.ver -rw-r--r-- 1 root root 215 Apr 26 09:29 part2_info.ver -rw-r--r-- 1 root root 4096 Apr 26 09:36 random_seed -rw-r--r-- 1 root root 3 Jun 30 2017 rxtx_mode -rw-r--r-- 1 root root 64 Apr 15 11:11 sensord_CSPRNG0 -rw-r--r-- 1 root root 64 Apr 15 11:11 sensord_CSPRNG1 drwxr-xr-x 3 support root 224 Jun 30 2017 support drwxr-xr-x 2 root root 2176 Apr 15 11:10 syslogs --------------------------------------------------------------------------- Filesystem Size Used Available Use% Mounted on flash 57.5M 372.0K 54.1M 1% /storage
Store and send syslogs
The syslog folder stores the syslog output from previous reboots. The command "show log" only shows syslog since the last reboot.
At each reboot cycle, the syslogs are written on incremental files.
artaki# show flash syslogs Directory of /storage/syslogs/ total 128 -rw-r--r-- 1 root root 11963 Jul 6 15:23 1 -rw-r--r-- 1 root root 20406 Jan 1 2000 1.0 -rw-r--r-- 1 root root 313 Jul 6 15:23 1.last_write -rw-r--r-- 1 root root 20364 Jan 1 2000 1.start -rw-r--r-- 1 root root 33 Jul 6 15:23 1.watchdog_status -rw-r--r-- 1 root root 19788 Jul 6 16:46 2 -rw-r--r-- 1 root root 20481 Jul 6 15:23 2.0 -rw-r--r-- 1 root root 313 Jul 6 16:46 2.last_write -rw-r--r-- 1 root root 20422 Jul 6 15:23 2.start --------------------------------------------------------------------------- Filesystem Size Used Available Use% Mounted on flash 57.6M 88.0K 54.5M 0% /storage artaki# show flash cores Directory of /storage/cores/ total 0 --------------------------------------------------------------------------- Filesystem Size Used Available Use% Mounted on flash 57.6M 88.0K 54.5M 0% /storage
The first output after initial boot is file 1.0 and a file 1.1 is created if 1.0 becomes too long. After reboot, a new file 2.0 is created and so on.
From the WLC, you can configure the Syslog destination if you want your APs to send their syslog messages unicast to a specific server.
By default, APs send their syslogs to a broadcast address which can cause quite some broadcast storm, so ensure to configure a syslog server.
The AP sends via syslog by default whatever prints on its console output.
On 9800 Controller, you can change these parameters in the Configuration -> AP Join profile, under Management.
You can change the Log Trap Value to also send debugs via syslog. You can then enable debugs on the AP CLI and the output of these are sent via syslog messages to your configured server .
Due to Cisco Bug ID CSCvu75017 ,only when you set the syslog facility to KERN (the default value) does the AP send syslog messages out.
If you are troubleshooting issues where an AP possibly loses network connectivity (or on a WGB for example), syslog is not as reliable as no messages are sent if the AP loses its uplink connectivity.
Therefore, reliance on the stored syslog files in flash is a great way to debug and store the output on the AP itself and then periodically upload it later on.
AP Support Bundle
Some commonly collected diagnostic information of various types can be made available in a single bundle that you can upload from Access Points.
The diagnostic information that can you can include in the bundle are:
- AP show tech
- AP syslogs
- AP Capwapd Brain logs
- AP Startup & Message logs
- AP Coredump files
To get the AP support bundle you can go into the AP CLI and enter the command "copy support-bundle tftp: x.x.x.x".
After this you can check for the file named with AP name appended with the support.apversion.date.time.tgz as shown subsequently :
When you "untar" the file you can view the various files collected:
Collect AP Core Files Remotely
To collect AP core files remotely, please enable core dump to be included in support bundle and then Upload support bundle from the AP, or send directly to tftp server. The subsequent examples use tftp server 192.168.1.100.
AireOS CLI
AireOS GUI
Cisco IOS® CLI
eWLC-9800-01(config)#ap profile TiagoOffice
eWLC-9800-01(config-ap-profile)#core-dump tftp-server 192.168.1.100 file apCores uncompress
Cisco IOS® GUI
As from Cisco IOS® XE 17.3.1, you have a Support Bundle tab and can download the AP SB from the WLC GUI.
All it does is execute “copy support-bundle” command on the AP and sends it via SCP to the WLC (because WLC can be an SCP server).
And then you can download it from your browser:
This means you can manually do the same trick in eWLC releases before 17.3.1:
Copy the support bundle from AP via SCP to eWLC IP if you don’t have a TFTP server reachable to the AP.
The eWLC is usually reachable via SSH from the AP, so that’s a good trick for pre-17.3.
Step 1. Enable SSH on 9800 v17.2.1
Step 2. Enable SCP on Cisco IOS® XE v17.2.1
This example shows how to configure the server-side functionality of SCP. This example uses a locally defined username and password:
Step 3. Use the command “copy support-bundle” and we need to specify the filename to be created in the SCP server.
Tip: You can run the command once to get a meaningful filename, and then copy/paste that filename in the command:
Step 4. Then you can go into the eWLC GUI and get the file under: Administration > Management > File Manager:
IoT and Bluetooth
The gRPC server logs can be checked on the AP with :
AP# show grpc server log
time="2020-04-01T01:36:52Z" level=info msg="[DNAS] spaces conn url 10.22.243.33:8000"
time="2020-04-01T01:36:52Z" level=info msg="[DNAS] entering stopDNAspacesTmpTokenRoutine"
time="2020-04-01T01:36:52Z" level=info msg="[DNAS] exiting stopDNAspacesTmpTokenRoutine"
time="2020-04-01T01:36:52Z" level=info msg="[DNAS] entering startDNAspacesTmpTokenRoutine"
time="2020-04-01T01:36:52Z" level=info msg="[DNAS] launching token request cycle"
time="2020-04-01T01:36:52Z" level=info msg="[DNAS] exiting startDNAspacesTmpTokenRoutine"
time="2020-04-01T01:36:52Z" level=info msg="[DNAS] spaces token expiration time 2020-04-02 01:36:52 +0000 UTC"
time="2020-04-01T01:36:52Z" level=info msg=" Calling startDNASpacesConn routine "
time="2020-04-01T01:36:52Z" level=info msg="[DNAS] Receive Success status"
time="2020-04-01T01:36:52Z" level=info msg="[DNAS] Connection not in ready state sleeping for 10 seconds"
time="2020-04-01T01:37:02Z" level=info msg="[DNAS] Setup Stream for the gRPC connection"
time="2020-04-01T01:37:02Z" level=info msg="[DNAS] Connect RPC Succeeded."
time="2020-04-01T01:37:02Z" level=info msg="[DNAS] RX routine got enabled "
time="2020-04-01T01:37:02Z" level=info msg="[DNAS] TX routine got enabled "
Connectivity to Cisco DNA Spaces connector can be verified with :
AP# show cloud connector key access
Token Valid : Yes
Token Stats :
Number of Attempts : 44
Number of Failures : 27
Last Failure on : 2020-03-28 02:02:15.649556818 +0000 UTC m=+5753.097022576
Last Failure reason : curl: SSL connect error
Last Success on : 2020-04-01 00:48:37.313511596 +0000 UTC m=+346934.760976625
Expiration time : 2020-04-02 00:48:37 +0000 UTC
Connection Retry Interval : 30
AP# show cloud connector connection detail
Connection State : READY
Connection Url : 10.22.243.33:8000
Certificate Available : true
Controller Ip : 10.22.243.31
Stream Setup Interval : 30
Keepalive Interval : 30
Last Keepalive Rcvd On : 2020-04-01 00:32:47.891433113 +0000 UTC m=+345985.338898246
Number of Dials : 2
Number of Tx Pkts : 2788175
Number of Rx Pkts : 11341
Number of Dropped Pkts : 0
Number of Rx Keepalive : 11341
Number of Tx Keepalive : 11341
Number of Rx Cfg Request : 0
Number of Tx AP Cfg Resp : 0
Number of Tx APP Cfg Resp : 0
Number of Tx APP state pkts : 5
Number of Tx APP data pkts : 2776829
To see the current BLE broadcasting config of the AP :
AP# show controllers ioTRadio ble 0 broadcast
BLE Profile Config
-------------------
Active profile : v-iBeacon
Profile 0 (iBeacon)
UUID : 00001000000000000000000000000000
Interval (ms) : 100
Power (dBm) : -21
Advertised Power (dBm) : -65
Minor : 0
Major : 0
TxPower byte : bfbfbfbfbfbfbfbfbfbfbfbfbf
Profile 1 (Eddystone UID)
Namespace (hex) : 0000000000005446089c
Instance-ID (hex) : 7f0000001f00
Profile 2 (Eddystone URL)
URL : http://www.
To see the scanned results :
AP# show controllers ioTRadio ble 0 scan brief
Profile MAC RSSI(-dBm) RSSI@1meter(-dBm) Last-heard
Unknown 3C:1D:AF:62:EC:EC 88 0 0000D:00H:00M:01S
iBeacon 18:04:ED:04:1C:5F 86 65 0000D:00H:00M:01S
Unknown 18:04:ED:04:1C:5F 78 65 0000D:00H:00M:01S
Unknown 04:45:E5:28:8E:E7 85 65 0000D:00H:00M:01S
Unknown 2D:97:FA:0F:92:9A 91 65 0000D:00H:00M:01S
iBeacon E0:7D:EA:16:35:35 68 65 0000D:00H:00M:01S
Unknown E0:7D:EA:16:35:35 68 65 0000D:00H:00M:01S
iBeacon 04:EE:03:53:74:22 45 256 0000D:00H:00M:01S
Unknown 04:EE:03:53:74:22 45 256 0000D:00H:00M:01S
04:EE:03:53:6A:3A 72 N/A 0000D:00H:00M:01S
Unknown 04:EE:03:53:6A:3A 72 65 0000D:00H:00M:01S
iBeacon E0:7D:EA:16:35:35 68 65 0000D:00H:00M:01S
Unknown E0:7D:EA:16:35:35 67 65 0000D:00H:00M:01S
iBeacon 04:EE:03:53:74:22 60 256 0000D:00H:00M:01S
Unknown 04:EE:03:53:74:22 60 256 0000D:00H:00M:01S
Eddystone URL 04:EE:03:53:6A:3A 72 N/A 0000D:00H:00M:01S
When the AP acts in Advanced BLE gateway mode where an app is deployed, you can check the status of the IoX application with :
AP#show iox applications
Total Number of Apps : 1
--------------------------
App Name : cisco_dnas_ble_iox_app
App Ip : 192.168.11.2
App State : RUNNING
App Token : 02fb3e98-ac02-4356-95ba-c43e8a1f4217
App Protocol : ble
App Grpc Connection : Up
Rx Pkts From App : 3878345
Tx Pkts To App : 6460
Tx Pkts To Wlc : 0
Tx Data Pkts To DNASpaces : 3866864
Tx Cfg Resp To DNASpaces : 1
Rx KeepAlive from App : 11480
Dropped Pkts : 0
App keepAlive Received On : Mar 24 05:56:49
You can connect to the IOX application with these commands and then monitor the logs during floor beacon configuration :
AP#connect iox application
/ #
/# tail -F /tmp/dnas_ble.log
Tue Mar 24 06:55:21 2020 [INFO]: Starting DNA Spaces BLE IOx Application
Tue Mar 24 06:55:21 2020 [INFO]: Auth token file contents: db26a8ab-e800-4fe9-a128-80683ea17b12
Tue Mar 24 06:55:21 2020 [INFO]: Setting gRPC endpoint to: 1.1.7.101:57777
Tue Mar 24 06:55:21 2020 [INFO]: Auth with token: db26a8ab-e800-4fe9-a128-80683ea17b12
Tue Mar 24 06:55:21 2020 [INFO]: Attempt to connect to DNAS Channel
Tue Mar 24 06:55:21 2020 [INFO]: Starting to run metrics
Tue Mar 24 06:55:21 2020 [INFO]: Starting to run Channel Keepalive
Tue Mar 24 06:55:21 2020 [INFO]: Initialize DNAS Reader Channel
Tue Mar 24 06:55:21 2020 [INFO]: Start listener for messages
Tue Mar 24 06:55:21 2020 [INFO]: Running BLE scan thread
Conclusion
There are many troubleshooting tools available to help us in the resolutions of problems related to COS APs.
This document lists the most commonly used ones and is regularly updated.
Revision History
| Revision | Publish Date | Comments |
|---|---|---|
6.0 |
04-Sep-2024 |
Added client debug bundle |
5.0 |
19-Aug-2024 |
Recertification |
3.0 |
17-Nov-2021 |
Added IoT section |
2.0 |
12-Nov-2021 |
Added a section for boot time debugging |
1.0 |
21-Jun-2019 |
Initial Release |
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)