Field Notice: FN63942 - Cisco Wireless Lightweight Access Points and WLAN Controllers Fail to Create CAPWAP Connections Due to Certificate Expiration - Software Upgrade Recommended

Due to the expiration of Manufacturer Installed Certificates (MICs) in Cisco Wireless LAN Controllers (WLCs) and Cisco Access Points (APs), the following connections types might fail to establish:

• AP-to-WLC Control and Provisioning of Wireless Access Points (CAPWAP) connections
• WLC-to-WLC encrypted mobility connections (CAPWAP)
• WLC-to-Mobility Services Engine (MSE) or Connected Mobile Experience (CMX) Network Mobility Services Protocol (NMSP) connections

When a Cisco AP attempts to establish a new connection, it fails to join. When mobility tunnels between controllers are configured, they fail to establish a connection.

The likelihood that this issue will be encountered is 100 percent for Cisco Wireless APs and WLCs that are more than 10 years old and were manufactured before 2017.

To help determine the potential future impact, see the list of Cisco APs and Cisco WLCs and their release dates in the Cisco Community article Lightweight AP - Fail to create CAPWAP/LWAPP connection due to certificate expiration.

Before July 18, 2005, Cisco APs were not manufactured with MICs.

All Cisco wireless products that were manufactured after July 18, 2005, have SHA-1 MICs that expire after 10 years.

Starting in 2017, Cisco manufactured all wireless devices with additional SHA-2 MICs that will expire in 2037 or later.

Starting in 2019, all Cisco 9800 WLCs and Cisco 9100 APs were manufactured with SHA-2 ("SUDI99") certificates that will expire in 2099.

Customers may see one or more of the following error messages:

The Cisco Wireless APs fail to connect to the Cisco WLCs that are running Cisco AireOS. At the time of the join failure, the Cisco WLC msglog might show messages similar to this example:

Jul 10 16:13:52.443 spam_lrad.c:6164 LWAPP-3-PAYLOAD_ERR: Join request does not contain valid certificate in certificate payload - AP 00:11:22:33:44:55

The following is an example of a Cisco WLC MIC expiration case as seen on a Cisco AP that is running Cisco IOS Software:

*Aug 1 05:16:27.127: %PKI-3-CERTIFICATE_INVALID_EXPIRED: Certificate chain 
 validation has failed.
The certificate (SN: 423F49A800000005FEAE) has expired. Validity period ended
 on 14:39:18 UTC Sep 3 2023 Peer certificate verification failed 001A
*Aug 1 05:16:27.127: %CAPWAP-3-ERRORLOG: Certificate verification failed!

Cisco AP-COS fails to join a Cisco WLC that is running Cisco AireOS and whose MIC is expired, as seen on the Cisco AP-COS:

[*12/12/2035 12:34:50.0099] Cert Verification FAILED with error 10 (certificate has expired) at 0 depth...

[*12/12/2035 12:34:50.0099]

[*12/12/2035 12:34:50.0099] /C=US/ST=California/L=San Jose/O=Cisco Systems/CN=AIR-CT5508-K9-00211bfe58c0/emailAddress=support@cisco.com

Cisco APs that are running Cisco IOS Software fail to join a Cisco WLC that is running Cisco AireOS whose MIC is expired, as seen on a Cisco AP:

*Aug 1 05:16:27.127: %PKI-3-CERTIFICATE_INVALID_EXPIRED: Certificate chain validation has failed. The certificate (SN: 423F49A800000005FEAE) has expired. Validity period ended on 14:39:18 UTC Sep 3 2023 Peer certificate verification failed 001A

*Aug 1 05:16:27.127: %CAPWAP-3-ERRORLOG: Certificate verification failed!

Cisco APs that are running Cisco IOS Software fail to join a Cisco C9800 WLC, as seen on a Cisco C9800:

Syslog messages that would appear when a problematic Cisco AP joins Cisco 9800 WLCs:

.Jan  1 00:01:05.630: %PKI-3-CERTIFICATE_INVALID_EXPIRED: Certificate chain validation has failed.  The certificate (SN: 0159BC17) has expired.    Validity period ended on 2027-02-23T17:02:59Z

.Jan  1 00:01:05.633: %DTLS_TRACE_MSG-3-WLC_DTLS_ERR: Chassis 1 R0/0: wncd: DTLS Error, session:x.x.x.x[5272] MAC: aaaa.bbbb.cccc.dddd Certificate validation failed

.Jan  1 00:01:05.633: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: sangudla-3802I Session-IP:x.x.x.x[5272] CAPWAP DTLS session closed for AP, cause: DTLS handshake error

Cisco C9800 fails to connect to an Cisco WLC that is running Cisco AireOS, as seen on a Cisco C9800:

.Jan  1 00:01:20.620: %PKI-3-CERTIFICATE_INVALID_EXPIRED: Certificate chain validation has failed.  The certificate (SN: 019A376F) has expired.    Validity period ended on 2027-06-08T18:09:45Z

Cisco WLC that is running Cisco AireOS fails to establish a Network Mobility Services Protocol (NMSP) connection to a Cisco CMX/MSE, as seen on a Cisco MSE:

6/10/20 14:21:48 ERROR[com.aes] [100] [ConnectHandler:handle-09] THROW javax.net.ssl.SSLHandshakeException: Handshake failed

...

Caused by: java.security.cert.CertificateExpiredException: Certificate expired at Thu Apr 23 22:03:27 PDT 2020 (compared to Wed Jun 10 14:21:48 PDT 2020) at com.cisco.ciscossl.provider.ciscojce.ssl.OpenSSLX509Certificate.checkValidity(OpenSSLX509Certificate.java:458)

Workaround for Cisco APs That Fail to Join a Cisco WLC Due to an Expired Certificate

Note: This workaround should be used only to allow Cisco APs to join a Cisco WLC just long enough to upgrade the software and implement the solution that is provided in this section.

If the Cisco AP or Cisco WLC certificates have expired, complete the following steps:

1. Disable Network Time Protocol (NTP).
2. Change the Cisco WLC clock time to a recent earlier time when the certificates were still valid. If you set the clock back too far, newer APs might not be able to join.

Note: Temporarily disabling NTP and changing the Cisco WLC time settings can adversely affect other time-dependent WLC features such as Management Frame Protection (MFP), Simple Network Management Protocol Version 3 (SNMPv3), and Location.

Solutions for Expired WLC Certificates

Situation: The Cisco WLC does not run a fixed software release and some Cisco APs cannot join. Complete the following steps:

1. Upgrade to a fixed software release.
2. Enter the config ap cert-expiry-ignore {mic|ssc} enable command.

If any of the Cisco APs that cannot join have not downloaded the fixed software, do the following:

1. Disable NTP.
2. Set the clock back to a time before the Cisco WLC certificate expired (this might keep newer Cisco APs from joining).
3. Have all Cisco APs join the Cisco WLC, download new software, and rejoin.
4. Set the clock to the correct time and re-enable NTP.

Situation: The Cisco WLC runs fixed software, but some Cisco APs cannot join. Complete the following steps:

1. Enter the config ap cert-expiry-ignore {mic|ssc} enable command.

If any of the APs that cannot join have not downloaded the fixed software:

1. Disable NTP.
2. Set the clock back to a time before the Cisco WLC certificate expired (this might keep newer Cisco APs from joining).
3. Have all Cisco APs join the Cisco WLC, download new software, and rejoin.
4. Set the clock to the correct time and re-enable NTP.

How to Identify Which Software Releases Have the Fix

For Cisco AireOS, the required fixes depend on the Cisco APs in the network:

For devices that are running Cisco IOS XE Software Release 16.10 or later, no software upgrade is required. 

Solution for Expired AP Certificates or for Encrypted Mobility Tunnels That Fail to Form

To address issues with expired AP certificates or encrypted mobility tunnels that fail to form, complete the following steps:

1. Use the Cisco C9800 command to accept expired certificates

configure terminal
crypto pki certificate map map1 1
 issuer-name co cisco manufacturing ca
crypto pki certificate map map1 2
 issuer-name co act2 sudi ca

crypto pki trustpool policy
 match certificate map1 allow expired-certificate
    
exit

2. Create a certificate map and add the rules

configure terminal
crypto pki certificate map map1 1
issuer-name co Cisco Manufacturing CA

3. Use the certificate map under the trustpool policy

configure terminal
crypto pki trustpool policy
match certificate map1 allow expired-certificate

Note: More rules and filters can be added under the same map. The rule mentioned in the previous configuration specifies that any certificate whose issuer-name contains Cisco Manufacturing CA (case insensitive) is selected under this map.

Situation: NMSP does not come up on MSE, complete the following step:

On MSE, enter the config unauthenticated-nmsp command.

Additional Resources

For recommended releases, see the following documents: 

• Recommended AireOS Wireless LAN Controller Releases
• Recommended Cisco IOS XE Releases for Catalyst 9800 Wireless LAN Controllers
• Guidelines for Cisco Wireless Software Release Product Bulletin

For APs that use an MIC, the approximate expiration date can be found using the device serial number (SN).

To see the serial numbers of all Cisco APs that are joined to a Cisco WLC, collect the following output:

• show tech support wireless (9800)
• show run-config (AireOS)

Load the output into Wireless Config Analyzer Express(WCAE).

MIC

Use the SN to determine the approximate date that the MIC will expire.

The Cisco AP MIC will expire, on average, 10 years after the date of manufacture. Note that some Cisco APs might have earlier or more recently created MICs under some conditions. For example, consider a scenario where the Cisco AP motherboard was manufactured and stored, but not assembled until later, or when the Cisco AP was subject to a Return Material Authorization (RMA) and a refurbishing process.

To determine when the Cisco AP was manufactured, enter this command on the Cisco WLC to find the Cisco AP SN:

(Cisco Controller) >show ap inventory all
Inventory for lap1130-sw3-9
NAME: "Cisco AP" , DESCR: "Cisco Wireless Access Point"
PID: AIR-LAP1131AG-E-K9, VID: V01, SN: FCZ1128Q0PE
NAME: "Dot11Radio0" , DESCR: "802.11G Radio"
PID: UNKNOWN, VID: , SN: GAM112706LC?
NAME: "Dot11Radio1" , DESCR: "802.11A Radio"
PID: UNKNOWN, VID: , SN: ALP112706LC

The Cisco AP chassis SN is in the first section of the output. For example: PID: AIR-LAP1131AG-E-K9, VID: V01, SN: FCZ1128Q0PE.

Note: For more information, see the Derive the Manufactured Date from the Product SN section that follows. 

Alternatively, the exact date that the MIC expires can be found with the output of the show crypto pki certificates (Cisco IOS Software) command or show crypto (Cisco AP-COS) command from the Cisco AP CLI. Look for the Certificate entry and ignore the CA Certificate entries. The end date that is associated with the Validity Date section is the expiration date for the MIC certificate:

AP_CLI#show crypto pki certificates
CA Certificate
Status: Available...
...
Certificate
Status: Available
Certificate Serial Number: 728AF4350000001E4C89
Certificate Usage: General Purpose
Issuer:
cn=Cisco Manufacturing CA
o=Cisco Systems
Subject:
Name: C1130-001c58b5b3a4
ea=support@cisco.com
cn=C1130-001c58b5b3a4
o=Cisco Systems
l=San Jose
st=California
c=US
CRL Distribution Points:
http://www.cisco.com/security/crl/cmca.crl
Validity Date:
start date: 04:22:10 UTC Jul 11 2007
end date: 04:32:10 UTC Jul 11 2017
Associated Trustpoints: Cisco_IOS_MIC_cert

AireOS WLC

For Cisco AireOS Release 8.0 or later, enter the following command and look for the Cisco SHA1 device cert entry to determine when the WLC certificate expires:

WLC_CLI: show certificate all
Certificate Name: Cisco SHA1 device cert

Subject Name :
C=US, ST=California, L=San Jose, O=Cisco Systems, CN=AIR-CT2504-K9-d0c282d65a20,
 MAILTO=support@cisco.com
Issuer Name :
O=Cisco Systems, CN=Cisco Manufacturing CA
Serial Number :
454384735992863371807890
Validity :
Start : 2011 Jul 26th, 20:17:17 GMT
End : 2021 Jul 26th, 20:27:17 GMT
Signature Algorithm :
rsa-pkcs1-sha1
Hash key :
SHA1 Fingerprint : 98:89:eb:12:2a:98:bc:fe:ad:5b:8f:23:63:0f:47:d1:36:ce:f5:be
MD5 Fingerprint : ba:f3:98:9a:cd:f8:01:08:84:b8:66:3c:6a:6c:d3:05

Note: This command is not available in Cisco AireOS releases earlier than Release 8.0. There is no similarly straightforward command that can be used to find this date in earlier Cisco AireOS releases. Alternatively, use the Cisco WLC SNs to determine the earliest possible MIC expiration date.

Derive the Manufactured Date from the Product SN

The SN format is LLLYYWWSSSS. The YY is the year of manufacture, and the WW is the week of manufacture. The date code can be found in the four middle digits of the SN.

Note: Since the SN format for the AIR-CT7510 is different, you cannot use it to derive the manufacturing date. Instead, for the AIR-CT7510, enter the show certificate all command as noted previously.

The manufacturing year codes are:

01 = 1997
02 = 1998
03 = 1999
04 = 2000
05 = 2001
06 = 2002
07 = 2003
08 = 2004
09 = 2005
10 = 2006
11 = 2007
12 = 2008
13 = 2009
14 = 2010
15 = 2011
16 = 2012
17 = 2013
18 = 2014
19 = 2015
20 = 2016
21 = 2017

The manufacturing week codes are:

01-05 = January
06-09 = February
10-14 = March
15-18 = April
19-22 = May
23-27 = June
28-31 = July
32-35 = August
36-40 = September
41-44 = October
45-48 = November
49-52 = December

For example, the SN FCZ1128xxxx has a manufacturing year code of 11, which means that it was manufactured in the year 2007. The week code is 28, which means that it was manufactured in July of that year.